Image super-resolution method and device based on bidirectional focusing enhancement

Abstract

The application relates to an image super-resolution method and device based on bidirectional focus enhancement. The method comprises designing a bidirectional focus enhancement module comprising a feature fusion module and a regional focus module in a bidirectional focus enhancement super-resolution model, wherein the feature fusion module fuses local enhancement and a context-aware graph neural attention mechanism, and the dynamic key region attention mechanism of the regional focus module, and the image features are enhanced and optimized from two dimensions of local details and global semantics through the two mechanisms, so that efficient reconstruction from a degraded low-resolution image to a high-quality high-resolution image is realized. The method can effectively improve the quality and efficiency of super-resolution reconstruction, and can be better applied to actual scenes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an image super-resolution method and device based on bidirectional focus enhancement. BACKGROUND

[0002] In today's era of rapid digital information development, image data, as an important carrier of information dissemination, plays a key role in many fields. Whether it is photo sharing, video calls in daily life, or medical imaging, satellite remote sensing, security monitoring in professional fields, the quality of the image directly relates to the accuracy and effectiveness of information transmission. However, in practical applications, due to the resolution limit of imaging devices, compression loss in the transmission process, poor shooting environment and other factors, the acquired images often have low resolution, blurred details and other problems, which seriously restricts the further application and value mining of images.

[0003] Image super-resolution technology emerges as the times require, and its core goal is to convert low-resolution images into high-resolution images through algorithms, thereby improving the visual quality of images and restoring more detailed information. This technology is of great significance and has a wide range of applications. In the field of security monitoring, super-resolution technology can convert blurred monitoring images into clear images, helping to more accurately identify key targets such as personnel and vehicles, and improving the effectiveness of security monitoring; in the field of medical imaging, by performing super-resolution reconstruction on low-resolution medical images, the details of tissues and organs can be observed more clearly, providing more accurate basis for disease diagnosis; in the field of satellite remote sensing, super-resolution technology can enhance the detailed resolution of remote sensing images, enabling the images of the Earth's surface obtained from space to more finely reflect the features of ground objects, providing higher quality data support for geographic information mapping, environmental monitoring, resource exploration, etc.; in the consumer electronics field, for low-resolution photos or videos taken by mobile phones, cameras and other devices, super-resolution technology can improve their clarity and improve users' visual experience.

[0004] Early super-resolution technologies mainly rely on traditional methods, such as interpolation-based algorithms. Interpolation methods generate high-resolution images by inserting intermediate pixels based on low-resolution images. Common methods include nearest neighbor interpolation, bilinear interpolation and bicubic interpolation. The advantages of these methods are simplicity of implementation and high computational efficiency, which can quickly enlarge images. However, they also have obvious limitations. Because interpolation algorithms mainly rely on the correlation between adjacent pixels to estimate the value of new pixels, they cannot capture more complex structure and texture information in images. In the reconstructed high-resolution images, there are often phenomena such as blurring, artifacts, and poor detail restoration, making it difficult to meet the requirements of high-quality image applications.

[0005] With the deepening of research in the field of computer vision, reconstruction-based methods have been gradually proposed. These methods attempt to recover high-resolution images by establishing image degradation models and utilizing prior knowledge of images. For example, some sparse representation-based methods assume that image patches can be represented as a sparse linear combination of a set of basis functions, and perform super-resolution reconstruction by learning the sparse representation relationship between low-resolution and high-resolution image patches. There are also gradient statistics-based methods that use the gradient prior information of high-resolution images to guide the reconstruction process. Although reconstruction-based methods can generate clearer images to some extent, they require high prior knowledge of images and involve complex optimization calculations in the solving process, which takes a long time to run and is difficult to meet the real-time requirements, limiting their widespread application in practical applications.

[0006] In recent years, with the rapid development of deep learning technology, convolutional neural networks (CNNs) have made breakthrough progress in the field of computer vision. Super-resolution methods based on CNNs can automatically learn the complex mapping relationship between low-resolution images and high-resolution images due to their powerful feature extraction and representation capabilities, thus achieving efficient reconstruction of images. For example, SRCNN (Super-Resolution Convolutional Neural Network), as one of the earliest deep learning-based super-resolution methods, pioneered the introduction of CNNs into the super-resolution reconstruction task, learned the feature representation of low-resolution images and mapped them to high-resolution image space, achieving better reconstruction results than traditional methods. Subsequently, researchers have continuously improved and optimized the network structure of CNNs, such as FSRCNN (Fast Super-Resolution Convolutional Neural Network), which adjusts the number of network layers and filter sizes to ensure reconstruction quality while improving network running speed; EDSR (Enhanced Deep Super-Resolution) further improves the performance of super-resolution models by deepening the network structure, increasing the number of channels, and adopting residual learning strategies. These CNN-based methods have achieved significant performance improvements on multiple benchmark datasets, becoming a research hotspot in the field of image super-resolution.

[0007] However, as the performance of CNN-based super-resolution methods continues to improve, new challenges arise. To further enhance the reconstruction accuracy of the models, many researchers have increased the number and complexity of network parameters to strengthen their expressive power, leading to a significant increase in computational and storage requirements. For example, some high-performance super-resolution models may contain millions or even tens of millions of parameters, placing extremely high demands on the computing power and memory capacity of hardware devices. In practical applications, such computationally intensive models are difficult to deploy effectively and run in real time on platforms with limited hardware performance, such as mobile devices and embedded systems. Taking mobile image super-resolution applications as an example, users want to quickly convert low-resolution photos into high-quality high-resolution images, but currently, some complex super-resolution models experience slow processing speeds or even fail to run on mobile devices due to computational resource limitations, seriously affecting user experience and the promotion of applications.

[0008] To address this issue, researchers have begun exploring lightweight super-resolution methods. These methods attempt to reduce the number of model parameters and computational complexity by designing simpler network structures, enabling efficient super-resolution reconstruction on low-performance devices. Common strategies include building shallow network structures, employing recursive network designs, and utilizing operations such as depthwise separable convolutions to reduce computational overhead. However, while these methods reduce the burden on the model, they often fail to fully utilize the deep expressive capabilities of CNNs, resulting in a significant gap in reconstruction performance compared to large, complex models, and still failing to meet the high image quality requirements of practical applications. Summary of the Invention

[0009] Therefore, it is necessary to provide an image super-resolution method and apparatus based on bidirectional focus enhancement that can simultaneously balance the clarity of image restoration and computational complexity, addressing the aforementioned technical problems.

[0010] An image super-resolution method based on bidirectional focusing enhancement, the method comprising:

[0011] The bidirectional focus enhancement super-resolution model comprises an input module, a bidirectional focus enhancement module and an output module; the input module performs shallow feature extraction on an input low-resolution image to obtain shallow features; the bidirectional focus enhancement module comprises a plurality of cascaded enhancement units, each enhancement unit comprising a feature fusion module and a regional focus module; the feature fusion module extracts local information of the shallow features or fusion features output by a previous-level enhancement unit to obtain local enhancement features, introduces a multi-head attention mechanism to process the local enhancement features to obtain global attention features, and fuses the shallow features, the local enhancement features and the global attention features to obtain fusion features; the regional focus module extracts effective information of a key region contained in the fusion features to obtain fusion key features; and the output module reconstructs to obtain a super-resolution image according to the fusion key features output by a last-level enhancement unit in the bidirectional focus enhancement module.

[0012] The low-resolution image to be processed is input into the pre-trained bidirectional focus enhancement super-resolution model, and a super-resolution image is output.

[0013] In one of the embodiments, the input module performs shallow feature extraction on the input low-resolution image to obtain shallow features.

[0014]

[0015] The shallow features are represented as The low-resolution image is represented as The shallow features are represented as The convolution layer is represented as

[0016] In one of the embodiments, the input module performs shallow feature extraction on the input low-resolution image to obtain shallow features. The convolution layer is used to compress the channel dimension, and the The convolution is used to extract local spatial information, and the ReLU activation function is fused in the process to obtain local enhancement features.

[0017] ;

[0018] The shallow features or the fusion features output by a previous-level enhancement unit are represented as The shallow features or the fusion features output by a previous-level enhancement unit are represented as The convolution layer is represented as The convolution layer is represented as The convolution layer is represented as The convolution layer is represented as The ReLU activation function is represented as

[0019] In one embodiment, the method further includes: projecting the local enhancement features onto the query vector, key vector, and value vector in the multi-head attention mechanism, respectively:

[0020] ;

[0021] in, For trainable weight matrix, For each attention head, the feature dimensions, Representing feature dimension, These represent the query vector, key vector, and value vector, respectively.

[0022] The query vector and key vector are concatenated, and the attention weights are calculated as follows:

[0023] ;

[0024] | indicates a vector concatenation operation. It is the attention parameter vector. It is a node The set of neighboring nodes, This represents the attention weight between node i and its neighbor node j. Let i and j represent the query vector and key vector, respectively. This represents the activation function, where exp is the exponential function. Represents the set of neighboring nodes of node i;

[0025] The attention weights are weighted and aggregated based on the features of neighboring nodes to obtain the graph attention output for each head:

[0026] ;

[0027] in, This represents the attention output of the h-th attention head;

[0028] The global attention feature obtained by concatenating multiple attention heads is as follows:

[0029] ;

[0030] in, Represents global attention features. H This indicates the number of attention heads.

[0031] In one embodiment, the method further includes fusing the shallow features, local enhancement features, and global attention features to obtain a fused feature:

[0032] ;

[0033] in, representing a fusion feature.

[0034] In one embodiment, the method further comprises: unfolding the fusion feature into a feature sequence in attention form, and calculating dense attention weights of the feature sequence as:

[0035]

[0036] wherein, is a feature dimension, are a query vector of an i-th node and a key vector of a j-th node in the feature sequence, respectively, represents a dense attention weight between the node i and the node j, and N is a total number of nodes;

[0037] According to the dense attention weights, a dense branch output is obtained as:

[0038]

[0039] wherein, represents the dense branch output, is a value vector in the feature sequence, represents a number of pixels;

[0040] calculating sparse attention weights of the feature sequence as:

[0041]

[0042] wherein, represents a sparse attention weight between the node i and the node j, represents a neighbor node set of the node i;

[0043] According to the sparse attention weights, a sparse branch output is obtained as:

[0044]

[0045] wherein, represents the sparse branch output;

[0046] weighting the dense branch output and the sparse branch output to obtain a fusion key feature as:

[0047]

[0048] wherein, represents the fusion key feature, is a weighting coefficient.

[0049] In one embodiment, a feature updating process of the bidirectional focus enhancement module is: ​​​​​

[0050] ;

[0051] ;

[0052] wherein K represents the number of enhancement units, represents a connection function, represents a connection feature, represents that the feature fusion module processes the input feature , represents that the region focusing module processes the output of the feature fusion module, represents the kth feature.

[0053] An image super-resolution device based on bidirectional focusing enhancement, the device comprising:

[0054] a model construction module, configured to construct a bidirectional focusing enhancement super-resolution model; the bidirectional focusing enhancement super-resolution model comprising an input module, a bidirectional focusing enhancement module and an output module; the input module performing shallow feature extraction on an input low-resolution image to obtain shallow features, the bidirectional focusing enhancement module comprising a plurality of cascaded enhancement units, each enhancement unit comprising a feature fusion module and a region focusing module; the feature fusion module extracting local information of the shallow features or fusion features output by a previous enhancement unit to obtain local enhancement features, introducing a multi-head attention mechanism to process the local enhancement features to obtain global attention features, and fusing the shallow features, the local enhancement features and the global attention features to obtain fusion features, the region focusing module extracting effective information of key regions contained in the fusion features to obtain fusion key features, and the output module reconstructing a super-resolution image according to the fusion key features output by a last enhancement unit in the bidirectional focusing enhancement module;

[0055] an image output module, configured to input a low-resolution image to be processed into a pre-trained bidirectional focusing enhancement super-resolution model and output a super-resolution image.

[0056] A computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the following steps when executing the computer program:

[0057] The bidirectional focus enhancement super-resolution model comprises an input module, a bidirectional focus enhancement module and an output module; the input module performs shallow feature extraction on an input low-resolution image to obtain shallow features; the bidirectional focus enhancement module comprises a plurality of cascaded enhancement units, each enhancement unit comprising a feature fusion module and a regional focus module; the feature fusion module extracts local information of the shallow features or fusion features output by a previous enhancement unit to obtain local enhancement features, introduces a multi-head attention mechanism to process the local enhancement features to obtain global attention features, and fuses the shallow features, the local enhancement features and the global attention features to obtain fusion features; the regional focus module extracts effective information of key regions contained in the fusion features to obtain fusion key features; and the output module reconstructs the fusion key features output by a last enhancement unit in the bidirectional focus enhancement module to obtain a super-resolution image.

[0058] A low-resolution image to be processed is input into the pre-trained bidirectional focus enhancement super-resolution model, and a super-resolution image is output.

[0059] A computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the following steps:

[0060] The bidirectional focus enhancement super-resolution model comprises an input module, a bidirectional focus enhancement module and an output module; the input module performs shallow feature extraction on an input low-resolution image to obtain shallow features; the bidirectional focus enhancement module comprises a plurality of cascaded enhancement units, each enhancement unit comprising a feature fusion module and a regional focus module; the feature fusion module extracts local information of the shallow features or fusion features output by a previous enhancement unit to obtain local enhancement features, introduces a multi-head attention mechanism to process the local enhancement features to obtain global attention features, and fuses the shallow features, the local enhancement features and the global attention features to obtain fusion features; the regional focus module extracts effective information of key regions contained in the fusion features to obtain fusion key features; and the output module reconstructs the fusion key features output by a last enhancement unit in the bidirectional focus enhancement module to obtain a super-resolution image.

[0061] A low-resolution image to be processed is input into the pre-trained bidirectional focus enhancement super-resolution model, and a super-resolution image is output.

[0062] The aforementioned image super-resolution method and apparatus based on bidirectional focusing enhancement incorporates a bidirectional focusing enhancement module within the super-resolution model. This module integrates a feature fusion module and a region focusing module. The feature fusion module combines local enhancement with a context-aware graph neural attention mechanism, while the region focusing module utilizes a dynamic key region attention mechanism. These two mechanisms enhance and optimize image features from both local detail and global semantic dimensions, respectively, thereby achieving efficient reconstruction from degraded low-resolution images to high-quality high-resolution images. This method aims to maintain a lightweight model while effectively improving the quality and efficiency of super-resolution reconstruction through the synergistic fusion of local feature enhancement and global semantic focusing mechanisms. This allows for better application in real-world scenarios, particularly in resource-constrained environments such as mobile devices. Attached Figure Description

[0063] Figure 1 This is a flowchart illustrating an image super-resolution method based on bidirectional focusing enhancement in one embodiment;

[0064] Figure 2 This is a framework diagram of a bidirectional focusing-enhanced super-resolution model in one embodiment;

[0065] Figure 3 This is a framework diagram of the feature fusion module in another embodiment;

[0066] Figure 4 This is a structural block diagram of an image super-resolution device based on bidirectional focusing enhancement in one embodiment;

[0067] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0069] In one embodiment, such as Figure 1 As shown, an image super-resolution method based on bidirectional focusing enhancement is provided, including the following steps:

[0070] Step 102: Construct a bidirectional focusing-enhanced super-resolution model.

[0071] The bidirectional focus enhancement super-resolution model comprises an input module, a bidirectional focus enhancement module and an output module; the input module performs shallow feature extraction on an input low-resolution image to obtain shallow features; the bidirectional focus enhancement module comprises a plurality of cascaded enhancement units, each enhancement unit comprising a feature fusion module and a region focus module; the feature fusion module extracts local information of the shallow features or fusion features output by a previous-level enhancement unit to obtain local enhanced features, introduces a multi-head attention mechanism to process the local enhanced features to obtain global attention features, and fuses the shallow features, the local enhanced features and the global attention features to obtain fusion features; the region focus module extracts effective information of a key region contained in the fusion features to obtain fusion key features; and the output module reconstructs a super-resolution image according to the fusion key features output by a last-level enhancement unit in the bidirectional focus enhancement module.

[0072] In step 104, a low-resolution image to be processed is input into the pre-trained bidirectional focus enhancement super-resolution model, and a super-resolution image is output.

[0073] In the above bidirectional focus enhancement-based image super-resolution method, a bidirectional focus enhancement module comprising a feature fusion module and a region focus module is designed in the bidirectional focus enhancement super-resolution model, wherein the feature fusion module fuses local enhancement and context-aware graph neural attention mechanisms, and the region focus module has a dynamic key region attention mechanism; through these two mechanisms, image features are enhanced and optimized from two dimensions of local details and global semantics, thereby realizing efficient reconstruction from a degraded low-resolution image to a high-quality high-resolution image. The above method aims to maintain the lightweight of the model, effectively improve the quality and efficiency of super-resolution reconstruction by synergistically fusing local feature enhancement and global semantic focusing mechanisms, and make it better applied to actual scenarios, especially in resource-constrained environments such as mobile devices.

[0074] In one embodiment, as shown in FIG. 1, the overall framework diagram of the bidirectional focus enhancement super-resolution model is provided, the input module performs shallow feature extraction on the input low-resolution image to obtain shallow features, and the bidirectional focus enhancement module comprises a plurality of cascaded enhancement units, each enhancement unit comprising a feature fusion module and a region focus module. Figure 2

[0075]

[0076] represents the shallow features, represents the low-resolution image, represents convolutional layers.

[0077] ​In one of the embodiments, to improve the fusion ability between local details and global semantics in the feature map, a fusion mechanism of local enhancement and context-aware graph attention network (LECAG) is designed, which is also called a feature fusion module. The core idea of the module is to use convolution operation to enhance the modeling ability of local spatial continuity features, and introduce a multi-head graph attention mechanism to capture long-distance semantic dependencies and context associations, and realize efficient collaborative expression of local-global features.

[0078] Specifically, as shown in Figure 3 , a model framework diagram of a feature fusion module is provided, and the input feature map is , where represents the number of nodes in the graph, represents the feature dimension. For the input shallow feature or the fusion feature output by the upper-level enhancement unit, a convolution layer is used to compress the channel dimension, and a convolution is used to extract local spatial information. In the process, a ReLU activation function is fused to obtain the local enhancement feature as:

[0079] ;

[0080] wherein, represents the shallow feature or the fusion feature output by the upper-level enhancement unit, represents a convolution layer, represents a convolution layer, represents a ReLU activation function.

[0081] Specifically, the ReLU activation function improves the nonlinear fitting ability, is used for dimension reduction and linear transformation, is used to extract context features within the local receptive field, and the final is an intermediate feature with local enhancement capability.

[0082] In one of the embodiments, on the basis of local modeling, a multi-head graph attention mechanism (Multi-Head Graph Attention) is introduced to further introduce non-local modeling capability, which is used to capture long-distance dependencies between nodes.

[0083] Specifically, the local enhancement feature is projected into the query vector, key vector and value vector in the multi-head attention mechanism as:

[0084] ;

[0085] wherein, is a trainable weight matrix, is the feature dimension of each attention head, denotes the feature dimension, denote the query vector, the key vector and the value vector respectively, the query vector and the key vector are spliced, and the attention weight is calculated as:

[0086] ;

[0087] denotes a vector splicing operation, is an attention parameter vector, is a neighbor node set of a node denotes the attention weight between the node i and the neighbor node j, denote the query vector and the key vector of the node i and the node j respectively, denotes an activation function, and exp is an exponential function, denotes the neighbor node set of the node i, the neighbor node features are weighted and aggregated according to the attention weight, and the graph attention output of each head is:

[0088] ;

[0089] wherein, denotes the attention output of the hth attention head; the global attention feature is obtained by splicing multiple attention heads as:

[0090] ;

[0091] wherein, denotes the global attention feature. Wherein is the number of graph attention heads. By introducing the multi-head mechanism, the modeling ability of the model for different attention paths can be improved, and the diversity expression of the feature space is enhanced.

[0092] In one of the embodiments, the shallow layer feature, the local enhanced feature and the global attention feature are fused to obtain a fusion feature as:

[0093] ;

[0094] wherein, denotes the fusion feature. Such a fusion strategy not only retains the original semantics in the input feature, but also supplements the local spatial continuity and non-local context dependence, so that the output feature has better discriminative ability and structural expression ability. The design of the entire LECAG module is simple and effective, and is suitable for task scenarios that are sensitive to local details and global information in graph structure modeling.​

[0095] In one embodiment, in computer vision tasks such as image restoration and reconstruction, the standard self-attention mechanism (Self-Attention) relied on by the traditional Transformer has strong global modeling ability, but its computational complexity is , which often brings huge resource overhead when processing high-resolution images, and calculating attention for all image regions equally can easily introduce redundant interference information, which is not conducive to accurately extracting effective features from degraded images. To overcome the above problems, a dynamic key region attention mechanism (DKA, region focusing module) is proposed, which effectively improves the efficiency and discriminability of the attention mechanism in the image degradation modeling process by introducing a sparse-dense parallel structure and a dynamic sparse selection strategy.

[0096] Specifically, the fusion features are expanded into a feature sequence in the form of attention, and the dense attention weight of the feature sequence is calculated as:

[0097] ;

[0098] wherein, is the feature dimension, are the query vector of the i-th node and the key vector of the j-th node in the feature sequence, denotes the dense attention weight between node i and node j, and N is the total number of nodes; according to the dense attention weight, the dense branch output is obtained as:

[0099] ;

[0100] wherein, denotes the dense branch output, is the value vector in the feature sequence, denotes the number of pixels; on the other hand, to reduce the computational load and highlight key region modeling, a sparse attention branch is introduced, which only selects the top key vectors most similar to each query vector to form a local candidate set, and the sparse attention weight of the feature sequence is calculated as:

[0101] ;

[0102] wherein, denotes the sparse attention weight between node i and node j; it is worth noting that the selection of the local candidate set is dynamically and adaptively determined based on the input features, i.e. for different input images or different spatial positions, the sparse neighborhood is variable. This selection method not only ensures the preservation of high response regions, but also effectively suppresses the influence of irrelevant interference regions such as raindrops, scratches, and fog in degraded images.

[0103] According to the sparse attention weight, the sparse branch output is obtained as:

[0104] ;

[0105] wherein, denotes the sparse branch output; the dense branch output and the sparse branch output are weighted to obtain the fused key feature as:

[0106] ;

[0107] wherein, denotes the fused key feature, is a weighting coefficient.

[0108] wherein the fusion coefficient may be set as a fixed value, or can be obtained by adaptive learning of the network to adapt to the complexity of different tasks or input features. This fusion mechanism realizes a good trade-off between precision and efficiency: the dense branch ensures the global dependence modeling capability, and the sparse branch greatly compresses the calculation amount and improves the robustness, especially showing stronger discriminative ability and generalization ability under high-resolution and low-quality input images.

[0109] In one of the embodiments, the feature updating process of the bidirectional focus enhancement module is as follows:

[0110] ;

[0111] ;

[0112] wherein K denotes the number of enhancement units, denotes a connection function, denotes a connection feature, denotes that the feature fusion module processes the input feature , denotes that the region focus module processes the output of the feature fusion module, denotes the kth feature.

[0113] It should be understood that although each step in the flowchart of Figure 1 is displayed in sequence according to the direction of the arrow, these steps are not necessarily executed in sequence according to the direction of the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, Figure 1At least one of the steps in the method can comprise a plurality of sub-steps or a plurality of stages, which sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the order of the sub-steps or stages is not necessarily sequential, but can be performed alternately or in rotation with other steps or sub-steps or stages of other steps.

[0114] In one embodiment, as shown in FIG. 4, there is provided a bidirectional focus enhancement based image super-resolution device, comprising a model construction module 402 and an image output module 404, wherein: Figure 4

[0115] The model construction module 402 is configured to construct a bidirectional focus enhancement super-resolution model, wherein the bidirectional focus enhancement super-resolution model comprises an input module, a bidirectional focus enhancement module and an output module; the input module is configured to perform shallow feature extraction on an input low-resolution image to obtain shallow features; the bidirectional focus enhancement module comprises a plurality of cascaded enhancement units, each enhancement unit comprising a feature fusion module and a regional focus module; the feature fusion module is configured to extract local information of the shallow features or fusion features output by a previous level enhancement unit to obtain local enhancement features, introduce a multi-head attention mechanism to process the local enhancement features to obtain global attention features, and fuse the shallow features, the local enhancement features and the global attention features to obtain fusion features; the regional focus module is configured to extract effective information of key regions contained in the fusion features to obtain fusion key features; and the output module is configured to reconstruct a super-resolution image according to the fusion key features output by a last level enhancement unit in the bidirectional focus enhancement module.

[0116] The image output module 404 is configured to input a low-resolution image to be processed into the pre-trained bidirectional focus enhancement super-resolution model to output a super-resolution image.

[0117] In one embodiment, the model construction module 402 is further configured to perform shallow feature extraction on an input low-resolution image by the input module to obtain shallow features as follows:

[0118]

[0119] represents the shallow features, represents the low-resolution image, represents a convolution layer.

[0120] In one embodiment, the model construction module 402 is further configured to compress the channel dimension of the shallow features input or the fusion features output by a previous level enhancement unit by a convolution layer to obtain local enhancement features as follows: ​​Convolution extracts local spatial information, and the ReLU activation function is fused during the process to obtain the local enhanced features:

[0121] ;

[0122] in, This represents shallow features or fused features output by the previous enhancement unit. express Convolutional layer express Convolutional layer This represents the ReLU activation function.

[0123] In one embodiment, the model building module 402 is further configured to project the local enhancement features onto the query vector, key vector, and value vector in the multi-head attention mechanism, respectively:

[0124] ;

[0125] in, For trainable weight matrix, For each attention head, the feature dimensions, Representing feature dimension, These represent the query vector, key vector, and value vector, respectively.

[0126] The query vector and key vector are concatenated, and the attention weights are calculated as follows:

[0127] ;

[0128] | indicates a vector concatenation operation. It is the attention parameter vector. It is a node The set of neighboring nodes, This represents the attention weight between node i and its neighbor node j. Let i and j represent the query vector and key vector, respectively. This represents the activation function, where exp is the exponential function. Represents the set of neighboring nodes of node i;

[0129] The attention weights are weighted and aggregated based on the features of neighboring nodes to obtain the graph attention output for each head:

[0130] ;

[0131] in, This represents the attention output of the h-th attention head;

[0132] The global attention feature obtained by concatenating multiple attention heads is as follows:

[0133] ;

[0134] wherein, denotes the global attention feature, H denotes the number of attention heads.

[0135] In one of the embodiments, the model construction module 402 is further configured to fuse the shallow feature, the local enhanced feature and the global attention feature to obtain a fused feature as follows:

[0136] ;

[0137] wherein, denotes the fused feature.

[0138] In one of the embodiments, the model construction module 402 is further configured to expand the fused feature into an attention-form feature sequence, and calculate a dense attention weight of the feature sequence as follows:

[0139] ;

[0140] wherein, is a feature dimension, are a query vector of an i-th node and a key vector of a j-th node in the feature sequence respectively, denotes a dense attention weight between the node i and the node j, and N is a total number of nodes;

[0141] According to the dense attention weight, a dense branch output is obtained as follows:

[0142] ;

[0143] wherein, denotes the dense branch output, is a value vector in the feature sequence;

[0144] A sparse attention weight of the feature sequence is calculated as follows:

[0145] ;

[0146] wherein, denotes a sparse attention weight between the node i and the node j, denotes a neighbor node set of the node i;

[0147] According to the sparse attention weight, a sparse branch output is obtained as follows:

[0148] ;

[0149] wherein, Indicates sparse branch output;

[0150] The dense branch output and the sparse branch output are weighted to obtain the fused key features as follows:

[0151] ;

[0152] in, Indicates key features of fusion. These are the weighting coefficients.

[0153] In one embodiment, the feature update process of the bidirectional focusing enhancement module is as follows:

[0154] ;

[0155] ;

[0156] Where K represents the number of enhancement units, Represents the join function. Indicates connection characteristics, The feature fusion module represents the input features. To process, This indicates that the region focusing module processes the output of the feature fusion module. This represents the k-th feature.

[0157] Specific limitations regarding the bidirectional focus enhancement-based image super-resolution device can be found in the limitations of the bidirectional focus enhancement-based image super-resolution method described above, and will not be repeated here. Each module in the aforementioned bidirectional focus enhancement-based image super-resolution device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0158] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5As shown in the figure. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium, an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store low-resolution image data to be recovered. The network interface of the computer device is used to communicate with the external terminal through the network connection. The computer program is executed by the processor to implement an image super-resolution method based on bidirectional focus enhancement.

[0159] Those skilled in the art can understand that, Figure 5 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0160] In one embodiment, a computer device is provided, including a memory and a processor, the memory stores a computer program, and the processor executes the computer program to implement the steps of the method in the above-mentioned embodiments.

[0161] In one embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by the processor to implement the steps of the method in the above-mentioned embodiments.

[0162] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, databases, or other media in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0163] The technical features of the above embodiments can be combined in any way. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, but as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0164] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, some modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A bidirectional focus enhancement-based image super-resolution method, characterized in that, The method includes: A bidirectional focusing enhanced super-resolution model is constructed. This model includes an input module, a bidirectional focusing enhancement module, and an output module. The input module extracts shallow features from the input low-resolution image to obtain shallow features. The bidirectional focusing enhancement module includes multiple cascaded enhancement units, each of which includes a feature fusion module and a region focusing module. The feature fusion module extracts local information from the shallow features or the fused features output by the previous enhancement unit to obtain local enhanced features. A multi-head attention mechanism is introduced to process the local enhanced features to obtain global attention features. The shallow features, local enhanced features, and global attention features are fused to obtain fused features. The region focusing module extracts effective information from key regions contained in the fused features to obtain fused key features. The output module reconstructs the super-resolution image based on the fused key features output by the last enhancement unit in the bidirectional focusing enhancement module. The low-resolution image to be processed is input into a pre-trained bidirectional focusing enhancement super-resolution model, and the super-resolution image is output. Extracting the effective information of key regions contained in the fusion features yields the fusion key features, including: The fused features are expanded into a feature sequence in the form of attention, and the dense attention weights of the feature sequence are calculated as follows: wherein, is a feature dimension, are a query vector of the i-th node and a key vector of the j-th node in the feature sequence, respectively, denotes a dense attention weight between node i and node j, and N is the total number of nodes. Based on the dense attention weights, the dense branch output is obtained as follows: wherein, denotes a dense branch output, is a value vector of node j in the feature sequence; The sparse attention weights of the feature sequence are calculated as follows: wherein, denotes a sparse attention weight of node i and node j, denotes a set of neighbor nodes of node i; Based on the sparse attention weights, the sparse branch output is obtained as follows: wherein denotes a sparse branch output; The dense branch output and the sparse branch output are weighted to obtain the fused key features as follows: wherein, denotes the fusion key features, is a weighting coefficient.

2. The method of claim 1, wherein, The input module performs shallow feature extraction on the input low-resolution image to obtain shallow features, including: The input module performs shallow feature extraction on the input low-resolution image, obtaining the following shallow features: representing shallow features, representing low resolution images, representing convolutional layers.

3. The method of claim 1, wherein, Extracting local information from the shallow features or the fused features output by the previous enhancement unit to obtain local enhancement features, including: For the input of the shallow features or the fusion features output by the upper level enhancement unit, the method adopts The convolution layer compresses the channel dimension, and obtains the local enhancement features by The convolution extracts the local spatial information, and the ReLU activation function is fused in the process, and the local enhancement features are obtained wherein, denotes a fusion feature of the shallow feature or the output of the upper level enhancement unit, denotes a convolutional layer, denotes a convolutional layer, denotes a ReLU activation function.

4. The method of claim 3, wherein, A multi-head attention mechanism is introduced to process the local enhancement features to obtain global attention features, including: The local enhancement features are projected onto the query vector, key vector, and value vector in the multi-head attention mechanism, respectively, as follows: wherein, is a trainable weight matrix, is a feature dimension for each attention head, denotes a feature dimension, denote a query vector, a key vector, and a value vector, respectively. The query vector and key vector are concatenated, and the attention weights are calculated as follows: | denotes a vector concatenation operation, is an attention parameter vector, is a node is a set of neighbor nodes of the node denotes an attention weight between the node i and the neighbor node j, denote a query vector and a key vector of the node i and the node j, respectively, denotes an activation function, exp is an exponential function, denotes a set of neighbor nodes of the node i; The attention weights are weighted and aggregated based on the features of neighboring nodes to obtain the graph attention output for each head: wherein, denotes the attention output of the h-th attention head, denotes the value vector of the neighbor node j ; The global attention feature obtained by concatenating multiple attention heads is as follows: wherein, represents global attention features, H represents the number of attention heads.

5. The method of claim 4, wherein, The shallow features, local enhancement features, and global attention features are fused to obtain fused features, including: The shallow features, local enhancement features, and global attention features are fused to obtain the fused features as follows: wherein denotes a fusion feature.

6. The method according to any one of claims 1 to 5, characterized in that, The feature update process of the bidirectional focusing enhancement module is as follows: wherein K represents the number of enhancement units, denotes a connection function, denotes a connection feature, denotes that the feature fusion module processes the input features denotes that the region focusing module processes the output of the feature fusion module, denotes that the region focusing module processes the output of the feature fusion module, denotes the kth feature.

7. A bidirectional focus+ enhancement based image super-resolution apparatus for implementing the bidirectional focus+ enhancement based image super-resolution method according to any one of claims 1 to 6, characterized in that, The device includes: The model construction module is configured to construct a bidirectional focus enhancement super-resolution model, wherein the bidirectional focus enhancement super-resolution model comprises an input module, a bidirectional focus enhancement module and an output module; the input module is configured to perform shallow feature extraction on an input low-resolution image to obtain shallow features; the bidirectional focus enhancement module comprises a plurality of cascaded enhancement units, each of which comprises a feature fusion module and a region focus module; the feature fusion module is configured to extract local information of the shallow features or fusion features output by a previous enhancement unit to obtain local enhancement features, introduce a multi-head attention mechanism to process the local enhancement features to obtain global attention features, and fuse the shallow features, the local enhancement features and the global attention features to obtain fusion features; the region focus module is configured to extract effective information of key regions contained in the fusion features to obtain fusion key features; and the output module is configured to reconstruct a super-resolution image according to the fusion key features output by a last enhancement unit in the bidirectional focus enhancement module. The image output module is configured to input a low-resolution image to be processed into the pre-trained bidirectional focus enhancement super-resolution model to output a super-resolution image.

Images