Image super-resolution method and device based on bidirectional focusing enhancement

Abstract

The invention relates to an image super-resolution method and device based on bidirectional focusing enhancement. The method comprises the steps that a bidirectional focusing enhancement module comprising a feature fusion module and a region focusing module is designed in a bidirectional focusing enhancement super-resolution model, and a graph nerve attention mechanism of local enhancement and context perception and a dynamic key region attention mechanism of the region focusing module are fused in the feature fusion module; through the two mechanisms, image features are enhanced and optimized from two dimensions of local details and global semantics, so that efficient reconstruction from a degraded low-resolution image to a high-quality high-resolution image is realized. By adopting the method, the quality and efficiency of super-resolution reconstruction can be effectively improved, so that the method can be better applied to an actual scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an image super-resolution method and device based on bidirectional focus enhancement. BACKGROUND

[0002] In today's era of rapid digital information development, image data, as an important carrier of information dissemination, plays a key role in many fields. Whether it is photo sharing, video calls in daily life, or medical imaging, satellite remote sensing, security monitoring in professional fields, the quality of the image directly relates to the accuracy and effectiveness of information transmission. However, in practical applications, due to the resolution limit of imaging devices, compression loss in the transmission process, poor shooting environment and other factors, the acquired images often have low resolution, blurred details and other problems, which seriously restricts the further application and value mining of images.

[0003] Image super-resolution technology emerges as the times require, and its core goal is to convert low-resolution images into high-resolution images through algorithms, thereby improving the visual quality of images and restoring more detailed information. This technology is of great significance and has a wide range of applications. In the field of security monitoring, super-resolution technology can convert blurred monitoring images into clear images, helping to more accurately identify key targets such as personnel and vehicles, and improving the effectiveness of security monitoring; in the field of medical imaging, by performing super-resolution reconstruction on low-resolution medical images, the details of tissues and organs can be observed more clearly, providing more accurate basis for disease diagnosis; in the field of satellite remote sensing, super-resolution technology can enhance the detailed resolution of remote sensing images, enabling the images of the Earth's surface obtained from space to more finely reflect the features of ground objects, providing higher quality data support for geographic information mapping, environmental monitoring, resource exploration, etc.; in the consumer electronics field, for low-resolution photos or videos taken by mobile phones, cameras and other devices, super-resolution technology can improve their clarity and improve users' visual experience.

[0004] Early super-resolution technologies mainly rely on traditional methods, such as interpolation-based algorithms. Interpolation methods generate high-resolution images by inserting intermediate pixels based on low-resolution images. Common methods include nearest neighbor interpolation, bilinear interpolation and bicubic interpolation. The advantages of these methods are simplicity of implementation and high computational efficiency, which can quickly enlarge images. However, they also have obvious limitations. Because interpolation algorithms mainly rely on the correlation between adjacent pixels to estimate the value of new pixels, they cannot capture more complex structure and texture information in images. In the reconstructed high-resolution images, there are often phenomena such as blurring, artifacts, and poor detail restoration, making it difficult to meet the requirements of high-quality image applications.

[0005] With the deepening of research in the field of computer vision, reconstruction-based methods have been gradually proposed. These methods attempt to recover high-resolution images by establishing image degradation models and utilizing prior knowledge of images. For example, some sparse representation-based methods assume that image patches can be represented as a sparse linear combination of a set of basis functions, and perform super-resolution reconstruction by learning the sparse representation relationship between low-resolution and high-resolution image patches. There are also gradient statistics-based methods that use the gradient prior information of high-resolution images to guide the reconstruction process. Although reconstruction-based methods can generate clearer images to some extent, they require high prior knowledge of images and involve complex optimization calculations in the solving process, which takes a long time to run and is difficult to meet the real-time requirements, limiting their widespread application in practical applications.

[0006] In recent years, with the rapid development of deep learning technology, convolutional neural networks (CNNs) have made breakthrough progress in the field of computer vision. Super-resolution methods based on CNNs can automatically learn the complex mapping relationship between low-resolution images and high-resolution images due to their powerful feature extraction and representation capabilities, thus achieving efficient reconstruction of images. For example, SRCNN (Super-Resolution Convolutional Neural Network), as one of the earliest deep learning-based super-resolution methods, pioneered the introduction of CNNs into the super-resolution reconstruction task, learned the feature representation of low-resolution images and mapped them to high-resolution image space, achieving better reconstruction results than traditional methods. Subsequently, researchers have continuously improved and optimized the network structure of CNNs, such as FSRCNN (Fast Super-Resolution Convolutional Neural Network), which adjusts the number of network layers and filter sizes to ensure reconstruction quality while improving network running speed; EDSR (Enhanced Deep Super-Resolution) further improves the performance of super-resolution models by deepening the network structure, increasing the number of channels, and adopting residual learning strategies. These CNN-based methods have achieved significant performance improvements on multiple benchmark datasets, becoming a research hotspot in the field of image super-resolution.

[0007] However, as the performance of CNN-based super-resolution methods continues to improve, new challenges arise. To further enhance the reconstruction accuracy of the models, many researchers have increased the number and complexity of network parameters to strengthen their expressive power, leading to a significant increase in computational and storage requirements. For example, some high-performance super-resolution models may contain millions or even tens of millions of parameters, placing extremely high demands on the computing power and memory capacity of hardware devices. In practical applications, such computationally intensive models are difficult to deploy effectively and run in real time on platforms with limited hardware performance, such as mobile devices and embedded systems. Taking mobile image super-resolution applications as an example, users want to quickly convert low-resolution photos into high-quality high-resolution images, but currently, some complex super-resolution models experience slow processing speeds or even fail to run on mobile devices due to computational resource limitations, seriously affecting user experience and the promotion of applications.

[0008] To address this issue, researchers have begun exploring lightweight super-resolution methods. These methods attempt to reduce the number of model parameters and computational complexity by designing simpler network structures, enabling efficient super-resolution reconstruction on low-performance devices. Common strategies include building shallow network structures, employing recursive network designs, and utilizing operations such as depthwise separable convolutions to reduce computational overhead. However, while these methods reduce the burden on the model, they often fail to fully utilize the deep expressive capabilities of CNNs, resulting in a significant gap in reconstruction performance compared to large, complex models, and still failing to meet the high image quality requirements of practical applications. Summary of the Invention

[0009] Therefore, it is necessary to provide an image super-resolution method and apparatus based on bidirectional focus enhancement that can simultaneously balance the clarity of image restoration and computational complexity, addressing the aforementioned technical problems.

[0010] An image super-resolution method based on bidirectional focusing enhancement, the method comprising: A bidirectional focusing enhanced super-resolution model is constructed. This model includes an input module, a bidirectional focusing enhancement module, and an output module. The input module extracts shallow features from the input low-resolution image to obtain shallow features. The bidirectional focusing enhancement module includes multiple cascaded enhancement units, each of which includes a feature fusion module and a region focusing module. The feature fusion module extracts local information from the shallow features or the fused features output by the previous enhancement unit to obtain local enhanced features. A multi-head attention mechanism is introduced to process the local enhanced features to obtain global attention features. The shallow features, local enhanced features, and global attention features are fused to obtain fused features. The region focusing module extracts effective information from key regions contained in the fused features to obtain fused key features. The output module reconstructs the super-resolution image based on the fused key features output by the last enhancement unit in the bidirectional focusing enhancement module. The low-resolution image to be processed is input into a pre-trained bidirectional focusing enhancement super-resolution model, which outputs a super-resolution image.

[0011] In one embodiment, the method further includes: the input module performing shallow feature extraction on the input low-resolution image to obtain the shallow features as follows:

[0012] Indicates shallow features. Indicates a low-resolution image. express Convolutional layer.

[0013] In one embodiment, the method further includes: applying the input shallow features or the fused features output by the previous enhancement unit to... Convolutional layers compress channel dimensions, through Convolution extracts local spatial information, and the ReLU activation function is fused during the process to obtain the local enhanced features: ; in, This represents shallow features or fused features output by the previous enhancement unit. express Convolutional layer express Convolutional layer This represents the ReLU activation function.

[0014] In one embodiment, the method further includes: projecting the local enhancement features onto the query vector, key vector, and value vector in the multi-head attention mechanism, respectively: ; wherein, is a trainable weight matrix, is a feature dimension of each attention head, denotes a feature dimension, denote query vector, key vector and value vector, respectively; the query vector and the key vector are spliced, and an attention weight is calculated as: ; denotes a vector splicing operation, is an attention parameter vector, is a neighbor node set of a node denotes an attention weight between the node i and the neighbor node j, denote query vector and key vector of the node i and the node j, respectively, denotes an activation function, and exp is an exponential function, denotes a neighbor node set of the node i; the attention weight is used to weight and aggregate neighbor node features, and a graph attention output of each head is obtained as: ;

[0015] wherein, denotes an attention output of the hth attention head; the global attention features are obtained by splicing multiple attention heads as: ; wherein, denotes global attention features, H denotes a number of attention heads.

[0016] In one of the embodiments, the method further comprises: fusing the shallow layer features, the local enhanced features and the global attention features to obtain fusion features as: ; wherein, denotes fusion features.

[0017] In one of the embodiments, the method further comprises: expanding the fusion features into an attention-form feature sequence, and calculating dense attention weights of the feature sequence as: ; wherein, is a feature dimension, denote query vector and key vector of the ith node in the feature sequence, respectively, denotes a dense attention weight between the node i and the node j, and N is a total number of nodes.​ According to the dense attention weight, a dense branch output is obtained as: ; wherein, denotes the dense branch output, is a value vector in a feature sequence, denotes the number of pixels; A sparse attention weight of the feature sequence is calculated as: ; wherein, denotes the sparse attention weight of node i and node j, denotes a neighbor node set of node i; According to the sparse attention weight, a sparse branch output is obtained as: ; wherein, denotes the sparse branch output; The dense branch output and the sparse branch output are weighted to obtain a fusion key feature as: ; wherein, denotes the fusion key feature, is a weighting coefficient.

[0018] In one of the embodiments, a feature updating process of the bidirectional focus enhancement module is: ; ; wherein, K denotes the number of enhancement units, denotes a connection function, denotes a connection feature, denotes that a feature fusion module processes an input feature , and denotes that a region focus module processes an output of the feature fusion module, denotes a kth feature.

[0019] An image super-resolution device based on bidirectional focus enhancement, the device comprising: The model construction module is configured to construct a bidirectional focus enhancement super-resolution model; the bidirectional focus enhancement super-resolution model comprises an input module, a bidirectional focus enhancement module and an output module; the input module is configured to perform shallow feature extraction on an input low-resolution image to obtain shallow features; the bidirectional focus enhancement module comprises a plurality of cascaded enhancement units, each of which comprises a feature fusion module and a regional focus module; the feature fusion module is configured to extract local information of the shallow features or fusion features output by a previous enhancement unit to obtain local enhancement features, introduce a multi-head attention mechanism to process the local enhancement features to obtain global attention features, and fuse the shallow features, the local enhancement features and the global attention features to obtain fusion features; the regional focus module is configured to extract effective information of key regions contained in the fusion features to obtain fusion key features; and the output module is configured to reconstruct a super-resolution image according to the fusion key features output by a last enhancement unit in the bidirectional focus enhancement module. The image output module is configured to input a low-resolution image to be processed into the pre-trained bidirectional focus enhancement super-resolution model and output a super-resolution image.

[0020] A computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program: The model construction module is configured to construct a bidirectional focus enhancement super-resolution model; the bidirectional focus enhancement super-resolution model comprises an input module, a bidirectional focus enhancement module and an output module; the input module is configured to perform shallow feature extraction on an input low-resolution image to obtain shallow features; the bidirectional focus enhancement module comprises a plurality of cascaded enhancement units, each of which comprises a feature fusion module and a regional focus module; the feature fusion module is configured to extract local information of the shallow features or fusion features output by a previous enhancement unit to obtain local enhancement features, introduce a multi-head attention mechanism to process the local enhancement features to obtain global attention features, and fuse the shallow features, the local enhancement features and the global attention features to obtain fusion features; the regional focus module is configured to extract effective information of key regions contained in the fusion features to obtain fusion key features; and the output module is configured to reconstruct a super-resolution image according to the fusion key features output by a last enhancement unit in the bidirectional focus enhancement module. The image output module is configured to input a low-resolution image to be processed into the pre-trained bidirectional focus enhancement super-resolution model and output a super-resolution image.

[0021] A computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the following steps: The bidirectional focus enhancement super-resolution model comprises an input module, a bidirectional focus enhancement module and an output module; the input module performs shallow feature extraction on an input low-resolution image to obtain shallow features; the bidirectional focus enhancement module comprises a plurality of cascaded enhancement units, each enhancement unit comprising a feature fusion module and a regional focus module; the feature fusion module extracts local information of the shallow features or fusion features output by a previous enhancement unit to obtain local enhancement features, introduces a multi-head attention mechanism to process the local enhancement features to obtain global attention features, and fuses the shallow features, the local enhancement features and the global attention features to obtain fusion features; the regional focus module extracts effective information of key regions contained in the fusion features to obtain fusion key features; and the output module reconstructs a super-resolution image according to the fusion key features output by a last enhancement unit in the bidirectional focus enhancement module. The low-resolution image to be processed is input into the pre-trained bidirectional focus enhancement super-resolution model, and a super-resolution image is output.

[0022] The bidirectional focus enhancement super-resolution model comprises an input module, a bidirectional focus enhancement module and an output module; the input module performs shallow feature extraction on an input low-resolution image to obtain shallow features; the bidirectional focus enhancement module comprises a plurality of cascaded enhancement units, each enhancement unit comprising a feature fusion module and a regional focus module; the feature fusion module extracts local information of the shallow features or fusion features output by a previous enhancement unit to obtain local enhancement features, introduces a multi-head attention mechanism to process the local enhancement features to obtain global attention features, and fuses the shallow features, the local enhancement features and the global attention features to obtain fusion features; the regional focus module extracts effective information of key regions contained in the fusion features to obtain fusion key features; and the output module reconstructs a super-resolution image according to the fusion key features output by a last enhancement unit in the bidirectional focus enhancement module. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 A flowchart of the bidirectional focus enhancement-based image super-resolution method in one embodiment; Figure 2 A framework diagram of the bidirectional focus enhancement super-resolution model in one embodiment; Figure 3 A framework diagram of the feature fusion module in another embodiment; Figure 4 A structural block diagram of the bidirectional focus enhancement-based image super-resolution device in one embodiment; Figure 5 An internal structure diagram of a computer device in one embodiment. DETAILED DESCRIPTION

[0024] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application and not to limit the present application.

[0025] In one embodiment, as shown in Figure 1 A bidirectional focus enhancement based image super-resolution method is provided, comprising the following steps: Step 102, constructing a bidirectional focus enhancement super-resolution model.

[0026] The bidirectional focus enhancement super-resolution model comprises an input module, a bidirectional focus enhancement module and an output module; the input module performs shallow feature extraction on the input low-resolution image to obtain shallow features; the bidirectional focus enhancement module comprises a plurality of cascaded enhancement units, each enhancement unit comprising a feature fusion module and a regional focus module; the feature fusion module extracts local information of the shallow features or fusion features output by the previous level enhancement unit to obtain local enhanced features, introduces a multi-head attention mechanism to process the local enhanced features to obtain global attention features, and fuses the shallow features, the local enhanced features and the global attention features to obtain fusion features; the regional focus module extracts effective information of key regions contained in the fusion features to obtain fusion key features; and the output module reconstructs a super-resolution image according to the fusion key features output by the last level enhancement unit in the bidirectional focus enhancement module.

[0027] Step 104, inputting a low-resolution image to be processed into the pre-trained bidirectional focus enhancement super-resolution model to output a super-resolution image.

[0028] In the bidirectional focus enhancement based image super-resolution method described above, a bidirectional focus enhancement module comprising a feature fusion module and a regional focus module is designed in the bidirectional focus enhancement super-resolution model, wherein the feature fusion module fuses local enhancement and context-aware graph neural attention mechanisms, and the regional focus module has a dynamic key region attention mechanism; through these two mechanisms, image features are enhanced and optimized from two dimensions of local details and global semantics, thereby realizing efficient reconstruction from a degraded low-resolution image to a high-quality high-resolution image. The above method aims to maintain the lightweight of the model, effectively improve the quality and efficiency of super-resolution reconstruction by synergistically fusing local feature enhancement and global semantic focusing mechanisms, and make it better applied to actual scenarios, especially in resource-constrained environments such as mobile devices.

[0029] In one embodiment, as shown in Figure 2As shown, the overall framework diagram of the bidirectional focus enhancement super-resolution model is provided, the input module performs shallow feature extraction on the input low-resolution image to obtain shallow features:

[0030] representing shallow features, representing low-resolution images, representing convolutional layers.

[0031] In one embodiment, to improve the fusion ability between local details and global semantics in the feature map, a graph neural attention mechanism that fuses local enhancement and context awareness, called Local Enhancement and Context-Aware Graph Attention Network (LECAG), is designed, which is a feature fusion module. The core idea of this module is: use convolution operation to enhance the ability of local spatial continuity feature modeling, and introduce multi-head graph attention mechanism to capture long-distance semantic dependence and context association, and realize efficient collaborative expression of local-global features.

[0032] Specifically, as shown in Figure 3 , a model framework diagram of a feature fusion module is provided, and the input feature map is , where represents the number of nodes in the graph, represents the feature dimension. For the input shallow feature or the fusion feature output by the previous enhancement unit, a convolutional layer is used to compress the channel dimension, and a convolution is used to extract local spatial information, and a ReLU activation function is fused in the process, and the local enhancement feature is: ; wherein, represents the shallow feature or the fusion feature output by the previous enhancement unit, represents convolutional layers, represents convolutional layers, represents a ReLU activation function.

[0033] Specifically, the ReLU activation function improves the nonlinear fitting ability, is used for dimension reduction and linear transformation, is used to extract context features within the local receptive field, and the final is an intermediate feature with local enhancement capability.

[0034] In one of the instances, on the basis of local modeling, a multi-head graph attention mechanism is introduced to further introduce non-local modeling capability, which is used to capture long-distance dependency between nodes.

[0035] Specifically, the local enhanced features are projected into the query vector, key vector and value vector in the multi-head attention mechanism as follows: ; wherein, is a trainable weight matrix, is the feature dimension of each attention head, denotes the feature dimension, denote the query vector, the key vector and the value vector respectively, the query vector and the key vector are spliced, and the attention weight is calculated as: ; denotes the vector splicing operation, is an attention parameter vector, is a neighbor node set of node , denotes the attention weight between node i and neighbor node j, denotes the query vector and the key vector of node i and node j respectively, denotes an activation function, and exp is an exponential function, denotes the neighbor node set of node i, the attention weight is weighted and aggregated on the neighbor node feature, and the graph attention output of each head is obtained as: ;

[0036] wherein, denotes the attention output of the hth attention head; the global attention feature is obtained by splicing multiple attention heads as: ; wherein, denotes the global attention feature. Wherein is the number of graph attention heads. By introducing the multi-head mechanism, the modeling capability of different attention paths can be improved, and the diversity expression of the feature space can be enhanced.

[0037] In one of the embodiments, the shallow layer feature, the local enhanced feature and the global attention feature are fused to obtain a fused feature as: ; wherein, ​represent the fused features. Such fusion strategy not only preserves the original semantics in the input features, but also supplements the local spatial continuity and non-local context dependency, making the output features more discriminative and structurally expressive. The design of the entire LECAG module is simple and effective, and is suitable for task scenarios in graph structure modeling that are sensitive to both local details and global information.

[0038] In one of the embodiments, in computer vision tasks such as image restoration and reconstruction, although the standard self-attention mechanism (Self-Attention) relied on by the traditional Transformer has strong global modeling capability, its computational complexity is O(N2), which often brings huge resource overhead when processing high-resolution images, and the attention is calculated uniformly for all image regions, which easily introduces redundant interference information and is not conducive to accurately extracting effective features from degraded images. To overcome the above problems, a dynamic key region attention mechanism (DKA, region focusing module) is proposed, which effectively improves the efficiency and discriminative ability of the attention mechanism in the image degradation modeling process by introducing a sparse-dense parallel structure and a dynamic sparse selection strategy.

[0039] Specifically, the fused features are expanded into a feature sequence in the form of attention, and the dense attention weight of the feature sequence is calculated as follows: ; wherein, is the feature dimension, are the query vector of the i-th node and the key vector of the j-th node in the feature sequence, denotes the dense attention weight between node i and node j, and N is the total number of nodes; according to the dense attention weight, the dense branch output is obtained as follows: ; wherein, denotes the dense branch output, is the value vector in the feature sequence, denotes the number of pixels; on the other hand, in order to reduce the computational load and highlight the key region modeling, a sparse attention branch is introduced, which only selects the first similar key vectors to form a local candidate set for each query vector, and the sparse attention weight of the feature sequence is calculated as follows: ; wherein, denotes the sparse attention weight of node i and node j; it is worth noting that the selection of the local candidate set is dynamically adaptive based on the input features, that is, the sparse neighborhood is variable for different input images or different spatial positions. This selection not only guarantees the preservation of the high response region, but also effectively suppresses the influence of irrelevant interference regions such as raindrops, scratches, and fog in degraded images.

[0040] According to the sparse attention weight, the sparse branch output is obtained as: ; wherein, denotes the sparse branch output; the dense branch output and the sparse branch output are weighted to obtain the fused key feature as: ; wherein, denotes the fused key feature, is a weighting coefficient.

[0041] wherein the fusion coefficient can be set as a fixed value, or can be adaptively learned by the network to adapt to the complexity of different tasks or input features. This fusion mechanism achieves a good trade-off between precision and efficiency: the dense branch ensures the global dependence modeling capability, and the sparse branch greatly compresses the computational amount and improves the robustness, especially in high-resolution, low-quality input images, it shows stronger discriminative ability and generalization ability.

[0042] In one of the embodiments, the feature updating process of the bidirectional focus enhancement module is as follows: ; ; wherein K denotes the number of enhancement units, denotes the connection function, denotes the connection feature, denotes the input feature processed by the feature fusion module, denotes the output of the feature fusion module processed by the region focus module, denotes the kth feature.

[0043] It should be understood that although each step in the flowchart of Figure 1 is displayed in sequence according to the direction of the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, there is no strict order limitation for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1At least one of the steps in the method can comprise a plurality of sub-steps or stages which are not necessarily performed at the same time but can be performed at different times and in which case the order of the sub-steps or stages can be varied.

[0044] In one embodiment, as shown in Figure 4 A bidirectional focus enhancement based image super-resolution device is provided, comprising: a model construction module 402 and an image output module 404, wherein: The model construction module 402 is configured to construct a bidirectional focus enhancement super-resolution model, wherein the bidirectional focus enhancement super-resolution model comprises an input module, a bidirectional focus enhancement module, and an output module; the input module is configured to perform shallow feature extraction on an input low-resolution image to obtain shallow features; the bidirectional focus enhancement module comprises a plurality of cascaded enhancement units, each enhancement unit comprising a feature fusion module and a regional focus module; the feature fusion module is configured to extract local information of the shallow features or fusion features output by a previous-level enhancement unit to obtain local enhancement features, introduce a multi-head attention mechanism, process the local enhancement features to obtain global attention features, and fuse the shallow features, the local enhancement features, and the global attention features to obtain fusion features; the regional focus module is configured to extract effective information of key regions contained in the fusion features to obtain fusion key features; and the output module is configured to reconstruct a super-resolution image based on the fusion key features output by a last-level enhancement unit in the bidirectional focus enhancement module. The image output module 404 is configured to input a low-resolution image to be processed into a pre-trained bidirectional focus enhancement super-resolution model and output a super-resolution image.

[0045] In one embodiment, the model construction module 402 is further configured to perform shallow feature extraction on an input low-resolution image by the input module to obtain shallow features as follows:

[0046] wherein x represents the shallow features, wherein x represents the low-resolution image, wherein x represents a convolution layer.

[0047] In one embodiment, the model construction module 402 is further configured to compress the channel dimension by a convolution layer on the input shallow features or fusion features output by a previous-level enhancement unit, extract local spatial information by a convolution, fuse a ReLU activation function in the process, and obtain local enhancement features as follows: ; wherein, denotes the shallow feature or the fused feature output by the upper-level enhancement unit, denotes a convolutional layer, denotes a convolutional layer, denotes a ReLU activation function.

[0048] In one of the embodiments, the model construction module 402 is further configured to project the local enhanced features into query vectors, key vectors and value vectors in the multi-head attention mechanism respectively as follows: ; wherein, is a trainable weight matrix, is the feature dimension of each attention head, denotes the feature dimension, denote the query vector, the key vector and the value vector respectively; the query vector and the key vector are spliced, and the attention weight is calculated as follows: ; denotes a vector splicing operation, is an attention parameter vector, is a neighbor node set of a node , denotes the attention weight between the node i and the neighbor node j, denote the query vector and the key vector of the node i and the node j respectively, denotes an activation function, and exp is an exponential function, denotes the neighbor node set of the node i; the attention weight is used to weight and aggregate the neighbor node features to obtain the graph attention output of each head as follows: ; wherein, denotes the attention output of the hth attention head; the global attention features are obtained by splicing multiple attention heads as follows: ; wherein, denotes the global attention features, denotes the number of attention heads. H In one of the embodiments, the model construction module 402 is further configured to fuse the shallow features, the local enhanced features and the global attention features to obtain the fused features as follows:

[0049] ​ ; wherein, denotes the fusion feature.

[0050] In one of the embodiments, the model construction module 402 is further configured to expand the fusion feature into a feature sequence in the form of attention, and calculate dense attention weights of the feature sequence as: ; wherein, is a feature dimension, are a query vector of an i-th node and a key vector of a j-th node in the feature sequence respectively, denotes a dense attention weight between the node i and the node j, and N is a total number of nodes; According to the dense attention weights, a dense branch output is obtained as: ; wherein, denotes the dense branch output, is a value vector in the feature sequence; The sparse attention weights of the feature sequence are calculated as: ; wherein, denotes a sparse attention weight between the node i and the node j, denotes a neighbor node set of the node i; According to the sparse attention weights, a sparse branch output is obtained as: ; wherein, denotes the sparse branch output; The dense branch output and the sparse branch output are weighted to obtain a fusion key feature as: ; wherein, denotes the fusion key feature, is a weighting coefficient.

[0051] In one of the embodiments, the feature updating process of the bidirectional focusing enhancement module is as follows: ; ; wherein, K denotes a number of enhancement units, denotes a connection function, denotes a connection feature, denotes that the feature fusion module processes an input feature ; The area focusing module processes the output of the feature fusion module, The kth feature is represented.

[0052] The specific definitions of the image super-resolution device based on bidirectional focus enhancement can refer to the definitions of the image super-resolution method based on bidirectional focus enhancement in the above, which will not be repeated here. Each module in the above image super-resolution device based on bidirectional focus enhancement can be realized by software, hardware and their combinations in whole or in part. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to call and execute the operations corresponding to the above modules by the processor.

[0053] In one embodiment, a computer device, which can be a server, is provided, and an internal structure diagram of the computer device can be as shown in Figure 5 The computer device includes a processor, a memory, a network interface and a database connected by a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the computer device is used to store low-resolution image data to be recovered. The network interface of the computer device is used to communicate with external terminals through network connection. The computer program is executed by the processor to implement an image super-resolution method based on bidirectional focus enhancement.

[0054] Those skilled in the art can understand that Figure 5 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0055] In one embodiment, a computer device is provided, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the method in the above embodiments.

[0056] In one embodiment, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the steps of the method in the above embodiments.

[0057] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, databases, or other media in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0058] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0059] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A bidirectional focus enhancement-based image super-resolution method, characterized in that, The method comprises: The method comprises: The input module extracts shallow features from the input low-resolution image, and the shallow features are obtained.

2. The method of claim 1, wherein, The input module extracts shallow features from the input low-resolution image, and the shallow features are obtained. The local information of the shallow features or the fusion features output by the upper-level enhancement unit is extracted to obtain local enhanced features, and the local enhanced features are processed by introducing a multi-head attention mechanism to obtain global attention features. representing shallow features, representing low resolution images, representing convolutional layers.

3. The method of claim 1, wherein, The local enhanced features are projected into query vectors, key vectors and value vectors in the multi-head attention mechanism. For the input of the shallow features or the fusion features output by the upper level enhancement unit, the method adopts The convolution layer compresses the channel dimension, and obtains the local enhancement features by The convolution extracts the local spatial information, and the ReLU activation function is fused in the process, so as to obtain the local enhancement features wherein, denotes a fusion feature of the shallow feature or the output of the upper level enhancement unit, denotes a convolutional layer, denotes a convolutional layer, denotes a ReLU activation function.

4. The method of claim 3, wherein, The query vectors and the key vectors are spliced to calculate attention weights. The neighbor node features are weighted and aggregated according to the attention weights to obtain the graph attention output of each head. wherein, is a trainable weight matrix, is a feature dimension for each attention head, denotes a feature dimension, denote a query vector, a key vector, and a value vector, respectively; The global attention features are obtained by splicing multiple attention heads. | denotes a vector concatenation operation, is an attention parameter vector, is a node set of neighbor nodes of node i, denotes an attention weight between node i and neighbor node j, , denote the query vector and key vector of node i and node j, respectively, denotes an activation function, exp is the exponential function, denotes a set of neighbor nodes of node i; The shallow features, the local enhanced features and the global attention features are fused to obtain fusion features. wherein, represents the attention output of the h-th attention head; The shallow features, the local enhanced features and the global attention features are fused to obtain fusion features. wherein, represents global attention features, H represents the number of attention heads.

5. The method of claim 4, wherein, The effective information of the fusion features containing key regions is extracted to obtain fusion key features. The fusion features are unfolded into an attention-form feature sequence, and dense attention weights of the feature sequence are calculated. wherein represents a fusion feature.

6. The method according to any one of claims 1 to 5, characterized in that, The dense branch output is obtained according to the dense attention weights. Sparse attention weights of the feature sequence are calculated. wherein, is a feature dimension, , are a query vector of the i-th node and a key vector of the j-th node in the feature sequence, respectively, denotes a dense attention weight between node i and node j, and N is the total number of nodes. The sparse branch output is obtained according to the sparse attention weights. wherein, denotes a dense branch output, is a value vector of node j in the feature sequence; The dense branch output and the sparse branch output are weighted to obtain the fusion key features. wherein, denotes a sparse attention weight of node i and node j, denotes a set of neighbor nodes of node i; The feature updating process of the bidirectional focus enhancement module is: wherein denotes a sparse branch output; The device comprises: wherein, denotes the fusion of key features, is a weighting coefficient.

7. The method according to any one of claims 1 to 5, characterized in that, The device comprises: wherein K represents the number of enhancement units, denotes a connection function, denotes a connection feature, denotes that the feature fusion module processes the input features denotes that the region focusing module processes the output of the feature fusion module, denotes that the region focusing module processes the output of the feature fusion module, denotes the kth feature.

8. An image super-resolution device based on bidirectional focus enhancement, characterized in that, ​ The model construction module is configured to construct a bidirectional focus enhancement super-resolution model, wherein the bidirectional focus enhancement super-resolution model comprises an input module, a bidirectional focus enhancement module and an output module; the input module is configured to perform shallow feature extraction on an input low-resolution image to obtain shallow features; the bidirectional focus enhancement module comprises a plurality of cascaded enhancement units, each of which comprises a feature fusion module and a region focus module; the feature fusion module is configured to extract local information of the shallow features or fusion features output by a previous enhancement unit to obtain local enhancement features, introduce a multi-head attention mechanism to process the local enhancement features to obtain global attention features, and fuse the shallow features, the local enhancement features and the global attention features to obtain fusion features; the region focus module is configured to extract effective information of key regions contained in the fusion features to obtain fusion key features; and the output module is configured to reconstruct a super-resolution image according to the fusion key features output by a last enhancement unit in the bidirectional focus enhancement module. The image output module is configured to input a low-resolution image to be processed into the pre-trained bidirectional focus enhancement super-resolution model to output a super-resolution image.

Images