Image arbitrary-scale super-division method based on block matching and dynamic sparse attention

Abstract

The invention discloses an image arbitrary-scale super-division method based on block matching and dynamic sparse attention, and the method specifically comprises the steps: 1, image feature extraction and aggregation: extracting image block features based on a coding network, and aggregating local context information into potential codes through feature expansion and splicing operation; 2, non-local feature extraction is carried out based on block matching or dynamic sparse attention; step 3, non-local feature fusion is carried out based on coordinate attention, and pixel value prediction is carried out; and carrying out pixel value prediction on the pixel value of each point in the high-resolution spatial scale according to the three steps, and reconstructing a high-resolution image. According to the method, organic unification of local detail keeping and global semantic modeling is realized, the problems of fuzzy details, structural distortion, long-range dependency loss and the like in any-scale reconstruction of a traditional super-resolution method are effectively solved, and the visual quality and geometric consistency of super-resolution reconstruction are remarkably improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to an image arbitrary scale super-resolution reconstruction method based on block matching and dynamic sparse attention. BACKGROUND

[0002] Image super-resolution reconstruction is one of the core tasks in the field of computer vision and image processing, and the goal is to restore a high-resolution version from a low-resolution image to enhance the detail information and visual quality of the image. With the popularity of high-resolution display devices and the increasing demand for professional image analysis such as remote sensing, medical treatment, and historical images, the importance of super-resolution technology in practical applications is increasingly prominent. In image downstream application fields such as target detection and identification, the resolution of the image directly affects the accuracy and effect of the downstream task. For example, in the task of intelligent identification of goods in a self-service vending cabinet, the image obtained by the camera of the vending cabinet usually has low resolution, which affects the accuracy of identification. Therefore, before performing the identification of goods, super-resolution processing needs to be performed. Traditional methods usually train a model independently for each specific scale, which is not only inefficient, but also difficult to adapt to flexible and variable application scenarios. In recent years, the arbitrary scale super-resolution technology based on implicit field has provided a new idea to solve this problem.

[0003] Methods based on implicit representation, such as LIIF (Local Implicit Image Function), map image coordinates to corresponding RGB pixel values through a continuous function, realizing continuous reconstruction of a single model supporting arbitrary scaling factors. This paradigm effectively reduces the burden of multi-model storage and exhibits excellent scale extrapolation capability. However, existing methods still have obvious bottlenecks in reconstruction quality. The core problem lies in that they mainly rely on local neighborhood features to predict target pixels, lacking the ability to model the overall semantic structure and long-distance dependency of the image. For example, when processing images with regular repeated textures (such as building window grids, repeated patterns on goods) or complex structural content (such as facial features), such "local implicit" models are difficult to utilize similar patterns in other areas of the image to assist in reconstruction, easily leading to phenomena such as texture blurring, detail distortion, or structural incoherence in the output results.

[0004] To make up for the shortcomings of local modeling, existing research attempts to introduce global attention mechanisms such as Transformers into the super-resolution task. Although this type of method can capture global context, its computational complexity grows with the square of the image size, and it faces severe efficiency challenges when processing larger scale images, making it difficult to be put into practical application. In addition, the standard attention mechanism often ignores the key prior information of the spatial position of the pixels when calculating the feature similarity, which may lead to misalignment or distortion when reconstructing geometric structures.

[0005] Therefore, the key to improving the performance of this type of method lies in how to efficiently and accurately introduce and integrate global information while maintaining the advantages of implicit field reconstruction at any scale. Summary of the Invention

[0006] To overcome the shortcomings of existing methods, this invention proposes an arbitrary-scale super-resolution image reconstruction method based on block matching and dynamic sparse attention. This invention aims to construct a novel network architecture that integrates local implicit modeling and non-local feature enhancement. By introducing an efficient non-local information mining and fusion mechanism, it significantly enhances the model's ability to recover the global structure and details of images while ensuring computational efficiency, thus overcoming the challenge of insufficient super-resolution reconstruction quality in complex scenes using existing methods.

[0007] First, based on the LIFT implicit field super-resolution reconstruction theory, this invention utilizes an EDSR (Enhanced Deep Super-Resolution Network) encoding network to extract image features and aggregates local context through feature expansion and concatenation operations to enhance the detailed representation of features. Subsequently, to overcome the limitations of local implicit modeling, this invention proposes a non-local feature extraction method based on block matching. By calculating global similarity, the most relevant semantic information is introduced into each local point to enhance the model's ability to model long-distance dependencies. Furthermore, to adapt to large-scale super-resolution reconstruction, this invention proposes a hierarchical dynamic sparse attention mechanism to achieve efficient non-local feature extraction, further improving computational efficiency while mining more effective global information. Finally, this invention performs non-local feature fusion based on coordinate attention. Pixel coordinates are introduced as explicit priors in the standard attention mechanism. By comprehensively measuring feature similarity and positional differences, more robust feature fusion is achieved. A trainable implicit function is used to predict the pixel value of the query point, thereby realizing super-resolution reconstruction of the entire image.

[0008] The technical solution of this invention is: an image super-resolution method based on block matching and dynamic sparse attention, comprising the following steps: Step 1, Image Feature Extraction and Aggregation: Based on the coding network, image block features are extracted, and then local contextual information is aggregated into the latent code through feature expansion and stitching operations; Step 2: Extract nonlocal features based on block matching or dynamic sparse attention; Step 3: Perform non-local feature fusion based on coordinate attention and predict pixel values; The pixel value of each point in the high-resolution spatial scale is predicted according to the above three steps to reconstruct the high-resolution image.

[0009] Furthermore, the image block features extracted by the coding network are represented as follows:

[0010] wherein, represents a convolutional layer encoding the network, denotes a residual connection layer without normalization, and M represents the latent encoding output.

[0011] Further, the new feature obtained by the feature expansion and concatenation operation is denoted as:

[0012] wherein, M represents the latent encoding output of the feature extraction, denotes the feature vector at coordinates on the new feature map, and Concat denotes the concatenation operation along the channel dimension; -1, 0, 1 represent the position offset relative to the center point (i, j), specifically, -1 represents a unit translation to the left along the x-axis or up along the y-axis; 0 represents no translation; 1 represents a unit translation to the right along the x-axis or down along the y-axis.

[0013] Further, the non-local feature extraction based on block matching is to calculate the similarity index between feature blocks in the full image range, and then select a number of non-local positions most relevant to each local feature block according to the similarity, and finally only perform attention operation in the most relevant area, calculate the most relevant reference image position for each position; after finding the most similar image block, input this image block into the offset network to generate offset information, and add the original position to obtain more accurate spatial position information of similar features, then select the corresponding texture features from the image feature extraction and aggregated features, introduce attention mechanism to calculate texture confidence as attention weight, integrate texture features and encoding features as the non-local features finally obtained by block matching.

[0014] Further, the non-local features finally obtained by block matching are denoted as :

[0015] wherein, denotes a convolution operation, denotes feature concatenation, denotes element-wise multiplication, S refers to the texture confidence at all positions, T is the overall texture feature after migration, and F is the feature obtained by image feature extraction and aggregation.

[0016] Further, the specific implementation of non-local feature extraction based on dynamic sparse attention includes: Feature encoding and downsampling: Multiple downsampling-based feature extraction modules are used to downsample the spatial dimension of the features obtained from image feature extraction and aggregation. The downsampling-based feature extraction modules include a CNN convolutional network, a ReLU activation function, and a BN layer. Similarity calculation and hierarchical mapping: First, initial self-attention matching is performed at the smallest scale. Then, for each query position, a set of points most relevant to the query point is selected, and then a layer-by-layer block mapping operation is performed to obtain the first... l Attention scores of similar points between the layer and the query point and the first l front of the layer A set of similar points; Soft attention feature fusion: Extract the soft attention weight matrix and global candidate features from the selected set of points, and concatenate the candidate features with the query features, as shown in the following formula:

[0017]

[0018]

[0019] in, This represents several features that are most relevant to the query point at level 1. Attention scores representing these features This indicates the first feature extraction module based on downsampling. The attention score represents the similarity score between the first layer and the query point. Represents the front of the first layer A set of similar points Indicates query characteristics, The concatenated features are represented by the pre-calculated embedding similarity. These features are then fed into a lightweight soft attention module for soft attention aggregation. This process can be formalized as follows:

[0020] Where Conv represents the convolution operation. This represents element-wise multiplication. This represents the nonlocal features of the final output.

[0021] Furthermore, the specific implementation methods for similarity calculation and hierarchical mapping are as follows: In the lowest layer feature map Above, for each query point When performing self-attention operations, the attention similarity matrix is ​​represented by the following formula:

[0022] in, , , , represent the attention scores of other positions to the query point, H3 and W3 represent the length and width of the feature map F3; Then, the top points with the highest similarity are selected for each query position:

[0023] represent the point set with the highest relevance to the query point on the minimum scale feature map, represent the top similar points to form the most relevant point set; Next, the layer-by-layer block mapping operation is performed: each matching point is mapped to the 4 pixels in the corresponding 2x2 region of the upper level, and for these new candidate points, the similarity embedding operation is repeated and the top most relevant points are extracted, as follows:

[0024]

[0025] wherein, , , respectively represent the query feature and the key feature of the i-th layer, l l denotes the feature channel of the i-th layer, C l and l l+1 denote the key features of the i-th layer and the i+1-th layer, respectively; K represent the set of the expanded 2x range mapped by the i+1-th layer, K represent the attention scores of the i-th layer to the query point, l represent the top l most relevant point set in the i-th layer. l Further, the specific implementation of step three includes: Coordinate attention mechanism: for a given query coordinate in the high-resolution HR space l , select the known low-resolution pixels in the local 2x2 region centered on as a local set, and denote the coordinates of the key vector Key and the value vector Value in the local neighborhood as l ,

[0026] Further, the specific implementation of step three includes: Coordinate attention mechanism: for a given query coordinate in the high-resolution HR space , select the known low-resolution pixels in the local 2x2 region centered on as a local set, and denote the coordinates of the key vector Key and the value vector Value in the local neighborhood as , ​​The corresponding latent encoding is denoted as Q, K, V; based on the coordinates and the latent encoding pair (Q, ), (K, ), (V, ), the size of each pixel grid is c, and the coordinate attention mechanism is defined to predict the RGB value at the query position The specific formula is as follows:

[0027] Wherein represents the RGB value at the query position , is a multi-layer perceptron, representing an implicit function of pixel value prediction, and Attention() refers to feature fusion based on coordinate attention; Non-local feature fusion and pixel value prediction: replace the value vector Value in the attention mechanism with the non-local feature vector obtained through block matching or dynamic sparse attention to obtain more rich context information, which is no longer limited to the feature response in the local area; Specifically, for each local feature point in the 2x2 neighborhood near the query point, more non-local information is fused through the block matching attention mechanism or the dynamic sparse attention mechanism, thereby realizing non-local feature fusion and pixel value prediction, and the specific calculation formula is as follows:

[0028] Wherein is a query network, i.e. MLP, denotes a Softmax function, denotes a local 2x2 region centered on the query point , Q , and are the query, key and value respectively, and are formalized as:

[0029] Wherein and are network modules for generating keys and values respectively, is the resolution feature on the nearest neighbor grid corresponding to the query coordinate , is the local feature extracted from the latent encoding through the unfolding operation, which is equivalent to being centered on i, j and spliced within its neighborhood 3x3, denotes the non-local feature corresponding to the position i, j ; the parameter denotes the size of the image grid, which is used to introduce unit decoding information in feature modeling; and respectively represent the relative coordinate distance of the key and value in the local area, and the specific formula is as follows: .

[0030] The application also provides an image arbitrary scale super-resolution system based on block matching and dynamic sparse attention, comprising a processor and a memory, the memory is used for storing program instructions, and the processor is used for calling the program instructions in the memory to execute the image arbitrary scale super-resolution method based on block matching and dynamic sparse attention as described in the above technical solution.

[0031] The application also provides a computer readable storage medium, comprising a readable storage medium, and the readable storage medium stores a computer program, and the computer program is executed to realize the image arbitrary scale super-resolution method based on block matching and dynamic sparse attention as described in the above technical solution.

[0032] The application realizes the organic unification of local detail preservation and global semantic modeling by synergistically utilizing the advantages of implicit field on continuous representation and the non-local feature enhancement mechanism, effectively overcomes the problems of detail blur, structure distortion and long-range dependence loss existing in the traditional super-resolution method in arbitrary scale reconstruction, and significantly improves the visual quality and geometric consistency of super-resolution reconstruction. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 The processing flowchart of the image super-resolution reconstruction method provided by the application; Figure 2 The image feature extraction network structure used by the application is shown in the schematic diagram; Figure 3 The flowchart of the non-local feature extraction method based on block matching provided by the application is shown in the schematic diagram; Figure 4 The flowchart of the non-local feature extraction method based on sparse attention provided by the application is shown in the schematic diagram; Figure 5 The query point pixel value prediction based on feature fusion provided by the application is shown in the schematic diagram, (a) pixel value prediction based on coordinate attention mechanism, (b) pixel value prediction based on non-local feature fusion and coordinate attention; Figure 6 The comparison diagram of the results of the method and other methods on visual image super-resolution reconstruction is shown in the schematic diagram. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical scheme and advantages of the application more clear, the application will be further described in detail below with reference to the drawings. It should be understood that the implementation examples described herein are only used to illustrate and explain the application, and are not used to limit the application.

[0035] Example 1 An image arbitrary scale super-resolution reconstruction method based on block matching and dynamic sparse attention is provided, and the overall process is as shown in the figure. Figure 1 The method takes a low-resolution image as input, aims to output a high-resolution image with rich details and clear structure, and supports any scaling factor. Specifically, the present application first extracts the deep feature map of the input image using the EDSR-based encoding network; then, through feature unfolding and channel splicing operations, local neighborhood information is aggregated into the latent code of each target point to enhance its detail representation capability. To introduce global context information, the present application then performs non-local feature extraction based on block matching, finds the most relevant semantic blocks for each query point in the entire image, and performs feature fusion. For the efficiency requirement of high-resolution image reconstruction, the present application further proposes a hierarchical dynamic sparse attention mechanism to filter key information nodes at multiple scales to approximate the effect of global attention while greatly reducing the computational complexity. Finally, in the feature decoding and pixel prediction stage, the present application uses an attention mechanism that integrates coordinate priors to weight and fuse non-local features, and through a lightweight implicit function network, outputs the final high-resolution pixel values according to the query coordinates. The technical scheme of the present application includes the following steps: Step 1, image feature extraction and aggregation. The present application designs a feature extraction module based on encoding network and local context aggregation. The module takes a low-resolution image as input, aims to extract deep features with rich detail representation capability, and provides high-quality latent code for subsequent arbitrary scale super-resolution reconstruction. It specifically includes the following sub-steps: Step 1.1, feature extraction. Similar to the LIIF feature extraction method, the present application uses EDSR without upsampling module as the encoding network. EDSR itself is an excellent image super-resolution method, and its encoding network is widely used in various super-resolution models as a key feature extraction module. EDSR builds a deep residual network and removes the standard Batch Normalization (BN) layer, enhancing image detail preservation and improving model training stability. Its core consists of multiple residual blocks without normalization operations, which can efficiently extract deep semantic features from low-resolution images. In the super-resolution task, the EDSR encoding network can provide rich and stable feature representation for subsequent modules, laying a solid foundation for various implicit modeling and arbitrary scale reconstruction methods. The network structure of EDSR is as shown in the figure. Figure 2 Since the present application only uses the encoding module of the network, the final upsampling and convolution operations are discarded, and only the residual blocks and convolution operations before them are retained. Specifically, the low-resolution image I The process of obtaining deep features through the EDSR encoding network is formalized as: (1) wherein, denotes the input low-resolution image, denotes the convolutional layer of the encoding network, and denotes the residual connection layer without normalization with the same structure, and M denotes the output latent encoding.

[0036] Step 1.2, feature expansion. In order to aggregate the context information of the local neighborhood to the target point, the deep feature map extracted in step 1.1 is expanded and reorganized in the spatial dimension in this step. Specifically, for an input RGB image with a size of HxWx3, after being extracted by the EDSR encoding network, a deep feature map M with a size of HxWx64 is obtained, which aggregates deep semantic information of the image. In order to enrich the local context contained in each latent encoding of the feature map M, the feature expansion operation is performed on it to generate a new enhanced feature map F. In this process, the latent encoding of each target point on the new feature map F is obtained by concatenating all feature vectors in the 3x3 neighborhood of the corresponding position point on the previous feature M along the channel dimension. The specific calculation formula of this operation is as follows: (2) wherein, denotes the feature vector at the coordinate on the new feature map, Concat denotes the concatenation operation along the channel dimension, and -1, 0, 1 denote the position offset relative to the center point (i, j). Specifically, -1 denotes a left (up) shift of one unit along the x-axis (or y-axis); 0 denotes no shift; and 1 denotes a right (down) shift of one unit along the x-axis (or y-axis). This operation explicitly encodes the local spatial context information into the feature channel, so that the feature dimension of each point is expanded from 64 to 576.

[0037] Step 2, non-local feature extraction based on block matching. In LIIF, the prediction coordinates mainly rely on the feature information of the four pixels in the nearby 2x2 region. However, in the actual low-resolution image, in addition to the local region, other positions of the image may also contain latent useful features related to the query point. Therefore, a correlation-guided block matching attention screening mechanism is introduced in the present application, and the specific process is as follows: Figure 3As shown. First, the similarity index between the feature blocks is calculated in the full image range, and then several non-local positions most relevant to each local feature block are selected according to the similarity, and finally the attention operation is only performed in the most relevant area. This strategy effectively filters irrelevant information, improves the performance and reasoning efficiency of the model in long-distance dependence modeling. However, the most similar local block often contains rich context information, and due to the geometric deformation, scale change or viewing angle difference in the image, the fixed rule neighborhood sampling method often fails to accurately capture the truly corresponding semantic area, limiting the expression ability of the model. Therefore, the present application introduces a coordinate offset network (Coordinate offset network), which dynamically predicts the corresponding similar feature position of the current query position according to the context, thereby realizing adaptive feature alignment and context modeling. Specifically, the following sub-steps are included: Step 2.1, image block matching. The present application divides the input low-resolution image into several local image blocks Patch (block division on the image is equivalent to block division on the unfolded feature, and the size of the unfolded feature map and the input image is consistent), and finds the most similar local area to the query point in the entire image. Unlike traditional patch matching methods based on propagation strategy and random search, the similarity is calculated by normalized inner product, thereby realizing more efficient and more robust feature matching. Specifically, the following sub-steps are included: Step 2.1.1, similarity calculation. The network divides the input feature map F with size (H, W, C) after encoding into a two-dimensional image block sequence with size (h, w), where H and W represent the length and width of the feature map F respectively, and the image block sequence contains a total of image blocks with channel number C, The calculation formula is as follows: (3) After that, these image block sequences are flattened into a vector with size (1, h×w×C). Similarity calculation aims to estimate the similarity between each Patch, thereby establishing effective feature association. In the feature space, cosine similarity is calculated for each Patch to find the most relevant Patch. Specifically, let the query feature and the key feature be and respectively, which are obtained by processing the features (from F) of the i-th query image block and the j-th query image block through two multi-layer perceptron MLPs respectively; wherein i, j ∈[1, ] and the correlation between each and is calculated by normalized dot product: (4) wherein, represent and the feature similarity between them.

[0038] Step 2.1.2, most relevant non-local image block screening. The standard attention mechanism can introduce a blurring effect by weighted summation of feature values, weakening the ability to preserve texture details. To this end, the invention only selects the most relevant features for each query location for migration, according to the relevance matrix for each location i to calculate the most relevant reference image location : (5) where represents the position of the image block most relevant to the i th image block, indicates the set of points calculated in the j dimension, and the set of points calculated in the

[0039] Step 2.2, offset network and attention fusion. Considering that the most similar local block often contains rich contextual information, the invention introduces a coordinate offset network to further improve the positioning accuracy of similar features. This network predicts a set of fine-grained offset vectors for each query point, which are used to correct the initial matching candidate positions, so that the attention calculation can more accurately focus on the feature regions with potential correspondence.

[0040] Specifically, after finding the most similar image block, the image block is input into the offset network (a standard MLP with an output dimension of 3, used to predict the offset coordinates (dx, dy, dz) for each query point) to generate offset information, and the original position is added to obtain more accurate spatial position information of similar features. This process can be formalized as: (6) where represents the coordinate offset of the i th sequence, represents the coordinates obtained after offset. Then the corresponding texture features are selected from the low-resolution encoding features F to obtain the texture representation at the migrated position i : (7) To further integrate the texture information and global features from the low-resolution image, the invention introduces an attention mechanism to achieve fusion optimization, which can be formalized as: (8)​ where, denotes the texture confidence at position i , q denotes the feature vector of the query position (obtained from the query position in the feature map F), Softmax denotes the normalization function, d k denotes the dimension. In the feature fusion stage, T is denoted as the overall feature after migration, T and F are spliced, the convolution layer is input to extract the fused feature, then the soft attention weight is applied by element multiplication, and finally the original feature is added to obtain the output representation of the aggregated global feature, and this process can be formalized as: (9) where, denotes the convolution operation, denotes feature splicing, denotes element-wise multiplication, represents the fused feature finally obtained through block matching, and S refers to the texture confidence at all positions.

[0041] Step 3, non-local feature extraction based on dynamic sparse attention. In high-resolution visual tasks or long sequence modeling, non-global feature extraction based on block matching has strong global modeling ability, but its computational complexity increases quadratically with the sequence length or spatial resolution. When the input scale is high or the sequence is long, the calculation and memory overhead will increase sharply, making it difficult to apply to actual large-scale tasks. Some existing sparse attention (such as local attention, fixed sparse mode, window division, long and short range combination, etc.) can reduce complexity, but it is often a static or regular sparse mode that cannot adjust the attention path flexibly according to the input content, which may miss the long-range dependency information across regions and thus limit the global information capture ability. To solve the above problems, the present application further proposes a hierarchical dynamic sparse attention mechanism. The algorithm structure is shown in Figure 4, and the operation process specifically includes the following sub-steps: Step 3.1, feature encoding and downsampling. For a low-resolution image LR, according to the processing flow of step 1, it is sent to the EDSR encoding network to extract the basic feature map , then the feature map is unfolded to obtain , and then it is sent to the feature extraction module based on downsampling to extract the global context by continuously reducing the spatial size. Each down-sampled feature extraction module is composed of a CNN convolution network, a ReLU activation function, and a Batch Normalization (BN) part, which down-samples the spatial dimension, i.e. , , while the number of channels increases layer by layer to improve the feature expression ability and alleviate information loss. This process can be formalized as: (10) (11) Step 3.2, Similarity calculation and hierarchical mapping. The multi-level feature maps are extracted in the previous step. Then, the initial self-attention matching is performed on the smallest scale, i.e., the bottom layer of the feature maps The self-attention operation is performed on each query point The attention similarity matrix can be represented as follows: (12) where, , = , represents the attention score of other positions to the query point, and H3 and W3 represent the length and width of the feature map F3.

[0042] Then, the top points with the largest similarity are selected for each query position: (13) represents the set of points most relevant to the query point on the 3rd layer feature map (the smallest scale), represents the top similar points that make up the most relevant point set. Next, the patch mapping operation is performed layer by layer. Each matching point is mapped to the corresponding 2x2 region of the next level of 4 pixels. For these new candidate points, the similarity embedding operation is repeated and the top most relevant points are extracted, as follows: (14) (15) where, , , respectively represent the query feature and key feature of the l layer, C l denotes the feature channel (dimension) of the l layer, K l and K l+1 denote the key feature of the l layer and l +1 layer, respectively. represents the set of expanded 2 times range mapped from the l +1 layer, represents the attention score of the l layer to the query point, represents thel Front of the layer A set of similar points.

[0043] Step 3.3, Soft Attention Feature Fusion. Since the attention weights between similar points and query points directly determine the subsequent feature aggregation process, it is not necessary to recalculate the attention scores for both query and similar points. Simply performing Softmax normalization on the obtained similarity scores yields a stable and efficient sequence of attention scores. This soft attention-based design not only simplifies the computation process but also improves modeling efficiency.

[0044] Specifically, the soft attention weight matrix and global candidate features are extracted from the selected set of points, and the candidate features are concatenated with the query features, as shown in the following formula: (16) (17) (18) in, This represents several features that are most relevant to the query point at level 1. Attention scores representing these features This represents the fused features. Using the calculated embedding similarity, the concatenated features are fed into a lightweight soft attention module for soft attention aggregation. This process can be formulated as follows: (19) Where Conv represents the convolution operation. This represents element-wise multiplication. This represents the fused features of the final output. The final output representation of each query point is obtained after fusing global features. This module, through layer-by-layer expansion of candidate point regions and fusion of multi-scale semantic similarity, ensures that the final query features fully integrate global contextual information with broad spatial scope and high semantic relevance, significantly enhancing the model's expressive power.

[0045] Step 4: Non-local feature fusion and pixel value prediction based on coordinate attention. The LIIF method only uses the area of ​​the diagonal rectangle formed between the query point and its four surrounding feature points as weights, and performs pixel value interpolation based solely on spatial geometric relationships, thus ignoring the potential visual semantic correlation between the query point and its neighboring feature points. To overcome this limitation, this invention introduces an attention mechanism during the query process, dynamically allocating weights by calculating the attention score between the query point and its surrounding feature points, thereby making fuller use of visual feature information to improve prediction accuracy. However, coordinate information itself is also a variable that cannot be ignored when performing pixel queries. To achieve pixel prediction at any scale—that is, to output the corresponding RGB value given only one image coordinate—this invention further explicitly introduces coordinate information into the attention mechanism. Specifically, it includes the following sub-steps: Step 4.1, Coordinate Attention Mechanism. For a given query coordinate in a high-resolution HR space... ( Figure 5 (the dark spot in (a)) Within a 2×2 local region centered on the key, select known low-resolution pixels as a local set. Denote the coordinates of the key vector (Key) and value vector (Value) within this local neighborhood as follows: , The corresponding latent codes are denoted as Q, K, and V. Based on the coordinates and the latent code pair (Q, K, V), ), (K, ), (V, Each pixel grid (cell) has a size of c. A coordinate attention mechanism is defined to predict the query location. The RGB value at that location is calculated using the following formula: (20) in Represents the query location RGB values ​​at that location It is a multilayer perceptron (MLP), representing an implicit function for pixel value prediction, and Attention() refers to feature fusion based on coordinate attention.

[0046] Step 4.2, Non-local Feature Fusion and Pixel Value Prediction. To further enhance feature aggregation capabilities, this invention replaces the Value in the attention mechanism with a feature vector that has undergone block matching in Step 2 (or sparse attention in Step 3) to fuse non-local features, thereby obtaining richer contextual information and no longer being limited to feature responses within local regions, such as... Figure 5As shown in (b). Specifically, for each local feature point in the 2x2 neighborhood near the query point, more non-local information can be fused through the block matching attention mechanism (or multi-level dynamic sparse attention mechanism) (block), so as to realize non-local feature fusion and pixel value prediction. The specific calculation formula is as follows: (21) wherein is the query network, namely MLP, denotes the Softmax function, denotes the local 2x2 region centered on the query point , Q , and are the query, key and value respectively, and can be formulated as: (22) wherein and are network modules for generating keys (Key) and values (Value) respectively, is the resolution feature on the nearest neighbor grid corresponding to the query coordinate , is the local feature extracted from the latent encoding through the unfolding operation (Unfold), which is equivalent to being centered on (x, y) and being spliced in its neighborhood 3x3, i, j denotes the non-local feature corresponding to the (x, y) position (in the block in (b) in (b)). The parameter denotes the size of the image grid, which is used to introduce unit decoding information in feature modeling. i, j and Figure 5 denote the relative coordinate distance of Key and Value in the local region respectively, and the specific formula is as follows: (23) By fusing the spatial position relationship and the semantic feature, the coordinate attention mechanism not only retains the learnability of the standard attention, but also enhances the modeling ability of the non-uniformly sampled coordinates. Further, the pixel value of each point in the high-resolution spatial scale is predicted according to the above steps, and the high-resolution image is reconstructed. Thus, the non-local feature fusion and pixel value prediction based on the coordinate attention are realized. The present application carries out image super-resolution reconstruction experiment on the public data set DIV2K. The comparison results of the method proposed in the present application and the currently widely used method are as follows:

[0047]

[0048] Figure 6 ​​​The evaluation index and visual result of the method are optimal, which fully verifies that the method can effectively solve the problem of image super-resolution reconstruction, can more accurately reconstruct clear image edges and natural textures, and realizes the reconstruction effect of more details and more natural vision.

[0049] In the implementation, the above process can be realized by computer software technology to automatically run the process.

[0050] Embodiment 2 The embodiment of the application also provides an image arbitrary scale super-resolution system based on block matching and dynamic sparse attention, including a processor and a memory, the memory is used for storing program instructions, and the processor is used for calling the program instructions in the memory to execute the image arbitrary scale super-resolution method based on block matching and dynamic sparse attention.

[0051] Embodiment 3 The embodiment of the application also provides a computer readable storage medium, including a readable storage medium, and the readable storage medium stores a computer program, and the computer program is executed to realize the image arbitrary scale super-resolution method based on block matching and dynamic sparse attention.

[0052] It should be understood that the above description of the preferred embodiments is more detailed, and therefore should not be considered as a limitation on the scope of patent protection of the application. Those skilled in the art can make substitutions or modifications without departing from the scope of protection of the claims of the application, and all fall within the scope of protection of the application. The scope of protection of the application should be subject to the appended claims.

Claims

1. An image super-resolution method based on block matching and dynamic sparse attention, characterized in that, Includes the following steps: Step 1, Image Feature Extraction and Aggregation: Based on the coding network, image block features are extracted, and then local contextual information is aggregated into the latent code through feature expansion and stitching operations; Step 2: Extract nonlocal features based on block matching or dynamic sparse attention; Step 3: Perform non-local feature fusion based on coordinate attention and predict pixel values; The pixel value of each point in the high-resolution spatial scale is predicted according to the above three steps to reconstruct the high-resolution image.

2. The image arbitrary-scale super-resolution method based on block matching and dynamic sparse attention as described in claim 1, characterized in that: The image block features extracted by the coding network are represented as follows: in, Represents the convolutional layer of the encoding network. This indicates the removal of the normalized residual connection layer, where M represents the latent encoding of the output.

3. The image arbitrary-scale super-resolution method based on block matching and dynamic sparse attention as described in claim 1, characterized in that: The new features obtained by feature expansion and concatenation operations are represented as follows: Where M represents the latent encoding of the feature extraction output. Represents the coordinates on the new feature map The feature vector at the location is represented by Concat, which indicates a concatenation operation along the channel dimension; -1, 0, 1 represent the position offset relative to the center point (i, j). Specifically, -1 means shifting one unit to the left along the x-axis or one unit upward along the y-axis; 0 means no shift; 1 means shifting one unit to the right along the x-axis or one unit downward along the y-axis.

4. The image arbitrary-scale super-resolution method based on block matching and dynamic sparse attention as described in claim 1, characterized in that: Non-local feature extraction based on block matching calculates the similarity index between feature blocks across the entire image. Then, based on similarity, it selects several non-local locations most relevant to each local feature block. Finally, it performs an attention operation only within the most relevant region, calculating the most relevant reference image location for each location. After finding the most similar image block, it uses this image block as input to an offset network to generate offset information, which is added to the original location to obtain more accurate spatial location information of similar features. Then, it selects the corresponding texture features from the low-resolution encoded features, i.e., the features extracted and aggregated from image features, and introduces an attention mechanism to calculate texture confidence as attention weights. It integrates texture features and encoded features as the final non-local features obtained after block matching.

5. The image arbitrary-scale super-resolution method based on block matching and dynamic sparse attention as described in claim 4, characterized in that: The nonlocal features ultimately obtained from block matching are represented as follows: : in, This represents the convolution operation. Indicates feature splicing, This indicates element-wise multiplication, S refers to the texture confidence at all locations, T is the overall texture feature after migration, and F is the feature obtained by image feature extraction and aggregation.

6. The image arbitrary-scale super-resolution method based on block matching and dynamic sparse attention as described in claim 1, characterized in that: Specific implementations of nonlocal feature extraction based on dynamic sparse attention include: Feature encoding and downsampling: Multiple downsampling-based feature extraction modules are used to downsample the spatial dimension of the features obtained from image feature extraction and aggregation. The downsampling-based feature extraction modules include a CNN convolutional network, a ReLU activation function, and a BN layer. Similarity calculation and hierarchical mapping: First, initial self-attention matching is performed at the smallest scale. Then, for each query position, a set of points most relevant to the query point is selected, and then a layer-by-layer block mapping operation is performed to obtain the first... l Attention scores of similar points between the layer and the query point and the first l Front of the layer A set of similar points; Soft attention feature fusion: Extract the soft attention weight matrix and global candidate features from the selected set of points, and concatenate the candidate features with the query features, as shown in the following formula: in, This represents several features that are most relevant to the query point at level 1. Attention scores representing these features This indicates the first feature extraction module based on downsampling. The attention score represents the similarity score between the first layer and the query point. Represents the front of the first layer A set of similar points Indicates query characteristics, The concatenated features are represented by the pre-calculated embedding similarity. These features are then fed into a lightweight soft attention module for soft attention aggregation. This process can be formalized as follows: Where Conv represents the convolution operation. This represents element-wise multiplication. This represents the nonlocal features of the final output.

7. The image arbitrary-scale super-resolution method based on block matching and dynamic sparse attention as described in claim 6, characterized in that: The specific implementation methods for similarity calculation and hierarchical mapping are as follows: In the lowest layer feature map Above, for each query point When performing self-attention operations, the attention similarity matrix is ​​represented by the following formula: in, , = , H3 and W3 represent the attention scores for other locations and the query point, respectively, and H3 and W3 represent the length and width of feature map F3. Then, for each query location, the top results with the highest similarity are selected. Points: This represents the set of points most relevant to the query point on the smallest-scale feature map. Represents taking the former The most relevant points are formed from 10 similar points; next, a layer-by-layer block mapping operation is performed: each matching point is mapped to 4 pixels in the corresponding 2×2 region of the previous level. For these new candidate points, the similarity embedding operation is repeated and the previous similarity is extracted. The most relevant point, the formula is as follows: in, , , respectively representing the l Query features and key features of the layer C l Refers to the first l The characteristic channels of the layer, K l and K l+1 They refer to the first l Layers and l +1 layer key features; Representative by the first l A set whose range is doubled after +1 layer mapping. Representing the l Attention scores of similar points between the layer and the query point Representing the l Front of the layer A set of similar points.

8. The image arbitrary-scale super-resolution method based on block matching and dynamic sparse attention as described in claim 1, characterized in that: The specific implementation of step three includes: Coordinate attention mechanism: For a given query coordinate in a high-resolution HR space In the Within a 2×2 local region centered on the key, select known low-resolution pixels as a local set. Denote the coordinates of the key vector (Key) and value vector (Value) within this local neighborhood as follows: , The corresponding latent codes are denoted as Q, K, and V; based on the coordinates and the latent code pair (Q, ), (K, ), (V, Given a grid of size c for each pixel, a coordinate attention mechanism is defined to predict the query location. The specific formula for the RGB value at that location is as follows: in Represents the query location RGB values ​​at that location It is a multilayer perceptron, representing the implicit function for pixel value prediction, and Attention() refers to feature fusion based on coordinate attention; Non-local feature fusion and pixel value prediction: The value vector in the attention mechanism is replaced with a non-local feature vector obtained through block matching or dynamic sparse attention to obtain richer contextual information, no longer limited to feature responses within local regions. Specifically, for each local feature point within a 2×2 neighborhood of the query point, more non-local information is fused through block matching attention or dynamic sparse attention mechanisms to achieve non-local feature fusion and pixel value prediction. The specific calculation formula is as follows: in It is a query network, i.e., MLP. This represents the Softmax function. Indicates query point A local 2×2 region centered on the center, Q , and The query, key, and value are respectively represented, and the formula is as follows: in and Network modules used to generate keys and values ​​respectively. Is it related to the query coordinates? The corresponding resolution features on the nearest neighbor grid, It is a local feature extracted from the latent encoding through an unfolding operation, which is equivalent to ( i, j Centered on ), it is obtained by splicing within its 3×3 neighborhood. Indicates and ( i,j The non-local features corresponding to the location; parameters This indicates the size of the image grid, used to incorporate unit decoding information in feature modeling; and These represent the relative coordinate distances between keys and values ​​within a local region, as shown in the following formulas: 。 9. An image super-resolution system at any scale based on block matching and dynamic sparse attention, characterized in that: It includes a processor and a memory, the memory being used to store program instructions, and the processor being used to call the program instructions in the memory to execute the image arbitrary-scale super-resolution method based on block matching and dynamic sparse attention as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, It includes a readable storage medium on which a computer program is stored, and when the computer program is executed, it implements the image arbitrary-scale super-resolution method based on block matching and dynamic sparse attention as described in any one of claims 1-8.

Images