Lightweight image super-resolution method based on adaptive feature aggregation network

Abstract

The embodiment of the invention provides a lightweight image super-resolution method based on an adaptive feature aggregation network. The method comprises the steps of obtaining a low-resolution input image, performing convolution processing on the low-resolution input image, inputting the low-resolution input image into a pre-constructed aggregation network based on adaptive features, and performing super-resolution processing on the low-resolution input image to obtain a super-resolution output image; the aggregation network comprises an image reconstruction module and a plurality of efficient feature extraction modules connected in series, and the efficient feature extraction modules are used for feature extraction and comprise lightweight omni-directional feature aggregation modules and lightweight adaptive gating modules; the image reconstruction module is used for image reconstruction; and outputting the super-resolution output image. According to the method, an efficient feature extraction module is obtained by constructing a lightweight omnibearing feature aggregation module and a lightweight self-adaptive gating module, so that optimal balance of performance and efficiency is realized; and the image reconstruction module is combined with the image reconstruction module to obtain an aggregation network, so that super-resolution output of a low-resolution input image is realized, and the visual effect is greatly enhanced.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of image processing technology, and in particular to a lightweight image super-resolution method based on an adaptive feature aggregation network. Background Art

[0002] With the rapid development of information technology and the increasing use of digital images, image quality has become a key factor in measuring information transmission efficiency and visual experience. The demand for high-resolution images is increasingly urgent in fields such as medical image analysis, security monitoring, satellite remote sensing, and high-definition display. However, due to limitations in the physical performance of imaging equipment, information loss during transmission, and storage conditions, the actual images obtained often lack resolution, making them difficult to meet the requirements of high-precision analysis and visual appreciation.

[0003] Image super-resolution technology, an effective algorithmic approach for enhancing image resolution, has garnered widespread attention in recent years from both academia and industry. This technology aims to recover more detail from low-resolution images, generating high-resolution images that approach or even surpass the original scene's resolution. It not only significantly improves image clarity and detail, but also overcomes the physical limitations of imaging devices, opening up new possibilities for image processing and applications.

[0004] Super-resolution technology plays a vital role in image and video compression and transmission. First, by introducing super-resolution technology, low-resolution images or videos can be transmitted and then super-resolution processed at the receiving end to restore high-quality images or videos. This allows the image or video size to be further compressed while minimizing image quality loss, saving storage space and transmission bandwidth. This technology can significantly improve image or video transmission efficiency, particularly in bandwidth-constrained environments. Secondly, super-resolution technology is also applicable to video surveillance. When low-resolution cameras need to display more video detail, super-resolution technology can magnify the image or video and restore the details, making the details in the surveillance video clearer and facilitating better monitoring and recording of the situation. Furthermore, super-resolution technology can enhance the information and details in low-resolution images, thereby improving the accuracy of image recognition algorithms. For example, by improving the clarity and detail of faces in videos, super-resolution technology can help banks more accurately identify customers, reduce the risks of remote account opening, and ensure the security and reliability of identity verification.

[0005] However, traditional super-resolution technology has complex models and high hardware requirements. Therefore, how to make super-resolution technology balance performance and efficiency has become a key issue that needs to be solved urgently. Summary of the Invention

[0006] In response to the above-mentioned problems in the prior art, the purpose of the embodiments of this specification is to provide a lightweight image super-resolution method based on an adaptive feature aggregation network to solve the problem that the super-resolution technology in the prior art is difficult to strike a balance between performance and efficiency.

[0007] In order to solve the above technical problems, the specific technical solutions of the embodiments of this specification are as follows:

[0008] In a first aspect, the embodiments of this specification provide a lightweight image super-resolution method based on an adaptive feature aggregation network, comprising:

[0009] Get a low-resolution input image and perform convolution processing;

[0010] The low-resolution input image after convolution processing is input into a pre-built adaptive feature-based aggregation network for super-resolution processing to obtain a super-resolution output image; the adaptive feature-based aggregation network includes an image reconstruction module and a plurality of series-connected efficient feature extraction modules, the efficient feature extraction module is used for feature extraction, and the efficient feature extraction module includes a lightweight omnidirectional feature aggregation module and a lightweight adaptive gating module; the image reconstruction module is used to reconstruct the feature extraction results obtained by the plurality of series-connected efficient feature extraction modules and the low-resolution input image after convolution processing;

[0011] The super-resolution output image is output.

[0012] Specifically, the lightweight omnidirectional feature aggregation module includes an input module, a first convolution, a channel segmentation module, a local feature extraction module, a non-local feature extraction module, a second convolution and an output module;

[0013] The input module is used to receive input features and feed them back to the first convolution;

[0014] The first convolution is used to increase the number of channels of the input feature to 2 times;

[0015] The channel segmentation module is used to segment the input features after the number of channels is increased to obtain local input features and non-local input features, and input them into the local feature extraction module and the non-local feature extraction module respectively;

[0016] The local feature extraction module and the non-local feature extraction module are used to extract features from the local input features and the non-local input features respectively;

[0017] The second convolution is used to perform convolution processing on the element-wise addition result of the first output of the local feature extraction module and the second output of the non-local feature extraction module, and output the result after the convolution processing through the output module.

[0018] Furthermore, the local feature extraction module includes a 1×1 convolution, a first GELU activation function, a 3×3 convolution, a second GELU activation function, a channel attention unit and a 1×1 convolution connected in sequence;

[0019] The output of the local feature extraction unit is:

[0020]

[0021] in, is the local input feature segmented by the channel segmentation module and input to the local feature extraction module; 1×1 (·),f 3×3 (·), θ(·) and CA(·) are 1×1 convolution, 3×3 convolution, GELU activation function and channel attention module respectively.

[0022] Specifically, the non-local feature extraction module includes an inverse sub-pixel convolution, a channel segmentation unit, a variance extraction unit, a 3×3 depth-separable convolution, a 1×1 convolution, a third GELU activation function, a sub-pixel convolution, a channel splicing unit, and a multiplication unit;

[0023] The inverse sub-pixel convolution and the variance extraction unit are used to receive the non-local input features segmented by the channel segmentation unit and input to the non-local feature extraction module;

[0024] The channel segmentation unit is used to generate four identical feature groups from the high-dimensional features output by the inverse sub-pixel convolution and input them into the 3×3 depth-separable convolution;

[0025] The 1×1 convolution is connected to the third GELU activation function and receives the result of element-wise addition of the output of the variance extraction unit and the output of the 3×3 depth-wise separable convolution;

[0026] The sub-pixel convolution unit is connected to the channel stitching unit to perform sub-pixel convolution processing and stitching processing on the result of element addition;

[0027] The multiplication unit is used to multiply the results after sub-pixel convolution processing and splicing processing by the input image information input to the non-local feature extraction module element by element and output the multiplication result.

[0028] The output of the non-local feature extraction unit is:

[0029]

[0030] in, is the non-local input feature segmented by the channel segmentation unit and input to the non-local feature extraction module; Ω(·) and are the splicing operation of the channel splicing unit and the feature segmentation operation of the channel segmentation unit respectively; ↑(·) and ↓(·) are the sub-pixel convolution and anti-sub-pixel convolution respectively; express The variance of f 1×1 (·), and θ(·) are 1×1 convolution, 3×3 depthwise separable convolution, and GELU activation function, respectively.

[0031] Furthermore, the lightweight adaptive gating module includes an input module, a 1×1 convolution, a channel segmentation module, a 3×3 convolution, a multiplication module, a GELU activation function, a channel attention module, a spatial attention module, an addition module and an output module;

[0032] The input module is used to receive input image information and input it into the channel segmentation module through the first 1×1 convolution and the fourth GELU activation function;

[0033] The channel segmentation module is used to segment the input image information after the number of channels is increased by the first 1×1 convolution to obtain a first feature and a second feature;

[0034] 3×3 convolution is used to perform convolution processing on the first feature;

[0035] The multiplication module is used to multiply the result obtained by multiplying the output of the 3×3 convolution by the first learnable parameter on a pixel basis, and the result obtained by multiplying the second feature and the second learnable parameter on a pixel basis to obtain a multiplication result;

[0036] The second 1×1 convolution is used to perform convolution processing on the multiplication result and input it into the parallel channel attention module and spatial attention module through the fifth GELU activation function;

[0037] The addition module is used to add the multiplication result of the second feature and the second learnable parameter, the output of the channel attention module and the output of the spatial attention module on a pixel basis to obtain an addition result;

[0038] The output module is used to output the addition result.

[0039] Specifically, the output of the lightweight adaptive gating module is:

[0040] F out =SA(F c )+CA(F d )+βF b ;

[0041] Among them, SA(·) and CA(·) are the spatial attention module and channel attention module respectively;

[0042] F c,F d =θ(f 1×1 (F2));

[0043] F2=αβF b ·f 3×3 (F a );

[0044] F a and F b are the first and second characteristics respectively, F1=θ(f 1×1 (F in )), F in To input image information, It is the feature segmentation operation of the channel segmentation unit;

[0045] α and β are the first and second learnable parameters respectively; f 1×1 (·),f 3×3 (·) and θ(·) are 1×1 convolution, 3×3 convolution and GELU activation function, respectively.

[0046] Specifically, the output of the efficient feature extraction module is:

[0047] F EFEM-out =f AGB (f OFAM (F EFEM-in )+F EFEM-in )+f OFAM (F EFEM-in )+F EFEM-in ;

[0048] Among them, f AGB (·) is a lightweight adaptive gating module, f OFAM (·) is a lightweight all-round feature aggregation module, F EFEM-in and F EFEM-out They are the input and output of the efficient feature extraction module respectively.

[0049] Specifically, the image reconstruction module is implemented by sub-pixel convolution upsampling, and the super-resolution image output image is:

[0050]

[0051] Among them, ↑(·) is sub-pixel convolution, It is 7 efficient feature extraction modules connected in series, f 3×3 (·) is a 3×3 convolution, I LR and I SR are low-resolution input images and super-resolution output images, respectively.

[0052] In a second aspect, the embodiments of this specification provide a lightweight image super-resolution device based on an adaptive feature aggregation network, comprising:

[0053] The acquisition module is used to obtain low-resolution input images and perform convolution processing;

[0054] a processing module for inputting the low-resolution input image after convolution processing into a pre-built adaptive feature-based aggregation network for super-resolution processing; the adaptive feature-based aggregation network includes an image reconstruction module and multiple series-connected efficient feature extraction modules, the efficient feature extraction modules are used for feature extraction, and the efficient feature extraction modules include a lightweight omnidirectional feature aggregation module and a lightweight adaptive gating module; the image reconstruction module is used to reconstruct the feature extraction results obtained by the multiple series-connected efficient feature extraction modules and the low-resolution input image after convolution processing to obtain a super-resolution output image;

[0055] An output module is used to output the super-resolution output image.

[0056] In a third aspect, an embodiment of this specification provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method provided by the above technical solution when executing the computer program.

[0057] In a fourth aspect, an embodiment of this specification provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method provided by the above technical solution is implemented.

[0058] In a fifth aspect, an embodiment of this specification provides a computer program product, comprising at least one instruction or at least one program segment, wherein the at least one instruction or the at least one program segment is loaded and executed by a processor to implement the method provided by the above technical solution.

[0059] By adopting the above technical solution, the embodiment of this specification provides a lightweight image super-resolution method based on an adaptive feature aggregation network. By constructing a lightweight all-round feature aggregation module and a lightweight adaptive gating module, an efficient feature extraction module is obtained, which successfully achieves an optimal trade-off between performance and efficiency; and then combined with the image reconstruction module to obtain an aggregation network based on adaptive features, which realizes super-resolution output of low-resolution input images, making it widely applicable to various practical scenarios. Among them, the lightweight all-round feature aggregation module can effectively capture local and non-local features in the image while maintaining low computational complexity to ensure fast processing speed. In addition, the lightweight adaptive gating module can realize dynamic adjustment of feature weights, selectively enhance beneficial features and suppress irrelevant features, improve the effectiveness and robustness of feature representation, so that the output super-resolution image can significantly improve the resolution and make the details clearer while maintaining the original image content, so that the visual effect is greatly enhanced.

[0060] In order to make the above and other purposes, features and advantages of the embodiments of this specification more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0062] Figure 1 A schematic diagram of the steps of a lightweight image super-resolution method based on an adaptive feature aggregation network provided in an embodiment of this specification is shown;

[0063] Figure 2 A schematic diagram of the framework of the adaptive feature-based aggregation network is shown;

[0064] Figure 3 Shows a schematic diagram of the structure of the efficient feature extraction module;

[0065] Figure 4 shows a schematic structural diagram of a lightweight all-round feature aggregation module;

[0066] Figure 5 shows a schematic structural diagram of a lightweight adaptive gating module;

[0067] Figure 6 A schematic structural diagram of a lightweight image super-resolution device based on an adaptive feature aggregation network provided in an embodiment of this specification is shown;

[0068] Figure 7 A schematic structural diagram of a computer device provided in an embodiment of this specification is shown.

[0069] Description of the accompanying symbols:

[0070] 702. Computer equipment;

[0071] 704, processor;

[0072] 706. Memory;

[0073] 708, driving mechanism;

[0074] 710, input / output module;

[0075] 712. Input devices;

[0076] 714. Output device;

[0077] 716. Presentation equipment;

[0078] 718. Graphical User Interface;

[0079] 720, network interface;

[0080] 722, communication link;

[0081] 724. Communication bus. DETAILED DESCRIPTION

[0082] The following will be combined with the drawings in the embodiments of this specification to clearly and completely describe the technical solutions in the embodiments of this specification. Obviously, the embodiments described are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this specification.

[0083] It should be noted that the terms "first," "second," and the like in this specification, the claims, and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this specification described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, apparatus, product, or device comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes, methods, products, or devices.

[0084] In order to solve the above problems, the embodiments of this specification provide a lightweight image super-resolution method based on an adaptive feature aggregation network, which can solve the problem that the super-resolution technology in the existing technology is difficult to strike a balance between performance and efficiency. Figure 1 This is a step diagram of a lightweight image super-resolution method based on an adaptive feature aggregation network provided in an embodiment of this specification. This specification provides method operation steps as described in the embodiment or flowchart, but may include more or fewer operation steps based on conventional or non-creative labor. The order of steps listed in the embodiment is only one way of executing the steps among many, and does not represent the only execution order. When the actual system or device product is executed, it can be executed in sequence or in parallel according to the method shown in the embodiment or the accompanying drawings. Specifically, Figure 1 As shown, the method may include:

[0085] S101: Obtain a low-resolution input image and perform convolution processing.

[0086] S102: Input the low-resolution input image after convolution processing into a pre-built adaptive feature-based aggregation network for super-resolution processing; the adaptive feature-based aggregation network includes an image reconstruction module and multiple series-connected efficient feature extraction modules, the efficient feature extraction module is used for feature extraction, and the efficient feature extraction module includes a lightweight all-round feature aggregation module and a lightweight adaptive gating module; the image reconstruction module is used to reconstruct the feature extraction results obtained by the multiple series-connected efficient feature extraction modules and the low-resolution input image after convolution processing to obtain a super-resolution output image.

[0087] like Figure 2 The structure of the adaptive feature-based aggregation network is shown in Figure 2. Specifically, the image reconstruction module is implemented by sub-pixel convolution upsampling, and the super-resolution image output image is:

[0088]

[0089] Among them, ↑(·) is sub-pixel convolution, It is 7 efficient feature extraction modules connected in series, f 3×3 (·) is a 3×3 convolution, I LR and I SR are low-resolution input images and super-resolution output images, respectively.

[0090] S103: Outputting the super-resolution output image.

[0091] The embodiment of this specification provides a lightweight image super-resolution method based on an adaptive feature aggregation network. By constructing a lightweight all-round feature aggregation module and a lightweight adaptive gating module, an efficient feature extraction module is obtained, which successfully achieves an optimized trade-off between performance and efficiency; and then combined with an image reconstruction module to obtain an aggregation network based on adaptive features, which realizes super-resolution output of low-resolution input images, making it widely applicable to various practical scenarios. Among them, the lightweight all-round feature aggregation module can effectively capture local and non-local features in the image while maintaining low computational complexity to ensure fast processing speed. In addition, the lightweight adaptive gating module can realize dynamic adjustment of feature weights, selectively enhance beneficial features and suppress irrelevant features, improve the effectiveness and robustness of feature representation, so that the output super-resolution image has significantly improved resolution, clearer details, and greatly enhanced visual effects while maintaining the original image content.

[0092] Furthermore, in the embodiment of this specification, the structure of the efficient feature extraction module is as follows: Figure 3 As shown in Figure 2, it consists of a lightweight omnidirectional feature aggregation module and a lightweight adaptive gating module. The output of the efficient feature extraction module is:

[0093] F EFEM-out =f AGB (f OFAM (F EFEM-in )+F EFEM-in )+f OFAM (F EFEM-in )+F EFEM-in ;

[0094] Among them, f AGB (·) is a lightweight adaptive gating module, f OFAM (·) is a lightweight all-round feature aggregation module, F EFEM-in and F EFEM-out They are the input and output of the efficient feature extraction module respectively.

[0095] The efficient feature extraction module of the embodiment of this specification is composed of a lightweight all-round feature aggregation module and a lightweight adaptive gating module, wherein the lightweight all-round feature aggregation module can effectively capture local and non-local features in the image while maintaining low computational complexity to ensure processing speed; the lightweight adaptive gating module can dynamically adjust the weight of the feature according to its importance, selectively enhance features that are beneficial to image super-resolution reconstruction, and suppress irrelevant features, thereby further improving the effectiveness and robustness of the feature representation.

[0096] Furthermore, in the embodiment of this specification, the structure of the lightweight all-round feature aggregation module is as follows: Figure 4As shown, the lightweight omnidirectional feature aggregation module includes an input module, a first convolution, a channel segmentation module, a local feature extraction module, a non-local feature extraction module, a second convolution and an output module;

[0097] The input module is used to receive input features and feed them back to the first convolution; wherein the first convolution is specifically a 1×1 convolution.

[0098] The first convolution is used to increase the number of channels of the input feature to 2 times;

[0099] That is, when the input feature is When H, W, and C represent the length, width, and number of channels of the input feature, respectively, the feature will be obtained after the first convolution. That is, X1 = f 1×1 (X in ).

[0100] The channel segmentation module is used to segment the input features after the number of channels is increased to obtain local input features and non-local input features, and input them into the local feature extraction module and the non-local feature extraction module respectively; that is, the channel segmentation module segments the feature X1 to obtain local input features and non-local input features Then, they are input into the local feature extraction module and the non-local feature extraction module respectively to enhance the feature expression ability, that is, ξ(·) is the segmentation operation of the channel segmentation module.

[0101] The local feature extraction module and the non-local feature extraction module are used to extract features from the local input features and the non-local input features respectively;

[0102] The second convolution is used to perform convolution processing on the element-wise addition result of the first output of the local feature extraction module and the second output of the non-local feature extraction module, and output the convolution result through the output module, wherein the second convolution is specifically a 1×1 convolution.

[0103] Thus, the output of the lightweight omnidirectional feature aggregation module is:

[0104]

[0105] in, and are the first output of the local feature extraction module and the second output of the non-local feature extraction module respectively.

[0106] like Figure 4As shown, the local feature extraction module includes a sequentially connected 1×1 convolution, a first GELU activation function, a 3×3 convolution, a second GELU activation function, a channel attention unit and a 1×1 convolution;

[0107] The first output of the local feature extraction unit is:

[0108]

[0109] in, is the local input feature segmented by the channel segmentation module and input to the local feature extraction module; 1×1 (·),f 3×3 (·), θ(·) and CA(·) are 1×1 convolution, 3×3 convolution, GELU activation function and channel attention module respectively.

[0110] As well as Figure 4 As shown, the non-local feature extraction module includes inverse sub-pixel convolution, channel segmentation unit, variance extraction unit, 3×3 depth-separable convolution, 1×1 convolution, the third GELU activation function, sub-pixel convolution, channel splicing unit and multiplication unit;

[0111] The inverse sub-pixel convolution and the variance extraction unit are used to receive the non-local input features. In order to capture more refined features in the non-local operation, in the embodiment of this specification, the low-dimensional feature map is expanded to a high dimension using inverse sub-pixel convolution, thereby extracting key information of the feature map dimension, and obtaining

[0112] The channel segmentation unit is used to generate 4 identical feature groups from the high-dimensional features output by the inverse sub-pixel convolution and input them into the 3×3 depth-separable convolution; that is, feature X a It is split again on the channel, so that each subspace can learn different features independently to enhance the diversity and expression ability of features, and then extract richer information in different channel subspaces. This operation will generate 4 features of the same shape The feature groups are sequentially passed through 3×3 depth-wise separable convolutions with shared weights.

[0113] The 1×1 convolution is connected to the third GELU activation function and receives the result of element-wise addition of the output of the variance extraction unit and the output of the 3×3 depth-separable convolution; that is, the variance of the non-local input feature extracted by the variance extraction module is added element-wise to the output result of the 3×3 depth-separable convolution, and is activated by the 1×1 convolution and the third GELU.

[0114] The sub-pixel convolution and the channel stitching unit are connected to perform sub-pixel convolution processing and stitching processing on the result after element addition, so that the characteristic shape is restored to the original dimension.

[0115] The multiplication unit is used to multiply the result after sub-pixel convolution and splicing processing with the non-local input feature element by element and output it; the final output is consistent with the dimension of the non-local input feature, that is,

[0116] The second output of the non-local feature extraction unit is:

[0117]

[0118]

[0119] in, is the input image information segmented by the channel segmentation unit and input to the non-local feature extraction module; Ω(·) and are the splicing operation of the channel splicing unit and the feature segmentation operation of the channel segmentation unit respectively; ↑(·) and ↓(·) are the sub-pixel convolution and anti-sub-pixel convolution respectively; express The variance of x i , μ and N are the i-th pixel, the average value of pixel feature information and the number of pixels respectively; f 1×1 (·), and θ(·) are 1×1 convolution, 3×3 depthwise separable convolution, and GELU activation function, respectively.

[0120] The structure of the lightweight adaptive gating module is as follows Figure 5 As shown, the lightweight adaptive gating module includes an input module, a 1×1 convolution, a channel segmentation module, a 3×3 convolution, a multiplication module, a GELU activation function, a channel attention module, a spatial attention module, an addition module and an output module;

[0121] The input module is used to receive input image information and input it to the channel segmentation module through the first 1×1 convolution and the fourth GELU activation function; when the input image information is After the first 1×1 convolution and the fourth GELU activation function, the feature is generated. The above process can be defined as:

[0122] F1=θ(f 1×1 (F in )), F in Input image information.

[0123] The channel segmentation module is used to segment the input image information after the first 1×1 convolution channel number is increased to obtain the first feature and the second feature; that is, the channel segmentation module segments the feature F1 to obtain the first feature and the second feature It is the feature segmentation operation of the channel segmentation unit.

[0124] 3×3 convolution is used to perform convolution processing on the first feature;

[0125] The multiplication module is used to multiply the output of the 3×3 convolution by the first learnable parameter on a pixel-by-pixel basis, and multiply the result by the second feature and the second learnable parameter on a pixel-by-pixel basis to obtain a multiplication result; the multiplication result is denoted as F2, then:

[0126] F2=αβF b ·f 3×3 (F a ), α and β are the first learnable parameter and the second learnable parameter respectively;

[0127] The above operations enable the network to adaptively adjust the importance of different channels and spatial positions, which helps the module extract higher quality features.

[0128] The second 1×1 convolution is used to perform convolution processing on the multiplication result and input it into the parallel channel attention module and spatial attention module through the fifth GELU activation function; the output of the channel attention module and the input of the spatial attention module are respectively F c and F d , then F c ,F d =θ(f 1×1 (F2)).

[0129] The addition module is used to add the multiplication result of the second feature and the second learnable parameter, the output of the channel attention module and the output of the spatial attention module on a pixel basis to obtain an addition result;

[0130] The output module is used to output the addition result, that is, the output of the lightweight adaptive gating module is:

[0131] F out =SA(F c )+CA(F d )+βF b ;

[0132] Among them, SA(·) and CA(·) are the spatial attention module and channel attention module respectively;

[0133] F aand F b are the first and second characteristics respectively, F1=θ(f 1×1 (F in ));

[0134] f 1×1 (·),f 3×3 (·) and θ(·) are 1×1 convolution, 3×3 convolution and GELU activation function, respectively.

[0135] Table 1

[0136] Parameter quantity Number of floating-point operations Set5 Set14 EDSR-baseline 1518K 114G 32.09 28.58 CARN 1592K 91G 32.13 28.60 SMSR 1006K 42G 32.12 28.55 This method 324K 19.88G 32.10 28.45

[0137] As shown in Table 1, this is a comparison table of the results of comparing the lightweight image super-resolution method based on the adaptive feature aggregation network provided in the embodiment of this specification with three other advanced lightweight image SR methods in a 4x magnification experiment. The public Set5 and Set14 data test sets were used in the experiment. As shown in Table 1, the method provided in the embodiment of this specification can significantly reduce the number of model parameters while ensuring excellent peak signal-to-noise ratio and structural similarity indicators, and maintaining a high level of super-resolution reconstruction quality. Compared with the SMSR method, the method provided in the embodiment of this specification can still achieve comparable performance while using only 32.2% (i.e., 324K / 1006K=32.2%) of the parameters.

[0138] In summary, the embodiment of this specification provides a proposed lightweight image super-resolution method based on an adaptive feature aggregation network, which obtains an efficient feature extraction module by constructing a lightweight all-round feature aggregation module and a lightweight adaptive gating module, and successfully achieves an optimized trade-off between performance and efficiency; and then combines it with the image reconstruction module to obtain an aggregation network based on adaptive features, which realizes super-resolution output of low-resolution input images, making it widely applicable to various practical scenarios. Among them, the lightweight all-round feature aggregation module can effectively capture local and non-local features in the image while maintaining low computational complexity to ensure fast processing speed. In addition, the lightweight adaptive gating module adjusts the importance of different channels and spatial positions by setting two learnable parameters, which is conducive to the module extracting higher quality features, thereby realizing dynamic adjustment of feature weights, selectively enhancing beneficial features and suppressing irrelevant features, and improving the effectiveness and robustness of feature representation.

[0139] Based on the above-mentioned lightweight image super-resolution method based on an adaptive feature aggregation network, the embodiments of this specification also provide a lightweight image super-resolution device based on an adaptive feature aggregation network. The device may include a system (including a distributed system), software (application), module, component, server, client, etc. using the method described in the embodiments of this specification and combined with the necessary implementation hardware. Based on the same innovative concept, the devices in one or more embodiments provided in the embodiments of this specification are as described in the following embodiments. Since the implementation scheme and method of the device to solve the problem are similar, the implementation of the specific device in the embodiments of this specification can refer to the implementation of the aforementioned method, and the repetitions will not be repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements the predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0140] like Figure 6 As shown, the lightweight image super-resolution device based on the adaptive feature aggregation network includes:

[0141] An acquisition module 61 is used to acquire a low-resolution input image and perform convolution processing;

[0142] a processing module 62 configured to input the low-resolution input image after convolution processing into a pre-built adaptive feature-based aggregation network for super-resolution processing; the adaptive feature-based aggregation network includes an image reconstruction module and multiple series-connected efficient feature extraction modules, the efficient feature extraction modules are used for feature extraction, and the efficient feature extraction modules include a lightweight omnidirectional feature aggregation module and a lightweight adaptive gating module; the image reconstruction module is configured to reconstruct the feature extraction results obtained by the multiple series-connected efficient feature extraction modules and the low-resolution input image after convolution processing to obtain a super-resolution output image;

[0143] The output module 63 is configured to output the super-resolution output image.

[0144] The beneficial effects achieved by the device provided in the embodiments of this specification are consistent with the beneficial effects achieved by the above-mentioned method and will not be repeated here.

[0145] like Figure 7As shown, a computer device provided in an embodiment of this specification is provided. The lightweight image super-resolution apparatus based on an adaptive feature aggregation network in this specification can be a computer device in this embodiment, which performs the above-mentioned method of this specification. The computer device 702 may include one or more processors 704, such as one or more central processing units (CPUs), each of which may implement one or more hardware threads. The computer device 702 may also include any memory 706 for storing any type of information, such as code, settings, data, etc. For example, without limitation, the memory 706 may include any one or more combinations of the following: any type of RAM, any type of ROM, a flash memory device, a hard disk, an optical disk, etc. More generally, any memory may use any technology to store information. Furthermore, any memory may provide volatile or non-volatile retention of information. Furthermore, any memory may represent a fixed or removable component of the computer device 702. In one embodiment, when the processor 704 executes associated instructions stored in any memory or combination of memories, the computer device 702 may perform any operation of the associated instructions. The computer device 702 also includes one or more drive mechanisms 708 for interacting with any storage, such as a hard disk drive mechanism, an optical disk drive mechanism, and the like.

[0146] The computer device 702 may also include an input / output module 710 (I / O) for receiving various inputs (via input devices 712) and for providing various outputs (via output devices 714). A specific output mechanism may include a presentation device 716 and an associated graphical user interface (GUI) 718. In other embodiments, the input / output module 710 (I / O), input devices 712, and output devices 714 may not be included, and the computer device 702 may simply be a computer device in a network. The computer device 702 may also include one or more network interfaces 720 for exchanging data with other devices via one or more communication links 722. One or more communication buses 724 couple the components described above together.

[0147] The communication link 722 may be implemented in any manner, for example, via a local area network, a wide area network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. The communication link 722 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc., governed by any protocol or combination of protocols.

[0148] Corresponding to Figure 1 In addition to the method shown, an embodiment of this specification also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are executed.

[0149] The embodiment of this specification also provides a computer-readable instruction, wherein when the processor executes the instruction, the program therein causes the processor to execute the following Figure 1 method.

[0150] The embodiment of this specification also provides a computer program product, including at least one instruction or at least one program, which is loaded and executed by a processor to implement the following Figure 1 method.

[0151] It should be understood that in the various embodiments of this specification, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this specification.

[0152] It should also be understood that in the embodiments of this specification, the term "and / or" is merely a description of the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. Furthermore, the character " / " in this specification generally indicates that the associated objects are in an "or" relationship.

[0153] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this specification can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the composition and steps of each example according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this specification.

[0154] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0155] In the several embodiments provided in this specification, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or can be an electrical, mechanical or other form of connection.

[0156] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of this specification.

[0157] In addition, the functional units in the various embodiments of this specification may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0158] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this specification is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of this specification. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0159] Specific embodiments are used in this specification to illustrate the principles and implementation methods of this specification. The description of the above embodiments is only used to help understand the methods and core ideas of this specification. At the same time, for those skilled in the art, based on the ideas of this specification, there will be changes in the specific implementation methods and application scope. In summary, the contents of this specification should not be understood as limiting this specification.

Claims

1. A lightweight image super-resolution method based on an adaptive feature aggregation network, characterized in that: include: Get a low-resolution input image and perform convolution processing; The low-resolution input image after convolution processing is input into a pre-built adaptive feature-based aggregation network for super-resolution processing to obtain a super-resolution output image; the adaptive feature-based aggregation network includes an image reconstruction module and a plurality of series-connected efficient feature extraction modules, the efficient feature extraction module is used for feature extraction, and the efficient feature extraction module includes a lightweight omnidirectional feature aggregation module and a lightweight adaptive gating module; the image reconstruction module is used to reconstruct the feature extraction results obtained by the plurality of series-connected efficient feature extraction modules and the low-resolution input image after convolution processing; The super-resolution output image is output.

2. The method according to claim 1, characterized in that The lightweight omnidirectional feature aggregation module includes an input module, a first convolution, a channel segmentation module, a local feature extraction module, a non-local feature extraction module, a second convolution and an output module; The input module is used to receive input features and feed them back to the first convolution; The first convolution is used to increase the number of channels of the input feature to 2 times; The channel segmentation module is used to segment the input features after the number of channels is increased to obtain local input features and non-local input features, and input them into the local feature extraction module and the non-local feature extraction module respectively; The local feature extraction module and the non-local feature extraction module are used to extract features from the local input features and the non-local input features respectively; The second convolution is used to perform convolution processing on the element-wise addition result of the first output of the local feature extraction module and the second output of the non-local feature extraction module, and output the result after the convolution processing through the output module.

3. The method according to claim 2, characterized in that The local feature extraction module includes a sequentially connected 1×1 convolution, a first GELU activation function, a 3×3 convolution, a second GELU activation function, a channel attention unit and a 1×1 convolution; The output of the local feature extraction unit is: in, is the local input feature segmented by the channel segmentation module and input to the local feature extraction module; 1×1 (·),f 3×3 (·), and CA(·) are 1×1 convolution, 3×3 convolution, GELU activation function and channel attention module respectively.

4. The method according to claim 2, characterized in that The non-local feature extraction module includes an inverse sub-pixel convolution, a channel segmentation unit, a variance extraction unit, a 3×3 depth-separable convolution, a 1×1 convolution, a third GELU activation function, a sub-pixel convolution, a channel splicing unit, and a multiplication unit; The inverse sub-pixel convolution and the variance extraction unit are used to receive the non-local input features segmented by the channel segmentation unit and input to the non-local feature extraction module; The channel segmentation unit is used to generate four identical feature groups from the high-dimensional features output by the inverse sub-pixel convolution and input them into the 3×3 depth-separable convolution; The 1×1 convolution is connected to the third GELU activation function and receives the result of element-wise addition of the output of the variance extraction unit and the output of the 3×3 depth-wise separable convolution; The sub-pixel convolution unit is connected to the channel stitching unit to perform sub-pixel convolution processing and stitching processing on the result of element addition; The multiplication unit is used to multiply the results after sub-pixel convolution processing and splicing processing by the input image information input to the non-local feature extraction module element by element and output the multiplication result.

5. The method according to claim 4, characterized in that The output of the non-local feature extraction unit is: in, is the non-local input feature segmented by the channel segmentation unit and input to the non-local feature extraction module; Ω(·) and are the splicing operation of the channel splicing unit and the feature segmentation operation of the channel segmentation unit respectively; ↑(·) and ↓(·) are the sub-pixel convolution and anti-sub-pixel convolution respectively; express The variance of f 1×1 (·), and They are 1×1 convolution, 3×3 depth-separable convolution and GELU activation function respectively.

6. The method according to claim 2, characterized in that The lightweight adaptive gating module includes an input module, a 1×1 convolution, a channel segmentation module, a 3×3 convolution, a multiplication module, a GELU activation function, a channel attention module, a spatial attention module, an addition module and an output module; The input module is used to receive input image information and input it into the channel segmentation module through the first 1×1 convolution and the fourth GELU activation function; The channel segmentation module is used to segment the input image information after the number of channels is increased by the first 1×1 convolution to obtain a first feature and a second feature; 3×3 convolution is used to perform convolution processing on the first feature; The multiplication module is used to multiply the output of the 3×3 convolution by the first learnable parameter on a pixel basis, and multiply the result by the second feature and the second learnable parameter on a pixel basis to obtain a multiplication result; The second 1×1 convolution is used to perform convolution processing on the multiplication result and input it into the parallel channel attention module and spatial attention module through the fifth GELU activation function; The addition module is used to add the multiplication result of the second feature and the second learnable parameter, the output of the channel attention module and the output of the spatial attention module on a pixel basis to obtain an addition result; The output module is used to output the addition result.

7. The method according to claim 6, characterized in that The output of the lightweight adaptive gating module is: F out =SA(F c )+CA(F d )+βF b ; Among them, SA(·) and CA(·) are the spatial attention module and channel attention module respectively; F2=αβF b ·f 3×3 (F a ); F a and F b are the first and second characteristics respectively, F a , F in To input image information, It is the feature segmentation operation of the channel segmentation unit; α and β are the first and second learnable parameters respectively; f 1×1 (·),f 3×3 (·)and They are 1×1 convolution, 3×3 convolution and GELU activation functions respectively.

8. The method according to claim 6, characterized in that The output of the efficient feature extraction module is: F EFEM-out =f AGB (f OFAM (F EFEM-in )+F EFEM-in )+f OFAM (F EFEM-in )+F EFEM-in ; Among them, f AGB (·) is a lightweight adaptive gating module, f OFAM (·) is a lightweight all-round feature aggregation module, F EFEM-in and F EFEM-out They are the input and output of the efficient feature extraction module respectively.

9. The method according to claim 1, characterized in that The image reconstruction module is implemented by sub-pixel convolution upsampling, and the super-resolution image output image is: Among them, ↑(·) is sub-pixel convolution, It is 7 efficient feature extraction modules connected in series, f 3×3 (·) is a 3×3 convolution, I LR and I SR are low-resolution input images and super-resolution output images, respectively.

10. A lightweight image super-resolution device based on an adaptive feature aggregation network, characterized in that: include: The acquisition module is used to obtain low-resolution input images and perform convolution processing; a processing module for inputting the low-resolution input image after convolution processing into a pre-built adaptive feature-based aggregation network for super-resolution processing; the adaptive feature-based aggregation network includes an image reconstruction module and multiple series-connected efficient feature extraction modules, the efficient feature extraction modules are used for feature extraction, and the efficient feature extraction modules include a lightweight omnidirectional feature aggregation module and a lightweight adaptive gating module; the image reconstruction module is used to reconstruct the feature extraction results obtained by the multiple series-connected efficient feature extraction modules and the low-resolution input image after convolution processing to obtain a super-resolution output image; An output module is used to output the super-resolution output image.

11. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 9 is implemented.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

13. A computer program product, characterized in that The method comprises at least one instruction or at least one program, wherein the at least one instruction or the at least one program is loaded and executed by a processor to implement the method according to any one of claims 1 to 9.

Images