Image super-resolution method and system based on frequency information feedback

Abstract

The present invention relates to the field of image processing, and proposes an image super-resolution method and system based on frequency information feedback. A deep feature extraction module is designed to obtain features of multiple levels of depth, enhance the acquisition of texture detail feature information, and thus improve the detail accuracy of overall super-resolution reconstruction. A frequency information feedback module connecting adjacent time steps is then designed, and deep features of different levels in a previous time step are adjusted by frequency information and fed back to a subsequent time step, so as to provide guidance for the step of extracting deep features according to shallow features in the subsequent time step. After multiple time steps are sequentially processed, high-frequency information in the features is effectively utilized, and detail information in the features is greatly enhanced. The present invention greatly improves the texture detail reconstruction accuracy of the image super-resolution method.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to an image super-resolution method and system based on frequency information feedback. Background Art

[0002] With the continuous development of computer vision technology, the requirements for image quality are also increasing. However, the high-quality image acquisition equipment itself is expensive, so in most cases the consistency of image quality used for computer vision technology cannot be guaranteed, resulting in uneven results. Therefore, super-resolution processing of low-quality and low-resolution images becomes particularly critical.

[0003] In the prior art, super-resolution processing of images is usually performed according to deep learning algorithms, and the good performance of convolutional neural networks in deep learning is used to obtain high-quality and high-resolution images. However, when designing existing super-resolution networks based on deep learning algorithms, whether using skip connections or dense connections, they do not effectively utilize the high-frequency information in the features, resulting in the problem of texture detail distortion in the reconstructed high-resolution images.

[0004] Therefore, how to design an image super-resolution method to avoid the problem of texture detail distortion and enhance the texture detail information of the reconstructed high-resolution image has become an urgent problem to be solved. Summary of the invention

[0005] Based on this, the purpose of the present invention is to provide an image super-resolution method and system based on frequency information feedback, by designing a deep feature extraction module to obtain features of multiple levels of depth, enhancing the acquisition of texture detail feature information, and thus improving the detail accuracy of the overall super-resolution reconstruction, and then designing a frequency information feedback module connecting adjacent time steps, the deep features of different levels in the previous time step are adjusted by frequency information, and then fed back to the next time step, so as to provide guidance for the step of extracting deep features according to shallow features in the next time step, and then as multiple time steps are processed sequentially, the high-frequency information in the features is effectively utilized, and the detail information in the features is greatly enhanced. The present invention greatly improves the texture detail reconstruction accuracy of the image super-resolution method.

[0006] The present invention proposes an image super-resolution method based on frequency information feedback, comprising:

[0007] Obtaining a low-resolution image to be processed and preprocessing it, and inputting the preprocessed low-resolution image into an image super-resolution network, wherein the image super-resolution network includes a plurality of time-step cycle structures and is expanded in a time-step sequence;

[0008] Input the low-resolution image into a shallow feature extraction module to obtain shallow features, and then input the low-resolution image into a deep feature extraction module;

[0009] The deep feature extraction module obtains multi-level deep features according to the feature layer depth sequence and inputs them into the frequency information feedback module;

[0010] The frequency information feedback module performs feature adjustment on the multi-level deep features according to the frequency information to obtain key deep features, and feeds back to the deep feature extraction module of the next time step;

[0011] When the cycle reaches the final time step, the reconstruction module extracts the features of the deep feature module at the current time step to reconstruct the final high-resolution image.

[0012] In summary, according to the above-mentioned image super-resolution method based on frequency information feedback, a deep feature extraction module is designed to obtain features of multiple levels of depth, enhance the acquisition of texture detail feature information, and thus improve the detail accuracy of the overall super-resolution reconstruction, and then a frequency information feedback module connecting adjacent time steps is designed, and the deep features of different levels in the previous time step are adjusted by frequency information and fed back to the next time step to guide the step of extracting deep features according to shallow features in the next time step, and then as multiple time steps are processed sequentially, the high-frequency information in the features is effectively utilized, and the detail information in the features is greatly enhanced. The present invention greatly improves the texture detail reconstruction accuracy of the image super-resolution method. Specifically, a low-resolution image to be processed is obtained and preprocessed, and the preprocessed low-resolution image is input into an image super-resolution network, the image super-resolution network includes a plurality of time-step loop structures and is expanded in a time-step order, the low-resolution image is input into a shallow feature extraction module to obtain shallow features, and then input into a deep feature extraction module, the deep feature extraction module obtains multi-level deep features according to the feature level depth order, and inputs into a frequency information feedback module to enhance the acquisition of texture detail feature information, thereby improving the detail accuracy of the overall super-resolution reconstruction, the frequency information feedback module performs feature adjustment on the multi-level deep features according to the frequency information to obtain key deep features, and feeds back to the deep feature extraction module of the next time step, effectively utilizing the high-frequency information in the features, greatly enhancing the detail information in the features, and when the cycle reaches the final time step, the reconstruction module reconstructs and generates a final high-resolution image according to the features of the deep feature extraction module of the current time step, and the present invention greatly improves the texture detail reconstruction accuracy of the image super-resolution method.

[0013] Furthermore, the step of inputting the low-resolution image into a shallow feature extraction module to obtain shallow features, and then inputting the low-resolution image into a deep feature extraction module specifically includes:

[0014] Input the low-resolution image into the shallow feature extraction module;

[0015] The shallow feature extraction module includes two serially connected 3×3 convolutional layers, and the shallow features of the low-resolution image are extracted according to the convolutional layers. The algorithm of the shallow feature extraction module is as follows:

[0016]

[0017] in, Represents shallow features, represents shallow feature extraction, represents a low-resolution image, Represents the time step, 0 means the feature level is shallow;

[0018] The acquired shallow features are input into the deep feature extraction module.

[0019] Furthermore, the deep feature extraction module obtains multi-level deep features according to the feature layer depth order and inputs the steps of the frequency information feedback module, specifically including:

[0020] The deep feature extraction module obtains the shallow features, the deep feature extraction module includes a plurality of residual aggregation blocks, each residual aggregation block corresponds to a feature level, and the plurality of residual aggregation blocks are linearly connected in series from shallow to deep according to the depth order of the corresponding feature level;

[0021] The residual aggregation block obtains deep features based on shallow features. The algorithm for obtaining deep features is as follows:

[0022]

[0023] in, Represents deep features, represents deep feature extraction, Represents shallow features, Represents the time step, 0 means the feature level is shallow, Represents the feature level corresponding to the current residual aggregation block, [1, N];

[0024] The deep features obtained from multiple residual aggregation blocks are selected as multi-level deep features and input into the frequency information feedback module.

[0025] Furthermore, the residual aggregation block specifically includes:

[0026] The residual aggregation block includes multiple residual structures, each of which includes a residual branch and an original branch, the residual branch includes a single residual block, the latter residual structure is contained in the residual branch of the previous residual structure to form a residual aggregation structure, and the residual branch of the outermost residual structure of the residual aggregation structure also includes a 1×1 convolution block, and the residual block includes two 3×3 convolution blocks and a ReLU layer.

[0027] Furthermore, the frequency information feedback module performs feature adjustment on the multi-level deep features according to the frequency information to obtain key deep features, and feeds back the key deep features to the deep feature extraction module of the next time step, specifically including:

[0028] The frequency information feedback module obtains multi-level deep features, the frequency information feedback module includes a frequency information feedback block and a 1×1 convolution block, the 1×1 convolution block is connected to the deep feature extraction module of the next time step, and the frequency information feedback block performs feature adjustment on the multi-level deep features;

[0029] The frequency information feedback module selects key deep features and feeds them back to the deep feature extraction module of the next time step. The algorithm for selecting key deep features is as follows:

[0030]

[0031] in, represents the key deep features, represents the key deep feature selection, represents the multi-level deep features selected, M represents the number of multi-level deep features selected, , represents the time step;

[0032] After being fed back to the first residual aggregation block of the deep feature extraction module of the next time step, the output deep features are defined as follows:

[0033]

[0034] in, represents the deep features output by the first residual aggregation block of the deep feature extraction module at the next time step, represents the deep feature extraction of the first residual aggregation block of the deep feature extraction module at the next time step, Represents a merge operation. represents a 1×1 convolution operation, Represents shallow features.

[0035] Furthermore, the frequency information feedback block performs feature adjustment on the multi-level deep features, specifically including:

[0036] The frequency information feedback block divides the multi-level deep features into i The features after division ,in i represents the feature ordinal number after partitioning, represents the feature dimension after division, H represents the feature height, W Indicates feature width;

[0037] The two-dimensional DCT frequency components corresponding to all the divided features are obtained. The specific formula for obtaining the two-dimensional DCT frequency components corresponding to all the divided features is as follows:

[0038]

[0039] in, express dimensional vector, , 2DDCT represents the two-dimensional DCT frequency component, is corresponding to The two-dimensional frequency components of , ;

[0040] The overall vector of multi-level deep features is obtained according to the merging operation, and the algorithm of the merging operation is as follows:

[0041]

[0042] in, Freq Represents the overall vector of multi-level deep features, , , represents the overall feature dimension, Indicates a merge operation;

[0043] The feature after frequency information adjustment is finally obtained, and the formula for finally obtaining the feature after frequency information adjustment is as follows:

[0044]

[0045]

[0046] in, Represents the vector value after sigmoid transformation, Indicates converting the vector value to between 0 and 1. Represents the features after adjustment of frequency information.

[0047] Furthermore, when the cycle reaches the final time step, the reconstruction module extracts the features of the deep feature module at the current time step to reconstruct and generate the final high-resolution image, which specifically includes:

[0048] The reconstruction module includes a reconstruction branch and an upsampling branch, the reconstruction branch includes two 3×3 convolution blocks and a deconvolution layer, and the upsampling branch includes an upsampling branch;

[0049] The final high-resolution image is reconstructed and generated by the following algorithm:

[0050]

[0051] in, represents the final high-resolution image, Represents a rebuild operation. represents the upsampling operation, represents the features of the deep feature extraction module at the final time step, represents a low-resolution image, T represents the final time step.

[0052] The present invention proposes an image super-resolution system based on frequency information feedback, comprising:

[0053] An acquisition module is used to acquire a low-resolution image to be processed and perform preprocessing, and input the preprocessed low-resolution image into an image super-resolution network, wherein the image super-resolution network includes a plurality of time-step cycle structures and is expanded in a time-step sequence;

[0054] A shallow feature extraction module, used for inputting the low-resolution image into the shallow feature extraction module to obtain shallow features, and then inputting the shallow features into the deep feature extraction module;

[0055] A deep feature extraction module is used for the deep feature extraction module to obtain multi-level deep features according to the feature level depth sequence and input them into the frequency information feedback module;

[0056] A frequency information feedback module, which is used for the frequency information feedback module to adjust the multi-level deep features according to the frequency information to obtain key deep features, and feed back to the deep feature extraction module of the next time step;

[0057] The reconstruction module is used to reconstruct and generate a final high-resolution image based on the features of the module according to the residual features of the current time step when the cycle reaches the final time step.

[0058] The present invention also provides a storage medium, wherein the storage medium stores one or more programs, and when the programs are executed by a processor, the image super-resolution method based on frequency information feedback as described above is implemented.

[0059] The present invention also provides a computer device, the computer device comprising a memory and a processor, wherein:

[0060] The memory is used to store computer programs;

[0061] When the processor is used to execute the computer program stored in the memory, the image super-resolution method based on frequency information feedback as described above is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 A flowchart of an image super-resolution method based on frequency information feedback proposed in the first embodiment of the present invention;

[0063] Figure 2 A flowchart of an image super-resolution method based on frequency information feedback proposed in the second embodiment of the present invention;

[0064] Figure 3 This is a schematic diagram of the structure of an image super-resolution system based on frequency information feedback proposed in the third embodiment of the present invention.

[0065] The following specific implementation manner will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0066] In order to facilitate understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Several embodiments of the present invention are provided in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0067] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be a central element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be a central element at the same time. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are for illustrative purposes only.

[0068] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0069] See also Figure 1, which is a flow chart of an image super-resolution method based on frequency information feedback proposed in the first embodiment of the present invention, the image super-resolution method based on frequency information feedback comprises steps S01 to S05, wherein:

[0070] Step S01: obtaining a low-resolution image to be processed and preprocessing it, and inputting the preprocessed low-resolution image into an image super-resolution network;

[0071] It should be noted that the image super-resolution network in this embodiment includes multiple time step cycle structures and is expanded in the order of time steps.

[0072] Step S02: input the low-resolution image into a shallow feature extraction module to obtain shallow features, and then input the shallow features into a deep feature extraction module;

[0073] It should be noted that in this embodiment, the low-resolution image is input into the shallow feature extraction module;

[0074] The shallow feature extraction module includes two serially connected 3×3 convolutional layers, and the shallow features of the low-resolution image are extracted according to the convolutional layers. The algorithm of the shallow feature extraction module is as follows:

[0075]

[0076] in, Represents shallow features, represents shallow feature extraction, represents a low-resolution image, Represents the time step, 0 means the feature level is shallow;

[0077] The acquired shallow features are input into the deep feature extraction module.

[0078] Step S03: the deep feature extraction module obtains multi-level deep features according to the feature layer depth sequence and inputs them into the frequency information feedback module;

[0079] It should be noted that in this embodiment, the deep feature extraction module obtains shallow features, and the deep feature extraction module includes multiple residual aggregation blocks, each residual aggregation block corresponds to a feature level, and the multiple residual aggregation blocks are linearly connected in series from shallow to deep according to the depth order of the corresponding feature level;

[0080] The residual aggregation block obtains deep features based on shallow features. The algorithm for obtaining deep features is as follows:

[0081]

[0082] in, Represents deep features, represents deep feature extraction, Represents shallow features, Represents the time step, 0 means the feature level is shallow, Represents the feature level corresponding to the current residual aggregation block, [1, N];

[0083] The deep features obtained by multiple residual aggregation blocks are selected as multi-level deep features and input into the frequency information feedback module;

[0084] The residual aggregation block includes multiple residual structures, each of which includes a residual branch and an original branch, the residual branch includes a single residual block, the latter residual structure is contained in the residual branch of the previous residual structure to form a residual aggregation structure, and the residual branch of the outermost residual structure of the residual aggregation structure also includes a 1×1 convolution block, and the residual block includes two 3×3 convolution blocks and a ReLU layer.

[0085] Step S04: The frequency information feedback module performs feature adjustment on the multi-level deep features according to the frequency information to obtain key deep features, and feeds back to the deep feature extraction module of the next time step;

[0086] It should be noted that in this embodiment, the frequency information feedback module obtains multi-level deep features, and the frequency information feedback module includes a frequency information feedback block and a 1×1 convolution block, and the 1×1 convolution block is connected to the deep feature extraction module of the next time step, and the frequency information feedback block performs feature adjustment on the multi-level deep features;

[0087] The frequency information feedback module selects key deep features and feeds them back to the deep feature extraction module of the next time step. The algorithm for selecting key deep features is as follows:

[0088]

[0089] in, represents the key deep features, represents the key deep feature selection, represents the multi-level deep features selected, M represents the number of multi-level deep features selected, , represents the time step;

[0090] After being fed back to the first residual aggregation block of the deep feature extraction module of the next time step, the output deep features are defined as follows:

[0091]

[0092] in, represents the deep features output by the first residual aggregation block of the deep feature extraction module at the next time step, represents the deep feature extraction of the first residual aggregation block of the deep feature extraction module at the next time step, Represents a merge operation. represents a 1×1 convolution operation, Indicates shallow features;

[0093] The frequency information feedback block divides the multi-level deep features into i The features after division ,in i represents the feature ordinal number after partitioning, represents the feature dimension after division, H represents the feature height, W Indicates feature width;

[0094] The two-dimensional DCT frequency components corresponding to all the divided features are obtained. The specific formula for obtaining the two-dimensional DCT frequency components corresponding to all the divided features is as follows:

[0095]

[0096] in, express dimensional vector, , 2DDCT represents the two-dimensional DCT frequency component, is corresponding to The two-dimensional frequency components of , ;

[0097] The overall vector of multi-level deep features is obtained according to the merging operation, and the algorithm of the merging operation is as follows:

[0098]

[0099] in, Freq Represents the overall vector of multi-level deep features, , , represents the overall feature dimension, Indicates a merge operation;

[0100] The feature after frequency information adjustment is finally obtained, and the formula for finally obtaining the feature after frequency information adjustment is as follows:

[0101]

[0102]

[0103] in, Represents the vector value after sigmoid transformation, Indicates converting the vector value to between 0 and 1. Represents the features adjusted by frequency information.

[0104] Step S05: When the cycle reaches the final time step, the reconstruction module reconstructs and generates a final high-resolution image based on the features of the deep feature extraction module at the current time step;

[0105] It should be noted that the reconstruction module in this embodiment includes a reconstruction branch and an upsampling branch, the reconstruction branch includes two 3×3 convolution blocks and a deconvolution layer, and the upsampling branch includes an upsampling branch;

[0106] The final high-resolution image is reconstructed and generated by the following algorithm:

[0107]

[0108] in, represents the final high-resolution image, Represents a rebuild operation. represents the upsampling operation, represents the features of the deep feature extraction module at the final time step, represents a low-resolution image, T represents the final time step.

[0109] In summary, according to the above-mentioned image super-resolution method based on frequency information feedback, a deep feature extraction module is designed to obtain features of multiple levels of depth, enhance the acquisition of texture detail feature information, and thus improve the detail accuracy of the overall super-resolution reconstruction, and then a frequency information feedback module connecting adjacent time steps is designed, and the deep features of different levels in the previous time step are adjusted by frequency information and fed back to the next time step to guide the step of extracting deep features according to shallow features in the next time step, and then as multiple time steps are processed sequentially, the high-frequency information in the features is effectively utilized, and the detail information in the features is greatly enhanced. The present invention greatly improves the texture detail reconstruction accuracy of the image super-resolution method. Specifically, a low-resolution image to be processed is obtained and preprocessed, and the preprocessed low-resolution image is input into an image super-resolution network, the image super-resolution network includes a plurality of time-step loop structures and is expanded in a time-step order, the low-resolution image is input into a shallow feature extraction module to obtain shallow features, and then input into a deep feature extraction module, the deep feature extraction module obtains multi-level deep features according to the feature level depth order, and inputs into a frequency information feedback module to enhance the acquisition of texture detail feature information, thereby improving the detail accuracy of the overall super-resolution reconstruction, the frequency information feedback module performs feature adjustment on the multi-level deep features according to the frequency information to obtain key deep features, and feeds back to the deep feature extraction module of the next time step, effectively utilizing the high-frequency information in the features, greatly enhancing the detail information in the features, and when the cycle reaches the final time step, the reconstruction module reconstructs and generates a final high-resolution image according to the features of the deep feature extraction module of the current time step, and the present invention greatly improves the texture detail reconstruction accuracy of the image super-resolution method.

[0110] See also Figure 2 , which is a flow chart of an image super-resolution method based on frequency information feedback proposed in the second embodiment of the present invention, the image super-resolution method based on frequency information feedback comprises steps S11 to S16, wherein:

[0111] Step S11: obtaining a low-resolution image to be processed and preprocessing it, and inputting the preprocessed low-resolution image into an image super-resolution network;

[0112] Step S12: inputting the low-resolution image into a shallow feature extraction module, extracting shallow features of the low-resolution image according to the convolution layer, and inputting the obtained shallow features into a deep feature extraction module;

[0113] Step S13: the deep feature extraction module obtains the shallow features, the residual aggregation block obtains the deep features according to the shallow features, and the deep features obtained by multiple residual aggregation blocks are selected as multi-level deep features and input into the frequency information feedback module;

[0114] Step S14: the frequency information feedback module obtains multi-level deep features, selects key deep features and feeds them back to the deep feature extraction module of the next time step;

[0115] Step S15: The frequency information feedback block divides the multi-level deep features into i , obtain the two-dimensional DCT frequency components corresponding to all the divided features, and obtain the overall vector of multi-level deep features according to the merging operation, so as to finally obtain the features adjusted by the frequency information;

[0116] Step S16: When the cycle reaches the final time step, the reconstruction module generates a final high-resolution image based on the feature reconstruction of the deep feature extraction module at the current time step.

[0117] In summary, according to the above-mentioned image super-resolution method based on frequency information feedback, a deep feature extraction module is designed to obtain features of multiple levels of depth, enhance the acquisition of texture detail feature information, and thus improve the detail accuracy of the overall super-resolution reconstruction, and then a frequency information feedback module connecting adjacent time steps is designed, and the deep features of different levels in the previous time step are adjusted by frequency information and fed back to the next time step to guide the step of extracting deep features according to shallow features in the next time step, and then as multiple time steps are processed sequentially, the high-frequency information in the features is effectively utilized, and the detail information in the features is greatly enhanced. The present invention greatly improves the texture detail reconstruction accuracy of the image super-resolution method. Specifically, a low-resolution image to be processed is obtained and preprocessed, and the preprocessed low-resolution image is input into an image super-resolution network, the image super-resolution network includes a plurality of time-step loop structures and is expanded in a time-step order, the low-resolution image is input into a shallow feature extraction module to obtain shallow features, and then input into a deep feature extraction module, the deep feature extraction module obtains multi-level deep features according to the feature level depth order, and inputs into a frequency information feedback module to enhance the acquisition of texture detail feature information, thereby improving the detail accuracy of the overall super-resolution reconstruction, the frequency information feedback module performs feature adjustment on the multi-level deep features according to the frequency information to obtain key deep features, and feeds back to the deep feature extraction module of the next time step, effectively utilizing the high-frequency information in the features, greatly enhancing the detail information in the features, and when the cycle reaches the final time step, the reconstruction module reconstructs and generates a final high-resolution image according to the features of the deep feature extraction module of the current time step, and the present invention greatly improves the texture detail reconstruction accuracy of the image super-resolution method.

[0118] See also Figure 3 , which is a schematic diagram of the structure of an image super-resolution system based on frequency information feedback proposed in the third embodiment of the present invention, the system comprises:

[0119] An acquisition module 10 is used to acquire a low-resolution image to be processed and perform preprocessing, and input the preprocessed low-resolution image into an image super-resolution network, wherein the image super-resolution network comprises a plurality of time-step cycle structures and is expanded in a time-step sequence;

[0120] A shallow feature extraction module 20, used for inputting the low-resolution image into the shallow feature extraction module to obtain shallow features, and inputting the shallow features into the deep feature extraction module;

[0121] A deep feature extraction module 30, which is used for the deep feature extraction module to obtain multi-level deep features according to the feature layer depth sequence and input them into the frequency information feedback module;

[0122] A frequency information feedback module 40, which is used for the frequency information feedback module to adjust the multi-level deep features according to the frequency information to obtain key deep features, and feed back to the deep feature extraction module of the next time step;

[0123] The reconstruction module 50 is used for reconstructing and generating a final high-resolution image based on the features of the module according to the residual features of the current time step when the cycle reaches the final time step.

[0124] The present invention also provides a computer storage medium on which one or more programs are stored, and when the programs are executed by a processor, the above-mentioned image super-resolution method based on frequency information feedback is implemented.

[0125] The present invention also proposes a computer device, comprising a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to implement the above-mentioned image super-resolution method based on frequency information feedback.

[0126] Those skilled in the art will appreciate that the logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For purposes of this specification, "computer-readable medium" may be any device that can contain storage, communication, propagation or transmission of a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0127] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0128] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or a combination thereof: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0129] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0130] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. An image super-resolution method based on frequency information feedback, characterized in that: include: Obtaining a low-resolution image to be processed and preprocessing it, and inputting the preprocessed low-resolution image into an image super-resolution network, wherein the image super-resolution network includes a plurality of time-step cycle structures and is expanded in a time-step sequence; Input the low-resolution image into a shallow feature extraction module to obtain shallow features, and then input the low-resolution image into a deep feature extraction module; The deep feature extraction module obtains multi-level deep features according to the feature layer depth sequence and inputs them into the frequency information feedback module; The deep feature extraction module obtains multi-level deep features according to the feature layer depth sequence and inputs them into the frequency information feedback module, specifically including: The deep feature extraction module obtains the shallow features, the deep feature extraction module includes a plurality of residual aggregation blocks, each residual aggregation block corresponds to a feature level, and the plurality of residual aggregation blocks are linearly connected in series from shallow to deep according to the depth order of the corresponding feature level; The residual aggregation block obtains deep features based on shallow features. The algorithm for obtaining deep features is as follows: in, Represents deep features, represents deep feature extraction, Represents shallow features, Represents the time step, 0 means the feature level is shallow, Represents the feature level corresponding to the current residual aggregation block, [1, N]; The deep features obtained by multiple residual aggregation blocks are selected as multi-level deep features and input into the frequency information feedback module; The residual aggregation block specifically includes: The residual aggregation block includes a plurality of residual structures, each of which includes a residual branch and an original branch, the residual branch includes a single residual block, the latter residual structure is contained in the residual branch of the former residual structure to form a residual aggregation structure, the residual branch of the outermost residual structure of the residual aggregation structure also includes a 1×1 convolution block, and the residual block includes two 3×3 convolution blocks and a ReLU layer; The frequency information feedback module performs feature adjustment on the multi-level deep features according to the frequency information to obtain key deep features, and feeds back to the deep feature extraction module of the next time step; The frequency information feedback module performs feature adjustment on the multi-level deep features according to the frequency information to obtain key deep features, and feeds back to the deep feature extraction module of the next time step, specifically including: The frequency information feedback module obtains multi-level deep features, the frequency information feedback module includes a frequency information feedback block and a 1×1 convolution block, the 1×1 convolution block is connected to the deep feature extraction module of the next time step, and the frequency information feedback block performs feature adjustment on the multi-level deep features; The frequency information feedback module selects key deep features and feeds them back to the deep feature extraction module of the next time step. The algorithm for selecting key deep features is as follows: in, represents the key deep features, represents the key deep feature selection, represents the multi-level deep features selected, M represents the number of multi-level deep features selected, , represents the time step; After being fed back to the first residual aggregation block of the deep feature extraction module of the next time step, the output deep features are defined as follows: in, represents the deep features output by the first residual aggregation block of the deep feature extraction module at the next time step, represents the deep feature extraction of the first residual aggregation block of the deep feature extraction module at the next time step, Represents a merge operation. represents a 1×1 convolution operation, Indicates shallow features; When the cycle reaches the final time step, the reconstruction module extracts the features of the deep feature module at the current time step to reconstruct the final high-resolution image.

2. The image super-resolution method based on frequency information feedback according to claim 1, characterized in that: The step of inputting the low-resolution image into a shallow feature extraction module to obtain shallow features, and then inputting the low-resolution image into a deep feature extraction module specifically includes: Input the low-resolution image into the shallow feature extraction module; The shallow feature extraction module includes two serially connected 3×3 convolutional layers, and the shallow features of the low-resolution image are extracted according to the convolutional layers. The algorithm of the shallow feature extraction module is as follows: in, Represents shallow features, represents shallow feature extraction, represents a low-resolution image, Represents the time step, 0 means the feature level is shallow; The acquired shallow features are input into the deep feature extraction module.

3. The image super-resolution method based on frequency information feedback according to claim 1, characterized in that: The step of the frequency information feedback block performing feature adjustment on the multi-level deep features specifically includes: The frequency information feedback block divides the multi-level deep features into i The features after division ,in i represents the feature ordinal number after partitioning, represents the feature dimension after division, H represents the feature height, W Indicates feature width; The two-dimensional DCT frequency components corresponding to all the divided features are obtained. The specific formula for obtaining the two-dimensional DCT frequency components corresponding to all the divided features is as follows: in, express dimensional vector, , 2DDCT represents the two-dimensional DCT frequency component, is corresponding to The two-dimensional frequency components of , ; The overall vector of multi-level deep features is obtained according to the merging operation, and the algorithm of the merging operation is as follows: in, Freq Represents the overall vector of multi-level deep features, , , represents the overall feature dimension, Indicates a merge operation; The feature after frequency information adjustment is finally obtained, and the formula for finally obtaining the feature after frequency information adjustment is as follows: in, Represents the vector value after sigmoid transformation, Indicates converting the vector value to between 0 and 1. Represents the features adjusted by frequency information.

4. The image super-resolution method based on frequency information feedback according to claim 1, characterized in that: When the cycle reaches the final time step, the reconstruction module extracts the features of the deep feature module of the current time step to reconstruct and generate the final high-resolution image, which specifically includes: The reconstruction module includes a reconstruction branch and an upsampling branch, the reconstruction branch includes two 3×3 convolution blocks and a deconvolution layer, and the upsampling branch includes an upsampling branch; The final high-resolution image is reconstructed and generated by the following algorithm: in, represents the final high-resolution image, Represents a rebuild operation. represents the upsampling operation, represents the features of the deep feature extraction module at the final time step, represents a low-resolution image, T represents the final time step.

5. An image super-resolution system based on frequency information feedback, characterized in that: include: An acquisition module is used to acquire a low-resolution image to be processed and perform preprocessing, and input the preprocessed low-resolution image into an image super-resolution network, wherein the image super-resolution network includes a plurality of time-step cycle structures and is expanded in a time-step sequence; A shallow feature extraction module, used for inputting the low-resolution image into the shallow feature extraction module to obtain shallow features, and then inputting the shallow features into the deep feature extraction module; A deep feature extraction module is used for the deep feature extraction module to obtain multi-level deep features according to the feature level depth sequence and input them into the frequency information feedback module; The deep feature extraction module obtains multi-level deep features according to the feature layer depth sequence and inputs them into the frequency information feedback module, specifically including: The deep feature extraction module obtains the shallow features, the deep feature extraction module includes a plurality of residual aggregation blocks, each residual aggregation block corresponds to a feature level, and the plurality of residual aggregation blocks are linearly connected in series from shallow to deep according to the depth order of the corresponding feature level; The residual aggregation block obtains deep features based on shallow features. The algorithm for obtaining deep features is as follows: in, Represents deep features, represents deep feature extraction, Represents shallow features, Represents the time step, 0 means the feature level is shallow, Represents the feature level corresponding to the current residual aggregation block, [1, N]; The deep features obtained by multiple residual aggregation blocks are selected as multi-level deep features and input into the frequency information feedback module; The residual aggregation block specifically includes: The residual aggregation block includes a plurality of residual structures, each of which includes a residual branch and an original branch, the residual branch includes a single residual block, the latter residual structure is contained in the residual branch of the former residual structure to form a residual aggregation structure, the residual branch of the outermost residual structure of the residual aggregation structure also includes a 1×1 convolution block, and the residual block includes two 3×3 convolution blocks and a ReLU layer; A frequency information feedback module, which is used for the frequency information feedback module to adjust the multi-level deep features according to the frequency information to obtain key deep features, and feed back to the deep feature extraction module of the next time step; The frequency information feedback module performs feature adjustment on the multi-level deep features according to the frequency information to obtain key deep features, and feeds back to the deep feature extraction module of the next time step, specifically including: The frequency information feedback module obtains multi-level deep features, the frequency information feedback module includes a frequency information feedback block and a 1×1 convolution block, the 1×1 convolution block is connected to the deep feature extraction module of the next time step, and the frequency information feedback block performs feature adjustment on the multi-level deep features; The frequency information feedback module selects key deep features and feeds them back to the deep feature extraction module of the next time step. The algorithm for selecting key deep features is as follows: in, represents the key deep features, represents the key deep feature selection, represents the multi-level deep features selected, M represents the number of multi-level deep features selected, , represents the time step; After being fed back to the first residual aggregation block of the deep feature extraction module of the next time step, the output deep features are defined as follows: in, represents the deep features output by the first residual aggregation block of the deep feature extraction module at the next time step, represents the deep feature extraction of the first residual aggregation block of the deep feature extraction module at the next time step, Represents a merge operation. represents a 1×1 convolution operation, Indicates shallow features; The reconstruction module is used to reconstruct and generate a final high-resolution image based on the features of the module according to the residual features of the current time step when the cycle reaches the final time step.

6. A storage medium, characterized in that: The storage medium stores one or more programs, which, when executed by a processor, implement the image super-resolution method based on frequency information feedback as described in any one of claims 1 to 4.

7. A computer device, characterized in that: The computer device comprises a memory and a processor, wherein: The memory is used to store computer programs; When the processor is used to execute the computer program stored in the memory, the image super-resolution method based on frequency information feedback described in any one of claims 1 to 4 is implemented.

Images