A lightweight image and video quality reconstruction method, device and equipment

Abstract

The present application provides a lightweight image and video quality reconstruction method and device, the method comprising performing X-fold downsampling processing on the image to be processed, and performing channel expansion to obtain a first feature map; performing low-level feature extraction on the first feature map; performing N-fold upsampling on the extracted low-level features, and performing high-level feature extraction on the sampling results of the N-fold upsampling to obtain reconstructed features; performing M-fold upsampling on the reconstructed features; using a preset high-frequency attention structure to extract a high-frequency feature map of the image to be processed and processing it with the sampling results of the M-fold upsampling to obtain a result map after enhancing the high frequency, providing an image processing solution with a low model method complexity and capable of being applied to mobile devices.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a lightweight image and video quality reconstruction method, device and equipment. Background Art

[0002] With the development of science and technology, mobile devices such as mobile phones and tablets have been fully integrated into people's daily lives, and the requirements for the imaging quality of such devices have become increasingly higher. However, due to compression, transmission, storage and other operations, the pictures or videos displayed on the mobile terminal will generate a lot of noise, loss of details and other distortions, which greatly affect the user's visual experience. In recent years, in order to improve the visual quality of pictures or videos displayed on the terminal, image or video denoising and super-resolution technologies have been widely used. In addition, with the rapid development of deep learning in the field of computer vision, image or video enhancement based on deep learning has become a new solution. However, in the industry, the super-resolution method based on deep learning faces two huge challenges in the implementation process: 1. The complexity of the image or video restoration model method is too high; 2. The computing power of mobile devices is limited. Summary of the invention

[0003] In view of this, embodiments of the present invention provide a lightweight image and video quality reconstruction method, apparatus and device to provide an image processing solution with a low model method complexity and applicable to mobile devices.

[0004] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0005] A lightweight image and video quality reconstruction method, comprising:

[0006] Get the image to be processed;

[0007] Downsampling the image to be processed by a factor of X, and performing channel expansion to obtain a first feature map;

[0008] Performing low-level feature extraction on the first feature map;

[0009] Upsampling the extracted low-level features by a factor of N, where N is not less than 2;

[0010] Performing high-level feature extraction on the N-fold upsampling sampling result to obtain reconstruction features;

[0011] Upsampling the reconstructed features by a factor of M, where M is not less than 2;

[0012] A preset high-frequency attention structure is used to extract a high-frequency feature map of the image to be processed and the high-frequency feature map is processed with the M-fold upsampling result to obtain an enhanced high-frequency result map.

[0013] Optionally, in the above lightweight image and video quality reconstruction method, the image to be processed is downsampled by X times to obtain a first feature map, and the number of channels of the first feature map is expanded;

[0014] A convolution operation with a step length of X is performed on the image to be processed using a preset convolution kernel to obtain a feature map that is downsampled by X times, which is recorded as a first feature map;

[0015] A nonlinear transformation is performed on the first feature map to expand the number of channels of the first feature map.

[0016] Optionally, performing a nonlinear transformation on the first feature map includes:

[0017] An information enhancement module is used to perform a nonlinear transformation on the first feature map.

[0018] Optionally, performing low-level feature extraction on the first feature map includes:

[0019] Performing a first distillation on the channels of the first feature map by using a feature enhancement distillation module, distilling feature maps of A channels from the channels of the first feature map, which are recorded as first distillation feature maps, where A is a preset positive integer;

[0020] Perform feature extraction and channel dimension reduction operations on the feature graph corresponding to the remaining channels of the first distillation, which is recorded as the first dimension reduction feature graph;

[0021] A feature enhancement distillation module is used to perform a second distillation on the channels of the first dimensionality reduction feature map, and feature maps of C channels are distilled from the channels of the first dimensionality reduction feature map, which are recorded as second distillation feature maps;

[0022] Perform feature extraction and channel dimension reduction operations on the feature maps corresponding to the remaining channels of the second distillation to obtain feature maps of B channels, which are recorded as second dimension reduction feature maps, where B is the number of channels after the image to be processed is downsampled by X times;

[0023] The first feature map, the first distillation feature map, the second distillation feature map, and the second dimensionality reduction feature map are fused in the channel dimension to obtain a first fused feature map;

[0024] Then, feature fusion and channel dimension reduction are performed on the first fused feature map to obtain a feature map with D channels, which is recorded as a second feature map;

[0025] Then, feature extraction is performed on the second feature map to obtain a third feature map, and the feature extraction result is the low-level feature extraction result.

[0026] Optionally, the extracted low-level features are upsampled by N times, where N is not less than 2, including:

[0027] The third feature map and the second feature map are fused in the channel dimension, and a 1*1 convolution is used to perform feature fusion on the fusion result to obtain a fourth feature map;

[0028] The fourth feature map is upsampled by a factor of 2.

[0029] Optionally, performing advanced feature extraction on the N-fold upsampling sampling result to obtain reconstruction features includes:

[0030] A feature enhancement distillation module for feature enhancement is used to perform high-level feature extraction on the 2-fold upsampling result of the fourth feature map to obtain reconstructed features.

[0031] Optionally, the reconstructed features are upsampled M times as follows:

[0032] The reconstructed features are upsampled by a factor of 2.

[0033] Optionally, the extracting a high-frequency feature map of the image to be processed by using a preset high-frequency attention structure and processing the image with the M-fold upsampling sampling result to obtain a high-frequency enhanced result map includes:

[0034] Using the Sobel operator to extract the high-frequency feature map of the image to be processed;

[0035] Fusing the extracted high-frequency feature map with the reconstructed feature in the channel dimension;

[0036] Using 1*1 convolution to perform channel dimension reduction processing on the fusion result of the high-frequency feature map and the reconstructed feature, so as to obtain a feature map with only one channel;

[0037] Performing a nonlinear transformation on the feature map with only one channel to obtain a value of each pixel position of the feature map after the feature map with only one channel is linearly transformed, and using the value of each pixel position as the weight of the high-frequency feature to obtain a high-frequency feature attention feature map;

[0038] Performing a dot multiplication of the high-frequency feature attention feature map and the reconstructed feature to obtain a feature map with enhanced high-frequency information;

[0039] Upsampling the image to be processed by using a bicubic interpolation method;

[0040] The sampling result of the reconstructed feature being upsampled by 2 times, the feature map with high-frequency information enhancement, and the upsampling processing result of the image to be processed are superimposed to obtain a result map with enhanced high frequency.

[0041] A lightweight image and video quality reconstruction device, comprising:

[0042] An acquisition unit, used for acquiring an image to be processed;

[0043] A processing unit is used to perform X-fold downsampling on the image to be processed and perform channel expansion to obtain a first feature map; perform low-level feature extraction on the first feature map; perform N-fold upsampling on the extracted low-level features, where N is not less than 2; perform high-level feature extraction on the sampling results of the N-fold upsampling to obtain reconstructed features; perform M-fold upsampling on the reconstructed features, where M is not less than 2; use a preset high-frequency attention structure to extract a high-frequency feature map from the image to be processed and process it with the sampling results of the M-fold upsampling to obtain a result map after enhancing the high frequency.

[0044] A lightweight image and video quality reconstruction system comprises: any one of the lightweight image and video quality reconstruction devices described above.

[0045] Based on the above technical scheme, the above scheme provided by the embodiment of the present invention obtains a first feature map by performing X-fold downsampling on the image to be processed and performing channel expansion; extracts low-level features from the first feature map; performs N-fold upsampling on the extracted low-level features; performs high-level feature extraction on the sampling results of the N-fold upsampling to obtain reconstructed features; performs M-fold upsampling on the reconstructed features; uses a preset high-frequency attention structure to extract the high-frequency feature map of the image to be processed and processes it with the sampling results of the M-fold upsampling to obtain an enhanced high-frequency result map, thereby providing an image processing scheme with a low model method complexity and capable of being applied to mobile devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0047] Figure 1 A network structure diagram of a lightweight image and video quality reconstruction method provided in an embodiment of the present application;

[0048] Figure 2 A schematic diagram of a process for lightweight image and video quality reconstruction method provided in an embodiment of the present application;

[0049] Figure 3 A schematic flow chart of a lightweight image and video quality reconstruction method provided by another embodiment of the present application;

[0050] Figure 4 A schematic flow chart of a lightweight image and video quality reconstruction method provided by another embodiment of the present application;

[0051] Figure 5 A schematic structural diagram of a lightweight image and video quality reconstruction device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] The existing lightweight image or video restoration based on deep learning still has many shortcomings, such as low generalization, complex network, large amount of calculation, and serious heating of mobile devices. Therefore, the innovation of the present invention is to fully consider the limited computing power of mobile devices, combine the real landing scenarios of mobile devices, and design a lightweight image and video quality reconstruction method that takes into account both denoising and super-resolution under complex image or video degradation, so as to achieve real-time processing on mobile devices and avoid device heating.

[0054] This application discloses a lightweight image and video quality reconstruction method, see Figure 1 and Figure 2 , the method may include:

[0055] Step S101: obtaining an image to be processed;

[0056] The image to be processed refers to an image that needs to be processed using the method disclosed in the embodiment of the present application, which may be a picture, and the picture may be an image of each frame in a video image;

[0057] See also Figure 1 In this step, the image to be processed is used as input data.

[0058] Step S102: performing X-fold downsampling processing on the image to be processed, and performing channel expansion to obtain a first feature map;

[0059] When performing an X-fold downsampling process on the image to be processed to obtain a feature map, a preset convolution kernel may be used to perform a convolution operation with a step length of X on the image to be processed to obtain a feature map that is downsampled X times;

[0060] Specifically, the PixelUnShuffle method (sub-pixel convolution downsampling method) can be used to downsample the input image to obtain a feature map;

[0061] For example, assuming that the shape of the image to be processed is [3, H, W], where H and W are the height and width of the image, the image to be processed is downsampled by 2 times using the PixelUnShuffle operation to obtain Figure 1 The first feature map F shown in d1 , the specific implementation of the PixelUnShuffle operation is as follows:

[0062] First, define a convolution kernel of size [12, 1, 2, 2] with all zeros, and then assign 1 to the positions [0::4,0,0,0], [1::4,0,D,1], [2::4,0,1,D], and [3::4,0,1,0] in the convolution kernel, and get the preset convolution kernel. Then use the defined preset convolution kernel to perform a convolution operation with a step length of 2 on the input image to be processed, so as to get the first feature map that is downsampled by 2 times, and its shape is [12,H / / 2,W / / 2], that is, a low-dimensional first feature map F with 12 channels and half the height and width is obtained. d1 ;

[0063] When expanding the channels, the number of channels of the first feature map may be expanded by performing a nonlinear transformation on the first feature map. The specific process may include:

[0064] The feature map downsampled by X times is processed by a feature enhancement module (hereinafter referred to as FEM) to expand the number of channels of the feature map to obtain a first feature map, wherein the FEM module is composed of a 3×3 convolution and an LReLU (leakage corrected linear) function. Assuming that the input feature map of X times has 12 channels, after being processed by the FEM module, an enhanced feature Fe with 48 feature maps will be obtained, which is recorded as the first feature map Fe;

[0065] Step S103: extracting low-level features from the first feature map;

[0066] In this step, if Figure 1 As shown in FIG. 1 , the low-level feature extraction is mainly realized by a feature enhanced distillation module (Feature Enhanced Distillation Block, hereinafter referred to as FEDB). Since the number of feature maps decreases step by step during the information distillation process, the reduction of features affects the expression ability of the network to a certain extent. Therefore, the present application discloses a specific low-level feature extraction scheme. For details, see Figure 3, the extraction scheme may include:

[0067] Step S201: In the channel dimension, a feature enhancement distillation module is used to perform a first distillation operation on the channels of the first feature map Fe, and feature maps of A channels are distilled from the channels of the first feature map Fe, which are recorded as the first distillation feature map F dis1 , wherein the value of A can be 3 or other values;

[0068] Step S202: Perform feature extraction and channel dimension reduction operations on the feature graphs corresponding to the remaining channels in the first distillation to obtain the first dimension reduction feature graph F re1 ;

[0069] Specifically, a 3×3 convolution and an LReLU function may be used to perform feature extraction and channel dimension reduction on the feature map corresponding to the remaining channels of the first distillation;

[0070] The number of channels of the first reduced-dimensionality feature map can be selected according to user needs. For example, if the number of channels of the first feature map Fe is 48 and the value of A is 3, the number of channels of the first reduced-dimensionality feature map can be 33.

[0071] Step S203: Using a feature enhancement distillation module to enhance the first dimension reduction feature map F re1 The channels of the feature map are distilled for the second time, and the first dimension reduction feature map F is re1 The characteristic graphs of C channels are distilled out from the channels, which are recorded as the second distillation characteristic graph F dis2 ;

[0072] Step S204: Perform feature extraction and channel dimension reduction operations on the feature maps corresponding to the remaining channels in the second distillation to obtain feature maps of B channels, which are recorded as the second dimension reduction feature map F re2 ;

[0073] B is the number of channels after the image to be processed is downsampled by a factor of X. For example, the value of B is 12;

[0074] In this step, a 3×3 convolution and an LReLU function can be used to extract features and reduce channel dimensions of the feature maps corresponding to the remaining channels of the second distillation.

[0075] Step S205: fusing the first feature map, the first distillation feature map, the second distillation feature map, and the second dimensionality reduction feature map in the channel dimension to obtain a first fused feature map;

[0076] This step can be understood as performing a concatenate operation (fusion operation) on the first feature map, the first distillation feature map, the second distillation feature map, and the second dimensionality reduction feature map. The specific implementation is to concatenate the first feature map Fe, the first distillation feature map F dis1 , Second distillation characteristic diagram F dis2 And the second dimension reduction feature map F re2 , the first fusion feature map is obtained by fusion in the channel dimension. Further, in order to optimize the fusion result, the GCT (threshold channel transform) module can be used in this application to further optimize the fused features to obtain the output F o1 , at this time, the F o1 As the first fused feature map, GCT is a lightweight attention mechanism. Its specific implementation can be referred to the reference Yang Z, and Zhu L, Wu Y, et al. Gated Channel Transformation for Visual Recognition[C] / / 2020 IEEE Conference on Computer Vision and Pattern Recognition(CVPR),2020.

[0077] In this solution, when the number of channels of the first distillation feature map and the second distillation feature map is 3 and the number of channels of the second dimensionality reduction feature map is 12, the number of channels of the fused feature map is 30, which is the first fused feature map.

[0078] Step S206: Perform feature fusion and channel dimension reduction on the first fused feature map to obtain a feature map with D channels, which is recorded as the second feature map F d2 ;

[0079] In this step, a 1×1 convolution is specifically used to perform feature fusion and channel dimension reduction on the first fused feature map. For example, 30 feature maps are fused to obtain 12 second feature maps F d2 ,like Figure 1 F d2 ;

[0080] Step S207: Then, the second feature map F d2 Perform feature extraction to obtain the third feature map F d3 , the feature extraction result is the low-level feature extraction result;

[0081] In this step, see Figure 1 , the FEDB module can be used again to analyze the second feature map F d2 Processing to obtain the third feature map F d3 .

[0082] Step S104: upsampling the extracted low-level features by N times, where N is not less than 2;

[0083] Specifically, this step includes:

[0084] See also Figure 1 , for the third feature map F d3 With the second feature map F d2 Perform concatenate operation (fusion operation) on the channel dimension, and use 1*1 convolution to fuse the fusion results to obtain the fourth feature map F d4 ;

[0085] like Figure 1 As shown, for the fourth characteristic graph F d4 Perform a 2x upsampling, specifically using PixelShuffle (sub-pixel convolution upsampling) to upsample the fourth feature map F d4 Perform a 2x upsampling to obtain a feature map F with high-level representation 1 .

[0086] Step S105: performing high-level feature extraction on the N-fold upsampling sampling result to obtain reconstruction features;

[0087] This step is specifically as follows, see Figure 1 , the feature enhancement distillation module is used again to perform a 2-fold upsampling of the fourth feature map (feature map F 1 ) to extract high-level features and obtain the reconstructed features F 2 .

[0088] Step S106: Upsample the reconstructed features by M times to obtain F u1 , said M is not less than 2;

[0089] See also Figure 1 This step specifically uses the PixelShuffle operation to reconstruct the feature F 2 Perform 2 times upsampling to obtain the feature map F u1 .

[0090] Step S107: using a preset high-frequency attention structure to extract a high-frequency feature map from the image to be processed and processing it with the sampling result of M-fold upsampling to obtain a result map after enhancing the high frequency;

[0091] In this step, in order to better restore the high-frequency details of the image, the present invention designs a high-frequency attention structure (High Frequency Attention, referred to as HFA) to enhance the high-frequency details of the restored image. For details, see Figure 4 , the process of this step is as follows:

[0092] Step S301: using the Sobel operator to extract the high-frequency feature map of the image to be processed;

[0093] The HFA is a high-frequency attention structure based on the Sobel operator.

[0094] Step S302: fusing the extracted high-frequency feature map with the reconstructed feature in the channel dimension;

[0095] In this step, the high-frequency feature map extracted in step S301 is combined with the reconstructed feature F 2 Cascaded through the concatenate operation in the channel dimension;

[0096] Step S303: using 1*1 convolution to perform channel dimension reduction processing on the fusion result of the high-frequency feature map and the reconstructed feature, so as to obtain a feature map having only one channel;

[0097] Step S304: performing a nonlinear transformation on the feature map having only one channel to obtain a value of each pixel position of the feature map after the feature map having only one channel is linearly transformed, and using the value of each pixel position as the weight of the high-frequency feature, thereby obtaining a high-frequency feature attention feature map;

[0098] In this step, a Sigmoid operation is used to perform a nonlinear transformation on the feature map having only one channel. The so-called Sigmoid operation performs a nonlinear transformation on the feature map, which refers to the use of a Sigmoid function to perform a nonlinear transformation on the feature map, wherein the Sigmoid function is an S-shaped function commonly seen in biology, also known as an S-shaped growth curve;

[0099] Step S305: See Figure 1 , perform dot multiplication of the high-frequency feature attention feature map and the reconstructed feature to obtain a feature map F with high-frequency information enhancement u2 ;

[0100] Step S306: upsampling the image to be processed by using a bicubic interpolation method;

[0101] In this step, see Figure 1 , the image to be processed is interpolated and upsampled using the bicubic interpolation method to obtain a feature map F u3 ;

[0102] Step S307: See Figure 3 , the sampling result F of upsampling the reconstructed features by 2 times u1 , feature map F with high-frequency information enhancement u2and the upsampling result F of the image to be processed u3 Superimpose to obtain the result image after enhancing high frequency;

[0103] In this step, the F u1 ,F u2 and F u3 The result image after high frequency enhancement is obtained by adding, and the result image after high frequency enhancement is output as the processing result of the image to be processed.

[0104] The technical solution disclosed in the above-mentioned embodiments of the present application has a low level of complexity in the processing model used when processing images, and thus has a small requirement on the size of the model, so that the physical device applying the solution can be applied to a mobile terminal device. Moreover, during the image processing process, the amount of data calculation of the model is small, and there will be no problem of device heating when running on a mobile terminal.

[0105] Corresponding to the above method, the present application also discloses a lightweight image and video quality reconstruction device. In this embodiment, for the specific working content of each unit in the device, please refer to the content of the above method embodiment.

[0106] The lightweight image and video quality reconstruction device for in-process information provided in an embodiment of the present invention is described below. The lightweight image and video quality reconstruction device for in-process information described below and the lightweight image and video quality reconstruction method described above can correspond to each other.

[0107] See also Figure 5 , the device may include: a collection unit 100 and a processing unit 200, and the specific implementations of the method and the device may refer to each other;

[0108] Corresponding to step S101 in the above method, the collector 100 is used to obtain the image to be processed;

[0109] Corresponding to steps S102-S107 in the above method, the processing unit 200 is used to perform X-fold downsampling processing on the image to be processed, and perform channel expansion to obtain a first feature map; perform low-level feature extraction on the first feature map; perform N-fold upsampling on the extracted low-level features, where N is not less than 2; perform high-level feature extraction on the sampling results of the N-fold upsampling to obtain reconstructed features; perform M-fold upsampling on the reconstructed features, where M is not less than 2; use a preset high-frequency attention structure to extract a high-frequency feature map from the image to be processed and process it with the sampling results of the M-fold upsampling to obtain a result map after enhancing the high frequency.

[0110] Correspondingly, the present application also provides a lightweight image and video quality reconstruction system, which may include: a lightweight image and video quality reconstruction device as described in any one of the above embodiments of the present application.

[0111] The present invention also provides a non-transitory electronic device readable storage medium, which includes: a program, which, when run by an electronic device, enables the lightweight image and video quality reconstruction method.

[0112] The present invention also provides an electronic device, which is applied with the non-transitory electronic device readable storage medium or the lightweight image and video quality reconstruction system.

[0113] For the convenience of description, the above system is described by dividing the functions into various modules. Of course, when implementing the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0114] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0115] Professionals may further appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0116] The methods or steps of the methods described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, software modules executed by a processor, or a combination of the two. The software modules may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0117] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0118] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A lightweight image and video quality reconstruction method, characterized in that: include: Get the image to be processed; Downsampling the image to be processed by a factor of X, and performing channel expansion to obtain a first feature map; Performing low-level feature extraction on the first feature map; Upsampling the extracted low-level features by a factor of N, where N is not less than 2; Performing high-level feature extraction on the N-fold upsampling sampling result to obtain reconstruction features; Upsampling the reconstructed features by a factor of M, where M is not less than 2; A preset high-frequency attention structure is used to extract a high-frequency feature map of the image to be processed and the high-frequency feature map is processed with the sampling result of M times upsampling to obtain a result map after enhancing the high frequency; The method uses a preset high-frequency attention structure to extract a high-frequency feature map from the image to be processed and processes the high-frequency feature map with the sampling result of M-fold upsampling to obtain a high-frequency enhanced result map, including: Using the Sobel operator to extract the high-frequency feature map of the image to be processed; Fusing the extracted high-frequency feature map with the reconstructed feature in the channel dimension; Using 1*1 convolution to perform channel dimension reduction processing on the fusion result of the high-frequency feature map and the reconstructed feature, so as to obtain a feature map with only one channel; Performing a nonlinear transformation on the feature map with only one channel to obtain the value of each pixel position of the feature map after the nonlinear transformation of the feature map with only one channel, and using the value of each pixel position as the weight of the high-frequency feature to obtain a high-frequency feature attention feature map; Performing a dot multiplication of the high-frequency feature attention feature map and the reconstructed feature to obtain a feature map with enhanced high-frequency information; Upsampling the image to be processed by using a bicubic interpolation method; The sampling result of the reconstructed feature being upsampled by 2 times, the feature map with high-frequency information enhancement, and the upsampling processing result of the image to be processed are superimposed to obtain a result map with enhanced high frequency.

2. The lightweight image and video quality reconstruction method according to claim 1, characterized in that: The image to be processed is downsampled by X times, and the channel is expanded to obtain a first feature map, including: A convolution operation with a step length of X is performed on the image to be processed using a preset convolution kernel to obtain a feature map that is downsampled by X times, which is recorded as a first feature map; A nonlinear transformation is performed on the first feature map to expand the number of channels of the first feature map.

3. The lightweight image and video quality reconstruction method according to claim 2, characterized in that: Performing a nonlinear transformation on the first feature map includes: An information enhancement module is used to perform a nonlinear transformation on the first feature map.

4. The lightweight image and video quality reconstruction method according to claim 2, characterized in that: Performing low-level feature extraction on the first feature map includes: Performing a first distillation on the channels of the first feature map by using a feature enhancement distillation module, distilling feature maps of A channels from the channels of the first feature map, which are recorded as first distillation feature maps, where A is a preset positive integer; Perform feature extraction and channel dimension reduction operations on the feature graph corresponding to the remaining channels of the first distillation, which is recorded as the first dimension reduction feature graph; A feature enhancement distillation module is used to perform a second distillation on the channels of the first dimensionality reduction feature map, and feature maps of C channels are distilled from the channels of the first dimensionality reduction feature map, which are recorded as second distillation feature maps; Perform feature extraction and channel dimension reduction operations on the feature maps corresponding to the remaining channels of the second distillation to obtain feature maps of B channels, which are recorded as second dimension reduction feature maps, where B is the number of channels after the image to be processed is downsampled by X times; The first feature map, the first distillation feature map, the second distillation feature map, and the second dimensionality reduction feature map are fused in the channel dimension to obtain a first fused feature map; Then, feature fusion and channel dimension reduction are performed on the first fused feature map to obtain a feature map with D channels, which is recorded as a second feature map; Then, feature extraction is performed on the second feature map to obtain a third feature map as a low-level feature extraction result.

5. The lightweight image and video quality reconstruction method according to claim 4, characterized in that: The extracted low-level features are upsampled by N times, where N is not less than 2, including: The third feature map and the second feature map are fused in the channel dimension, and a 1*1 convolution is used to perform feature fusion on the fusion result to obtain a fourth feature map; The fourth feature map is upsampled by a factor of 2.

6. The lightweight image and video quality reconstruction method according to claim 5, characterized in that: Performing advanced feature extraction on the N-fold upsampling sampling result to obtain reconstruction features, including: A feature enhancement distillation module for feature enhancement is used to perform high-level feature extraction on the 2-fold upsampling result of the fourth feature map to obtain reconstructed features.

7. The lightweight image and video quality reconstruction method according to claim 6, characterized in that: The reconstructed features are upsampled M times as follows: The reconstructed features are upsampled by a factor of 2.

8. A lightweight image and video quality reconstruction device, characterized in that: include: An acquisition unit, used for acquiring an image to be processed; A processing unit, configured to perform X-fold downsampling processing on the image to be processed and perform channel expansion to obtain a first feature map; Performing low-level feature extraction on the first feature map; performing N-fold upsampling on the extracted low-level features, where N is not less than 2; Performing high-level feature extraction on the N-fold upsampling sampling result to obtain reconstruction features; The reconstructed features are upsampled M times, where M is not less than 2; a preset high-frequency attention structure is used to extract a high-frequency feature map of the image to be processed and the high-frequency feature map is processed with the sampling result of the M-fold upsampling to obtain a high-frequency enhanced result map; The processing unit uses a preset high-frequency attention structure to extract a high-frequency feature map from the image to be processed and processes it with the sampling result of M-fold upsampling to obtain a result map after enhancing the high frequency, including: Using the Sobel operator to extract the high-frequency feature map of the image to be processed; Fusing the extracted high-frequency feature map with the reconstructed feature in the channel dimension; Using 1*1 convolution to perform channel dimension reduction processing on the fusion result of the high-frequency feature map and the reconstructed feature, so as to obtain a feature map with only one channel; Performing a nonlinear transformation on the feature map with only one channel to obtain the value of each pixel position of the feature map after the nonlinear transformation of the feature map with only one channel, and using the value of each pixel position as the weight of the high-frequency feature to obtain a high-frequency feature attention feature map; Performing a dot multiplication of the high-frequency feature attention feature map and the reconstructed feature to obtain a feature map with enhanced high-frequency information; Upsampling the image to be processed by using a bicubic interpolation method; The sampling result of the reconstructed feature being upsampled by 2 times, the feature map with high-frequency information enhancement, and the upsampling processing result of the image to be processed are superimposed to obtain a result map with enhanced high frequency.

9. A lightweight image and video quality reconstruction system, characterized in that: include: The lightweight image and video quality reconstruction device as described in claim 8.

Images