Training method and reconstruction method of image super-resolution reconstruction model

Abstract

The invention provides a training method and a reconstruction method of an image super-resolution reconstruction model, and the training method comprises the steps: obtaining a super-resolution training image, and carrying out the shallow feature extraction of the super-resolution training image, and obtaining a shallow feature; performing multi-level separation type deep feature extraction on the shallow features to obtain a plurality of deep features of different feature levels; performing first feature fusion complementation on all the deep features to obtain a first complementary feature; according to the first complementary feature, performing image reconstruction on the shallow feature to obtain a super-resolution reconstructed image; and according to the super-resolution reconstruction image, performing parameter updating on the initialized image super-resolution reconstruction model to obtain a trained image super-resolution reconstruction model. According to the method, by providing an image super-resolution reconstruction model, computing resources and storage space required during image super-resolution reconstruction can be effectively reduced, and the image super-resolution reconstruction effect is improved. The invention relates to the technical field of image processing.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a training method and a reconstruction method of an image super-resolution reconstruction model. Background Art

[0002] Image super-resolution reconstruction refers to the generation of high-resolution images by processing limited low-quality image data. It can enhance the edge sharpness, texture fineness, and visualization of microstructure of target objects in the image, which is beneficial to improving the recognition accuracy of machine vision systems and has attracted much attention.

[0003] At present, the existing technology usually adopts a deep learning-based approach to perform super-resolution reconstruction of images. This approach usually requires a large amount of computing resources and storage space when performing image super-resolution reconstruction, and the effect of image super-resolution reconstruction is not good.

[0004] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0005] The purpose of the present invention is to solve one of the technical problems existing in the related art to at least a certain extent.

[0006] The main purpose of the embodiments of the present application is to propose a training method and a reconstruction method for an image super-resolution reconstruction model, wherein the method can effectively reduce the computing resources and storage space required for image super-resolution reconstruction and improve the image super-resolution reconstruction effect by providing an image super-resolution reconstruction model.

[0007] To achieve the above object, an embodiment of the present application provides a training method for an image super-resolution reconstruction model, the training method comprising:

[0008] Acquire a super-resolution training image, and perform shallow feature extraction on the super-resolution training image to obtain shallow features;

[0009] Performing multi-level separation deep feature extraction on the shallow features to obtain deep features of several different feature levels;

[0010] Performing first feature fusion and complementation on all the deep features to obtain first complementary features;

[0011] Reconstructing the shallow features according to the first complementary features to obtain a super-resolution reconstructed image;

[0012] According to the super-resolution reconstructed image, parameters of the initialized image super-resolution reconstruction model are updated to obtain a trained image super-resolution reconstruction model.

[0013] In some embodiments, extracting shallow features from the super-resolution training image to obtain shallow features includes:

[0014] Performing a first point-by-point convolution on the super-resolution training image to obtain a plurality of channel features;

[0015] Performing grouped deep convolution on all the channel features to obtain grouped features corresponding to each of the channel features;

[0016] A second point-by-point convolution is performed on all the grouped features to obtain the shallow features.

[0017] In some embodiments, performing separate deep feature extraction on the input features to obtain target deep features includes:

[0018] Performing multi-layer separable residual convolution on the input features to obtain refined features at several different feature interaction levels;

[0019] Performing a first feature distillation on the input features to obtain a first distilled feature, and performing a second feature distillation on all first target refined features to obtain a second distilled feature corresponding to each of the first target refined features, wherein the first target refined features are refined features other than the refined features at the last feature interaction level;

[0020] Performing deep separable convolution on the second target refined feature to obtain a third target refined feature, where the second target refined feature is a refined feature of the last feature interaction level;

[0021] According to the input features, attention feature fusion and complementation are performed on the first distilled features, the second distilled features and the third target refined features to obtain the target deep features.

[0022] In some embodiments, the step of performing attention feature fusion and complementation on the first distilled feature, the second distilled feature, and the third target refined feature according to the input feature to obtain the target deep feature includes:

[0023] Performing second feature fusion and complementation on the first distillation feature, the second distillation feature and the third target refining feature to obtain a second complementary feature;

[0024] Performing perceptual channel attention extraction on the second complementary feature to obtain a channel attention feature;

[0025] Performing multi-scale spatial attention extraction on the channel attention feature to obtain a spatial attention feature;

[0026] According to the input features, feature concatenation is performed on the spatial attention features to obtain the target deep features.

[0027] In some embodiments, performing multi-scale spatial attention extraction on the channel attention feature to obtain the spatial attention feature includes:

[0028] Performing multi-scale attention distribution extraction on the channel attention feature to obtain a plurality of spatial attention distributions, each of which corresponds to a different spatial scale;

[0029] According to all the spatial attention distributions, spatial attention fusion is performed on the channel attention features to obtain the spatial attention features.

[0030] In some embodiments, the intermediate feature set is subjected to feature fusion and complementation to obtain target complementary features, including:

[0031] Perform branch attention feature fusion on the intermediate feature set to obtain the intermediate fusion feature;

[0032] Performing feature complementation processing on the intermediate fusion features to obtain the target complementary features;

[0033] The intermediate feature set is a set of all deep features or a set of the first distillation feature, the second distillation feature and the third target refined feature.

[0034] In some embodiments, the performing branch attention feature fusion on the intermediate feature set to obtain the intermediate fusion feature includes:

[0035] Performing feature channel splicing on all branch features in the intermediate feature set to obtain spliced ​​features;

[0036] Performing channel information interaction on the splicing features to obtain channel interaction features;

[0037] Performing maximum pooling on the channel interaction features to obtain maximum pooling features, and performing variance feature extraction on the channel interaction features to obtain variance features;

[0038] Generate a plurality of feature weights according to the maximum pooling feature and the variance feature, each of the feature weights corresponding to a branch feature in the intermediate feature set;

[0039] According to all the feature weights, the intermediate feature set is weighted fused to obtain the intermediate fused feature.

[0040] In some embodiments, performing feature complementation processing on the intermediate fusion features to obtain the target complementary features includes:

[0041] Performing spatial displacement on the intermediate fusion feature to obtain a spatial displacement feature;

[0042] According to the intermediate fusion feature, the spatial displacement feature is subjected to displacement feature fusion to obtain the target complementary feature;

[0043] Among them, the intermediate fusion feature includes several subdivided channel features, each subdivided channel feature includes several spatial feature regions, the spatial displacement feature includes several spatial displacement regions, and each of the spatial displacement regions is used to record the feature information of the spatial feature regions in all the subdivided channel features that do not correspond to the spatial displacement region.

[0044] To achieve the above object, another aspect of the embodiment of the present application provides a reconstruction method of an image super-resolution reconstruction model, comprising:

[0045] Acquire a resolution image to be reconstructed;

[0046] The resolution image is input into the image super-resolution reconstruction model trained as described above to perform image reconstruction to obtain a super-resolution reconstructed image.

[0047] To achieve the above object, another aspect of the embodiment of the present application provides a training system for an image super-resolution reconstruction model, comprising:

[0048] A first processing unit is used to obtain a super-resolution training image, and perform shallow feature extraction on the super-resolution training image to obtain shallow features;

[0049] A second processing unit is used to perform multi-level separation deep feature extraction on the shallow features to obtain deep features of several different feature levels;

[0050] A third processing unit is used to perform first feature fusion and complementation on all the deep features to obtain first complementary features;

[0051] a fourth processing unit, configured to perform image reconstruction on the shallow features according to the first complementary features to obtain a super-resolution reconstructed image;

[0052] The fifth processing unit is used to update the parameters of the initialized image super-resolution reconstruction model according to the super-resolution reconstructed image to obtain a trained image super-resolution reconstruction model.

[0053] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.

[0054] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0055] The embodiments of the present application include at least the following beneficial effects:

[0056] The present application provides a training method and a reconstruction method for an image super-resolution reconstruction model, wherein the training method obtains a super-resolution training image, and performs shallow feature extraction on the super-resolution training image to obtain shallow features; performs multi-level separation deep feature extraction on the shallow features to obtain deep features of several different feature levels; performs first feature fusion and complementation on all the deep features to obtain first complementary features; performs image reconstruction on the shallow features based on the first complementary features to obtain a super-resolution reconstructed image; and performs parameter update on an initialized image super-resolution reconstruction model based on the super-resolution reconstructed image to obtain a trained image super-resolution reconstruction model. This training method extracts deep features from shallow features in a multi-level separation manner, and performs feature fusion and complementation on all the obtained deep features. This allows the trained image super-resolution reconstruction model to fuse features of different scales using only a small number of small-size convolutions, that is, it can reduce the computing resources and storage space required for image super-resolution reconstruction by reducing the number of network parameters of the model. At the same time, this training method can increase the receptive field of the image super-resolution reconstruction model by performing feature fusion and complementation on all deep features, thereby improving the global feature extraction capability of the image super-resolution reconstruction model and effectively improving the image super-resolution reconstruction effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a flowchart of a training method for an image super-resolution reconstruction model provided in an embodiment of the present application;

[0058] Figure 2 It is a schematic diagram of the overall network framework of an image super-resolution reconstruction model provided in an embodiment of the present application;

[0059] Figure 3 It is a schematic diagram of a framework of a depth-separable convolution provided in an embodiment of the present application;

[0060] Figure 4 It is a schematic diagram of a network framework of a separated deep feature extraction module provided in an embodiment of the present application;

[0061] Figure 5 It is a schematic diagram of a network framework of a depth-separable convolutional residual block provided in an embodiment of the present application;

[0062] Figure 6Schematic diagram of a network framework of a multi-scale spatial attention module provided in an embodiment of the present application;

[0063] Figure 7 It is a schematic diagram of a network framework of a branch attention feature fusion module provided in an embodiment of the present application;

[0064] Figure 8 This is a schematic diagram of a network framework of a feature complementation module provided in an embodiment of the present application;

[0065] Fig. 9 It is a schematic diagram of a framework of a training system for an image super-resolution reconstruction model provided in an embodiment of the present application;

[0066] Fig.10 It is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices / equipment and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0068] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0069] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0070] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0071] At present, the existing technology usually uses a deep learning-based method to reconstruct images for super-resolution. In order to obtain a better super-resolution reconstructed image, this method often chooses to deepen the network layer or increase the size of the convolution kernel, which increases the model parameters and the computational complexity. When performing image super-resolution reconstruction, it usually requires a large amount of computing resources and storage space, which is difficult to implement on resource-constrained devices. In addition, there are also methods for image super-resolution reconstruction based on lightweight algorithms. Although the existing methods can alleviate the computing resources and storage space required for image super-resolution reconstruction to a certain extent, their image super-resolution reconstruction effects are often poor.

[0072] In view of this, the present application provides a training method and a reconstruction method for an image super-resolution reconstruction model, wherein the training method performs multi-level separation deep feature extraction on shallow features, and performs feature fusion and complementation on all obtained deep features, so that the trained image super-resolution reconstruction model only needs to use a small number of small-size convolutions to fuse features of different scales, that is, the computing resources and storage space required for image super-resolution reconstruction can be reduced by reducing the network parameters of the model.

[0073] At the same time, the training method can increase the receptive field of the image super-resolution reconstruction model and strengthen the global feature extraction capability of the model by performing feature fusion and complementation on all deep features, specifically by performing feature complementation processing on the intermediate fusion features, and can effectively improve the image super-resolution reconstruction effect; in addition, the training method can assign different feature weights to each branch feature according to its importance by performing branch attention feature fusion on the intermediate feature set, thereby realizing weighted fusion of multiple feature information, specifically based on dynamically adjusting the contribution of features of different branches, capturing the most significant and most changing features of each channel from different angles, enhancing the distinguishability and diversity of features, and enabling features of different branches to be better integrated, thereby improving the overall feature representation capability, and thus improving the image super-resolution reconstruction effect.

[0074] The training method and reconstruction method of an image super-resolution reconstruction model provided in the embodiment of the present application can be applied to image processing application scenarios. In image processing application scenarios, an image processing service provider can reconstruct low-resolution images through the image super-resolution reconstruction model provided in the embodiment of the present application to obtain a reconstructed image with a higher resolution, which can reduce the computing resources and storage space required for image super-resolution reconstruction and improve the image super-resolution reconstruction effect.

[0075] The training method and reconstruction method of an image super-resolution reconstruction model provided in the embodiments of the present application can be applied to a terminal or a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements the method, etc., but is not limited to the above forms.

[0076] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0077] Reference Figure 1 and Figure 2 , Figure 1 is an optional flow chart of a training method for an image super-resolution reconstruction model provided in an embodiment of the present application, Figure 1 The training method may include but is not limited to steps S110 to S150.

[0078] Step S110, obtaining a super-resolution training image, and performing shallow feature extraction on the super-resolution training image to obtain shallow features;

[0079] In the embodiment of the present application, the training image can be any image in the training image set publicly available on the Internet. Specifically, the embodiment of the present application can first obtain the publicly available DIV2 dataset and Flickr2k dataset, and merge the obtained DIV2 dataset and Flickr2k dataset to obtain a merged dataset; then, the merged dataset is Bicubic downsampled to obtain a number of low-resolution images, and these low-resolution images are determined as super-resolution training images. In addition, the shallow feature extraction in step S110 is used to extract fine-grained information such as color, texture, edge and angular information in the super-resolution training image, so as to obtain shallow features corresponding to the super-resolution training image.

[0080] Reference Figure 3 In some embodiments, the step S110 of extracting shallow features from the super-resolution training image to obtain shallow features includes:

[0081] A1. Performing a first point-by-point convolution on the super-resolution training image to obtain a plurality of channel features;

[0082] A2. Perform deep convolution on all the channel features to obtain group features corresponding to each channel feature;

[0083] A3. Perform a second point-by-point convolution on all the grouped features to obtain the shallow features.

[0084] In an embodiment of the present application, the shallow feature extraction in step S110 can be implemented by a depth-wise separable convolution (DSConv) with a 3×3 convolution kernel size. Specifically, step A1 can first be a point-by-point multiplication of the input super-resolution training image to obtain the features (i.e., channel features) in each channel dimension of the super-resolution training image; step A2 can be an independent depth-wise convolution of each channel feature in the feature dimension to obtain the grouping features corresponding to each channel feature; step A3 can be a point-by-point convolution of each grouping feature using a 1×1 convolution to aggregate each grouping feature in the channel dimension to obtain the shallow feature.

[0085] Step S120, performing multi-level separation deep feature extraction on the shallow features to obtain deep features of several different feature levels;

[0086] In an embodiment of the present application, step S120 may be to input the extracted shallow features into an image super-resolution reconstruction model for feature extraction by several cascaded separate deep feature extraction modules, so as to obtain deep features output by each separate deep feature extraction module, wherein the input of the first separate deep feature extraction module is the shallow features, and the input of the second or subsequent separate deep feature extraction modules is the features output by the previous separate deep feature extraction module, and the features output by each separate deep feature extraction module are deep features of a feature level.

[0087] Reference Figure 4 and Figure 5 In some embodiments, separate deep feature extraction is performed on the input features to obtain target deep features, including:

[0088] B1. Perform multi-layer separable residual convolution on the input features to obtain refined features at several different feature interaction levels;

[0089] B2. Performing a first feature distillation on the input features to obtain a first distilled feature, and performing a second feature distillation on all first target refined features to obtain a second distilled feature corresponding to each of the first target refined features, wherein the first target refined features are refined features other than the refined features at the last feature interaction level;

[0090] B3, performing deep separable convolution on the second target refined feature to obtain a third target refined feature, where the second target refined feature is a refined feature of the last feature interaction level;

[0091] In an embodiment of the present application, for a certain separable deep feature extraction module (SDFEM), the multi-layer separable residual convolution in step B1 can be to input the input features into a plurality of cascaded depthwise separable convolution residual blocks (Channel-Enhanced Depthwise Separable Convolutional Residual Block, CDSRB) for feature refinement, so as to obtain the refined features output by each depthwise separable convolution residual block, and each depthwise separable convolution residual block can perform feature channel interaction on the input features. Specifically, the embodiment of the present application takes the number of depthwise separable convolution residual blocks as 3 as an example. For the first depthwise separable convolution residual block, the refined features of its output correspond to the first feature interaction level; the input of the second depthwise separable convolution residual block is the refined features of the first feature interaction level, and the refined features of its output correspond to the second feature interaction level; the input of the third depthwise separable convolution residual block is the refined features of the second feature interaction level, and the refined features of its output correspond to the third feature interaction level. The same is true for the remaining number of depthwise separable convolution residual blocks, which can be simply inferred.

[0092] It can be understood that the first feature distillation in step B2 can be feature distillation of the input features, specifically, feature distillation of the input features based on 1×1 convolution to obtain the first distilled features; in addition, after obtaining the refined features of each feature interaction level, the refined features that are not the last feature interaction level can be determined as the first target refined features, and then, the first target refined features of different feature interaction levels are feature distilled using 1×1 convolution to obtain the second distilled features corresponding to the refined features of each feature interaction level.

[0093] It should be noted that the depthwise separable convolution in step B3 is used to improve the feature quality of the second target refined feature. Specifically, the second target refined feature can be input into a 3×3 depthwise separable convolution to obtain the second target refined feature with improved quality (i.e., the third target refined feature).

[0094] B4. According to the input features, the first distilled features, the second distilled features and the third target refined features are subjected to attention feature fusion and complementation to obtain the target deep features.

[0095] Furthermore, the step B4, according to the input feature, performs attention feature fusion and complementation on the first distillation feature, the second distillation feature and the third target refined feature to obtain the target deep feature, includes:

[0096] B41, performing second feature fusion and complementation on the first distillation feature, the second distillation feature and the third target refining feature to obtain a second complementary feature;

[0097] B42, performing perceptual channel attention extraction on the second complementary feature to obtain a channel attention feature;

[0098] In the embodiment of the present application, the second feature fusion and complementation in step B41 can be to fuse and complement the feature information contained in the first distillation feature, all second distillation features and the third target refining feature in the channel dimension and the spatial dimension, so as to obtain the second complementary feature.

[0099] It can be understood that the perceptual attention extraction in step B42 is used to add channel attention to the second complementary feature, which can specifically be inputting the second complementary feature into the contrast-aware channel attention module (Contrast-aware Channel Attention, CCA), calculating the contrast information in the second complementary feature, obtaining the channel attention corresponding to the second complementary feature, and adding the added channel attention to the second complementary feature, so as to enhance the feature representation in the second complementary feature, and finally obtain the channel attention feature.

[0100] B43, performing multi-scale spatial attention extraction on the channel attention feature to obtain a spatial attention feature;

[0101] Reference Figure 6 Further, the step B43, performing multi-scale spatial attention extraction on the channel attention feature to obtain the spatial attention feature, includes:

[0102] B431, performing multi-scale attention distribution extraction on the channel attention feature to obtain a plurality of spatial attention distributions, each of which corresponds to a different spatial scale;

[0103] B432. Based on all the spatial attention distributions, perform spatial attention fusion on the channel attention features to obtain the spatial attention features.

[0104] In an embodiment of the present application, step B43 may be to input the channel attention feature into a multi-scale spatial attention module in an image super-resolution reconstruction model, so as to add spatial attention features to the channel attention feature. Specifically, the multi-scale spatial attention module (MSSAM) in the embodiment of the present application includes a 1×1 Conv convolution layer, a maximum pooling layer (MaxPooling), several cascaded attention distribution extraction units, several linear extraction units and a normalization layer, wherein each attention distribution extraction module corresponds to a spatial scale, each linear extraction unit corresponds to an attention distribution extraction unit, each attention distribution extraction unit includes a GELU activation function layer and a Dialted and Depth-Separable Convolution (DDSConv), and each linear extraction unit includes a linear transformation layer (Bilinear) and a 1×1 Conv convolution layer.

[0105] It can be understood that the embodiment of the present application takes the number of cascaded attention distribution extraction units in the multi-scale spatial attention module as 3 as an example. Then, step B431 can be that after the channel attention feature passes through the 1×1 Conv convolution layer and the maximum pooling layer, the spatial attention distribution of the channel attention feature on a small spatial scale is extracted through the first attention distribution extraction unit and the corresponding linear extraction unit; then, the feature output by the first attention distribution extraction unit is subjected to spatial attention distribution on a medium spatial scale through the second attention distribution extraction unit and the corresponding linear extraction unit; then, the feature output by the second attention distribution extraction unit is subjected to spatial attention distribution on a large spatial scale through the third attention distribution extraction unit and the corresponding linear extraction unit. The example of the present application is for illustration only. The number of cascaded attention distribution extraction units in the multi-scale spatial attention module can be set according to actual conditions.

[0106] It should be noted that after obtaining all spatial attention distributions, step B432 can first be to perform an element-by-element sum operation on the spatial attention distributions of all different spatial scales to obtain the summed spatial attention distribution; then, based on the summed spatial attention distribution, the original channel attention features are element-by-element multiplied. By calculating the spatial attention at different spatial scales, the image super-resolution reconstruction model's perception of multi-scale information can be improved, so that the model can capture more spatial details, and finally the spatial attention features are obtained after the normalization layer.

[0107] B44. Based on the input features, perform feature concatenation on the spatial attention features to obtain the target deep features.

[0108] In an embodiment of the present application, after obtaining the spatial attention feature, a concatenation operation can be performed on the spatial attention feature and the input feature to obtain the target deep feature.

[0109] Step S130, performing first feature fusion and complementation on all the deep features to obtain first complementary features;

[0110] In the embodiment of the present application, the first feature fusion and complementation in step S130 may be to fuse and complement the feature information contained in all deep features in the channel dimension and the spatial dimension, so as to obtain the first complementary feature.

[0111] Reference Figure 7 In some embodiments, the intermediate feature set is subjected to feature fusion and complementation to obtain target complementary features, including:

[0112] C1, perform branch attention feature fusion on the intermediate feature set to obtain the intermediate fusion feature;

[0113] Furthermore, the step C1, performing branch attention feature fusion on the intermediate feature set to obtain the intermediate fusion feature, includes:

[0114] C11, performing feature channel splicing on all branch features in the intermediate feature set to obtain a spliced ​​feature;

[0115] C12, performing channel information interaction on the splicing features to obtain channel interaction features;

[0116] C13, performing maximum pooling on the channel interaction feature to obtain a maximum pooling feature, and performing variance feature extraction on the channel interaction feature to obtain a variance feature;

[0117] C14. Generate a plurality of feature weights according to the maximum pooling feature and the variance feature, each of the feature weights corresponding to a branch feature in the intermediate feature set;

[0118] C15. Perform weight fusion on the intermediate feature set according to all the feature weights to obtain the intermediate fused feature.

[0119] In the embodiment of the present application, the intermediate feature set can be a set of all deep features, or a set of the first distillation feature, the second distillation feature and the third target refined feature. The embodiment of the present application takes the intermediate feature set as a set of all deep features as an example. At this time, each branch feature in the intermediate feature set corresponds to a deep feature of a feature level. Specifically, step C1 can be to input the intermediate feature set into the branch attention feature fusion module (FFM) in the image super-resolution reconstruction model, and the branch attention feature fusion module allocates corresponding feature weights based on the importance of each branch feature, thereby realizing weighted fusion of multiple feature information.

[0120] It can be understood that step C11 can be a concatenation operation on the deep features of all different feature levels in the channel dimension, and the concatenated features are determined as concatenated features. The channel information interaction of step C12 can be inputting the concatenated features into a 1×1 convolution layer, realizing the channel information interaction of the concatenated features through the 1×1 convolution layer, and then obtaining the channel interaction features after the GELU activation function layer. Step C13 can be inputting the channel interaction features into two parallel branches, and completing the maximum pooling (Max Pooling) and variance (Variance) feature extraction of the channel interaction features through the two branches respectively, so as to obtain the maximum pooling features and variance features. The obtained maximum pooling features and variance features can measure the importance of each feature channel in the deep features from different angles.

[0121] It should be noted that after obtaining the maximum pooling feature and the variance feature, the two features can first be added together, and then multiple 1×1 convolutional layers and ReLU activation layers are used to obtain multiple feature weights, each of which corresponds to a deep feature. In addition, step C15 can be to multiply each branch feature by the corresponding feature weight to obtain several weight-multiplied branch features, and then concatenate all weight-multiplied branch features along the channel dimension to obtain the intermediate fusion feature.

[0122] It is worth mentioning that after obtaining the intermediate fusion feature, the intermediate fusion feature can also be input into the 1×1 convolution layer and the normalization layer (Layer Norm) in sequence. The data of the intermediate fusion feature is normalized by the normalization layer, which can improve the stability of the image super-resolution reconstruction model. In addition, when the intermediate feature set is a set of the first distillation feature, the second distillation feature, and the third target refined feature, the rest of the aforementioned intermediate feature sets are similar to the set of all deep features, which can be simply deduced by analogy, and this application will not repeat them here.

[0123] In addition, the embodiment of the present application can dynamically adjust the contribution of different branch features by inputting the intermediate feature set into the branch attention feature fusion module, capture the most significant and most changing features of each channel from different angles, enhance the feature discrimination and diversity, and enable the features of different branches to be better integrated together, thereby improving the overall feature representation capability, thereby improving the quality and effect of image super-resolution reconstruction.

[0124] C2. Perform feature complementation processing on the intermediate fusion features to obtain the target complementary features;

[0125] Reference Figure 8 Further, the step C2, performing feature complementation processing on the intermediate fusion features to obtain the target complementary features, includes:

[0126] C21, performing spatial displacement on the intermediate fusion feature to obtain a spatial displacement feature;

[0127] C22. According to the intermediate fusion feature, perform displacement feature fusion on the spatial displacement feature to obtain the target complementary feature;

[0128] Among them, the intermediate fusion feature includes several subdivided channel features, each subdivided channel feature includes several spatial feature regions, the spatial displacement feature includes several spatial displacement regions, and each of the spatial displacement regions is used to record the feature information of the spatial feature regions in all the subdivided channel features that do not correspond to the spatial displacement region.

[0129] In an embodiment of the present application, the intermediate fusion feature can be input into a feature complementation module (FCM) in an image super-resolution reconstruction model, and feature complementation processing of the intermediate fusion feature can be implemented through the feature complementation module. Specifically, step C2 can first be to input the intermediate fusion feature into a feature displacement layer (Feature Displacement, FD), and the intermediate fusion feature is evenly divided into a number of subdivided channel features in the channel dimension through the feature displacement layer, and each subdivided channel feature is further divided into a number of spatial feature regions of the same size in the spatial dimension; then, a spatial displacement operation of different amplitudes is performed on each spatial feature region in each subdivided channel feature, so as to obtain a spatial displacement feature, and each spatial displacement region in the spatial displacement feature is obtained by splicing the feature information of each subdivided channel feature in different spatial feature regions in the channel dimension.

[0130] It should be noted that step C22 can be to input the obtained intermediate fusion features into a 3×3 depth-separable convolution, a GELU activation function layer and two consecutive 3×3 depth-separable convolutions in sequence, and then sum them element-by-element with the original intermediate fusion features, so as to fuse the features before and after the displacement, so that the spatial feature areas of different subdivision channel features complement each other, thereby obtaining the target complementary features.

[0131] It is worth mentioning that the feature complementation processing in the embodiment of the present application can fuse the features of different parts of the feature map with only a small number of small-size convolutions, so that the image super-resolution reconstruction model can make full use of long-range features for image reconstruction, while also making the network of the image super-resolution reconstruction model lightweight, increasing the network's receptive field, and improving the global feature extraction capability.

[0132] Step S140: reconstructing the shallow features according to the first complementary features to obtain a super-resolution reconstructed image;

[0133] In an embodiment of the present application, step S140 may be to fuse the shallow features and the first complementary features through a jump connection in the image super-resolution reconstruction model, and then perform a Pixel Shuffle operation on the fused feature image to obtain a super-resolution reconstructed image output by the image super-resolution reconstruction model, and the super-resolution reconstructed image corresponds to the super-resolution training image input to the image super-resolution reconstruction model.

[0134] Step S150: updating parameters of an initialized image super-resolution reconstruction model according to the super-resolution reconstructed image to obtain a trained image super-resolution reconstruction model.

[0135] In an embodiment of the present application, before the image super-resolution reconstruction model is put into use, it is necessary to train it to adjust its internal parameters so as to achieve a better image super-resolution reconstruction effect. Specifically, when training the model, the label image of the super-resolution training image can be determined according to the merged data set, and the label image can be the original high-resolution image in the merged data set. Then each super-resolution training image and the corresponding label image are used as a set of training sample pairs, and the super-resolution training image is reconstructed as super-resolution through the model, and the output data of the model is a super-resolution reconstructed image. After obtaining the super-resolution reconstructed image output by the model, the image reconstruction effect of the model can be evaluated according to the super-resolution reconstructed image and the label image. Specifically, the parameters of the model can be updated based on the loss value, using the back propagation algorithm, the Adam optimization algorithm, the cosine annealing algorithm, etc., and the trained image super-resolution reconstruction model can be obtained by iterating several rounds. The specific number of iterations can be preset, or the training is considered to be completed when the test set meets the accuracy requirements.

[0136] The present application also provides a method for reconstructing an image super-resolution reconstruction model, comprising:

[0137] Step S160, obtaining a resolution image to be reconstructed;

[0138] Step S170: input the resolution image into the trained image super-resolution reconstruction model as described above to perform image reconstruction to obtain a super-resolution reconstructed image.

[0139] In an embodiment of the present application, in actual applications, the resolution image to be reconstructed can be input into a trained image super-resolution reconstruction model, and the input resolution image can be reconstructed with image super-resolution through the trained image super-resolution reconstruction model to obtain a super-resolution reconstructed image corresponding to the input resolution image.

[0140] See also Fig. 9 The present application also provides a training system for an image super-resolution reconstruction model, including:

[0141] The first processing unit 810 is used to obtain a super-resolution training image, and perform shallow feature extraction on the super-resolution training image to obtain shallow features;

[0142] The second processing unit 820 is used to perform multi-level separation deep feature extraction on the shallow features to obtain deep features of several different feature levels;

[0143] The third processing unit 830 is used to perform first feature fusion and complementation on all the deep features to obtain first complementary features;

[0144] A fourth processing unit 840 is used to perform image reconstruction on the shallow features according to the first complementary features to obtain a super-resolution reconstructed image;

[0145] The fifth processing unit 850 is used to update the parameters of the initialized image super-resolution reconstruction model according to the super-resolution reconstructed image to obtain a trained image super-resolution reconstruction model.

[0146] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0147] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0148] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0149] See also Fig.10 , Fig.10 The hardware structure of an electronic device of an embodiment is illustrated, and the electronic device includes:

[0150] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0151] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 902, and the processor 901 calls and executes the methods of the embodiments of this application;

[0152] Input / output interface 903, used to implement information input and output;

[0153] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0154] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0155] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0156] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program, and the computer program implements the above method when executed by a processor.

[0157] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0158] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0159] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0160] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0161] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0162] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0163] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0164] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0165] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0166] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0167] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0168] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0169] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A training method for an image super-resolution reconstruction model, characterized in that: include: Acquire a super-resolution training image, and perform shallow feature extraction on the super-resolution training image to obtain shallow features; Performing multi-level separation deep feature extraction on the shallow features to obtain deep features of several different feature levels; Performing first feature fusion and complementation on all the deep features to obtain first complementary features; Reconstructing the shallow features according to the first complementary features to obtain a super-resolution reconstructed image; According to the super-resolution reconstructed image, parameters of the initialized image super-resolution reconstruction model are updated to obtain a trained image super-resolution reconstruction model.

2. The training method according to claim 1, characterized in that: The extracting shallow features from the super-resolution training image to obtain shallow features includes: Performing a first point-by-point convolution on the super-resolution training image to obtain a plurality of channel features; Performing grouped deep convolution on all the channel features to obtain grouped features corresponding to each of the channel features; A second point-by-point convolution is performed on all the grouped features to obtain the shallow features.

3. The training method according to claim 1, characterized in that: Perform separate deep feature extraction on the input features to obtain the target deep features, including: Performing multi-layer separable residual convolution on the input features to obtain refined features at several different feature interaction levels; Performing a first feature distillation on the input features to obtain a first distilled feature, and performing a second feature distillation on all first target refined features to obtain a second distilled feature corresponding to each of the first target refined features, wherein the first target refined features are refined features other than the refined features at the last feature interaction level; Performing deep separable convolution on the second target refined feature to obtain a third target refined feature, where the second target refined feature is a refined feature of the last feature interaction level; According to the input features, attention feature fusion and complementation are performed on the first distilled features, the second distilled features and the third target refined features to obtain the target deep features.

4. The training method according to claim 3, characterized in that: The step of performing attention feature fusion and complementation on the first distilled feature, the second distilled feature and the third target refined feature according to the input feature to obtain the target deep feature includes: Performing second feature fusion and complementation on the first distillation feature, the second distillation feature and the third target refining feature to obtain a second complementary feature; Performing perceptual channel attention extraction on the second complementary feature to obtain a channel attention feature; Performing multi-scale spatial attention extraction on the channel attention feature to obtain a spatial attention feature; According to the input features, feature concatenation is performed on the spatial attention features to obtain the target deep features.

5. The training method according to claim 4, characterized in that: The performing multi-scale spatial attention extraction on the channel attention feature to obtain the spatial attention feature includes: Performing multi-scale attention distribution extraction on the channel attention feature to obtain a plurality of spatial attention distributions, each of which corresponds to a different spatial scale; According to all the spatial attention distributions, spatial attention fusion is performed on the channel attention features to obtain the spatial attention features.

6. The training method according to claim 3, characterized in that: The intermediate feature set is fused and complemented to obtain the target complementary features, including: Perform branch attention feature fusion on the intermediate feature set to obtain the intermediate fusion feature; Performing feature complementation processing on the intermediate fusion features to obtain the target complementary features; The intermediate feature set is a set of all deep features or a set of the first distillation feature, the second distillation feature and the third target refined feature.

7. The training method according to claim 6, characterized in that: The branch attention feature fusion is performed on the intermediate feature set to obtain the intermediate fusion feature, including: Performing feature channel splicing on all branch features in the intermediate feature set to obtain spliced ​​features; Performing channel information interaction on the splicing features to obtain channel interaction features; Performing maximum pooling on the channel interaction features to obtain maximum pooling features, and performing variance feature extraction on the channel interaction features to obtain variance features; Generate a plurality of feature weights according to the maximum pooling feature and the variance feature, each of the feature weights corresponding to a branch feature in the intermediate feature set; According to all the feature weights, the intermediate feature set is weighted fused to obtain the intermediate fused feature.

8. The training method according to claim 6, characterized in that: The performing feature complementation processing on the intermediate fusion features to obtain the target complementary features includes: Performing spatial displacement on the intermediate fusion feature to obtain a spatial displacement feature; According to the intermediate fusion feature, the spatial displacement feature is subjected to displacement feature fusion to obtain the target complementary feature; Among them, the intermediate fusion feature includes several subdivided channel features, each subdivided channel feature includes several spatial feature regions, the spatial displacement feature includes several spatial displacement regions, and each of the spatial displacement regions is used to record the feature information of the spatial feature regions in all the subdivided channel features that do not correspond to the spatial displacement region.

9. A reconstruction method of an image super-resolution reconstruction model, characterized in that: include: Acquire a resolution image to be reconstructed; The resolution image is input into the trained image super-resolution reconstruction model as described in any one of claims 1 to 8 to perform image reconstruction to obtain a super-resolution reconstructed image.

10. A training system for an image super-resolution reconstruction model, characterized in that: include: A first processing unit is used to obtain a super-resolution training image, and perform shallow feature extraction on the super-resolution training image to obtain shallow features; A second processing unit is used to perform multi-level separation deep feature extraction on the shallow features to obtain deep features of several different feature levels; A third processing unit is used to perform first feature fusion and complementation on all the deep features to obtain first complementary features; a fourth processing unit, configured to perform image reconstruction on the shallow features according to the first complementary features to obtain a super-resolution reconstructed image; The fifth processing unit is used to update the parameters of the initialized image super-resolution reconstruction model according to the super-resolution reconstructed image to obtain a trained image super-resolution reconstruction model.

Images