Super-resolution image generation method and apparatus

Abstract

Embodiments of this disclosure provide a method and apparatus for generating super-resolution images. The super-resolution image generation method includes: firstly, in response to acquiring a low-resolution image and a scaling factor, acquiring an image generation model for generating a super-resolution image, wherein the image generation model includes a feature extraction module for extracting high-frequency features and a feature reconstruction module for reconstructing high-frequency features; then, based on the low-resolution image, the scaling factor, and the feature extraction module, extracting high-frequency image features corresponding to the low-resolution image; subsequently, based on the high-frequency image features, the scaling factor, and the feature reconstruction module, reconstructing the pixel values ​​of the high-resolution image corresponding to the low-resolution image; and finally, based on the pixel values ​​of the high-resolution image, generating the high-resolution image corresponding to the low-resolution image, capable of generating high-resolution images of arbitrary scales based on the low-resolution image and the image generation model.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the fields of computer technology and internet technology, specifically to the fields of artificial intelligence technology and image processing technology, and particularly to a method and apparatus for generating super-resolution images. Background Technology

[0002] With the continuous development and widespread application of deep neural networks, artificial intelligence has made rapid progress in the field of computer vision. Research on single-image super-resolution algorithms has achieved great success, and super-resolution of images at arbitrary scales is crucial in computer vision and digital image processing. A pioneering work in deep learning image super-resolution is SRCNN, which uses three convolutional layers to learn the mapping between low-resolution (LR) and high-resolution (HR) images to achieve image super-resolution tasks based on CNN networks. Furthermore, since residual structures can alleviate the training difficulty of CNNs, EDSR and RDN methods introduce residual blocks to increase network depth, obtain better feature representations, and upsample end features through sub-pixel convolutional layers, thereby achieving better super-resolution performance.

[0003] However, these methods require designing specific upsampling modules for each scaling factor. Each module can only upscale images with fixed integer scaling factors, and subpixel convolution only works for integer scaling factors. Summary of the Invention

[0004] Embodiments of this disclosure provide a method, apparatus, electronic device, and computer-readable medium for generating super-resolution images.

[0005] In a first aspect, embodiments of this disclosure provide a super-resolution image generation method, the method comprising: in response to acquiring a low-resolution image and a scaling factor, acquiring an image generation model for generating a super-resolution image, wherein the image generation model includes a feature extraction module for extracting high-frequency features and a feature reconstruction module for reconstructing high-frequency features; extracting high-frequency image features corresponding to the low-resolution image based on the low-resolution image, the scaling factor and the feature extraction module; reconstructing pixel values ​​of a high-resolution image corresponding to the low-resolution image based on the high-frequency image features, the scaling factor and the feature reconstruction module; and generating a high-resolution image corresponding to the low-resolution image based on the pixel values ​​of the high-resolution image.

[0006] In some embodiments, the feature extraction module includes a feature extraction network and a scale-aware network; and, based on the low-resolution image, a scaling factor, and the feature extraction module, extracts high-frequency image features corresponding to the low-resolution image, including: extracting image features corresponding to the low-resolution image based on the low-resolution image and the feature extraction network; and generating high-frequency image features corresponding to the low-resolution image based on the image features, the scaling factor, and the scale-aware network.

[0007] In some embodiments, the image generation model includes multiple feature extraction modules based on a hierarchical architecture. Each feature extraction module receives the output of the previous layer feature extraction module and uses the output as the input of the next layer feature extraction module. Furthermore, based on the low-resolution image, a scaling factor, and the feature extraction modules, high-frequency image features corresponding to the low-resolution image are extracted, including: inputting the low-resolution image into the feature extraction network in the first feature extraction module to extract image features corresponding to the low-resolution image; inputting the image features and a scaling factor into the scale-aware network in the first feature extraction module to generate initial high-frequency image features corresponding to the low-resolution image; inputting the initial high-frequency image features and a scaling factor into the second feature extraction module, until the last feature extraction module outputs the high-frequency image features corresponding to the low-resolution image.

[0008] In some embodiments, the scale-aware network includes a scale-aware subnetwork and a frequency equalization subnetwork; and, based on image features, a scaling factor, and the scale-aware network, generating high-frequency image features corresponding to the low-resolution image includes: generating initial perceived image features corresponding to the low-resolution image based on image features, a scaling factor, and the scale-aware subnetwork; and generating high-frequency image features corresponding to the low-resolution image based on the initial perceived image features and the frequency equalization subnetwork.

[0009] In some embodiments, the scale-aware subnetwork includes a fully connected layer and a dynamic convolutional layer; and, based on image features, a scaling factor, and the scale-aware subnetwork, generating initial perceptual image features corresponding to a low-resolution image includes: inputting a scaling factor into the fully connected layer to generate dynamic convolution weights for convolutional processing of the image features; and inputting the dynamic convolution weights and image features into the dynamic convolutional layer to generate initial perceptual image features corresponding to the low-resolution image.

[0010] In some embodiments, the frequency equalization subnetwork includes a spectral channel layer and a spatial scaling layer; and, based on the initial perceived image features and the frequency equalization subnetwork, generating high-frequency image features corresponding to the low-resolution image includes: inputting the initial perceived image features into the spectral channel layer to generate frequency domain attention weights corresponding to the initial perceived image features; inputting the initial perceived image features into the spatial scaling layer to generate spatial attention weights corresponding to the initial perceived image features; obtaining high-frequency attention weights corresponding to the low-resolution image based on the frequency domain attention weights and the spatial attention weights; and generating high-frequency image features corresponding to the low-resolution image based on the initial perceived image features and the high-frequency attention weights.

[0011] In some embodiments, the feature reconstruction module includes a scale coding network, a scale-aware sub-network, and a residual multilayer perceptron; and, based on high-frequency image features, a scaling factor, and the feature reconstruction module, reconstructs the pixel values ​​of the high-resolution image corresponding to the low-resolution image, including: obtaining an encoded scaling factor based on the scaling factor and the scale coding network; obtaining a perceived scale feature based on the scaling factor, the encoded scaling factor, and the scale-aware sub-network; and inputting the high-frequency image features, the encoded scaling factor, and the perceived scale feature into the residual multilayer perceptron to reconstruct the pixel values ​​of the high-resolution image corresponding to the low-resolution image.

[0012] Secondly, embodiments of this disclosure provide a super-resolution image generation apparatus, comprising: an acquisition module configured to acquire an image generation model for generating a super-resolution image in response to acquiring a low-resolution image and a scaling factor, wherein the image generation model includes a feature extraction module for extracting high-frequency features and a feature reconstruction module for reconstructing high-frequency features; an extraction module configured to extract high-frequency image features corresponding to the low-resolution image based on the low-resolution image, the scaling factor, and the feature extraction module; a reconstruction module configured to reconstruct pixel values ​​of a high-resolution image corresponding to the low-resolution image based on the high-frequency image features, the scaling factor, and the feature reconstruction module; and a generation module configured to generate a high-resolution image corresponding to the low-resolution image based on the pixel values ​​of the high-resolution image.

[0013] In some embodiments, the feature extraction module includes a feature extraction network and a scale-aware network; and the extraction module is further configured to: extract image features corresponding to the low-resolution image based on the low-resolution image and the feature extraction network; and generate high-frequency image features corresponding to the low-resolution image based on the image features, the scale factor and the scale-aware network.

[0014] In some embodiments, the image generation model includes multiple feature extraction modules based on a hierarchical architecture. Each feature extraction module receives the output of the previous layer feature extraction module and uses the output as the input of the next layer feature extraction module. The extraction module is further configured to: input a low-resolution image into the feature extraction network in the first feature extraction module to extract image features corresponding to the low-resolution image; input the image features and a scaling factor into the scale-aware network in the first feature extraction module to generate initial high-frequency image features corresponding to the low-resolution image; input the initial high-frequency image features and the scaling factor into the second feature extraction module, until the last feature extraction module outputs the high-frequency image features corresponding to the low-resolution image.

[0015] In some embodiments, the scale-aware network includes a scale-aware subnetwork and a frequency equalization subnetwork; and the extraction module is further configured to: generate initial perceived image features corresponding to the low-resolution image based on image features, a scaling factor, and the scale-aware subnetwork; and generate high-frequency image features corresponding to the low-resolution image based on the initial perceived image features and the frequency equalization subnetwork.

[0016] In some embodiments, the scale-aware subnetwork includes a fully connected layer and a dynamic convolutional layer; and the extraction module is further configured to: input a scaling factor into the fully connected layer to generate dynamic convolutional weights for convolutional processing of image features; and input the dynamic convolutional weights and image features into the dynamic convolutional layer to generate initial perceptual image features corresponding to the low-resolution image.

[0017] In some embodiments, the frequency equalization subnetwork includes a spectral channel layer and a spatial scaling layer; and the extraction module is further configured to: input initial perceived image features into the spectral channel layer to generate frequency domain attention weights corresponding to the initial perceived image features; input the initial perceived image features into the spatial scaling layer to generate spatial attention weights corresponding to the initial perceived image features; obtain high-frequency attention weights corresponding to the low-resolution image based on the frequency domain attention weights and the spatial attention weights; and generate high-frequency image features corresponding to the low-resolution image based on the initial perceived image features and the high-frequency attention weights.

[0018] In some embodiments, the feature reconstruction module includes a scale coding network, a scale-aware sub-network, and a residual multilayer perceptron; and the reconstruction module is further configured to: obtain an encoded scale factor based on the scale factor and the scale coding network; obtain a perceived scale feature based on the scale factor, the encoded scale factor, and the scale-aware sub-network; and input the high-frequency image features, the encoded scale factor, and the perceived scale feature into the residual multilayer perceptron to reconstruct the pixel values ​​of the high-resolution image corresponding to the low-resolution image.

[0019] Thirdly, embodiments of this disclosure provide an electronic device comprising: one or more processors; a storage device having one or more programs stored thereon; wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the super-resolution image generation method as described in any embodiment of the first aspect.

[0020] Fourthly, embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon that, when executed by a processor, implements the super-resolution image generation method as described in any embodiment of the first aspect.

[0021] The super-resolution image generation method provided in the embodiments of this disclosure first obtains a low-resolution image and a scaling factor, and then obtains an image generation model for generating the super-resolution image. The image generation model includes a feature extraction module for extracting high-frequency features and a feature reconstruction module for reconstructing high-frequency features. Then, based on the low-resolution image, the scaling factor, and the feature extraction module, high-frequency image features corresponding to the low-resolution image are extracted. Next, based on the high-frequency image features, the scaling factor, and the feature reconstruction module, the pixel values ​​of the high-resolution image corresponding to the low-resolution image are reconstructed. Finally, based on the pixel values ​​of the high-resolution image, the high-resolution image corresponding to the low-resolution image is generated. This method can generate high-resolution images of arbitrary scales based on the low-resolution image and the image generation model, achieving arbitrary scale super-resolution tasks. It improves the accuracy and flexibility of super-resolution tasks, and can adaptively change the feature extraction capability according to different scaling factors, achieving adaptive generation of super-resolution images at arbitrary scales. It can achieve super-resolution images corresponding to arbitrary scaling factors without adjusting the network structure of the image generation model. Attached Figure Description

[0022] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0023] Figure 1 This is an exemplary system architecture diagram to which one embodiment of this disclosure can be applied;

[0024] Figure 2 This is a flowchart of an embodiment of the super-resolution image generation method according to the present disclosure;

[0025] Figure 3 This is a flowchart of yet another embodiment of the super-resolution image generation method according to the present disclosure;

[0026] Figure 4 This is a flowchart of an embodiment of generating high-frequency image features based on image features according to the present disclosure;

[0027] Figure 5 This is a flowchart of an embodiment of generating high-frequency image features based on initial perceived image features according to the present disclosure;

[0028] Figure 6 This is a flowchart of an embodiment of reconstructing pixel values ​​of a high-resolution image according to the present disclosure;

[0029] Figure 7 This is a schematic diagram of a structure of an embodiment of the super-resolution image generation apparatus according to the present disclosure;

[0030] Figure 8This is a schematic diagram of the structure of an electronic device suitable for implementing embodiments of the present disclosure. Detailed Implementation

[0031] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the relevant disclosure and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the disclosure are shown in the accompanying drawings.

[0032] It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0033] Figure 1 An exemplary system architecture 100 is shown for a super-resolution image generation method and apparatus to which embodiments of the present disclosure can be applied.

[0034] like Figure 1 As shown, system architecture 100 may include terminal devices 104, 105, and 106, network 107, and servers 101, 102, and 103. Network 107 serves as the medium for providing communication links between terminal devices 104, 105, and 106 and servers 101, 102, and 103. Network 107 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0035] Users can interact with servers 101, 102, and 103 belonging to the same server cluster via network 107 through terminal devices 104, 105, and 106 to receive or send information. Various applications can be installed on terminal devices 104, 105, and 106, such as product display applications, data analysis applications, and search applications.

[0036] Terminal devices 104, 105, and 106 can be either hardware or software. When the terminal device is hardware, it can be any electronic device with a display screen that supports communication with the server, including but not limited to smartphones, tablets, laptops, and desktop computers. When the terminal device is software, it can be installed in the electronic devices listed above. It can be implemented as multiple software programs or software modules, or as a single software program or software module. No specific limitations are made here.

[0037] Servers 101, 102, and 103 can be servers that provide various services, such as backend servers that receive requests sent by terminal devices with which they have established communication connections. The backend servers can receive and analyze the requests sent by the terminal devices and generate processing results.

[0038] Servers 101, 102, and 103 can acquire low-resolution images and scale factors, and obtain an image generation model for generating super-resolution images. This image generation model can include a feature extraction module for extracting high-frequency features and a feature reconstruction module for reconstructing high-frequency features. Then, based on the low-resolution image, scale factor, and feature extraction module, high-frequency image features corresponding to the low-resolution image are extracted. After that, based on the high-frequency image features, scale factor, and feature reconstruction module, the pixel values ​​of the high-resolution image corresponding to the low-resolution image are reconstructed. Finally, based on the pixel values ​​of the high-resolution image, the high-resolution image corresponding to the low-resolution image is generated.

[0039] It should be noted that a server can be either hardware or software. When a server is hardware, it can be any electronic device that provides various services to terminal devices. When a server is software, it can be implemented as multiple software programs or software modules providing various services to terminal devices, or it can be implemented as a single software program or software module providing various services to terminal devices. No specific limitations are made here.

[0040] It should be noted that the super-resolution image generation method provided in the embodiments of this disclosure can be executed by servers 101, 102, and 103. Accordingly, the super-resolution image generation device is disposed in servers 101, 102, and 103.

[0041] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0042] Continue to refer to Figure 2 The diagram illustrates a flow 200 of an embodiment of a super-resolution image generation method according to the present disclosure. The super-resolution image generation method includes the following steps:

[0043] Step 210: In response to obtaining the low-resolution image and the scaling factor, obtain an image generation model for generating the super-resolution image.

[0044] In this step, the super-resolution image generation method runs on the execution entity (e.g., Figure 1 Servers 101, 102, and 103 can receive or retrieve low-resolution images from terminal devices via a network or from local storage. These low-resolution images can include any object, and this disclosure does not impose any specific limitations on them. The aforementioned executing entities can also receive a scaling factor sent by the terminal device. This scaling factor can represent the resolution ratio between the low-resolution image and the high-resolution image, and can be the factor by which the user needs to enlarge the low-resolution image.

[0045] After obtaining the low-resolution image and the scaling factor, the aforementioned execution entity can obtain an image generation model for generating super-resolution images. This image generation model is a pre-trained model for generating high-resolution images from low-resolution images, including a feature extraction module for extracting high-frequency features and a feature reconstruction module for reconstructing high-frequency features.

[0046] Step 220: Based on the low-resolution image, scaling factor, and feature extraction module, extract the high-frequency image features corresponding to the low-resolution image.

[0047] In this step, after the execution entity obtains the image generation model, it can input the low-resolution image and the scaling factor into the image generation model. The feature extraction module in the image generation model processes the low-resolution image and the scaling factor to extract the high-frequency image features corresponding to the low-resolution image. These high-frequency image features can characterize the image features of the low-resolution image that generate a high-resolution image based on the scaling factor.

[0048] The aforementioned execution entity can input the low-resolution image and the scaling factor into the feature extraction module. The feature extraction module can determine the high-resolution image corresponding to the low-resolution image based on the scaling factor, extract features from the low-resolution image to obtain the corresponding initial image features, and process the initial image features based on the high-resolution image to generate high-frequency image features from the initial image features used to generate the high-resolution image.

[0049] Step 230: Based on high-frequency image features, scaling factors, and feature reconstruction modules, reconstruct the pixel values ​​of the high-resolution image corresponding to the low-resolution image.

[0050] In this step, after the execution entity obtains the high-frequency image features corresponding to the low-resolution image through the feature extraction module, it can input the high-frequency image features and the scaling factor into the feature reconstruction module of the image generation model. The feature reconstruction module processes the high-frequency image features and the scaling factor, and reconstructs the pixel values ​​of the low-resolution image for the high-resolution image corresponding to the scaling factor based on the high-frequency image features and the scaling factor.

[0051] The aforementioned execution entity can input high-frequency image features and scaling factors into the feature reconstruction module. The feature reconstruction module can adaptively adjust the receptive field range of the current scale-related texture region according to the scaling factor, determine the high-resolution image corresponding to the low-resolution image based on the scaling factor, and restore and reconstruct the high-frequency image features to generate the pixel values ​​of the low-resolution image for the high-resolution image corresponding to the scaling factor.

[0052] Step 240: Generate a high-resolution image corresponding to the low-resolution image based on the pixel values ​​of the high-resolution image.

[0053] In this step, after the execution entity obtains the pixel values ​​of the high-resolution image corresponding to the low-resolution image, it can generate the high-resolution image corresponding to the low-resolution image based on the pixel values ​​of the high-resolution image.

[0054] The aforementioned executing entity can also send the generated high-resolution image to the terminal device, so that the terminal device can display the high-resolution image to the user through the screen.

[0055] Therefore, the aforementioned executing entity can generate a high-resolution image corresponding to a low-resolution image based on the pixel values ​​of the high-resolution image using any relevant technology that supports it.

[0056] The super-resolution image generation method provided in the embodiments of this disclosure first obtains a low-resolution image and a scaling factor, and then obtains an image generation model for generating the super-resolution image. The image generation model includes a feature extraction module for extracting high-frequency features and a feature reconstruction module for reconstructing high-frequency features. Then, based on the low-resolution image, the scaling factor, and the feature extraction module, high-frequency image features corresponding to the low-resolution image are extracted. Next, based on the high-frequency image features, the scaling factor, and the feature reconstruction module, the pixel values ​​of the high-resolution image corresponding to the low-resolution image are reconstructed. Finally, based on the pixel values ​​of the high-resolution image, the high-resolution image corresponding to the low-resolution image is generated. This method can generate high-resolution images of arbitrary scales based on the low-resolution image and the image generation model, achieving arbitrary scale super-resolution tasks. It improves the accuracy and flexibility of super-resolution tasks, and can adaptively change the feature extraction capability according to different scaling factors, achieving adaptive generation of super-resolution images at arbitrary scales. It can achieve super-resolution images corresponding to arbitrary scaling factors without adjusting the network structure of the image generation model.

[0057] refer to Figure 3 , Figure 3 A flowchart illustrating yet another embodiment of a super-resolution image generation method is provided, which may include the following steps:

[0058] Step 310: In response to obtaining the low-resolution image and the scaling factor, obtain an image generation model for generating the super-resolution image.

[0059] Step 310 of this embodiment can be performed in accordance with... Figure 2 Step 210 in the illustrated embodiment is performed in a similar manner, and will not be described in detail here.

[0060] Step 320: Extract image features corresponding to the low-resolution image based on the low-resolution image and the feature extraction network.

[0061] The aforementioned feature extraction module may include a feature extraction network for extracting image features and a scale-aware network for adjusting image features based on a scaling factor.

[0062] In this step, after acquiring the low-resolution image and scale factor, the execution entity can input the low-resolution image into a feature extraction network for feature extraction. This feature extraction network can be a backbone network for extracting image features, i.e., a deep neural feature extraction network, which can be a residual extraction block composed of different convolutional layers and pooling layers. Simultaneously, a feature extraction module can be used to extract corresponding feature information. The execution entity inputs the low-resolution image into the feature extraction network, which extracts features from the low-resolution image and outputs the image features corresponding to the low-resolution image.

[0063] Step 330: Based on image features, scaling factor and scale-aware network, generate high-frequency image features corresponding to the low-resolution image.

[0064] In this step, after the execution entity obtains the image features corresponding to the low-resolution image through the feature extraction network, it can input the obtained image features and scale factor into the scale-aware network. The scale-aware network can adjust the image features according to the input scale factor. The adjusted features can characterize the image features in the high-resolution image corresponding to the scale factor, thereby generating high-frequency image features for generating the high-resolution image corresponding to the low-resolution image.

[0065] The scale-aware network described above can adaptively adjust image features based on the received scaling factor, thereby generating high-frequency image features corresponding to the scaling factor for low-resolution images.

[0066] Step 340: Based on high-frequency image features, scaling factors, and feature reconstruction modules, reconstruct the pixel values ​​of the high-resolution image corresponding to the low-resolution image.

[0067] Step 340 of this embodiment can be performed in accordance with... Figure 2 Step 230 in the illustrated embodiment is performed in a similar manner, and will not be described in detail here.

[0068] Step 350: Generate a high-resolution image corresponding to the low-resolution image based on the pixel values ​​of the high-resolution image.

[0069] Step 350 of this embodiment can be performed in accordance with... Figure 2 Step 240 in the illustrated embodiment is performed in a similar manner, and will not be described in detail here.

[0070] In this embodiment, by using a feature extraction module that includes a feature extraction network and a scale-aware network to process the low-resolution image and the scale factor, the image features can be adaptively adjusted according to the received scale factor, thereby generating high-frequency image features corresponding to the low-resolution image under the scale factor. Therefore, the feature extraction module can obtain different high-frequency image features according to different scale factors, improving the flexibility of super-resolution image generation and no longer being limited by the scale factor in the prior art.

[0071] As an optional implementation, the image generation model described above may further include multiple feature extraction modules based on a hierarchical architecture. Each feature extraction module receives the output of the previous layer's feature extraction module and uses the output as the input of the next layer's feature extraction module. That is, the image generation model includes multiple feature extraction modules and a feature reconstruction module based on a hierarchical architecture. These multiple feature extraction modules are used to repeatedly process the image features and scale factor of the low-resolution image to obtain the optimal high-frequency image features corresponding to the scale factor. The number of feature extraction modules can be set by the operator based on the neural network depth, for example, it could be 10. This disclosure does not specifically limit this. Furthermore, step 220 above, extracting high-frequency image features corresponding to the low-resolution image based on the low-resolution image, scale factor, and feature extraction modules, may include the following steps: inputting the low-resolution image into the feature extraction network of the first feature extraction module to extract the image features corresponding to the low-resolution image; inputting the image features and scale factor into the scale-aware network of the first feature extraction module to generate initial high-frequency image features corresponding to the low-resolution image; inputting the initial high-frequency image features and scale factor into the second feature extraction module, until the last feature extraction module outputs the high-frequency image features corresponding to the low-resolution image.

[0072] Specifically, the image generation model described above can include multiple feature extraction modules based on a hierarchical architecture. Each feature extraction module transmits data in a series manner, and each feature extraction module also includes a feature extraction network and a scale-aware network.

[0073] The aforementioned execution entity can input a low-resolution image into the first feature extraction module. The feature extraction network of the first feature extraction module extracts features from the low-resolution image, extracting the image features corresponding to the low-resolution image. Then, the image features and the scale factor are simultaneously input into the scale-aware network of the first feature extraction module. The scale-aware network of the first feature extraction module adjusts the image features according to the input scale factor. The adjusted features can characterize the initial high-frequency image features in the high-resolution image corresponding to the scale factor.

[0074] The aforementioned execution entity can input the initial high-frequency image features output by the first feature extraction module into the second feature extraction module. The second feature extraction module can perform the same processing operations as the first feature extraction module. The feature extraction network of the second feature extraction module extracts features from the initial high-frequency image features, extracts the image features corresponding to the initial high-frequency image features, and then simultaneously inputs the image features and scale factor into the scale-aware network of the second feature extraction module. The scale-aware network of the second feature extraction module adjusts the image features according to the input scale factor to obtain the initial high-frequency image features output by the second feature extraction module. The initial high-frequency image features output by the second feature extraction module are then input into the third feature extraction module to perform the same processing operations until the last feature extraction module outputs the high-frequency image features corresponding to the low-resolution image.

[0075] In this implementation, by setting up multiple feature extraction modules, high-frequency image features under the scaling factor are repeatedly extracted and adjusted, making the high-frequency image features corresponding to the low-resolution image more accurate, thus improving the accuracy of high-frequency image feature extraction, and making the high-resolution image generated based on the high-frequency image features more accurate.

[0076] refer to Figure 4 , Figure 4 A flowchart illustrating an embodiment of generating high-frequency image features based on image features is provided. The scale-aware network in the feature extraction module may include a scale-aware sub-network for adaptively learning corresponding degradation modes based on arbitrary scaling factors and a frequency equalization sub-network for attentionally extracting high-frequency features in low-resolution images. Step 330 above, generating high-frequency image features corresponding to the low-resolution image based on image features, scaling factors, and the scale-aware network, may include the following steps:

[0077] Step 410: Based on image features, scaling factor and scale-aware subnetwork, generate initial perceptual image features corresponding to the low-resolution image.

[0078] In this step, the scale-aware network in the image generation model can include a scale-aware subnetwork. After the executing entity obtains the image features corresponding to the low-resolution image, it can input the image features and scale factor into the scale-aware subnetwork. The scale-aware subnetwork can model and extract features based on the scale factor, achieving adaptive learning of the corresponding degradation mode according to any scale factor. The scale-aware subnetwork can process the image features and scale factor to generate initial perceived image features corresponding to the low-resolution image.

[0079] As an optional implementation, the scale-aware subnetwork described above may include fully connected layers and dynamically convolutional layers. Step 410, which generates initial perceptual image features corresponding to the low-resolution image based on image features, a scaling factor, and the scale-aware subnetwork, may include the following steps: inputting the scaling factor into the fully connected layer to generate dynamic convolution weights for convolutional processing of the image features; inputting the dynamic convolution weights and image features into the dynamically convolutional layer to generate the initial perceptual image features corresponding to the low-resolution image.

[0080] Specifically, the scale-aware subnetwork described above can be configured with a preset number of fully connected layers and dynamic convolutional layers, such as two fully connected layers; this disclosure does not impose a specific limitation on this. The execution entity can input a scaling factor into the fully connected layer, which processes the input scaling factor to generate dynamic convolution weights for convolutional processing of image features. Then, the execution entity can input the dynamic convolution weights and image features into the dynamic convolutional layer, which can adapt the image features according to the dynamic convolution weights, perform scale-aware feature mapping, and generate initial perceptual image features corresponding to the low-resolution image.

[0081] The operations performed by the aforementioned dynamic convolutional layer based on dynamic convolutional weights and image features can be represented as follows:

[0082]

[0083] Among them, F sda Let F represent the initial perceived image features, φ(.) represent the dynamic convolution weights, and F... rg Represents image features. This represents a degradation prediction network that uses a scaling factor as input, where θ represents the weight parameters of the prediction network, and K represents the convolution kernel.

[0084] Therefore, the scale-aware subnetwork can dynamically perceive the downsampling degradation model of various scaling factors, and achieve adaptive feature representation of arbitrary scaling factors.

[0085] In this implementation, the scale-aware subnetwork is composed of fully connected layers and dynamic convolutional layers. The fully connected layers can model scale degradation information during training and adaptively generate attention weights for the dynamic convolutional kernels based on the input scaling factor during testing. The dynamic convolutional layers can provide stronger feature extraction capabilities for super-resolution tasks with arbitrary scaling factors.

[0086] Step 420: Based on the initial perceptual image features and the frequency equalization sub-network, generate high-frequency image features corresponding to the low-resolution image.

[0087] In this step, the scale-aware network in the image generation model can also include a frequency equalization subnetwork. After the execution entity obtains the initial perceived image features corresponding to the low-resolution image, it can input the initial perceived image features into the frequency equalization subnetwork. The frequency equalization subnetwork can extract high-frequency components based on the initial perceived image features, thereby extracting the high-frequency image features corresponding to the low-resolution image.

[0088] In this implementation, by setting up a scale-aware network including a scale-aware subnetwork and a frequency equalization subnetwork, it is possible to achieve adaptive feature representation of arbitrary scaling factors and pay more attention to extracting high-frequency components, so that the extracted high-frequency image features can be more accurate and comprehensive.

[0089] refer to Figure 5 , Figure 5 A flowchart illustrating an embodiment of generating high-frequency image features based on initial perceived image features is shown, which may include the following steps:

[0090] The aforementioned frequency equalization subnetwork may include a spectrum channel layer and a spatial scaling layer. The spectrum channel layer can adaptively select useful high-frequency spectrum channels and fuse them according to a scaling factor. The spatial scaling layer can assign appropriate model receiver field locations to scaling-related high-frequency texture regions.

[0091] Step 510: Input the initial perceived image features into the spectral channel layer to generate the frequency domain attention weights corresponding to the initial perceived image features.

[0092] In this step, after the execution entity obtains the initial perceptual image features corresponding to the low-resolution image, it can input the initial perceptual image features into the spectrum channel layer. The spectrum channel layer can use two-dimensional discrete cosine transform (DCT) to convert the initial perceptual image features into frequency domain image features, and adaptively assign different weights to different frequency channels, fusing the spectrum components into spectrum channel attention, and generating frequency domain attention weights corresponding to the initial perceptual image features.

[0093] The specific process by which the spectral channel layer uses two-dimensional discrete cosine transform (DCT) to convert the initial perceptual image features into frequency domain image features can be represented as follows:

[0094]

[0095] Among them, DCT h,w Let represent the two-dimensional discrete cosine transform, h and W represent the magnitudes of the two-dimensional discrete cosine transform basis, and freq() represent the frequency domain image features of different frequency bands, which can be defined as freq = [freq()]. 0 ,freq 1 ,…,freq kThe frequency band K can be represented as:

[0096]

[0097] in, Represents the initial perceived image features. Specifically, this can be represented as a two-dimensional discrete cosine transform basis:

[0098]

[0099] Furthermore, the calculation method for spectrum channel attention can be as follows:

[0100] H FSCA (F)=sigmoid(fc(fuse(freq(F))))

[0101] Where fuse represents frequency domain fusion with the spectral components added, and its size ranges from C. out ×h×w becomes C×1×1.

[0102] Step 520: Input the initial perceived image features into the spatial scaling layer to generate spatial attention weights corresponding to the initial perceived image features.

[0103] In this step, after the execution entity obtains the initial perceptual image features corresponding to the low-resolution image, it can also input the initial perceptual image features into the spatial scaling layer. The spatial scaling layer can use three convolutional layers and two sampling layers to obtain the attention mask of the initial perceptual image features in the spatial dimension, process the initial perceptual image features, and generate the spatial attention weights corresponding to the initial perceptual image features.

[0104] Specifically, this can be manifested as:

[0105] H SsA (F)=conv(↑(conv(↓(conv(F))))

[0106] Here, conv represents a convolutional layer and activation function, and ↑ and ↓ represent upsampling and downsampling operations, respectively. Downsampling followed by upsampling expands the receptive field for use as an attention map, and spatial scaling operations are used to simulate spatial attention with different scaling factors, distinguishing between scaling factor-independent and scaling factor-dependent regions.

[0107] Step 530: Based on the frequency domain attention weight and the spatial attention weight, obtain the high-frequency attention weight corresponding to the low-resolution image.

[0108] In this step, after the execution entity generates the frequency domain attention weights corresponding to the initial perceived image features through the frequency channel layer and the spatial attention weights corresponding to the initial perceived image features through the spatial scaling layer, it can obtain the high-frequency attention weights corresponding to the low-resolution image based on the frequency domain attention weights and the spatial attention weights.

[0109] Specifically, this manifests as follows:

[0110]

[0111] Where F represents the initial perceived image features, H FSCA and H SSA These represent the operations of the spectral channel layer and the spatial scaling layer, respectively. ⊙ represents broadcast multiplication, and ⊙ represents Hadama multiplication.

[0112] Step 540: Based on the initial perceptual image features and high-frequency attention weights, generate high-frequency image features corresponding to the low-resolution image.

[0113] In this step, after the execution entity obtains the high-frequency attention weights corresponding to the low-resolution image, it can generate high-frequency image features corresponding to the low-resolution image based on the initial perceived image features and the high-frequency attention weights.

[0114] In this implementation, high-frequency attention weights are obtained by setting a spectral channel layer and a spatial scaling layer. Based on the initial perceptual image features and the high-frequency attention weights, high-frequency image features corresponding to the low-resolution image are generated. The spatial scaling layer can reasonably allocate the receptive field positions of texture and related regions according to different super-resolution scenarios. The spectral channel layer and the spatial scaling layer can perform self-optimization and mutual optimization to achieve balanced attention in spatial and channel dimensions. This enables the frequency equalization subnetwork to learn the differences in high-frequency texture recovery under different scaling factors, thereby improving the model's generalization performance.

[0115] refer to Figure 6 , Figure 6 A flowchart illustrating an embodiment of reconstructing pixel values ​​of a high-resolution image is provided. The aforementioned feature reconstruction module can avoid distortion in texture reconstruction by adaptively adjusting the receptive field range of the scale factor-related texture region based on an arbitrary scaling factor. It may include a scale-coding network, a scale-aware sub-network, and a residual multilayer perceptron. Step 230, based on high-frequency image features, the scaling factor, and the feature reconstruction module, reconstructs the pixel values ​​of the high-resolution image corresponding to the low-resolution image, and may include the following steps:

[0116] Step 610: Based on the scaling factor and scale coding network, obtain the encoded scaling factor.

[0117] In this step, the aforementioned executing entity can input the scaling factor into the scale encoding network, which encodes the scaling factor to generate the encoded scaling factor.

[0118] The scaling factor encoding by a scaling network can be represented as follows:

[0119] F SE =ν(ρ(S))

[0120] Here, S represents the input scaling factor, and ρ represents the standard coordinate projection function that projects the input scaling factor onto two-dimensional spatial coordinates. Since spatial coordinates are a very low-dimensional representation, they are insufficient to express rich color and geometric details. Therefore, v is used as an encoder function consisting of two convolutional layers to map the spatial coordinates to a higher-dimensional space to learn richer information expressions.

[0121] Step 620: Based on the scaling factor, the encoded scaling factor, and the scale-aware sub-network, the perceptual scale features are obtained.

[0122] In this step, after obtaining the encoded scaling factor, the execution entity inputs the scaling factor and the encoded scaling factor into the scale-aware subnetwork. This scale-aware subnetwork can have a preset number of fully connected layers and dynamic convolutional layers; for example, it can have two fully connected layers, but this disclosure does not specifically limit this. The execution entity can input the scaling factor into the fully connected layer, which processes the input scaling factor to generate dynamic convolution weights for convolutional processing of image features. Then, the execution entity can input the dynamic convolution weights and the encoded scaling factor into the dynamic convolutional layer, which can adapt the encoded scaling factor according to the dynamic convolution weights to generate scale-aware features.

[0123] The operations performed by the aforementioned dynamic convolutional layer based on dynamic convolutional weights and image features can be represented as follows:

[0124]

[0125] Among them, F sda,n+1 Represents the perceptual scale feature, φ(.) represents the dynamic convolution weights, and F SE This represents the scaling factor after encoding. This represents a degradation prediction network that uses a scaling factor as input, where θ represents the weight parameters of the prediction network, and K represents the convolution kernel.

[0126] Therefore, the scale-aware subnetwork can dynamically perceive the downsampling degradation model of various scaling factors, and achieve adaptive feature representation of arbitrary scaling factors.

[0127] Step 630: Input the high-frequency image features, the encoded scaling factor, and the perceptual scale features into the residual multilayer perceptron to reconstruct the pixel values ​​of the high-resolution image corresponding to the low-resolution image.

[0128] In this step, after the aforementioned execution entity obtains the high-frequency image features, the encoded scaling factor, and the perceptual scale features, it can input these features into a residual multilayer perceptron. The residual multilayer perceptron can be a Residual Multilayer Perceptron (ResMLP) that functions as an implicit image function. Compared to the standard multilayer perceptron (MLP), the ResMLP structure, which adds residual links between every two hidden layers, follows the conventions in ResNet. The residual multilayer perceptron can perform feature reconstruction on the high-frequency image features, the encoded scaling factor, and the perceptual scale features, reconstructing the pixel values ​​of the high-resolution image corresponding to the low-resolution image. The residual multilayer perceptron ensures that all hidden layers can directly obtain the target scaling factor, ensuring the accuracy of high-frequency information recovery.

[0129] The specific behavior of residual multilayer perceptrons is as follows:

[0130] HR rgb =f θ ([F saf a,n ,F SE ,F sda,n+1 ])

[0131] Among them, f θ F represents a residual multilayer perceptron that accurately reconstructs the pixel values ​​of a high-resolution image. saf a,n ,F SE ,F sda,n+1 As input, the pixel values ​​of the high-resolution image are used as output. F saf a,n F represents high-frequency image features. SE F represents the scale factor after encoding. sda,n+1 Represents perceptual scale features, HR rgb Represents the pixel value of a high-resolution image.

[0132] In this implementation, by setting up a feature reconstruction module that includes a scale coding network, a scale-aware sub-network, and a residual multilayer perceptron, distortion of reconstructed texture can be avoided. The receptive field range of the scale factor-related texture region can be adaptively adjusted according to any scale factor, thereby improving the accuracy of pixel values ​​in high-resolution images.

[0133] Further reference Figure 7 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a super-resolution image generation apparatus. This apparatus embodiment is similar to... Figure 2 The method embodiments shown correspond to those described.

[0134] like Figure 7 As shown, the super-resolution image generation apparatus 700 of this embodiment may include: an acquisition module 710, an extraction module 720, a reconstruction module 730, and a generation module 740.

[0135] The acquisition module 710 is configured to acquire an image generation model for generating a super-resolution image in response to acquiring a low-resolution image and a scaling factor. The image generation model includes a feature extraction module for extracting high-frequency features and a feature reconstruction module for reconstructing high-frequency features.

[0136] The extraction module 720 is configured to extract high-frequency image features corresponding to the low-resolution image based on the low-resolution image, the scaling factor, and the feature extraction module.

[0137] The reconstruction module 730 is configured to reconstruct the pixel values ​​of the high-resolution image corresponding to the low-resolution image based on high-frequency image features, scaling factors, and feature reconstruction modules.

[0138] The generation module 740 is configured to generate a high-resolution image corresponding to a low-resolution image based on the pixel values ​​of the high-resolution image.

[0139] In some optional implementations of this embodiment, the feature extraction module includes a feature extraction network and a scale-aware network; and the extraction module is further configured to: extract image features corresponding to the low-resolution image based on the low-resolution image and the feature extraction network; and generate high-frequency image features corresponding to the low-resolution image based on the image features, the scale factor and the scale-aware network.

[0140] In some optional implementations of this embodiment, the image generation model includes multiple feature extraction modules based on a hierarchical architecture. Each feature extraction module receives the output of the previous layer feature extraction module and uses the output as the input of the next layer feature extraction module. The extraction module is further configured to: input a low-resolution image into the feature extraction network of the first feature extraction module to extract image features corresponding to the low-resolution image; input the image features and a scaling factor into the scale-aware network of the first feature extraction module to generate initial high-frequency image features corresponding to the low-resolution image; input the initial high-frequency image features and the scaling factor into the second feature extraction module, until the last feature extraction module outputs the high-frequency image features corresponding to the low-resolution image.

[0141] In some optional implementations of this embodiment, the scale-aware network includes a scale-aware sub-network and a frequency equalization sub-network; and the extraction module is further configured to: generate initial perceived image features corresponding to the low-resolution image based on image features, scale factor and scale-aware sub-network; and generate high-frequency image features corresponding to the low-resolution image based on the initial perceived image features and frequency equalization sub-network.

[0142] In some optional implementations of this embodiment, the scale-aware subnetwork includes fully connected layers and dynamically convolutional layers; and the extraction module is further configured to: input a scaling factor into the fully connected layer to generate dynamically convolutional weights for convolutional processing of image features; and input the dynamically convolutional weights and image features into the dynamically convolutional layer to generate initial perceptual image features corresponding to the low-resolution image.

[0143] In some optional implementations of this embodiment, the frequency equalization subnetwork includes a spectral channel layer and a spatial scaling layer; and the extraction module is further configured to: input the initial perceived image features into the spectral channel layer to generate frequency domain attention weights corresponding to the initial perceived image features; input the initial perceived image features into the spatial scaling layer to generate spatial attention weights corresponding to the initial perceived image features; obtain high-frequency attention weights corresponding to the low-resolution image based on the frequency domain attention weights and the spatial attention weights; and generate high-frequency image features corresponding to the low-resolution image based on the initial perceived image features and the high-frequency attention weights.

[0144] In some optional implementations of this embodiment, the feature reconstruction module includes a scale coding network, a scale-aware sub-network, and a residual multilayer perceptron; and the reconstruction module is further configured to: obtain an encoded scale factor based on the scale factor and the scale coding network; obtain a perceived scale feature based on the scale factor, the encoded scale factor, and the scale-aware sub-network; and input the high-frequency image features, the encoded scale factor, and the perceived scale feature into the residual multilayer perceptron to reconstruct the pixel values ​​of the high-resolution image corresponding to the low-resolution image.

[0145] The super-resolution image generation apparatus provided in the above embodiments of this disclosure, wherein the executing entity first obtains an image generation model for generating a super-resolution image in response to acquiring a low-resolution image and a scaling factor, wherein the image generation model includes a feature extraction module for extracting high-frequency features and a feature reconstruction module for reconstructing high-frequency features, then extracts high-frequency image features corresponding to the low-resolution image based on the low-resolution image, the scaling factor and the feature extraction module, then reconstructs the pixel values ​​of the high-resolution image corresponding to the low-resolution image based on the high-frequency image features, the scaling factor and the feature reconstruction module, and finally generates a high-resolution image corresponding to the low-resolution image based on the pixel values ​​of the high-resolution image. It can generate high-resolution images of arbitrary scale based on low-resolution images and image generation models, realize super-resolution tasks of arbitrary scale, improve the accuracy and flexibility of super-resolution tasks, and can adaptively change the feature extraction capability according to different scaling factors, realizing the adaptive generation of super-resolution images of arbitrary scale without adjusting the network structure of the image generation model, thus realizing super-resolution images corresponding to arbitrary scaling factors.

[0146] Those skilled in the art will understand that the above-described apparatus also includes other well-known structures, such as processors and memories. To avoid unnecessarily obscuring the embodiments of this disclosure, these well-known structures are... Figure 7 Not shown in the image.

[0147] The following is for reference. Figure 8 The diagram illustrates a structural schematic of an electronic device 800 suitable for implementing embodiments of the present disclosure. Terminal devices in embodiments of the present disclosure may include, but are not limited to, mobile terminals such as smart screens, laptops, tablets, PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 8 The terminal device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0148] like Figure 8 As shown, the electronic device 800 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the electronic device 800. The processing device 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0149] Typically, the following devices can be connected to I / O interface 805: input devices 806 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 807 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 808 including, for example, magnetic tapes, hard disks, etc.; and communication devices 809. Communication device 809 allows electronic device 800 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 8 An electronic device 800 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 8 Each box shown can represent a device or multiple devices as needed.

[0150] Specifically, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by a processing device 801, it performs the functions defined in the methods of embodiments of this disclosure. It should be noted that the computer-readable medium of embodiments of this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0151] Computer program code for performing the operations of embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0152] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0153] The units described in the embodiments of this application can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor can be described as including an acquisition module, an extraction module, a reconstruction module, and a generation module, wherein the names of these modules do not necessarily limit the module itself.

[0154] In another aspect, this application also provides a computer-readable medium, which may be included in the aforementioned electronic device or may exist independently without being assembled into the electronic device. The computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: in response to acquiring a low-resolution image and a scaling factor, acquire an image generation model for generating a super-resolution image, wherein the image generation model includes a feature extraction module for extracting high-frequency features and a feature reconstruction module for reconstructing high-frequency features; extract high-frequency image features corresponding to the low-resolution image based on the low-resolution image, the scaling factor, and the feature extraction module; reconstruct the pixel values ​​of the high-resolution image corresponding to the low-resolution image based on the high-frequency image features, the scaling factor, and the feature reconstruction module; and generate a high-resolution image corresponding to the low-resolution image based on the pixel values ​​of the high-resolution image.

[0155] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A method for generating super-resolution images, the method comprising: In response to obtaining a low-resolution image and a scaling factor, an image generation model for generating a super-resolution image is obtained, wherein the image generation model includes a feature extraction module for extracting high-frequency features and a feature reconstruction module for reconstructing high-frequency features; the scaling factor represents the resolution ratio between the low-resolution image and the desired high-resolution image. Based on the low-resolution image, the scaling factor, and the feature extraction module, high-frequency image features corresponding to the low-resolution image are extracted, including: generating initial perceptual image features corresponding to the low-resolution image based on the image features corresponding to the low-resolution image, the scaling factor, and the scale-aware subnetwork of the scale-aware network in the feature extraction module; and generating high-frequency image features corresponding to the low-resolution image based on the initial perceptual image features and the frequency equalization subnetwork of the scale-aware network. Based on the high-frequency image features, the scaling factor, and the feature reconstruction module, the pixel values ​​of the high-resolution image corresponding to the low-resolution image are reconstructed. Based on the pixel values ​​of the high-resolution image, a high-resolution image corresponding to the low-resolution image is generated; The frequency equalization subnetwork includes a spectrum channel layer and a spatial scaling layer; and, The generation of high-frequency image features corresponding to the low-resolution image based on the initial perceptual image features and the frequency equalization subnetwork of the scale-aware network includes: The initial perceived image features are input into the spectral channel layer to generate frequency domain attention weights corresponding to the initial perceived image features; The initial perceived image features are input into the spatial scaling layer to generate spatial attention weights corresponding to the initial perceived image features; Based on the frequency domain attention weight and the spatial attention weight, the high-frequency attention weight corresponding to the low-resolution image is obtained; Based on the initial perceived image features and the high-frequency attention weights, high-frequency image features corresponding to the low-resolution image are generated.

2. The method of claim 1, wherein, The feature extraction module also includes a feature extraction network; The method further includes: Based on the low-resolution image and the feature extraction network, image features corresponding to the low-resolution image are extracted.

3. The method of claim 2, wherein, The image generation model includes multiple feature extraction modules based on a hierarchical architecture. Each feature extraction module receives the output of the feature extraction module in the previous layer and uses the output as the input of the feature extraction module in the next layer. as well as, The step of extracting high-frequency image features corresponding to the low-resolution image based on the low-resolution image, the scaling factor, and the feature extraction module includes: The low-resolution image is input into the feature extraction network in the first feature extraction module to extract the image features corresponding to the low-resolution image; The image features and the scaling factor are input into the scale-aware network in the first feature extraction module to generate the initial high-frequency image features corresponding to the low-resolution image. The initial high-frequency image features and the scaling factor are input into the second feature extraction module until the last feature extraction module outputs the high-frequency image features corresponding to the low-resolution image.

4. The method of claim 1, wherein, The scale-aware subnetwork includes fully connected layers and dynamically convolutional layers; and, The process of generating initial perceptual image features corresponding to the low-resolution image based on the image features corresponding to the low-resolution image, the scaling factor, and the scale-aware sub-network of the scale-aware network in the feature extraction module includes: The scaling factor is input into the fully connected layer to generate dynamic convolution weights for convolutional processing of the image features; The dynamic convolution weights and the image features are input into the dynamic convolution layer to generate the initial perceptual image features corresponding to the low-resolution image.

5. The method according to claim 1, wherein, The feature reconstruction module includes a scale coding network, a scale-aware sub-network, and a residual multilayer perceptron. as well as, The step of reconstructing the pixel values ​​of the high-resolution image corresponding to the low-resolution image based on the high-frequency image features, the scaling factor, and the feature reconstruction module includes: Based on the scaling factor and the scaling network, the encoded scaling factor is obtained; Based on the scaling factor, the encoded scaling factor, and the scale-aware subnetwork, the perceptual scale features are obtained. The high-frequency image features, the encoded scaling factor, and the perceptual scale features are input into the residual multilayer perceptron to reconstruct the pixel values ​​of the high-resolution image corresponding to the low-resolution image.

6. A super-resolution image generation apparatus, the apparatus comprising: The acquisition module is configured to acquire an image generation model for generating a super-resolution image in response to acquiring a low-resolution image and a scaling factor, wherein the image generation model includes a feature extraction module for extracting high-frequency features and a feature reconstruction module for reconstructing high-frequency features; the scaling factor represents the resolution ratio between the low-resolution image and the desired high-resolution image. The extraction module is configured to extract high-frequency image features corresponding to the low-resolution image based on the low-resolution image, the scaling factor, and the feature extraction module. This includes: generating initial perceptual image features corresponding to the low-resolution image based on the image features corresponding to the low-resolution image, the scaling factor, and the scale-aware sub-network of the scale-aware network in the feature extraction module; and generating high-frequency image features corresponding to the low-resolution image based on the initial perceptual image features and the frequency equalization sub-network of the scale-aware network. The reconstruction module is configured to reconstruct the pixel values ​​of the high-resolution image corresponding to the low-resolution image based on the high-frequency image features, the scaling factor, and the feature reconstruction module. The generation module is configured to generate a high-resolution image corresponding to the low-resolution image based on the pixel values ​​of the high-resolution image. The frequency equalization subnetwork includes a spectrum channel layer and a spatial scaling layer; and the extraction module is further configured to: The initial perceived image features are input into the spectral channel layer to generate frequency domain attention weights corresponding to the initial perceived image features; The initial perceived image features are input into the spatial scaling layer to generate spatial attention weights corresponding to the initial perceived image features; Based on the frequency domain attention weight and the spatial attention weight, the high-frequency attention weight corresponding to the low-resolution image is obtained; Based on the initial perceived image features and the high-frequency attention weights, high-frequency image features corresponding to the low-resolution image are generated.

7. The apparatus according to claim 6, wherein, The feature extraction module further includes a feature extraction network; and the extraction module is further configured to: Based on the low-resolution image and the feature extraction network, image features corresponding to the low-resolution image are extracted.

8. The apparatus according to claim 7, wherein, The image generation model includes multiple feature extraction modules based on a hierarchical architecture. Each feature extraction module receives the output of the previous layer's feature extraction module and uses the output as the input of the next layer's feature extraction module. Furthermore, the extraction modules are configured to: The low-resolution image is input into the feature extraction network in the first feature extraction module to extract the image features corresponding to the low-resolution image; The image features and the scaling factor are input into the scale-aware network in the first feature extraction module to generate the initial high-frequency image features corresponding to the low-resolution image. The initial high-frequency image features and the scaling factor are input into the second feature extraction module until the last feature extraction module outputs the high-frequency image features corresponding to the low-resolution image.

9. The apparatus according to claim 6, wherein, The scale-aware subnetwork includes fully connected layers and dynamically convolutional layers; and the extraction module is further configured to: The scaling factor is input into the fully connected layer to generate dynamic convolution weights for convolutional processing of the image features; The dynamic convolution weights and the image features are input into the dynamic convolution layer to generate the initial perceptual image features corresponding to the low-resolution image.

10. The apparatus according to claim 6, wherein, The feature reconstruction module includes a scale coding network, a scale-aware subnetwork, and a residual multilayer perceptron; and the reconstruction module is further configured to: Based on the scaling factor and the scaling network, the encoded scaling factor is obtained; Based on the scaling factor, the encoded scaling factor, and the scale-aware subnetwork, the perceptual scale features are obtained. The high-frequency image features, the encoded scaling factor, and the perceptual scale features are input into the residual multilayer perceptron to reconstruct the pixel values ​​of the high-resolution image corresponding to the low-resolution image.

11. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-5.

12. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method as described in any one of claims 1-5.

Images