Image super-division method and device, electronic equipment and storage medium

By acquiring the global and local features of the image in the image super-segment technology and fusion, the problem of global feature capture and calculation complexity in the prior art is solved, and efficient image super-segment effect and calculation cost savings are achieved. on different computing power equipment.

CN119090718APending Publication Date: 2024-12-06SHENZHEN SIYUAN ELECTRONICS TECH CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410879446.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-02
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The existing image super-scoring technology is difficult to take into account the capture and calculation complexity of image global features, resulting in limited image detail recovery ability.

Method used

By obtaining the initial feature map of the image to be processed and entering multiple feature extraction modules, each module divides the feature map into image blocks to obtain global and local features, and performs deconvolution operations to improve resolution after fusing. The feature extraction module The quantity can be varied to accommodate different computing power equipment.

Benefits of technology

It achieves richer feature expression and better image super-scoring effects, while saving computing costs, is suitable for different computing power devices, and improves the ability to restore image details.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119090718A_ABST
    Figure CN119090718A_ABST
Patent Text Reader

Abstract

The invention discloses an image super-division method and device, electronic equipment and a storage medium, and relates to the technical field of image super-division, and the method comprises the steps: obtaining an initial feature map of a to-be-processed image, and inputting the initial feature map into a plurality of feature extraction modules; dividing an input end feature map into a plurality of image blocks through each feature extraction module, scanning the image blocks along a plurality of scanning directions at a preset scanning interval to obtain a plurality of one-dimensional sequences corresponding to the scanning directions, and obtaining global features of the input end feature map based on the plurality of one-dimensional sequences; acquiring local features of the input end feature map; based on the global features and the local features, obtaining an output end feature map and outputting the output end feature map; and carrying out deconvolution operation on the output end feature map of the last feature extraction module to obtain a target image. According to the application, each feature extraction module fuses the global features and the local features of the input end feature map, so that the obtained output end feature map is richer in feature expression, and the image super-resolution effect is better.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of image super-resolution, and more specifically, to an image super-resolution method, device, electronic device and computer-readable storage medium. Background Art

[0002] Image super-resolution, also known as image super-resolution, is the process of converting a low-resolution image into a high-resolution image through an algorithm. While improving the image resolution, it restores and enriches the image texture details as much as possible to make the image clearer.

[0003] The current mainstream image super-resolution technology is mainly based on deep learning super-resolution reconstruction algorithms. However, due to the limitations of the existing algorithm structure, there is a problem that it is impossible to capture the global features of the image and have low computational complexity. Summary of the invention

[0004] The present application proposes an image super-resolution method, device, electronic device and storage medium to improve the above-mentioned defects.

[0005] In a first aspect, an embodiment of the present application provides an image super-resolution method, the method comprising: obtaining an initial feature map of an image to be processed, and inputting it into a plurality of feature extraction modules, wherein the input and output ends of each of the feature extraction modules are sequentially connected in series; dividing the input feature map into a plurality of image blocks by each of the feature extraction modules, scanning the image blocks along a plurality of scanning directions at a preset scanning interval, and obtaining a plurality of one-dimensional sequences corresponding to the scanning directions, wherein the one-dimensional sequences are used to characterize image features corresponding to the image blocks, and obtaining global features of the image to be processed based on the plurality of one-dimensional sequences; obtaining local features of the image to be processed based on the input feature map by each of the feature extraction modules; obtaining an output feature map based on the global and local features of the image to be processed by each of the feature extraction modules and outputting the map; performing a deconvolution operation on the output feature map of the last feature extraction module to obtain a target image, wherein the resolution of the target image is higher than the resolution of the image to be processed.

[0006] In the second aspect, an embodiment of the present application also provides an image super-resolution device, which includes: an initial feature acquisition module, which is used to acquire an initial feature map of an image to be processed, and input it into multiple feature extraction modules, which are sequentially connected in series through the input and output ends of each feature extraction module; a global feature extraction module, which is used to enable each feature extraction module to divide the input feature map into multiple image blocks, scan the image blocks along multiple scanning directions at a preset scanning interval, and obtain multiple one-dimensional sequences corresponding to the scanning directions, wherein the one-dimensional sequences are used to characterize the image features corresponding to the image blocks, and based on the multiple one-dimensional sequences, obtain the global features of the input feature map; a local feature extraction module, which is used to enable the feature extraction module to acquire the local features of the input feature map; a feature fusion module, which is used to enable the feature extraction module to obtain and output the output feature map based on the global features and local features of the input feature map; an image super-resolution module, which is used to perform a deconvolution operation on the output feature map of the last feature extraction module to obtain a target image, wherein the resolution of the target image is higher than the resolution of the image to be processed.

[0007] In a third aspect, an embodiment of the present application also provides an electronic device, comprising: one or more processors; a memory; one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to execute the above method.

[0008] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a program code executable by a processor, and when the program code is executed by the processor, the processor executes the above method.

[0009] Therefore, the present application provides an image super-resolution method, device, electronic device and storage medium, the method comprising: obtaining an initial feature map of an image to be processed, and inputting it into a plurality of feature extraction modules, wherein the input end and the output end of each of the feature extraction modules are sequentially connected in series; dividing the input end feature map into a plurality of image blocks through each of the feature extraction modules, scanning the image blocks along a plurality of scanning directions at a preset scanning interval, obtaining a plurality of one-dimensional sequences corresponding to the scanning directions, and obtaining global features of the input end feature map based on the plurality of one-dimensional sequences, wherein the one-dimensional sequences are used to characterize image features corresponding to the image blocks; obtaining local features of the input end feature map through each of the feature extraction modules; obtaining an output end feature map based on the global features and local features of the input end feature map through each of the feature extraction modules and outputting the same; performing a deconvolution operation on the output end feature map of the last feature extraction module to obtain a target image, wherein the resolution of the target image is higher than the resolution of the image to be processed. The present application enables each feature extraction module to fuse the global features and local features of the input feature map, so that the feature expression of the output feature map is richer and the image super-resolution effect is better. While maintaining the global image modeling capability, the computational cost of the state-space model is saved. Furthermore, the number of feature extraction modules is set to be variable, so that the method can be applied to devices with different computing powers and has a wider range of applications.

[0010] Other features and advantages of the embodiments of the present application will be described in the subsequent description, and partly become apparent from the description, or can be understood by practicing the embodiments of the present application. The purposes and other advantages of the embodiments of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0012] Figure 1 A method flow chart of an image super-resolution method proposed in an embodiment of the present application is shown.

[0013] Figure 2 A structural block diagram of a feature extraction module provided in an embodiment of the present application is shown.

[0014] Figure 3 A structural block diagram of an image super-resolution system provided in an embodiment of the present application is shown.

[0015] Figure 4 A method flow chart of another image super-resolution method proposed in an embodiment of the present application is shown.

[0016] Figure 5 A method flow chart of another image super-resolution method proposed in an embodiment of the present application is shown.

[0017] Figure 6 A structural block diagram of another feature extraction module provided in an embodiment of the present application is shown.

[0018] Figure 7 A structural block diagram of an image super-resolution device proposed in an embodiment of the present application is shown.

[0019] Figure 8 A structural block diagram of an electronic device proposed in an embodiment of the present application is shown.

[0020] Fig. 9 A structural block diagram of a computer-readable storage medium provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0021] In order to make those skilled in the art better understand the present application scheme, the technical scheme in the present application embodiment will be clearly and completely described below in conjunction with the drawings in the present application embodiment. Obviously, the described embodiment is only a part of the present application embodiment, rather than all the embodiments. The components of the present application embodiment usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the present application for protection, but merely represents the selected embodiment of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0022] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0023] At present, there are two main algorithms used in mainstream image super-resolution technology. One is the convolution-based generative adversarial network algorithm, which mainly includes a generator and a discriminator. The generator is responsible for generating high-resolution images, and the discriminator is responsible for guiding the generated images to be as close to the real images as possible. Although the algorithm has greatly enhanced the image generation effect, the training stability of the entire algorithm is difficult to guarantee. In addition, since the algorithm structure is based on a convolutional neural network, it is difficult to capture the global features of the image and is prone to losing long-distance details of the image, resulting in limited ability to restore image details. The other algorithm is an image super-resolution algorithm based on the Transformer deep learning model, which has the advantages of global image modeling, parallel computing, scalability and end-to-end training. It has achieved good image generation effects in academic research, but this algorithm has high computational complexity and high requirements on the computing power of the device, which makes it difficult to implement and apply to real life.

[0024] Therefore, this application proposes an image super-resolution method, device, electronic device and storage medium to improve the above defects. Figure 1 , Figure 1 A method flow chart of an image super-resolution method proposed in an embodiment of the present application is shown, and the method comprises the following steps:

[0025] S110: Acquire an initial feature map of the image to be processed, and input it into a plurality of feature extraction modules, wherein the input end and the output end of each feature extraction module are sequentially connected in series.

[0026] Specifically, the image to be processed has a low resolution. For example, the resolution of the image to be processed may be a common low resolution or medium resolution such as 640×480 pixels, 1920x1080 pixels, etc. Specifically, the initial feature map of the image to be processed may be understood as a two-dimensional image containing feature data, wherein the feature data may be expressed in a two-dimensional array, and further, each feature value represents the feature intensity of a specific area in the image to be processed. Further, the initial feature map may be used to reflect low-dimensional features such as edges and corners in the image to be processed.

[0027] As an implementation method, a method for obtaining an initial feature map of an image to be processed may be: obtaining the initial feature map by performing a convolution operation on the image to be processed. Exemplarily, the image to be processed is input into a convolution model, and the convolution model performs a convolution operation on the image to be processed. Specifically, each convolution kernel in the convolution model slides on the image to be processed and performs a dot product operation on each specific area, thereby outputting an initial feature map. It can be understood that each eigenvalue in the initial feature map is obtained based on each small area in the image to be processed, that is, the initial feature map can only be used to reflect the local features of the image to be processed, which has certain limitations.

[0028] As an implementation method, each feature extraction module includes an input end and an output end, and the feature extraction module is used to obtain an input feature map based on the input of the feature extraction module, obtain global features and local features of the input feature map, and fuse the global features and local features of the input feature map to obtain a fused new feature map as the output feature map of the feature extraction module, and output it from the output end. It can be obtained that, since the input end and the output end of each feature extraction module are connected in series in sequence, when there are n (n is a positive integer greater than or equal to 2) feature extraction modules, for the first feature extraction module, the input feature map input by its input end is the initial feature map of the image to be processed, for the i-th (i is a positive integer not equal to 1 and less than or equal to n) feature extraction module, its input end is connected to the output end of the i-1th feature extraction module, for re-extracting features from the output feature map of the i-1th feature extraction module, and its output end is connected to the input end of the i+1th feature extraction module, for inputting the new output feature map obtained after feature extraction into the i+1th feature extraction module.

[0029] It should be understood that in a deep learning structure, the deeper the network structure, the more high-dimensional features and abstract concepts the algorithm can extract. In other words, the more feature extraction modules are set, the more high-dimensional features can be extracted from the image. For example, the feature extraction modules with a higher ranking can extract low-dimensional features such as color, texture, and shape, while the feature extraction modules with a lower ranking can extract high-dimensional features such as semantic features (for example, whether the content in a certain area of ​​the image is a person or a car) and contextual features (for example, the type of scene in a certain area of ​​the image). Therefore, the output feature map output by the last feature extraction module can not only reflect the high-dimensional features of the image, but also because its feature elements are based on the fusion of the global features and local features of the image to be processed, the feature elements have a global correlation effect, and the feature map has global consistency.

[0030] S120: Divide the input-end feature map into multiple image blocks through each of the feature extraction modules, scan the image blocks along multiple scanning directions at a preset scanning interval to obtain multiple one-dimensional sequences corresponding to the scanning directions, and obtain global features of the input-end feature map based on the multiple one-dimensional sequences, where the one-dimensional sequences are used to characterize image features corresponding to the image blocks.

[0031] As an implementation method, an image patch refers to a sub-region in an input feature map, which is a square or rectangle with a predefined fixed size, and image features are extracted by processing multiple image patches. For example, when dividing a feature map of 1024x1024 pixels, the size of each image patch can be set to 256x256 pixels. It is understandable that when dividing the image patches, the pixel size of each image patch can be defined as needed, and there is no limitation here.

[0032] In this embodiment, a feature extraction module scans multiple image blocks along multiple scanning directions at a preset scanning interval to obtain multiple one-dimensional sequences corresponding to different scanning directions, thereby obtaining global features of the input feature map based on the multiple one-dimensional sequences.

[0033] As an implementation method, the preset scanning interval refers to the number of scanning steps used when extracting image block features, which can be set according to the different computing power of the device or the different requirements for image quality. For example, if the preset scanning interval is 2, then scanning multiple image blocks along the scanning direction at the preset scanning interval can refer to extracting image block features once every other image block along the scanning direction, and arranging the extracted image block features in a certain order to form a one-dimensional sequence. It can be understood that the smaller the preset scanning interval is set, the more image blocks need to be processed, the higher the computing power requirements for the device, the more complete the extracted image features, and the better the high-resolution image effect obtained after super-resolution. Therefore, if the computing power of the device is low, the preset scanning interval can be set to be relatively large, otherwise the preset scanning interval can be set to be relatively small. If the image quality requirements are high, the preset scanning interval can be set to be relatively small, otherwise the preset scanning interval can be set to be relatively large, and so on. Therefore, by setting the preset scanning interval, the number of image blocks that need to be scanned in the process of obtaining global features is reduced, while maintaining the global modeling capability, the computing cost is saved.

[0034] As an implementation manner, the scanning direction may include four directions: a scanning direction from the upper left to the lower right, a scanning direction from the lower right to the upper left, a scanning direction from the upper right to the lower left, and a scanning direction from the lower left to the upper right. Exemplarily, taking the scanning direction from the upper left to the lower right as an example, it means first scanning from left to right at a preset scanning interval, and then scanning from top to bottom at a preset scanning interval. Specifically, the scanning process may be: scanning the image block along the scanning direction from the upper left to the lower right at a preset scanning interval to obtain a one-dimensional sequence corresponding to the scanning direction from the upper left to the lower right; scanning the image block along the scanning direction from the lower right to the upper left at a preset scanning interval to obtain a one-dimensional sequence corresponding to the scanning direction from the lower right to the upper left; scanning the image block along the scanning direction from the upper right to the lower left at a preset scanning interval to obtain a one-dimensional sequence corresponding to the scanning direction from the upper right to the lower left; scanning the image block along the scanning direction from the lower left to the upper right at a preset scanning interval to obtain a one-dimensional sequence corresponding to the scanning direction from the lower left to the upper right.

[0035] As an implementation method, a one-dimensional sequence refers to an ordered set composed of a series of features, which is used to characterize the image features corresponding to the image block, that is, the elements contained in the one-dimensional sequence can correspond to the image features of each scanned image block. Therefore, after obtaining multiple one-dimensional sequences corresponding to multiple scanning directions, the multiple one-dimensional sequences can be fused to obtain a fused one-dimensional sequence, so as to obtain the global features of the image to be processed based on the fused one-dimensional sequence. Then, the global correlation of the features contained in the global features is better, and to a certain extent, it makes up for the problem of missing features caused by the scanning interval during the scanning process. Furthermore, the features in the one-dimensional sequence are arranged in a certain order, and each feature has its corresponding position. For example, the features can be arranged according to the image block corresponding to the feature, so that the features in different one-dimensional sequences can correspond one-to-one with the image block.

[0036] S130: Acquire local features of the input feature map through each of the feature extraction modules.

[0037] As an implementation method, the method of obtaining the local features of the input feature map through the feature extraction module can be: through the deep separable convolution (DWConv) algorithm, the scale-invariant feature transform (SIFT) algorithm, the speeded-up robust features (SURF) algorithm, etc., the input feature map of the input feature extraction module is further subjected to local feature extraction. Further, the local feature extraction algorithm used by each feature extraction module is generally the same.

[0038] As an implementation method, after the feature extraction module uses the DWConv algorithm to perform local image modeling on the input feature map, the result is input into the activation function Si LU (Significant-weighted Line Unit) for processing, and finally the local features of the input feature map are obtained.

[0039] S140: Obtaining an output-end feature map based on the global features and local features of the input-end feature map through each of the feature extraction modules and outputting the output-end feature map.

[0040] As an implementation method, the process of each feature extraction module acquiring global features and local features based on the input feature map is processed in parallel, that is, each feature extraction module can simultaneously execute the operations of acquiring global features and acquiring local features. Thus, after the global features and local features of the input feature map are acquired through the feature extraction module, the obtained global features and local features are fused. Exemplarily, the feature extraction module can realize the fusion of global features and local features by using a feature interaction module or a selective fusion module, thereby obtaining a new feature map as the output feature map.

[0041] In the present embodiment, it can be understood that by fusing the local features of the input feature map with the global features, the problem of feature loss caused by block jumping in the global features obtained due to the presence of a preset scanning interval in the process of obtaining the global features is solved. This is because the image is traversed when obtaining the local features, and the two are fused in this step to obtain the output feature map, which combines the overall information and local details of the image and has a more comprehensive and rich feature expression capability.

[0042] As an implementation method, the method of obtaining the output feature map based on the global features and local features of the input feature map by each feature extraction module can also be: after obtaining the global features and local features of the image to be processed, the global features are multiplied by the local features of the input feature map, so as to achieve the fusion of the global features and the local features and obtain the output feature map. For example, please refer to Figure 2 , Figure 2 A structural block diagram of a feature extraction module provided in an embodiment of the present application is shown. Figure 2 As shown, the feature extraction module 200 includes an input terminal 201 and an output terminal 202. After the feature extraction module 200 receives the input terminal feature map input by the input terminal 201, it extracts the corresponding global features and local features, and finally fuses them through a dot multiplication operation to obtain an output terminal feature map, and outputs it through the output terminal 202.

[0043] S150: performing a deconvolution operation on the output end feature map of the last feature extraction module to obtain a target image, wherein the resolution of the target image is higher than the resolution of the image to be processed.

[0044] Specifically, the deconvolution operation is used to achieve image super-resolution. The deconvolution operation can increase the resolution of the feature map to the resolution of the image that needs to be super-resolved. The convolution kernel is applied in a reverse manner so that the output size is larger than the input size. The process usually involves zero padding around the input data, and then using the conventional convolution operation. Furthermore, the convolution kernel weights of the deconvolution are learnable, which means that the network can automatically adjust the convolution kernel weights according to the training data to optimize the upsampling effect. Therefore, after the deconvolution operation is performed on the output feature map output from the output end of the last feature extraction module, a target image with a higher resolution than the image to be processed can be obtained.

[0045] For further information, see Figure 3 , Figure 3 FIG. 4 shows a structural block diagram of an image super-resolution system provided in an embodiment of the present application. Figure 2 As shown, the image super-resolution system 300 includes a convolution module 301, a deconvolution module 302 and a plurality of feature extraction modules 303, and the convolution module 301 and the plurality of feature extraction modules 303 (for ease of understanding, Figure 3 303) and a deconvolution module 302 are connected in sequence. Specifically, when the image to be processed is input into the image super-resolution system 300, the image to be processed generates an initial feature map through the convolution module 301. After the initial feature map is input into multiple feature extraction modules 303, multiple feature extraction modules 303 perform feature extraction operations, and the last feature extraction module 303 outputs the result to the deconvolution module 302, so that the output end feature map of the last feature extraction module 303 is subjected to the image super-resolution operation by the deconvolution module 302 to obtain the target image. Therefore, since the global features and local features of the image are integrated when each feature extraction module performs the feature extraction operation on the corresponding input end feature map, the feature expression of the output end feature map of each feature extraction module is richer and has good global consistency. As the number of feature extraction modules increases, the features contained in the output end feature map of the last feature extraction module have higher dimensions and abstract concepts, which improves the effect of the target image obtained after the image super-resolution.

[0046] Therefore, an image super-resolution method provided by an embodiment of the present application includes: obtaining an initial feature map of an image to be processed, and inputting it into multiple feature extraction modules, wherein the input end and the output end of each feature extraction module are sequentially connected in series; dividing the input end feature map into multiple image blocks through each feature extraction module, scanning the image blocks along multiple scanning directions at a preset scanning interval, obtaining multiple one-dimensional sequences corresponding to the scanning directions, and obtaining global features of the input end feature map based on the multiple one-dimensional sequences, wherein the one-dimensional sequence is used to characterize image features corresponding to the image blocks; obtaining local features of the input end feature map through each feature extraction module; obtaining an output end feature map based on the global features and local features of the input end feature map through each feature extraction module and outputting it; performing a deconvolution operation on the output end feature map of the last feature extraction module to obtain a target image, wherein the resolution of the target image is higher than the resolution of the image to be processed. The present application enables each feature extraction module to fuse the global features and local features of the input feature map, so that the feature expression of the output feature map is richer and the image super-resolution effect is better. In addition, the number of feature extraction modules is set to be variable, so that the method can be applied to devices with different computing powers and has a wider range of applications.

[0047] For example, if the computing power is low on edge devices or mobile phones, using 1-3 feature extraction modules is better than image super-resolution based on traditional convolutional neural networks, and the required calculation amount is lower. If 9 feature extraction blocks can be set on high-computing devices such as NVID IA GeForce RTX 4090, the calculation complexity is much lower than that of the algorithm based on the transformer structure, and the image super-resolution effect is better, especially in restoring image details.

[0048] See also Figure 4 , Figure 4 A method flow chart of another image super-resolution method proposed in an embodiment of the present application is shown. Specifically, the method comprises the following steps:

[0049] S210: Acquire an initial feature map of the image to be processed, and input it into a plurality of feature extraction modules, wherein the input end and the output end of each feature extraction module are sequentially connected in series.

[0050] S220: Divide the input feature map into multiple image blocks through each of the feature extraction modules, scan the image blocks along multiple scanning directions at a preset scanning interval, and obtain multiple one-dimensional sequences corresponding to the scanning directions, wherein the one-dimensional sequences are used to characterize image features corresponding to the image blocks.

[0051] The specific implementation of steps S210 - S220 may refer to some specific implementations of steps S110 - S120 in the above embodiment, which will not be described in detail here.

[0052] S230: Encode the one-dimensional sequence to obtain a corresponding one-dimensional feature vector, fuse elements of multiple one-dimensional feature vectors to obtain a fused feature vector, and reconstruct a global feature corresponding to the input feature map based on the fused feature vector and the positional relationship between the elements in the fused feature vector and the image block.

[0053] As an implementation method, each one-dimensional sequence can be encoded through linear transformation (Li near), nonlinear activation function Si LU operation and layer normalization (Layer Normalization, LN) operation, and the output result is used as a one-dimensional feature vector corresponding to each one-dimensional sequence.

[0054] As an implementation method, multiple one-dimensional feature vectors can be feature fused according to the strategy of adding elements at corresponding positions. According to the description of the one-dimensional sequence in the above embodiment, the positions of elements in different one-dimensional feature vectors can correspond one-to-one to the image blocks of the input feature map. Therefore, by adding the elements at corresponding positions in different one-dimensional feature vectors, the feature fusion corresponding to the same image block in the one-dimensional feature vectors (one-dimensional sequence) obtained along different scanning directions can be realized, thereby obtaining a fused feature vector. In other words, the image features obtained in multiple scanning directions are fused in the fused feature vector. Furthermore, element fusion can be realized by element-wise addition.

[0055] As an implementation method, based on the above analysis, it can be seen that the element positions in the fused feature vector correspond one-to-one to the image blocks of the input feature map. Therefore, based on the fused feature vector and the positional relationship between the elements in the fused feature vector and the image blocks, a new global feature can be constructed by rearranging the original positions.

[0056] Therefore, in this embodiment, each feature element in the global features obtained after fusion and reconstruction of multiple one-dimensional sequences integrates the information of feature elements at other positions from different directions, and the features at each position are associated with the previous and next features, that is, reflecting the global receptive field of the feature map at the input end.

[0057] S240: Acquire a feature missing region in the global feature, complete the missing feature values ​​in the feature missing region by interpolation, and update the global feature after the feature missing region is completed as a new global feature.

[0058] Since the image is scanned using a preset scanning interval in the process of acquiring global features, there may be a problem of missing features during the scanning process, that is, there may be a region with missing features in the global features finally obtained. In this embodiment, if there is a region with missing features in the global features, the missing feature values ​​of the region with missing features can be supplemented by interpolation. For example, based on the features in the region surrounding the region with missing features, interpolation functions such as spline interpolation or polynomial interpolation can be used to interpolate to obtain feature values ​​corresponding to the region with missing features, and the region with missing features can be supplemented, thereby updating the global features after the region with missing features is supplemented to new global features, and using the global features as the global features of the input feature map.

[0059] S250: Acquire local features of the input feature map through each of the feature extraction modules.

[0060] S260: Obtain an output-end feature map based on the global features and local features of the input-end feature map through each of the feature extraction modules, and output the output-end feature map.

[0061] S270: performing a deconvolution operation on the output end feature map of the last feature extraction module to obtain a target image, wherein the resolution of the target image is higher than the resolution of the image to be processed.

[0062] The specific implementation of steps S240-S270 may refer to the specific implementation of steps S130-S150 in the above embodiment, which will not be repeated here.

[0063] Therefore, an image super-resolution method provided in an embodiment of the present application, based on the aforementioned embodiment, obtains a fused feature vector by element fusion of multiple one-dimensional sequences, and reconstructs a global feature corresponding to the input feature map based on the fused feature vector, so that each feature element in the global feature integrates information of feature elements at other positions from different directions, and the feature at each position is associated with the previous and next features; at the same time, if there is a feature missing area in the global feature, the missing feature value of the feature missing area can be supplemented by interpolation, so that the image features reflected by the final global feature are more complete and rich.

[0064] See also Figure 5 , Figure 5 A method flow chart of another image super-resolution method proposed in an embodiment of the present application is shown. Specifically, the method includes the following steps:

[0065] S310: Acquire an initial feature map of the image to be processed, and input it into a plurality of feature extraction modules, wherein the input end and the output end of each feature extraction module are sequentially connected in series.

[0066] S320: Divide the input-end feature map into multiple image blocks through each of the feature extraction modules, scan the image blocks along multiple scanning directions at a preset scanning interval to obtain multiple one-dimensional sequences corresponding to the scanning directions, and obtain global features of the input-end feature map based on the multiple one-dimensional sequences, where the one-dimensional sequences are used to characterize image features corresponding to the image blocks.

[0067] S330: Obtain local features of the input feature map through each of the feature extraction modules.

[0068] S340: Obtain an output feature map based on the global features and local features of the input feature map through each of the feature extraction modules.

[0069] The specific implementation of steps S310 - S340 may refer to the specific implementation of steps S110 - S140 in the above embodiment, which will not be described in detail here.

[0070] S350: Add the input-end feature map to the output-end feature map through each of the feature extraction modules, update the output-end feature map output by the module to the addition result, and output the output-end feature map.

[0071] In this embodiment, by adding a residual structure to each feature extraction module, that is, not directly outputting the output feature map of the feature extraction module, but adding its output to the original input (or the input after some transformation), and then outputting it, that is, each feature extraction module adds the corresponding input feature map to its original output feature map to obtain the addition result, and outputs the addition result as a new output feature map. By adding a residual structure to each feature extraction module, the gradient vanishing problem caused by the deep network structure due to the setting of multiple feature extraction modules in the method of the present application can be avoided, and possible degradation phenomena can be avoided, and the expression ability of the model can also be improved.

[0072] For further information, see Figure 6 , Figure 6 FIG. 2 shows a structural block diagram of another feature extraction module provided in an embodiment of the present application. Figure 6 As shown, the feature extraction module 400 includes an input terminal 401 and an output terminal 402. After the feature extraction module 400 receives the input feature map input by the input terminal 401, it extracts the corresponding global features and local features, and finally fuses them through a dot multiplication operation to obtain an output feature map. Then, after adding the output feature map to the input feature map again, the addition result is used as a new output feature map and outputted through the output terminal 402.

[0073] S360: performing a deconvolution operation on the output end feature map of the last feature extraction module to obtain a target image, wherein the resolution of the target image is higher than the resolution of the image to be processed.

[0074] The specific implementation of step S360 may refer to the specific implementation of step S150 in the above embodiment, which will not be described in detail here.

[0075] Therefore, an image super-resolution method provided in an embodiment of the present application, based on the aforementioned embodiment, enables each feature extraction module to add the corresponding input feature map with its original output feature map to obtain an addition result, and outputs the addition result as a new output feature map, thereby avoiding the gradient vanishing problem caused by a deeper network structure due to the setting of multiple feature extraction modules in the method of the present application, avoiding possible degradation, and also improving the expressiveness of the model.

[0076] See also Figure 7 , Figure 7 The structure block diagram of an image super-resolution device proposed in an embodiment of the present application is shown. Specifically, the device 500 includes:

[0077] The initial feature acquisition module 501 is used to acquire the initial feature map of the image to be processed and input it into multiple feature extraction modules, and the input end and output end of each feature extraction module are connected in series in sequence.

[0078] The global feature extraction module 502 is used to divide the input-end feature map into multiple image blocks through each of the feature extraction modules, scan the image blocks along multiple scanning directions at a preset scanning interval, obtain multiple one-dimensional sequences corresponding to the scanning directions, and obtain the global features of the input-end feature map based on the multiple one-dimensional sequences, wherein the one-dimensional sequences are used to characterize the image features corresponding to the image blocks.

[0079] The local feature extraction module 503 is used to obtain the local features of the input feature map through each of the feature extraction modules.

[0080] The feature fusion module 504 is used to obtain and output the output feature map based on the global features and local features of the input feature map through each of the feature extraction modules.

[0081] The image super-resolution module 505 is used to perform a deconvolution operation on the output feature map of the last feature extraction module to obtain a target image, wherein the resolution of the target image is higher than the resolution of the image to be processed.

[0082] Optionally, the global feature extraction module 502 is also used to encode the one-dimensional sequence to obtain a corresponding one-dimensional feature vector; fuse elements of multiple one-dimensional feature vectors to obtain a fused feature vector; and reconstruct the global features corresponding to the input feature map based on the fused feature vector and the positional relationship between the elements in the fused feature vector and the image block.

[0083] Optionally, the global feature extraction module 502 is also used to obtain a feature missing area in the global feature after reconstructing the global feature corresponding to the input feature map based on the fused feature vector and the positional relationship between the elements in the fused feature vector and the image block; to complete the missing feature values ​​in the feature missing area by interpolation, and to update the global feature after the feature missing area is completed to a new global feature.

[0084] Optionally, the global feature extraction module 502 is further used to scan the image block along the scanning direction from upper left to lower right at a preset scanning interval to obtain a one-dimensional sequence corresponding to the scanning direction from upper left to lower right; scan the image block along the scanning direction from lower right to upper left at a preset scanning interval to obtain a one-dimensional sequence corresponding to the scanning direction from lower right to upper left; scan the image block along the scanning direction from upper right to lower left at a preset scanning interval to obtain a one-dimensional sequence corresponding to the scanning direction from upper right to lower left; scan the image block along the scanning direction from lower left to upper right at a preset scanning interval to obtain a one-dimensional sequence corresponding to the scanning direction from lower left to upper right.

[0085] Optionally, the feature fusion module 504 is further configured to perform a dot multiplication of the global feature of the input feature map with the local feature of the input feature map to obtain the output feature map.

[0086] Optionally, the feature fusion module 504 is also used to enable each of the feature extraction modules to add the input feature map to the output feature map after each of the feature extraction modules obtains the output feature map based on the global features and local features of the input feature map, and to update the output feature map output by the module to the addition result.

[0087] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0088] In several embodiments provided in the present application, the coupling between modules may be electrical, mechanical or other forms of coupling.

[0089] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0090] Please refer to Figure 8 , which shows a block diagram of an electronic device 600 provided in an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The electronic device 600 in the present application may include one or more of the following components: a processor 610, a memory 620, and one or more application programs, wherein one or more application programs may be stored in the memory 620 and configured to be executed by one or more processors 610, and one or more programs are configured to execute the method described in the aforementioned method embodiment.

[0091] The processor 610 may include one or more processing cores. The processor 610 uses various interfaces and lines to connect various parts of the entire electronic device 600, and executes various functions of the electronic device 600 and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 620, and calling data stored in the memory 620. Optionally, the processor 610 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 610 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It is understandable that the above-mentioned modem may not be integrated into the processor 610, but may be implemented by a communication chip alone. The memory 620 may include a random access memory (RAM) or a read-only memory (ROM). The memory 620 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 620 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as a phone book, audio and video data, chat record data) created by the electronic device 600 during use.

[0092] Please refer to Fig. 9 , which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium 700 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0093] The computer readable storage medium 700 can be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 700 includes a non-transitory computer-readable storage medium. The computer readable storage medium 700 has a storage space for program code 710 that performs any method step in the above method. These program codes can be read from or written into one or more computer program products. The program code 710 can be compressed, for example, in an appropriate form.

[0094] According to one aspect of an embodiment of the present application, a computer program product is provided, the computer program product including computer-readable instructions, the computer-readable instructions being stored in a computer-readable storage medium. A processor of an electronic device reads the computer-readable instructions from the computer-readable storage medium, and the processor executes the computer-readable instructions, so that the electronic device executes the method in any of the above embodiments.

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An image super-resolution method, characterized in that: include: Obtaining an initial feature map of the image to be processed and inputting it into a plurality of feature extraction modules, wherein the input end and the output end of each feature extraction module are sequentially connected in series; Dividing the input-end feature map into a plurality of image blocks by each of the feature extraction modules, scanning the image blocks along a plurality of scanning directions at a preset scanning interval to obtain a plurality of one-dimensional sequences corresponding to the scanning directions, and acquiring a global feature of the input-end feature map based on the plurality of one-dimensional sequences, wherein the one-dimensional sequences are used to characterize image features corresponding to the image blocks; Acquire local features of the input feature map through each of the feature extraction modules; Obtaining an output end feature map based on the global features and local features of the input end feature map by each of the feature extraction modules and outputting the output end feature map; A deconvolution operation is performed on the output end feature map of the last feature extraction module to obtain a target image, wherein the resolution of the target image is higher than the resolution of the image to be processed.

2. The method according to claim 1, characterized in that The acquiring the global features of the input feature map based on the multiple one-dimensional sequences includes: Encoding the one-dimensional sequence to obtain a corresponding one-dimensional feature vector; Element-wise fusion of the plurality of one-dimensional feature vectors to obtain a fused feature vector; Based on the fused feature vector and the positional relationship between the elements in the fused feature vector and the image block, a global feature corresponding to the input feature map is reconstructed.

3. The method according to claim 2, characterized in that After reconstructing the global feature corresponding to the input feature map based on the fused feature vector and the positional relationship between the elements in the fused feature vector and the image block, the method further includes: Obtaining a feature missing region in the global feature; The missing feature values ​​in the feature missing region are supplemented by an interpolation method, and the global features after the feature missing region is supplemented are updated as new global features.

4. The method according to claim 1, characterized in that: Scanning the image blocks along multiple scanning directions at a preset scanning interval to obtain multiple one-dimensional sequences corresponding to the scanning directions includes: Scanning the image block along a scanning direction from upper left to lower right at a preset scanning interval to obtain a one-dimensional sequence corresponding to the scanning direction from upper left to lower right; Scanning the image block along a scanning direction from lower right to upper left at a preset scanning interval to obtain a one-dimensional sequence corresponding to the scanning direction from lower right to upper left; Scanning the image block along a scanning direction from upper right to lower left at a preset scanning interval to obtain a one-dimensional sequence corresponding to the scanning direction from upper right to lower left; The image block is scanned along a scanning direction from the lower left to the upper right at a preset scanning interval to obtain a one-dimensional sequence corresponding to the scanning direction from the lower left to the upper right.

5. The method according to claim 1, characterized in that The step of obtaining an output feature map based on the global features and local features of the input feature map by each feature extraction module comprises: The global feature of the input feature map is dot-multiplied with the local feature of the input feature map to obtain the output feature map.

6. The method according to claim 1, characterized in that After obtaining the output end feature map based on the global features and local features of the input end feature map by each of the feature extraction modules, the method further includes: The input-end feature map and the output-end feature map are added by each of the feature extraction modules, and the output-end feature map outputted by the module is updated to the addition result.

7. An image super-resolution device, characterized in that: The device comprises: An initial feature acquisition module is used to acquire an initial feature map of an image to be processed and input it into a plurality of feature extraction modules, wherein the input end and the output end of each feature extraction module are sequentially connected in series; a global feature extraction module, configured to divide the input feature map into a plurality of image blocks through each of the feature extraction modules, scan the image blocks along a plurality of scanning directions at a preset scanning interval to obtain a plurality of one-dimensional sequences corresponding to the scanning directions, and obtain a global feature of the input feature map based on the plurality of one-dimensional sequences, wherein the one-dimensional sequences are used to characterize image features corresponding to the image blocks; A local feature extraction module, used for obtaining local features of the input feature map through each of the feature extraction modules; A feature fusion module, used to obtain and output an output feature map based on the global features and local features of the input feature map through each of the feature extraction modules; The image super-resolution module is used to perform a deconvolution operation on the output end feature map of the last feature extraction module to obtain a target image, wherein the resolution of the target image is higher than the resolution of the image to be processed.

8. An electronic device, characterized in that: include: one or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to execute the method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program code executable by a processor, and when the program code is executed by the processor, the processor executes the method according to any one of claims 1 to 6.

10. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Citation Information

Cited By

  • Image processing method and system and storage medium

    CN120931669A