Image super-resolution processing method and apparatus, device, program product, and chip

Abstract

Embodiments of the present application relate to the technical field of artificial intelligence, and provide an image super-resolution processing method and apparatus, a device, a program product, and a chip, improving the efficiency of image super-resolution processing. The method comprises: acquiring a first feature map corresponding to an image to be super-resolved; performing convolution calculation on the first feature map, to obtain convolution results in multiple dimensions; rearranging the convolution results in the multiple dimensions, a width dimension W of an image to be super-resolved in the rearranged convolution results being closely adjacent to a width dimension dw of a first convolution kernel or a super-resolution magnification dimension sw of the width, and the first convolution kernel being used for converting adjacent pixels in the same channel in the image to be super-resolved into different channels; performing upsampling on the rearranged convolution results, to obtain a second feature map, the resolution of an image corresponding to the second feature map being higher than the resolution of the image to be super-resolved corresponding to the first feature map.

Description

Image super-resolution processing method, device, equipment, program product and chip

[0001] The present application claims priority from the Chinese patent application No. 202411014812.1 filed on July 25, 2024, and entitled "Image super-resolution processing method, device, equipment, program product and chip", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence, and in particular to an image super-resolution processing method, device, equipment, program product and chip. BACKGROUND

[0003] With the wide application of artificial intelligence technology in the field of image processing, low-resolution images can be converted into high-resolution images through super-resolution calculation. However, when a large number of low-resolution images or large-scale low-resolution images are processed, a large amount of super-resolution calculation is required, which consumes a large amount of computing resources and time.

[0004] Currently, a low-resolution input image is divided into multiple low-resolution image blocks, which are then sent into a super-resolution model in parallel, and convolution calculation is performed on the backbone network of the super-resolution model. Finally, the super-resolved image blocks are merged and up-sampled to obtain a high-resolution output image.

[0005] In related technologies, a low-resolution input image is divided into multiple low-resolution image blocks, which are then super-resolved. After the blocks are spliced and up-sampled, only parallel calculation of the convolution network is relied on, since the parallelism of the convolution network is limited, the speed of super-resolution calculation is not greatly improved. SUMMARY

[0006] The present application provides an image super-resolution processing method, device, equipment, program product and chip, which improves the calculation efficiency of image super-resolution processing.

[0007] In a first aspect, the present application provides an image super-resolution processing method, comprising: obtaining a first feature map corresponding to a to-be-super-resolved image; performing convolution calculation on the first feature map to obtain convolution results in multiple dimensions; rearranging the convolution results in multiple dimensions, wherein the width dimension W of the to-be-super-resolved image in the rearranged convolution results is adjacent to the width dimension dw of a first convolution kernel or the super-resolution magnification dimension sw of the width, and the first convolution kernel is used to convert adjacent pixel points in the same channel of the to-be-super-resolved image to different channels; and performing up-sampling on the rearranged convolution results to obtain a second feature map, wherein the resolution of the image corresponding to the second feature map is higher than the resolution of the to-be-super-resolved image corresponding to the first feature map.

[0008] It can be understood that, by performing convolution calculation on the first feature map corresponding to the image to be super-resolved, a plurality of dimensional convolution results can be obtained, and the plurality of dimensional convolution results can be rearranged according to the arrangement order of the width dimension W of the image to be super-resolved, the width dimension dw of the first convolution kernel or the width super-resolution magnification dimension sw, so that the rearranged convolution results can be obtained by performing a transpose operation to obtain a feature map with increased resolution. In order to obtain a final super-resolved feature map with increased resolution, the convolution results need to be upsampled, and the upsampled process needs to perform a transpose operation on the convolution results. Since the number of transpositions performed on the rearranged convolution results is greatly reduced compared to the number of transpositions performed on the convolution results in the original arrangement order, the calculation resources used in the upsampled process and the operation time are reduced, thereby improving the efficiency of super-resolution processing of low-resolution images.

[0009] In a possible implementation, the first feature map corresponding to the image to be super-resolved is obtained by performing spatial-to-depth convolution calculation on the image to be super-resolved to obtain the first feature map, wherein the spatial-to-depth convolution calculation uses the first convolution kernel.

[0010] It can be understood that, by transferring the element points adjacent in space of the image to be super-resolved to different channels, a multi-channel mode is realized, the channel parallel processing capability of the computing acceleration card is utilized, parallel processing of feature convolution calculation corresponding to the element points adjacent in space is realized, and thus the running speed is improved, thereby improving the efficiency of image super-resolution processing.

[0011] In a possible implementation, the first feature map is subjected to convolution calculation to obtain a plurality of dimensional convolution results, including: using a second convolution kernel to perform convolution calculation on the first feature map to obtain a plurality of dimensional convolution results; the second convolution kernel is a convolution kernel adjusted according to the kernel size of the first convolution kernel, the kernel size of the convolution kernel before adjustment is greater than the kernel size of the second convolution kernel, and the number of channels of the convolution kernel before adjustment is less than the number of channels of the second convolution kernel.

[0012] It can be understood that, by adjusting the kernel size and the number of channels of the second convolution kernel according to the kernel size of the first convolution kernel, since the first feature map is obtained by performing convolution calculation on the image to be super-resolved by using the first convolution kernel, the size of the first feature map may be smaller and the number of channels may be increased compared to the size of the feature map generated by directly extracting features from the image to be super-resolved. Therefore, the kernel size of the original convolution kernel needs to be reduced and the number of channels of the original convolution kernel needs to be increased to achieve the purpose of adaptive adjustment of the convolution kernel, so as to ensure the effect and speed of convolution calculation on the first feature map.

[0013] In a possible implementation, before the convolution calculation on the first feature map using the second convolution kernel to obtain the convolution result in multiple dimensions, the method further includes: adjusting the convolution kernel in the super-resolution model to the second convolution kernel according to the kernel size of the first convolution kernel; and performing model training on the super-resolution model to obtain a trained super-resolution model; and the convolution calculation on the first feature map using the second convolution kernel to obtain the convolution result in multiple dimensions includes: inputting the first feature map into the trained super-resolution model, and performing the convolution calculation on the first feature map using the second convolution kernel in the trained super-resolution model to obtain the convolution result in multiple dimensions.

[0014] It can be understood that, in the process of performing super-resolution on the first feature map, the first feature map needs to be input into the super-resolution model first, and the super-resolution model includes a convolution kernel used for performing convolution calculation on the feature map. Since the first feature map can be a feature map processed by the first convolution kernel, the convolution kernel in the super-resolution model is adjusted according to the size and the number of channels of the first convolution kernel, and then the super-resolution model including the second convolution kernel is trained, so as to ensure the effect of performing super-resolution calculation using the super-resolution model after the convolution kernel is adjusted.

[0015] In a possible implementation, the ratio of the kernel size of the convolution kernel before adjustment in the super-resolution model to the kernel size of the first convolution kernel is the kernel size of the second convolution kernel, and the product of the number of channels of the convolution kernel before adjustment in the super-resolution model and the kernel size of the first convolution kernel is the number of channels of the second convolution kernel.

[0016] It can be understood that, in one case, the ratio of the kernel size of the convolution kernel before adjustment to the kernel size of the first convolution kernel can be used as the kernel size of the second convolution kernel, and the product of the number of channels of the convolution kernel before adjustment and the kernel size of the first convolution kernel can be used as the number of channels of the second convolution kernel. The kernel size and the number of channels of the second convolution kernel can be accurately and rapidly determined through a specific algorithm, so as to ensure the efficiency of image super-resolution processing.

[0017] In a possible implementation, the arrangement of the adjusted convolution result is N, the height dimension H of the image to be super-resolved, the width dimension W of the image to be super-resolved, the width dimension dw of the first convolution kernel, the width super-resolution magnification dimension sw, the output channel dimension c, the height dimension dh of the first convolution kernel, and the height super-resolution magnification dimension sh.

[0018] It can be understood that, arranging the multi-dimensional convolution result in the above implementation can facilitate subsequent transposition operation, thereby improving the efficiency of upsampling and further improving the efficiency of performing super-resolution on the low-resolution image.

[0019] In a possible implementation, the up-sampling is performed on the rearranged convolution result to obtain the second feature map, including: performing a transposition operation on the rearranged convolution result to obtain the second feature map; and the transposition operation corresponds to a transposition number smaller than the transposition number before the rearrangement of the convolution result.

[0020] It can be understood that the transposition operation performed on the rearranged convolution result corresponds to a transposition number smaller than the transposition number before the rearrangement of the convolution result, thereby improving the efficiency of the up-sampling process, and further improving the efficiency of the image super-resolution processing.

[0021] In a second aspect, an embodiment of the present application provides an image super-resolution processing apparatus, which is configured to perform any of the image super-resolution processing methods provided in the first aspect.

[0022] In a possible implementation, the image super-resolution processing apparatus can be divided into functional modules according to the method provided in the first aspect. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. For example, the image super-resolution processing apparatus can be divided into an obtaining module, a calculating module, a sorting module, and an up-sampling module according to functions. The possible technical solutions and beneficial effects of each functional module described above can be referred to the technical solutions provided in the first aspect or the corresponding possible implementation thereof, which will not be described herein.

[0023] In a third aspect, an embodiment of the present application provides a computing device, which includes a processor and a memory, the processor being coupled to the memory; the memory is configured to store computer instructions, the computer instructions being loaded and executed by the processor to enable the computing device to implement the image super-resolution processing method described in the above aspect.

[0024] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores at least one computer program instruction, the computer program instruction being loaded and executed by a processor to implement the image super-resolution processing method described in the above aspect.

[0025] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes computer instructions stored in a computer readable storage medium. A processor of a computing device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to enable the computing device cluster to perform the image super-resolution processing method provided in the various optional implementation manners of the first aspect.

[0026] In a sixth aspect, an embodiment of the present application provides a chip, comprising an interface circuit and a processor; the interface circuit and the processor are interconnected through a line; the processor receives and executes computer instructions from a memory through the interface circuit, so that the processor implements the image super-resolution processing method according to the above aspect.

[0027] The specific description of the second aspect to the sixth aspect and various implementation manners thereof in the present application can refer to the detailed description in the first aspect and various implementation manners thereof; and the beneficial effects of the second aspect to the sixth aspect and various implementation manners thereof can refer to the beneficial effect analysis in the first aspect and various implementation manners thereof, which will not be repeated here.

[0028] These aspects or other aspects of the present application will be more apparent in the following description. BRIEF DESCRIPTION OF DRAWINGS

[0029] FIG. 1 is a schematic diagram of an image super-resolution processing scenario according to an exemplary embodiment;

[0030] FIG. 2 is a schematic diagram of an architecture of an image super-resolution processing system according to an exemplary embodiment;

[0031] FIG. 3 is a schematic diagram of a flow of an image super-resolution processing method according to an exemplary embodiment;

[0032] FIG. 4 is a schematic diagram of a spatial-to-depth convolution calculation involved in the embodiment shown in FIG. 3;

[0033] FIG. 5 is a schematic diagram of convolution kernel rearrangement through a convolution kernel rearranger involved in the embodiment shown in FIG. 3;

[0034] FIG. 6 is a schematic diagram of a flow of PixelShuffle up-sampling involved in the embodiment shown in FIG. 3;

[0035] FIG. 7 is a schematic diagram of a structure of an image super-resolution processing apparatus according to an exemplary embodiment. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical scheme and advantages of the present application more clear, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0037] “Multiple” mentioned in the present application refers to two or more. “And / or” describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can mean that there are three cases of A alone, A and B together, and B alone. The character “ / ” generally represents that the associated objects before and after it are in an “or” relationship.

[0038] Also, in the description of the present application, unless otherwise specified, "a plurality of" refers to two or more than two. "At least one of the following (one)" or the like refers to any combination of the items, including any combination of a single item (one) or a plurality of items. For example, at least one of a, b, or c can mean a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, c can be a single or a plurality.

[0039] In addition, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the same items or similar items with basically the same functions and effects are distinguished by using "first", "second", etc. The skilled in the art can understand that "first", "second", etc. do not limit the quantity and execution order, and "first", "second", etc. also do not necessarily mean different. At the same time, in the embodiments of the present application, the words "exemplary" or "for example" are used to represent as an example, illustration or description. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the use of "exemplary" or "for example" is intended to present the relevant concept in a specific manner, for ease of understanding.

[0040] The following briefly introduces the related terms involved in the embodiments of the present application.

[0041] Super resolution or super resolution technology: Super resolution technology is a technology for improving the resolution of images or films, which can be applied to image restoration, image enhancement, ultra-high resolution cameras or microscopes, etc. That is, super resolution technology can be a technology for obtaining a high-resolution image from a low-resolution image through a specific algorithm. With the development of deep learning technology and chip computing power, deep learning-based super resolution technology is increasingly applied to offline rendering and real-time rendering scenarios. For example, the video image industry performs super resolution on existing videos to achieve image restoration or image enhancement, or deep learning super sampling (DLSS) technology performs real-time super resolution on games to improve game image quality and improve user experience, or deploys super resolution algorithm in mobile phones or end-side cameras to improve the resolution of shooting.

[0042] In the process of super resolution processing of a low-resolution image, a backbone network (such as a convolutional neural network) and an upsampling module can be included.

[0043] The upsampling method adopted by the upsampling module is an operation of mapping an image from a small resolution to a large resolution in a convolutional neural network (CNN). After an input image is extracted by the CNN, the size of the output is often reduced, and the image needs to be restored to the original size for further calculation (such as semantic segmentation of the image).

[0044] Specifically, the upsampling method can be pixel shuffle. PixelShuffle is a commonly used upsampling method in deep learning, and the main function can be to recombine a low-resolution feature map to obtain a high-resolution feature map. The PixelShuffle upsampling can be implemented based on matrix transposition and data transfer.

[0045] The matrix transposition (Transpose), that is, the transposition operation of the matrix, exchanges the data of two dimensions of the matrix for instructions or hardware circuits / units; for the Transpose operator of the deep learning model, it also contains the meaning of exchanging multiple dimensions of a multi-dimensional matrix.

[0046] First, the application scenario of the embodiment of the present application is exemplarily introduced.

[0047] At present, the process of using super-resolution technology to perform super-resolution processing on a low-resolution image to convert it into a high-resolution image can be applied to the image repair scene in the video image industry, and can also be applied to the scene of real-time improvement of game image quality in the game industry.

[0048] At present, the process of performing super-resolution processing on a low-resolution image can be summarized into two parts, that is, a convolution-based backbone network and a PixelShuffle-based upsampling post-processing module. First, a low-resolution image can be input into the convolution-based backbone network for convolution calculation, and then the obtained convolution result is output as a high-resolution image after PixelShuffle upsampling post-processing.

[0049] Since the computing device needs to process a large amount of image data, the super-resolution calculation requires a large amount of computing resources and operation time, especially for the super-resolution process of a large resolution image, the operation amount of convolution increases with the increase of the resolution. For a computing chip such as a neural network processing unit (NPU), a tensor processing unit (TPU), and the like, the PixelShuffle up-sampling in the super-resolution processing process is mostly realized based on matrix transposition and data transfer. At a high resolution, a large number of transposition operations are required, which also causes the up-sampling process to consume a large amount of computing resources and operation time, thereby becoming a performance bottleneck. Therefore, in order to meet the increasing requirements of low latency or high concurrency in the application scenarios of image processing, it is necessary to further improve the efficiency of super-resolution processing of images.

[0050] To solve the above technical problems, a reference table containing a one-to-one correspondence record of a complexity score reference value and an image quality reference value can be established, the low-resolution input image is divided into multiple low-resolution image blocks and then sent into the super-resolution model in parallel, before each convolution block (Block) in the model backbone network is calculated, a light scoring network is used to determine which sub-blocks are sent to the subsequent Block for calculation, and finally the super-resolution image blocks are merged to obtain a high-resolution output image.

[0051] In this way, the low-resolution input image is divided into multiple low-resolution image blocks for super-resolution, and then the blocks are spliced, which is easy to cause splitting at the boundary and affect the overall effect of the super-resolution algorithm. The scoring network is used to reduce the number of sub-blocks input into each Block of the super-resolution model backbone network, thereby reducing the calculation amount to speed up the super-resolution algorithm. The number of sub-blocks filtered out at each Block is uncontrollable, so the improvement of the parallel degree of convolution calculation is uncontrollable. Moreover, the up-sampling module of the super-resolution algorithm does not have an acceleration effect.

[0052] In addition, a graphics processing unit (GPU) acceleration method for deep learning super-resolution technology can also be used. The convolution of the super-resolution model is divided according to the output pixels, the divided convolution micro-tasks are calculated by GPU multi-thread parallel computing, and the data required for the convolution calculation of adjacent pixels is cached in the shared memory, thereby greatly optimizing the calculation speed of convolution.

[0053] Since it is a computing and storage allocation method for the convolution layer in the super-resolution model on the GPU, it does not have an acceleration effect on the super-resolution algorithm running on the NPU or TPU. Similarly, the up-sampling module of the super-resolution algorithm also does not have an acceleration effect.

[0054] Therefore, the embodiment of the present application provides an image super-resolution processing method. Convolution calculation on a first feature map corresponding to a to-be-super-resolved image can obtain convolution results in multiple dimensions. The convolution results in multiple dimensions are rearranged according to the arrangement order of the width dimension W of the to-be-super-resolved image, the width dimension dw of the first convolution kernel or the width super-resolution magnification dimension sw, so that the rearranged convolution results can be obtained by transposition operation to obtain a feature map with resolution magnification. In order to obtain a final super-resolved feature map with improved resolution, the convolution results need to be up-sampled. The up-sampling process needs to perform transposition operation on the convolution results. Since the number of transposition operations performed on the rearranged convolution results is greatly reduced compared with the number of transposition operations performed on the convolution results in the original arrangement order, the calculation resources used in the up-sampling process and the operation time are reduced, thereby improving the efficiency of super-resolution processing of low-resolution images.

[0055] In some embodiments, the method provided by the embodiment of the present application first obtains a first feature map corresponding to a to-be-super-resolved image. Then, convolution calculation is performed on the first feature map to obtain convolution results in multiple dimensions. Next, the convolution results in multiple dimensions are rearranged. In the rearranged convolution results, the width dimension W of the to-be-super-resolved image is adjacent to the width dimension dw of the first convolution kernel or the width super-resolution magnification dimension sw. The first convolution kernel is used to convert adjacent pixel points in the same channel of the to-be-super-resolved image to different channels. Finally, the rearranged convolution results are up-sampled to obtain a second feature map. The resolution of the image corresponding to the second feature map is higher than the resolution of the to-be-super-resolved image corresponding to the first feature map. Thus, the purpose of obtaining the second feature map can be achieved while reducing the operation amount of up-sampling the rearranged convolution results in multiple dimensions, simplifying the operation process of up-sampling, and thereby improving the efficiency of super-resolution processing of low-resolution images.

[0056] For example, FIG. 1 is a schematic diagram of an image super-resolution processing scene provided by the embodiment of the present application. In a scene where the resolution of an image needs to be improved, such as digital image processing, specifically, image super-resolution processing, it can be applied in video image repair, game picture enhancement, camera shooting picture quality enhancement and the like. For low-resolution images, a computing device can obtain high-resolution images by super-resolution processing on single or batch low-resolution images.

[0057] As shown in FIG. 1, the computing device 10 includes a super-resolution model 11 and an acceleration system 12. The super-resolution model 11 can be used to implement super-resolution processing and up-sampling processing on a single or batch of low-resolution images to output corresponding high-resolution images, and can include a convolutional neural network model. The acceleration system 12 can be used to pre-process the low-resolution images so that the computing device 10 can input the pre-processed low-resolution images to the super-resolution model 11; the acceleration system 12 can also be used to adjust and train the convolutional neural network in the super-resolution model 11 so that the convolutional neural network can be adaptive according to the adjustment of the input images; the acceleration system 12 can also adjust the convolution kernel output channel order in the convolutional neural network so that the arrangement order of the multi-dimensional convolution results output by the convolutional neural network is changed, thereby simplifying the subsequent operation process of the up-sampling process.

[0058] That is, the super-resolution model 11 can include a backbone network (such as a convolutional neural network) and an up-sampling module, the backbone network can include multiple layers of convolution kernels, and the up-sampling module can be a PixelShuffle up-sampling.

[0059] Wherein, the computing device can perform spatial-to-depth convolution processing on the low-resolution images according to the acceleration system 12 before processing the low-resolution images, and then the acceleration system 12 can update the kernel size and channel number of each layer and each convolution kernel in the backbone network in the super-resolution model 11, and after updating, the super-resolution model 11 is retrained to obtain a trained super-resolution model. Inputting the low-resolution images processed by the spatial-to-depth convolution calculation into the trained super-resolution model 11 can obtain the convolution results output by the backbone network under multiple dimensions, and the acceleration system can rearrange the convolution results under multiple dimensions according to a specified order, so that the specified order arranged convolution results under multiple dimensions are input into the PixelShuffle up-sampling. The data arranged in this specified order will undergo fewer transpose operations to obtain the second feature map compared with the originally arranged data, that is, the low-resolution image corresponding to the enlarged high-resolution image is output.

[0060] In the following, the system architecture of the embodiment of the present application is exemplarily introduced.

[0061] FIG. 2 is a schematic diagram of an architecture of an image super-resolution processing system according to an embodiment of the present application. As shown in FIG. 2, the image super-resolution processing system 20 can be applied in the computing device 10. The image super-resolution processing system 20 can include the acceleration system 12 and a super-resolution processing system. The super-resolution processing system can include a memory 21 and a computing acceleration card 22. The acceleration system 12 can include a module for accelerating convolution kernel group replacement and a module for accelerating data rearrangement. The module for accelerating convolution kernel group replacement can include a spatial-to-depth convolution module 121 and a convolution kernel adjustment module 122. The module for accelerating data rearrangement can include a convolution kernel rearranger 123 and an up-sampling folding module 124. The computing acceleration card 22 can be used to run the accelerated super-resolution model to implement a super-resolution algorithm and an up-sampling algorithm. The computing acceleration card 22 can be an NPU or a TPU. The memory 21 can be used to store an image to be super-resolved and a result image after super-resolution processing.

[0062] The acceleration system 12 obtains an original image to be super-resolved by the spatial-to-depth convolution module 121 in the module for accelerating convolution kernel group replacement, performs spatial-to-depth convolution calculation on the original image to be super-resolved, and stores the image to be super-resolved after convolution processing in the memory 21. The acceleration system 12 obtains an original super-resolution model by the convolution kernel adjustment module 122 in the module for accelerating convolution kernel group replacement, adjusts the size and the number of channels of the convolution kernel in the backbone network of the original super-resolution model, and performs model training on the adjusted super-resolution model. The acceleration system 12 adjusts the arrangement order of the convolution output channels of the backbone network of the trained super-resolution model by the convolution kernel rearranger 123 in the module for accelerating data rearrangement. The computing acceleration card 22 runs a corresponding super-resolution algorithm by the accelerated super-resolution model, calculates the image to be super-resolved in the memory 21 by the super-resolution algorithm, and uses the up-sampling folding module 124 in the module for accelerating data rearrangement to implement the up-sampling process in the super-resolution algorithm to obtain a super-resolution result (an image after super-resolution processing) output by the accelerated super-resolution model. The super-resolution result can be stored in the memory 21.

[0063] In the embodiments of the present application, the computing device can be a computer device, a terminal, a server, etc.

[0064] The server can be one physical or logical server, or two or more physical or logical servers sharing different responsibilities and cooperating with each other to implement the functions of the server.

[0065] For example, the server can be a blade server, a high-density server, a rack server, or a tower server, etc.

[0066] It should be noted that the application scenarios and system architectures described in the embodiments of the application are used to more clearly illustrate the technical solutions of the embodiments of the application, and do not constitute a limitation on the technical solutions provided by the embodiments of the application. Those skilled in the art can know that, with the evolution of system architecture and the appearance of new business scenarios, the technical solutions provided by the embodiments of the application are also applicable to similar technical problems.

[0067] For ease of understanding, the image super-resolution processing method provided by the application is exemplarily introduced below in combination with the accompanying drawings. The image super-resolution processing method is applicable to the computing device shown in FIG. 1 or the image super-resolution processing system shown in FIG. 2.

[0068] FIG. 3 shows a flowchart of an image super-resolution processing method provided by an example embodiment of the application. The image super-resolution processing method can be executed by a computing device. The image super-resolution processing method includes the following steps.

[0069] S101, the computing device obtains a first feature map corresponding to a to-be-super-resolved image.

[0070] In the embodiments of the application, the computing device can obtain a first feature map corresponding to a to-be-super-resolved image. The first feature map can be a feature map generated after feature extraction is performed on the to-be-super-resolved image.

[0071] In a possible implementation, the computing device can directly perform feature extraction on the to-be-super-resolved image to generate the first feature map. The first feature map can be a three-channel matrix.

[0072] In another possible implementation, the computing device performs spatial-to-depth convolution calculation on the to-be-super-resolved image to obtain the first feature map.

[0073] The first convolution kernel used in the spatial-to-depth convolution calculation is used to convert adjacent pixel points in the same channel in the to-be-super-resolved image to a convolution kernel of different channels. The kernel size of the first convolution kernel can be pre-set.

[0074] That is, the computing device can pre-insert a spatial-to-depth convolution module in front of the super-resolution algorithm. The spatial-to-depth convolution module can include multiple convolution layers. The weight of each convolution layer can be a specific parameter. The kernel size of the convolution kernel in the convolution layer is the step size (dh, dw) of the spatial-to-depth, which can be configured. The number of convolution kernels can be uniquely determined by the step size, C out = dh x dw x C in , where C out is the number of kernels, and C in is the number of input image channels.

[0075] For example, FIG. 4 is a schematic diagram of a spatial-to-depth convolution calculation according to an embodiment of the present application. As shown in FIG. 4, if the image to be super-resolved is a 3-channel 8*8 matrix, the first convolution kernel is a 3-channel 4*4 matrix, and the number of kernels is calculated according to the above formula to be 3*4*4, that is, 48 groups of 3-channel 4*4 matrices. The matrix under each convolution kernel has only one element of 1 and the rest of the elements are 0. For example, the matrix under the first channel of the first group has an element of 1 in the upper left corner and the rest of the elements are 0. Since the step size is the same as the kernel size, the step size is 4. After convolution calculation with the 3-channel 8*8 matrix of the image to be super-resolved, a 2*2 matrix composed of the elements in the first row and the first column, the first row and the fifth column, the fifth row and the first column, and the fifth row and the fifth column of the first channel of the 8*8 matrix is obtained. The matrix under the second channel of the second group has an element of 1 in the upper left corner and the rest of the elements are 0. Since the step size is the same as the kernel size, the step size is 4. After convolution calculation with the 3-channel 8*8 matrix of the image to be super-resolved, a 2*2 matrix composed of the elements in the first row and the first column, the first row and the fifth column, the fifth row and the first column, and the fifth row and the fifth column of the second channel of the 8*8 matrix is obtained. The 2*2 matrix of 48 channels is obtained by sequentially performing convolution calculation on the other convolution kernels. Similarly, after performing convolution calculation on the upper left corner elements of the three channels, convolution calculation is sequentially performed on the first row and the second column elements of the three channels, until the fourth row and the fourth column elements of the three channels are calculated, to complete the spatial-to-depth calculation process of converting the 3-channel 8*8 matrix into the 2*2 matrix of 48 channels. The 2*2 matrix of 48 channels can be the first feature map of the image to be super-resolved.

[0076] It can be understood that the spatial-to-depth calculation in the above manner can transfer adjacent elements in the same channel of the feature map of the image to be super-resolved to different channels. Since the computing acceleration card has affinity for parallel processing of channels, the super-resolution processing speed of the first feature map can be improved by parallel processing of different channels.

[0077] That is, the computing device obtains the first feature map, which can be a feature map obtained by directly extracting features of the image to be super-resolved in space, or a feature map obtained by extracting features of the image to be super-resolved through spatial-to-depth convolution calculation.

[0078] S102, the computing device performs convolution calculation on the first feature map to obtain convolution results in multiple dimensions.

[0079] In the embodiment of the present application, the super-resolution algorithm includes a convolutional neural network, and the first feature map needs to be convolved by the convolutional neural network to obtain convolution results in multiple dimensions. The convolution results in multiple dimensions can include convolution results in the height dimension H of the to-be-super-resolved image, convolution results in the width dimension W of the to-be-super-resolved image, convolution results in the width dimension dw of the first convolution kernel, convolution results in the width super-resolution magnification dimension sw, convolution results in the output channel dimension c, convolution results in the height dimension dh of the first convolution kernel, and convolution results in the height super-resolution magnification dimension sh.

[0080] Since the first feature map can be a feature map that is not obtained through spatial-to-depth convolution calculation, in this case, the multiple dimensions include the convolution results in the height dimension H of the to-be-super-resolved image, the convolution results in the width dimension W of the to-be-super-resolved image, the convolution results in the width super-resolution magnification dimension sw, the convolution results in the output channel dimension c, and the convolution results in the height super-resolution magnification dimension sh.

[0081] For example, after the computing device obtains the first feature map, the computing device can input the first feature map into the backbone network of the super-resolution model, and obtain convolution results in multiple dimensions through convolution calculation with the convolution kernel in the backbone network. Before inputting the first feature map into the backbone network of the super-resolution model, since the size and the number of channels of the first feature map change after the spatial-to-depth convolution processing on the to-be-super-resolved image, the size and the number of channels of the convolution kernel in the super-resolution model need to be adjusted accordingly. The computing device can adjust the size and the number of channels of the convolution kernel in the super-resolution algorithm layer by layer according to the step size of the spatial-to-depth convolution module. The adjustment manner is not limited, and the adjustment result is to reduce the size of the convolution kernel and increase the number of convolution channels, so as to transfer the spatial receptive field of the original super-resolution algorithm to the channel to realize the reduction of the number of matrix multiplication operations in the convolution process and the acceleration of the super-resolution convolution calculation while keeping the model parameter amount unchanged.

[0082] In a possible implementation, the computing device can adjust the convolution kernel in the super-resolution model to a second convolution kernel according to the kernel size of the first convolution kernel, then perform model training on the super-resolution model to obtain a trained super-resolution model, and then input the first feature map into the trained super-resolution model to perform convolution calculation through the second convolution kernel in the trained super-resolution model to obtain convolution results in multiple dimensions.

[0083] In a possible case, the ratio of the kernel size of the convolution kernel in the super-resolution model before adjustment to the kernel size of the first convolution kernel can be the kernel size of the second convolution kernel, and the product of the number of channels of the convolution kernel in the super-resolution model before adjustment and the kernel size of the first convolution kernel can be the number of channels of the convolution kernel of the second convolution.

[0084] In a possible implementation, the computing device can perform convolution calculation on the first feature map using a second convolution kernel to obtain convolution results in multiple dimensions; the second convolution kernel is a convolution kernel adjusted according to the kernel size of the first convolution kernel, the kernel size of the unadjusted convolution kernel is greater than the kernel size of the second convolution kernel, and the channel number of the unadjusted convolution kernel is less than the channel number of the second convolution kernel.

[0085] That is, the convolution kernel in the backbone network can be adjusted according to the formula as shown below. chn=chn ori ×dh×dw

[0086] wherein k size is the adjusted convolution kernel size, k size_ori is the unadjusted convolution kernel size, chn is the adjusted convolution channel number, and chn ori is the unadjusted convolution channel number.

[0087] For example, if the unadjusted convolution kernel size is 16*16, the adjusted convolution kernel size can be 4*4 according to the above calculation formula. If the unadjusted convolution kernel channel number is 3, the adjusted convolution kernel channel number is 48 channels.

[0088] In S103, the computing device rearranges the convolution results in multiple dimensions.

[0089] In the embodiments of the present application, the computing device can adjust the arrangement order of the convolution results in multiple dimensions according to a specified order.

[0090] In a possible implementation, the computing device can rearrange the order of the convolution output channels in the super-resolution model through the acceleration system, that is, the order of each convolution kernel outputting convolution results in different dimensions in the last convolution layer in the backbone network of the super-resolution model can be adjusted, so that the convolution results in multiple dimensions output by the backbone network can be rearranged.

[0091] wherein the convolution results in multiple dimensions are obtained after convolution calculation by the trained backbone network, and the original arrangement order of the convolution results in multiple dimensions after calculation can be C=dh×scale h ×dw×scale w ×c, wherein c is the channel number of the output image, scale h and scale wFor the super-resolution magnification, C is the number of channels of the PixelShuffle processing, and the order of × is the arrangement order of each dimension of the data in the memory. According to different implementations of the frame PixelShuffle, the original arrangement mode is different. The convolution kernel rearranger adjusts the output channel order of the convolution branch, so that a large number of operations of the upsampling processing can be folded out without introducing additional calculation. The rearranged channel order can be C = dw × scale w × c × dh × scale h .

[0092] The rearranged order, that is, the specified order, in the case that the image to be super-resolved is not subjected to spatial-to-depth convolution processing, can be the order of arranging the convolution result in the width super-resolution magnification dimension first, then arranging the convolution result in the output channel dimension, and then arranging the convolution result in the height super-resolution magnification dimension.

[0093] That is, the width dimension W of the image to be super-resolved in the rearranged convolution result needs to be adjacent to the width dimension dw of the first convolution kernel, or adjacent to the width super-resolution magnification dimension sw.

[0094] In a possible implementation, if the image to be super-resolved is subjected to spatial-to-depth convolution processing, the specified order is arranged in the order of the number N of images to be super-resolved, the height dimension H of the image to be super-resolved, the width dimension W of the image to be super-resolved, the width dimension dw of the first convolution kernel, the width super-resolution magnification dimension sw, the output channel dimension c, the height dimension dh of the first convolution kernel, and the height super-resolution magnification dimension sh.

[0095] That is, the arrangement of the adjusted convolution result is the number N of images to be super-resolved, the height dimension H of the image to be super-resolved, the width dimension W of the image to be super-resolved, the width dimension dw of the first convolution kernel, the width super-resolution magnification dimension sw, the output channel dimension c, the height dimension dh of the first convolution kernel, and the height super-resolution magnification dimension sh.

[0096] The convolution kernel rearranger is used to adjust the output channel order of the convolution branch, and the rearranged arrangement is C = dw × scale w × c × dh × scale h The implementation of the rearranger can be introduced in the deployment stage or in the training stage. The implementation modes of the two are different, but the output channel arrangement order is changed to C = dw × scale w × c × dh × scale h .

[0097] For example, FIG. 5 is a schematic diagram of rearranging the convolution kernel by the convolution kernel rearranger according to an embodiment of the present application. As shown in FIG. 5, the order of the output channels of the convolution branches before rearrangement in the super-resolution model can be the convolution branch in the height dimension dh of the first convolution kernel, the convolution branch in the height super-resolution magnification dimension sh, the convolution branch in the width dimension dw of the first convolution kernel, the convolution branch in the width super-resolution magnification dimension sw, and the convolution branch in the output channel dimension c. After rearrangement and adjustment of the order of the output channels of the last layer of the convolution branch by the convolution kernel rearranger, the order of the output channels of the rearranged convolution branch can be the convolution branch in the width dimension dw of the first convolution kernel, the convolution branch in the width super-resolution magnification dimension sw, the convolution branch in the output channel dimension c, the convolution branch in the height dimension dh of the first convolution kernel, and the convolution branch in the height super-resolution magnification dimension sh, so as to ensure that the convolution results in multiple dimensions of the subsequent output can be arranged in a specified order.

[0098] The specific implementation form of the convolution kernel rearrangement is to use the rearranger to process the last convolution layer of the super-resolution algorithm. In addition, the convolution kernel rearranger can independently realize the NPU or TPU acceleration of the super-resolution algorithm without S101 and S102, and the rearranged data arrangement can be C = scale w × c × scale h The implementation form of the convolution kernel rearranger can be program code, chip instruction, hardware circuit, etc.

[0099] S104, the computing device up-samples the rearranged convolution result to obtain a second feature map.

[0100] The resolution of the second feature map is higher than that of the first feature map.

[0101] In a possible implementation, the computing device can perform a transpose operation on the convolution result arranged in the adjusted arrangement order to obtain the second feature map.

[0102] That is, the computing device rearranges the order of the output channels of the convolution branch of the backbone network in the super-resolution model to obtain the convolution result in multiple dimensions arranged in a specified order, and up-samples the rearranged convolution result (for example, input PixelShuffle up-sampling) to obtain the second feature map.

[0103] For example, for the PixelShuffle upsampling implemented based on the matrix transpose instruction or the hardware unit, after the channel rearrangement operation is completed, a large number of operations of the PixelShuffle upsampling are folded in the convolution layer processed by the rearranger, and the upsampling of the super-resolution algorithm is replaced by a transpose operator or an efficient operator of a customized implementation, so that the super-resolution upsampling module can be accelerated.

[0104] For example, FIG. 6 is a flowchart of a PixelShuffle upsampling related to an embodiment of the present application. The process of the computing device upsampling the original data (convolution result before rearrangement) needs to perform a plurality of cycles of transpose operations, so as to output the second feature map of the target data arrangement, which is used to indicate that the high-resolution image corresponding to the to-be-super-resolved image can be directly converted to obtain the convolution result arrangement data. As shown in FIG. 5, the backbone network can output the rearranged convolution result, and then input the rearranged convolution result: N, H, W, dw, sw, c, dh, sh into the PixelShuffle upsampling (S31), and then the PixelShuffle performs the transpose operation on N, H, W, dw, sw, c, dh, sh with N, H as the axis (S32). The transpose operation can include multiple transpositions. In the case that the rearranged convolution result is obtained through the transpose operation to obtain the target data arrangement: N, H, dh, sh, W, dw, sw, c, the corresponding high-resolution image is output according to the target data arrangement: N, H, dh, sh, W, dw, sw, c (S33). Since the target data arrangement is to arrange the convolution result of the height dimension of the image, the convolution result of the height dimension of the first convolution kernel, and the convolution result of the height super-resolution magnification dimension in close proximity, and arrange the convolution result of the width dimension of the image, the convolution result of the width dimension of the first convolution kernel, and the convolution result of the width super-resolution magnification dimension in close proximity, the corresponding high-resolution image can be directly generated.

[0105] In summary, the convolution results in multiple dimensions can be obtained by performing convolution calculation on the first feature map corresponding to the image to be super-resolved, and the convolution results in multiple dimensions can be rearranged according to the arrangement order of the width dimension W of the image to be super-resolved, the width dimension dw of the first convolution kernel, or the width super-resolution magnification dimension sw, so that the rearranged convolution results can be obtained by performing a transpose operation to obtain a feature map with increased resolution. In order to obtain a final super-resolved feature map with increased resolution, the convolution results need to be upsampled, and the upsampled process needs to perform a transpose operation on the convolution results. Since the number of transpositions of the rearranged convolution results is greatly reduced compared to the number of transpositions of the convolution results in the original arrangement order, the calculation resources used in the upsampled process and the operation time are reduced, thereby improving the efficiency of super-resolution processing of low-resolution images.

[0106] The above mainly describes the scheme of the embodiments of the present application from the perspective of the method. It can be understood that the image super-resolution processing apparatus includes at least one of the corresponding hardware structure and software module for implementing each function. Those skilled in the art should easily realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is implemented in hardware or computer software driven hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0107] The embodiments of the present application can divide the functional units of the image super-resolution processing apparatus according to the above method examples. For example, each functional unit can be divided according to each function, or two or more functions can be integrated into one processing unit. The integrated unit can be implemented in the form of hardware or software functional unit. It should be noted that the division of units in the embodiments of the present application is illustrative, and is only a logical functional division. Actual implementation can have another division manner.

[0108] For example, FIG. 7 shows a structural schematic diagram of an image super-resolution processing apparatus 500 provided by an example embodiment of the present application. The image super-resolution processing apparatus 500 is applied to a computing device, or the image super-resolution processing apparatus 500 can be a computing device. The image super-resolution processing apparatus 500 includes:

[0109] The acquisition module 510 is configured to acquire a first feature map corresponding to an image to be super-resolved.

[0110] The computing module 520 is configured to perform convolution calculation on the first feature map to obtain convolution results in multiple dimensions.

[0111] The sorting module 530 is configured to rearrange the convolution results in the multiple dimensions, so that a width dimension W of the image to be super-resolved is adjacent to a width dimension dw of a first convolution kernel or a width super-resolution magnification dimension sw, and the first convolution kernel is used to convert adjacent pixel points in the same channel of the image to be super-resolved to different channels.

[0112] The up-sampling module 540 is configured to perform up-sampling on the rearranged convolution results to obtain a second feature map, and a resolution of the second feature map is higher than a resolution of the first feature map.

[0113] In a possible implementation, the obtaining module 510 is further configured to perform spatial-to-depth convolution calculation on the image to be super-resolved to obtain the first feature map, and the spatial-to-depth convolution calculation uses the first convolution kernel.

[0114] In a possible implementation, the computing module 520 is further configured to perform convolution calculation on the first feature map using a second convolution kernel to obtain the convolution results in the multiple dimensions, the second convolution kernel is a convolution kernel adjusted according to a kernel size of the first convolution kernel, a kernel size of a convolution kernel before adjustment is greater than a kernel size of the second convolution kernel, and a number of channels of the convolution kernel before adjustment is less than a number of channels of the second convolution kernel.

[0115] In a possible implementation, the device further includes:

[0116] The training module is configured to, before performing convolution calculation on the first feature map using a second convolution kernel to obtain the convolution results in the multiple dimensions, adjust a convolution kernel in a super-resolution model to the second convolution kernel according to a kernel size of the first convolution kernel, and perform model training on the super-resolution model to obtain a trained super-resolution model.

[0117] The computing module 520 is further configured to input the first feature map into the trained super-resolution model, perform convolution calculation on the first feature map through the second convolution kernel in the trained super-resolution model, and obtain the convolution results in the multiple dimensions.

[0118] In a possible implementation, a ratio of a kernel size of a convolution kernel before adjustment in the super-resolution model to a kernel size of the first convolution kernel is a kernel size of the second convolution kernel, and a product of a number of channels of the convolution kernel before adjustment in the super-resolution model and the kernel size of the first convolution kernel is a number of convolution kernel channels of the second convolution kernel.

[0119] In a possible implementation, the arrangement of the adjusted convolution result is N, a height dimension H of the image to be super-resolved, a width dimension W of the image to be super-resolved, a width dimension dw of the first convolution kernel, a super-resolution magnification dimension sw of the width, an output channel dimension c, a height dimension dh of the first convolution kernel, and a super-resolution magnification dimension sh of the height.

[0120] In a possible implementation, the upsampling module 540 is further configured to perform a transposition operation on the rearranged convolution result to obtain the second feature map, where the transposition operation corresponds to a transposition number smaller than the transposition number before the arrangement of the convolution result is adjusted.

[0121] For a specific description of the optional manner, refer to the foregoing method embodiments, which will not be repeated here. In addition, the explanation and beneficial effect of any one of the image super-resolution processing apparatuses provided above can refer to the corresponding method embodiments described above, which will not be repeated here.

[0122] The embodiments of the present application further provide a computer program product containing instructions. The computer program product can be software or a program product containing instructions, which can run on a computing device or be stored in any available medium. When the computer program product runs on at least one computing device, the at least one computing device is caused to perform the image super-resolution processing method.

[0123] The embodiments of the present application further provide a computer readable storage medium. The computer readable storage medium can be any available medium that a computing device can store or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk), etc. The computer readable storage medium contains instructions, which instruct the computing device to perform the digital resource management method or instruct the computing device to perform the image super-resolution processing method.

[0124] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. An image super-resolution processing method, characterized by, The method comprises: obtaining a first feature map corresponding to a to-be-super-resolution image; performing convolution calculation on the first feature map to obtain convolution results in multiple dimensions; rearranging the convolution results in multiple dimensions, wherein a width dimension W of the to-be-super-resolution image in the rearranged convolution results is adjacent to a width dimension dw of a first convolution kernel or a width super-resolution magnification dimension sw, and the first convolution kernel is used to convert adjacent pixel points in the same channel of the to-be-super-resolution image to different channels; performing up-sampling on the rearranged convolution results to obtain a second feature map, wherein a resolution of an image corresponding to the second feature map is higher than a resolution of the to-be-super-resolution image corresponding to the first feature map.

2. The method of claim 1, wherein, The method comprises: performing spatial-to-depth convolution calculation on the to-be-super-resolution image to obtain the first feature map; wherein the spatial-to-depth convolution calculation uses the first convolution kernel.

3. The method of claim 2, wherein, The method comprises: performing convolution calculation on the first feature map using a second convolution kernel to obtain the convolution results in multiple dimensions; the second convolution kernel is a convolution kernel adjusted according to the kernel size of the first convolution kernel, the kernel size of the convolution kernel before adjustment is larger than the kernel size of the second convolution kernel, and the number of channels of the convolution kernel before adjustment is smaller than the number of channels of the second convolution kernel.

4. The method of claim 3, wherein, The method further comprises: adjusting the convolution kernel in a super-resolution model to the second convolution kernel according to the kernel size of the first convolution kernel; performing model training on the super-resolution model to obtain a trained super-resolution model; The method comprises: inputting the first feature map into the trained super-resolution model, and performing convolution calculation on the first feature map through the second convolution kernel in the trained super-resolution model to obtain the convolution results in multiple dimensions.

5. The method of claim 4, wherein, The ratio of the kernel size of the convolution kernel before adjustment in the super-resolution model to the kernel size of the first convolution kernel is the kernel size of the second convolution kernel; and the product of the number of channels of the convolution kernel before adjustment in the super-resolution model and the kernel size of the first convolution kernel is the number of convolution kernel channels of the second convolution kernel.

6. The method according to any one of claims 1 to 5, characterized in that, The arrangement of the adjusted convolution results is the number N of to-be-super-resolution images, a height dimension H of the to-be-super-resolution image, a width dimension W of the to-be-super-resolution image, a width dimension dw of the first convolution kernel, a width super-resolution magnification dimension sw, an output channel dimension c, a height dimension dh of the first convolution kernel, and a height super-resolution magnification dimension sh.

7. The method according to any one of claims 1 to 6, characterized in that, The method comprises: performing a transpose operation on the rearranged convolution results to obtain the second feature map; wherein the transpose operation corresponds to a transpose number smaller than a transpose number before the arrangement of the convolution results is adjusted.

8. An image super-resolution processing apparatus characterized by comprising: The device comprises: The acquisition module is configured to acquire a first feature map corresponding to a to-be-super-resolved image; The calculation module is configured to perform convolution calculation on the first feature map to obtain convolution results in multiple dimensions; The sorting module is configured to rearrange the convolution results in the multiple dimensions, and a width dimension W of the to-be-super-resolved image in the rearranged convolution results is adjacent to a width dimension dw of a first convolution kernel or a width super-resolution magnification dimension sw, the first convolution kernel being used to convert adjacent pixel points in the same channel of the to-be-super-resolved image to different channels. The up-sampling module is configured to perform up-sampling on the rearranged convolution results to obtain a second feature map, the resolution of the second feature map being higher than that of the first feature map.

9. The apparatus of claim 8, wherein, The acquisition module is further configured to perform spatial-to-depth convolution calculation on the to-be-super-resolved image to obtain the first feature map, and the spatial-to-depth convolution calculation uses the first convolution kernel.

10. The apparatus of claim 9, wherein, The calculation module is further configured to perform convolution calculation on the first feature map using a second convolution kernel to obtain the convolution results in the multiple dimensions, the second convolution kernel being a convolution kernel adjusted according to the kernel size of the first convolution kernel, the kernel size of a convolution kernel before adjustment being larger than that of the second convolution kernel, and the number of channels of the convolution kernel before adjustment being smaller than that of the second convolution kernel.

11. The apparatus of claim 10, wherein, The device further comprises: The training module is configured to, before performing convolution calculation on the first feature map using a second convolution kernel to obtain the convolution results in the multiple dimensions, adjust a convolution kernel in a super-resolution model to the second convolution kernel according to the kernel size of the first convolution kernel, and perform model training on the super-resolution model to obtain a trained super-resolution model. The calculation module is further configured to input the first feature map into the trained super-resolution model and perform convolution calculation on the first feature map through the second convolution kernel in the trained super-resolution model to obtain the convolution results in the multiple dimensions.

12. The apparatus of claim 11, wherein, The ratio of the kernel size of the convolution kernel before adjustment in the super-resolution model to the kernel size of the first convolution kernel is the kernel size of the second convolution kernel, and the product of the number of channels of the convolution kernel before adjustment in the super-resolution model and the kernel size of the first convolution kernel is the number of convolution kernel channels of the second convolution kernel.

13. The apparatus of any one of claims 8 to 12, wherein, The arrangement of the adjusted convolution results is N, a height dimension H of the to-be-super-resolved image, a width dimension W of the to-be-super-resolved image, a width dimension dw of the first convolution kernel, a width super-resolution magnification dimension sw, an output channel dimension c, a height dimension dh of the first convolution kernel, and a height super-resolution magnification dimension sh.

14. The apparatus of any one of claims 8 to 13, wherein, The up-sampling module is further configured to perform a transpose operation on the rearranged convolution results to obtain the second feature map, and the transpose operation corresponds to a transpose number smaller than a transpose number before arrangement adjustment of the convolution results.

15. A computing device, comprising: The computing device includes a processor and a memory, and the processor of the computing device is configured to execute instructions stored in the memory of the computing device to enable the computing device to perform the image super-resolution processing method according to any one of claims 1 to 7.

16. A computer program product, characterised in that, The computer program product comprises instructions which, when executed by a computing device cluster, cause the computing device cluster to perform the image super-resolution processing method of any one of claims 1 to 7.

17. A chip, characterized by The chip comprises an interface circuit and a processor; the interface circuit and the processor are interconnected by a line; the processor receives and executes computer instructions from a memory through the interface circuit, so that the processor implements the image super-resolution processing method of any one of claims 1 to 7.

Images