Image processing method and device, computer equipment and storage medium

Abstract

The invention provides an image processing method and device, computer equipment and a storage medium. The method comprises the steps of simulating an initial convolution operation executed by a super-resolution model on an original image based on a plurality of first lookup tables LUT to obtain an intermediate image; executing an operation of the super-resolution model for the intermediate image to obtain a super-resolution image; wherein the initial convolution operation comprises a rising channel convolution operation and a point convolution operation which are sequentially and linearly executed, the convolution kernel size of the rising channel convolution operation is larger than that of the point convolution operation, and the positions of the convolution kernels of the point convolution operation and the rising channel convolution operation are in one-to-one correspondence; the plurality of first LUTs comprise one-dimensional LUTs in one-to-one correspondence with positions of convolution kernels of the ascending channel convolution operation, the one-dimensional LUTs are used for recording a mapping relation between values of pixels and point convolution results, and the point convolution results are products of the values of the pixels, weights of corresponding positions in the convolution kernels of the ascending channel convolution operation and weights of the convolution kernels of the point convolution operation.

Description

Technical Field

[0001] The present disclosure relates to the technical field of image super-resolution processing, and in particular to an image processing method, apparatus, computer equipment and storage medium. Background Art

[0002] Image super-resolution is abbreviated as super-resolution or super-resolution, which usually refers to improving the resolution of an image, for example, enlarging a picture with a resolution of 512×512 pixels to a picture with a resolution of 1024×1024 pixels. Traditional super-resolution algorithms usually include nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, etc. The advantages of these methods are high execution efficiency, but the disadvantages are poor results and the generated images are blurred although the resolution is improved. With the development of deep learning technology, deep learning models have also produced certain research results in the field of image super-resolution.

[0003] In related technologies, better deep learning models often mean huge model architectures and network data. Such deep learning models often require huge computing power to support them, which severely limits their scope of application. The super-resolution functions that users can actually use in daily life often have poor performance, resulting in poor super-resolution effects and poor user experience. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present disclosure provides an image processing method, apparatus, computer device and storage medium.

[0005] A first aspect of the present disclosure provides an image processing method, the method comprising:

[0006] Based on a plurality of first lookup tables LUT, simulating an initial convolution operation performed by a super-resolution model on an original image to obtain an intermediate image;

[0007] Executing an operation of the super-resolution model on the intermediate image to obtain a super-resolution image;

[0008] Among them, the initial convolution operation includes a channel-up convolution operation and a point convolution operation performed linearly in sequence, the convolution kernel size of the channel-up convolution operation is larger than the convolution kernel size of the point convolution operation, the point convolution operation corresponds one-to-one to each position of the convolution kernel of the channel-up convolution operation, the multiple first LUTs include a one-dimensional LUT corresponding one-to-one to each position of the convolution kernel of the channel-up convolution operation, the one-dimensional LUT is used to record the mapping relationship between the pixel value and the point convolution result, the point convolution result is the product of the pixel value, the weight of the corresponding position in the convolution kernel of the channel-up convolution operation, and the weight of the convolution kernel of the point convolution operation.

[0009] Optionally, simulating an initial convolution operation performed by a super-resolution model on an original image based on a plurality of first lookup tables LUTs includes:

[0010] Determine the receptive field pixel within the receptive field corresponding to each pixel in the original image, determine the point convolution result corresponding to the receptive field pixel in the first LUT corresponding to each receptive field pixel of the pixel, and sum the point convolution results corresponding to the receptive field pixels within the receptive field to obtain the simulated convolution operation result corresponding to the pixel, wherein the receptive field corresponding to the pixel is related to the convolution kernel size of the up-channel convolution operation.

[0011] Optionally, simulating an initial convolution operation performed by a super-resolution model on an original image based on a plurality of first lookup tables LUTs to obtain an intermediate image includes:

[0012] Based on the multiple first LUTs, simulating multiple initial convolution operations performed by the super-resolution model on each pixel of the original image to obtain multiple simulated convolution operation results, wherein the convolution kernel size of the channel-up convolution operation included in each initial convolution operation is different;

[0013] The multiple simulated convolution operation results of each pixel in the original image are averaged to obtain the value of the pixel in the intermediate image.

[0014] Optionally, the averaging of multiple simulated convolution operation results for each pixel in the original image to obtain the value of the pixel in the intermediate image includes:

[0015] The multiple simulated convolution operation results of each pixel in the original image and the value of the pixel in the original image are averaged to obtain the value of the pixel in the intermediate image.

[0016] Optionally, the performing of the operation of the super-resolution model on the intermediate image includes:

[0017] sequentially performing a first operation, at least one second operation, and a third operation on the intermediate image;

[0018] Among them, the first operation is performed by a convolutional neural network layer, a spatial attention network layer, a contrast channel attention network layer, a convolutional neural network layer and a spatial attention network layer deployed in sequence, the second operation is performed by a nonlinear activation function, a convolutional neural network layer and a spatial attention network layer deployed in sequence, and the third operation includes connecting the outputs of the first operation and at least one of the second operations and then performing a convolution operation.

[0019] Optionally, after sequentially performing the first operation, at least one second operation, and the third operation on the intermediate image, executing the operation of the super-resolution model on the intermediate image further includes:

[0020] sequentially performing a first operation, at least one second operation, and a fourth operation on the intermediate image;

[0021] The fourth operation includes connecting the outputs of the first operation and the at least one second operation, and then performing a convolution operation and an upsampling operation whose output channel number corresponds to the target super-resolution multiple.

[0022] Optionally, the performing of the operation of the super-resolution model on the intermediate image includes:

[0023] sequentially performing a first operation, at least one second operation, and a fourth operation on the intermediate image;

[0024] Among them, the first operation is performed by a convolutional neural network layer, a spatial attention network layer, a contrast channel attention network layer, a convolutional neural network layer and a spatial attention network layer deployed in sequence, the second operation is performed by a nonlinear activation function, a convolutional neural network layer and a spatial attention network layer deployed in sequence, and the fourth operation includes connecting the outputs of the first operation and at least one of the second operations, and then performing a convolution operation and an upsampling operation whose output channel number corresponds to the target super-resolution multiple.

[0025] Optionally, the output of each second operation includes:

[0026] The shallow residual of the input of the second operation and the result obtained after the input passes through the nonlinear activation function, the convolutional neural network layer and the spatial attention network layer.

[0027] Optionally, the performing of the operation of the super-resolution model on the intermediate image includes:

[0028] Based on at least one second LUT, part or all of the operation of the super-resolution model on the intermediate image is simulated.

[0029] Optionally, the super-resolution model is further processed as follows:

[0030] The super-resolution model is distilled as a decoder of a teacher model in a PISR method, wherein the PISR method is a privileged information method for super-resolution, and the encoder of the teacher model in the PISR method includes a downsampling algorithm.

[0031] A second aspect of the present disclosure provides a super-resolution model architecture, the architecture comprising:

[0032] An initial convolution layer, comprising a linearly arranged ascending channel convolution layer and a point convolution layer, wherein the convolution kernel size of the ascending channel convolution layer is larger than the convolution kernel size of the point convolution layer, and each weight of the convolution kernel of the point convolution layer corresponds one-to-one to that of the ascending channel convolution layer;

[0033] Among them, the initial convolution layer is used to generate multiple first LUTs, and the multiple first LUTs are used to simulate the initial convolution operation performed by the super-resolution model on the original image to obtain a super-resolution image. The multiple first LUTs include a one-dimensional LUT corresponding to each position of the convolution kernel of the up-channel convolution operation, and the one-dimensional LUT is used to record the mapping relationship between the pixel input and the point convolution result. The point convolution result is the product of the value of the pixel and the weight of the corresponding position in the convolution kernel of the up-channel convolution operation. The result is obtained by the point convolution operation.

[0034] Optionally, the architecture further includes:

[0035] A first network layer, at least one second network layer, and a third network layer are sequentially deployed, wherein the first network layer includes a convolutional neural network layer, a spatial attention network layer, a contrast channel attention network layer, a convolutional neural network layer, and a spatial attention network layer, the second network layer includes a nonlinear activation function, a convolutional neural network layer, and a spatial attention network layer, which are sequentially deployed, and the third network layer includes a connection layer and a convolution layer which are sequentially deployed, and the connection layer is used to connect the outputs of the first network layer and the at least one second network layer;

[0036] The first network layer, at least one of the second network layers, and a fourth network layer are deployed in sequence, wherein the fourth network layer includes the connection layer and the upsampling layer deployed in sequence, and the upsampling layer is used to perform convolution operations and upsampling operations whose output channel number corresponds to the target super-resolution multiple.

[0037] A third aspect of the present disclosure provides an image processing device, the device comprising:

[0038] A simulation module, used for simulating an initial convolution operation performed by a super-resolution model on an original image based on a plurality of first lookup tables LUTs to obtain an intermediate image;

[0039] A super-resolution module, used for performing the operation of the super-resolution model on the intermediate image to obtain a super-resolution image;

[0040] Among them, the initial convolution operation includes a channel-up convolution operation and a point convolution operation performed linearly in sequence, the convolution kernel size of the channel-up convolution operation is larger than the convolution kernel size of the point convolution operation, the point convolution operation corresponds one-to-one to each position of the convolution kernel of the channel-up convolution operation, the multiple first LUTs include a one-dimensional LUT corresponding one-to-one to each position of the convolution kernel of the channel-up convolution operation, the one-dimensional LUT is used to record the mapping relationship between the pixel input and the point convolution result, and the point convolution result is the product of the value of the pixel, the weight of the corresponding position in the convolution kernel of the channel-up convolution operation, and the weight of the convolution kernel of the point convolution operation.

[0041] Optionally, when the simulation module is used to simulate the initial convolution operation performed by the super-resolution model on the original image based on the multiple first lookup tables LUT, it is specifically used to:

[0042] Determine the receptive field pixel within the receptive field corresponding to each pixel in the original image, determine the point convolution result corresponding to the receptive field pixel in the first LUT corresponding to each receptive field pixel of the pixel, and sum the point convolution results corresponding to the receptive field pixels within the receptive field to obtain the simulated convolution operation result corresponding to the pixel, wherein the receptive field corresponding to the pixel is related to the convolution kernel size of the up-channel convolution operation.

[0043] Optionally, when the simulation module is used to simulate the initial convolution operation performed by the super-resolution model on the original image based on the multiple first lookup tables LUT to obtain the intermediate image, it is specifically used to:

[0044] Based on the multiple first LUTs, simulating multiple initial convolution operations performed by the super-resolution model on each pixel of the original image to obtain multiple simulated convolution operation results, wherein the convolution kernel size of the channel-up convolution operation included in each initial convolution operation is different;

[0045] The multiple simulated convolution operation results of each pixel in the original image are averaged to obtain the value of the pixel in the intermediate image.

[0046] Optionally, when the simulation module is used to average the results of multiple simulated convolution operations on each pixel in the original image to obtain the value of the pixel in the intermediate image, it is specifically used to:

[0047] The multiple simulated convolution operation results of each pixel in the original image and the value of the pixel in the original image are averaged to obtain the value of the pixel in the intermediate image.

[0048] Optionally, when the super-resolution module is used to perform the operation of the super-resolution model on the intermediate image, it is specifically used to:

[0049] sequentially performing a first operation, at least one second operation, and a third operation on the intermediate image;

[0050] Among them, the first operation is performed by a convolutional neural network layer, a spatial attention network layer, a contrast channel attention network layer, a convolutional neural network layer and a spatial attention network layer deployed in sequence, the second operation is performed by a nonlinear activation function, a convolutional neural network layer and a spatial attention network layer deployed in sequence, and the third operation includes connecting the outputs of the first operation and at least one of the second operations and then performing a convolution operation.

[0051] Optionally, after the super-resolution module is used to sequentially perform the first operation, at least one second operation, and the third operation on the intermediate image, it is further used to:

[0052] sequentially performing a first operation, at least one second operation, and a fourth operation on the intermediate image;

[0053] The fourth operation includes connecting the outputs of the first operation and the at least one second operation, and then performing a convolution operation and an upsampling operation whose output channel number corresponds to the target super-resolution multiple.

[0054] Optionally, when the super-resolution module is used to perform the operation of the super-resolution model on the intermediate image, it is specifically used to:

[0055] sequentially performing a first operation, at least one second operation, and a fourth operation on the intermediate image;

[0056] Among them, the first operation is performed by a convolutional neural network layer, a spatial attention network layer, a contrast channel attention network layer, a convolutional neural network layer and a spatial attention network layer deployed in sequence, the second operation is performed by a nonlinear activation function, a convolutional neural network layer and a spatial attention network layer deployed in sequence, and the fourth operation includes connecting the outputs of the first operation and at least one of the second operations, and then performing a convolution operation and an upsampling operation whose output channel number corresponds to the target super-resolution multiple.

[0057] Optionally, the output of each second operation includes:

[0058] The shallow residual of the input of the second operation and the result obtained after the input passes through the nonlinear activation function, the convolutional neural network layer and the spatial attention network layer.

[0059] Optionally, when the super-resolution module is used to perform the operation of the super-resolution model on the intermediate image, it is specifically used to:

[0060] Based on at least one second LUT, part or all of the operation of the super-resolution model on the intermediate image is simulated.

[0061] Optionally, the super-resolution model is further processed as follows:

[0062] The super-resolution model is distilled as a decoder of a teacher model in a PISR method, wherein the PISR method is a privileged information method for super-resolution, and the encoder of the teacher model in the PISR method includes a downsampling algorithm.

[0063] A fourth aspect of the present disclosure provides a computer program product, including a computer program / instruction, which implements the method described in the first aspect when executed by a processor.

[0064] A fifth aspect of the present disclosure provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in the first aspect is implemented.

[0065] A sixth aspect of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in the first aspect.

[0066] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0067] In the above implementation, the input and output of the convolution operation included in the deep learning model can be recorded by LUT, so the capabilities of the deep learning model can be introduced without deploying the convolution layer of the deep learning model and its complex weights in the device. At the same time, since the convolution kernel size of the ascending channel convolution operation is larger than the convolution kernel size of the point convolution operation, and the ascending channel convolution operation and the point convolution operation are performed linearly, the model has a strong feature-enhancing capability, and its results can be split into multiple one-dimensional LUTs; in other words, the LUT table no longer needs to record the convolution results corresponding to the various arrangement relationships of all pixels in the receptive field of the ascending channel convolution operation, but only needs to record the convolution result corresponding to one pixel, so that the training results of the convolution operation of the deep learning model can be recorded, thereby exponentially reducing the size of the LUT table, and enabling the simulated convolution operation based on the first LUT to achieve an effect similar to the capabilities of the above-mentioned deep learning model, without occupying a large amount of storage space and retrieval computing power of the device, greatly improving the accuracy of the super-resolution algorithm and the visual effect of the super-resolution image.

[0068] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0070] Figure 1 The figure is a flow chart of an image processing method shown in some exemplary embodiments.

[0071] Figure 2 is a schematic diagram of a LUT-based super-resolution method shown in some exemplary embodiments.

[0072] Figure 3 is a schematic diagram of an initial convolution operation shown in some exemplary embodiments.

[0073] Figure 4 FIG. 4 is a schematic diagram of an architecture of an initial convolution operation shown in some exemplary embodiments.

[0074] Figure 5 is an architectural diagram of a super-resolution model shown in some exemplary embodiments.

[0075] Figure 6 is an architecture diagram of another super-resolution model shown in some exemplary embodiments.

[0076] Figure 7 is a schematic diagram of a PISR method shown in some exemplary embodiments.

[0077] Figure 8 is a block diagram of an image processing apparatus shown in some exemplary embodiments.

[0078] Fig. 9 The figure is a hardware structure diagram of a computer device shown in some exemplary embodiments. DETAILED DESCRIPTION

[0079] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0080] As described in the background technology, the super-resolution effect is often positively correlated with the storage space and computing resources occupied. Deep learning models with more network layers and computing units often cannot play a role in super-resolution scenarios commonly seen in daily life, such as deployment in small-volume devices such as mobile phones and tablets. Therefore, the super-resolution functions used by users in daily life often have poor performance, resulting in poor super-resolution effects and poor user experience.

[0081] In view of this, the present disclosure provides an image processing method, apparatus, computer device and storage medium. Next, the embodiments of the present disclosure are described in detail.

[0082] The first aspect of the present disclosure provides an image processing method. Figure 1 , which may include step S101 to step S102.

[0083] Step S101, based on a plurality of first lookup tables LUT, simulating an initial convolution operation performed by a super-resolution model on an original image to obtain an intermediate image.

[0084] Among them, the initial convolution operation includes a channel-up convolution operation and a point convolution operation performed linearly in sequence, the convolution kernel size of the channel-up convolution operation is larger than the convolution kernel size of the point convolution operation, the point convolution operation corresponds one-to-one to each position of the convolution kernel of the channel-up convolution operation, the multiple first LUTs include a one-dimensional LUT corresponding one-to-one to each position of the convolution kernel of the channel-up convolution operation, the one-dimensional LUT is used to record the mapping relationship between the pixel value and the point convolution result, the point convolution result is the product of the pixel value, the weight of the corresponding position in the convolution kernel of the channel-up convolution operation, and the weight of the convolution kernel of the point convolution operation.

[0085] The following is a brief introduction to the concept of LUT (Lookup Table). Figure 2 , which shows an exemplary process for image super-resolution based on a four-dimensional LUT. It should be understood that in LUT, the concept of dimension means that the LUT includes inputs in several dimensions, that is, the concept of "one-dimensional" means that the lookup table has only one input variable in one dimension, and each input value corresponds to one (or a group of) output values. The four-dimensional LUT is a LUT with input variables in four dimensions. In addition, in order to minimize the size of the LUT, a multi-channel image can usually be split into multiple single-channel images for processing (the premise is that the channels of the image have a parallel relationship and can be processed using the same or similar processing strategies); taking an RGB (red, green, and blue) image as an example, in the process of processing the image, the image can be split into single-channel R (red) images, G (green) images, and B (blue) images for separate processing, and then synthesized after processing to generate a super-resolution RGB image.

[0086] exist Figure 2 , pixels 210, 220, 230 and 240 are four pixels of an original image, assuming that the super-resolution task is 2× super-resolution (i.e., the number of pixels is doubled both horizontally and vertically. For pixel 210, it can be understood that the super-resolution process is a process of expanding pixel 210 into pixels 211, 212, 213, and 214), and the receptive field of each pixel is the pixel itself and the pixels to the right, bottom right, and bottom of the pixel (for example, the receptive field of pixel 210 is pixels 210, 220, 230, and 240; it is worth noting that the receptive field here refers to the receptive field when applying LUT, which is related to the specifications and settings of LUT, and the receptive field of the training process of the super-resolution model may be different from this, and the receptive fields of different LUTs may also be different), then the super-resolution process of the original image can be to determine the corresponding result in LUT pixel by pixel, and use the result as the process of four pixels corresponding to the pixel in the super-resolution image. For example, for pixel 210, pixels 210, 220, 230 and 240 within its receptive field may be input into the LUT to determine pixels 211, 212, 213, 214, and so on, to obtain a super-resolution image corresponding to the original image.

[0087] Taking the grayscale range of the original image as 0 to 255 as an example, even if the original image is split into three channels of R, G, and B for separate processing, Figure 2 The LUT shown in also requires 256 4 Only by doing so can all possible situations be traversed. Therefore, illustratively, the LUT table can be downsampled, and then interpolation (such as tetrahedral interpolation) and other methods can be used to obtain super-resolution results in the subsequent use process, so as to reduce the storage space of the LUT. For example, a sampling interval W=16 can be set, and then the grayscale value of each point from 0 to 255 is downsampled, that is, the input space of the LUT table is divided into 17 sampling points of "0, 16, 32, 48, 64, 80, 96, 112, 128, 144, 160, 176, 192, 208, 224, 240, 255", thereby reducing the size of the LUT and greatly reducing the storage volume. Another example is that the storage space occupied by the LUT can also be reduced by reducing the dimension of the LUT. Next, the present disclosure will provide some exemplary implementation methods that can reduce the dimension of the LUT with little or no impact on the super-resolution effect of the LUT.

[0088] See also Figure 3 , which generally shows the architecture of an initial convolution operation. Next, the initial convolution operation including the channel-up convolution operation with a convolution kernel size of 3×3 is used as an example for explanation.

[0089] For Conv3×3 (Conv means convolutional neural network, Conv3×3 can represent a convolution operation with a convolution kernel size of 3×3), the initial convolution operation can be implemented based on the following exemplary code: (kernelsize 3×3, padding1, inputchannel1, outpuchannel 64; kernelsize 1×1, inputchannel64, outpuchannel 1). Among them, kernelsize is the size of the convolution kernel (the convolution operation with a convolution kernel size of 1×1 is a point convolution operation), for example, 3×3 means that the convolution kernel is a matrix of 3 rows and 3 columns; padding is a padding parameter used to fill additional 0 values ​​at the edge of the input data to better process edge pixels; inputchannel is the number of input channels, indicating the number of channels of the input data; outputchannel is the number of output channels, indicating the number of channels output by the convolution layer, that is, the number of convolution kernels or the number of output feature maps. The codes corresponding to the convolution operations of other sizes of convolution kernels, such as Conv5×5 and Conv7×7, can be consistent with the Conv3×3 module, except that the kernelsize is 5×5 and 7×7 respectively. It should be understood that the number of output channels of the channel-up convolution operation is not limited to 64, and it can be selected according to the actual situation (the number of input channels of the point convolution operation can be adapted to it); optionally, the number of output channels of the channel-up convolution operation can be set to a power of 2 to improve the computational efficiency; and the number of output channels of the point convolution operation can also be other values, as long as it is adapted to other network layers of the super-resolution model (referred to as the super-resolution model) and other operations performed.

[0090] In a 3×3 up-channel convolution operation for a pixel, the super-resolution model can extract the receptive field pixels of the pixel (for example, a 3×3 convolution kernel usually corresponds to a 3×3 receptive field, that is, when performing this operation, there is a 3×3 input data volume), and then multiply the value of each extracted receptive field pixel by the weight of the corresponding position on the 3×3 convolution kernel until the product results of all receptive field pixels are obtained, and then add these product results to get the convolution result of the pixel. For example, if the receptive field pixels of a certain pixel are set to the range centered on the pixel (the range corresponds to the convolution kernel size), then in Figure 3In the figure, the receptive field pixels of pixel 311 are pixels 311 to 319, which correspond to positions 321 to 329 on the 3×3 convolution kernel (for example, pixel 311 corresponds to position 321 of the 3×3 convolution kernel, and pixel 312 corresponds to position 322 of the 3×3 convolution kernel). After obtaining the product of each receptive field pixel and the weight of the corresponding position on the 3×3 convolution kernel, these products are added together to obtain the 3×3 convolution result of the pixel, and then the result is subjected to 1×1 convolution to obtain the result of the initial convolution operation.

[0091] In the present disclosure, the initial convolution operation is performed linearly internally, that is, during the execution of the initial convolution operation, no nonlinear operation such as a nonlinear activation function is introduced between the ascending channel convolution operation and the point convolution operation (which can also be understood as the stage for generating the first LUT in the super-resolution model). Therefore, in the application process, the addition operation included in the ascending channel convolution operation (that is, the operation of "adding these products" in the above text) can be moved to be executed after the 1×1 convolution is performed without affecting the calculated result: that is, after the training for the super-resolution model is completed, the weights at each position of the 3×3 convolution kernel and the 1×1 convolution kernel will be determined, then the value of a certain pixel, the weight of the corresponding position in the convolution kernel of the ascending channel convolution operation, and the product of the weight of the convolution kernel of the point convolution operation (that is, the point convolution result corresponding to the pixel) can be used as the mapping value of the pixel in the first LUT (that is, the value value, and the key value (key) is the value of the pixel).

[0092] Based on this, each first LUT can be made to correspond to each position of the convolution kernel of the ascending channel convolution operation; for example, a 3×3 convolution kernel can generate 9 first LUTs (similarly, a 5×5 convolution can generate 25 first LUTs, which will not be repeated here), and these 9 first LUTs correspond to Figure 3 The positions 321 to 329 shown in the figure are used to record the point convolution results corresponding to the receptive field pixels when the values ​​of the receptive field pixels corresponding to these positions undergo various changes. Further based on the linear characteristics and the convolution principle of the convolution operation described above, the result obtained after a single pixel passes through a 3×3 convolution kernel (without involving an addition operation) and a point convolution can be recorded based on a one-dimensional LUT.

[0093] In the process of applying the plurality of first LUTs to simulate the initial convolution operation, exemplarily, simulating the initial convolution operation performed by the super-resolution model on the original image based on the plurality of first lookup tables LUTs may include:

[0094] Determine the receptive field pixel within the receptive field corresponding to each pixel in the original image, determine the point convolution result corresponding to the receptive field pixel in the first LUT corresponding to each receptive field pixel of the pixel, and sum the point convolution results corresponding to the receptive field pixels within the receptive field to obtain the simulated convolution operation result corresponding to the pixel, wherein the receptive field corresponding to the pixel is related to the convolution kernel size of the up-channel convolution operation.

[0095] Here, the convolution kernel size of the ascending channel convolution operation in the initial convolution operation is 3×3, and the pixel targeted by the current convolution is Figure 3 311 in the image is used as an example. The receptive field pixels of pixel 311, i.e., pixels 311 to 319, can be first determined, and then the point convolution results corresponding to pixels 311 to 319 can be determined in the first LUT corresponding to the positions 321 to 329 of the convolution kernels, for example, the point convolution result corresponding to pixel 311 can be determined in the first LUT corresponding to the position 321 of the convolution kernel, and the point convolution result corresponding to pixel 312 can be determined in the first LUT corresponding to the position 322 of the convolution kernel, and so on. After the point convolution results corresponding to pixels 311 to 319 are determined respectively, these point convolution results can be added to obtain the value of pixel 311 in the intermediate image. Among them, the addition operation can be performed in a device that deploys the first LUT, that is, the device only needs to store a number of one-dimensional LUTs, and after reading in the pixel value, the point convolution results of the receptive field pixels of the pixel are determined in the corresponding one-dimensional LUT, and then these point convolution results are added to determine the simulated convolution operation result of the pixel.

[0096] This operation mode can inherit the operation effect of the initial convolution operation in the super-resolution model while consuming only a small amount of storage space and device computing power. Combined with the subsequent operations of the super-resolution model on the intermediate image (of course, these operations can also be simulated based on LUT), it can complete the image super-resolution processing with excellent effect. At the same time, the initial convolution operation of the super-resolution model not only involves the convolution operation of the large convolution kernel, but also involves channel up-convolution (for example, in the aforementioned code, the channel up-convolution operation actually obtains the convolution result of 64 channels after convolving the original image) and channel down-convolution (for example, in the aforementioned code, the point convolution operation reduces the convolution result of 64 channels to a single channel). These operations greatly improve the feature extraction capability of the super-resolution model and the receptive field in the feature extraction process, but they involve multi-channel feature maps, complex product operations and other operations. The output of this process is recorded based on a one-dimensional LUT, and the addition operation is completed by the device that applies the LUT (otherwise the LUT must record multi-dimensional input. This is because the addition operation inevitably involves the values ​​of multiple pixels, which will increase the dimension of the LUT and its total size will also increase exponentially due to the principles mentioned above). This completely avoids the above-mentioned complex operations, allowing the super-resolution model with high feature extraction capabilities and large receptive field to be actually applied to small-scale devices, greatly improving the image super-resolution effect of small-scale devices.

[0097] It should be understood that the super-resolution model may have multiple initial convolution operations (of course, there may be only one) including up-convolution operations with different convolution kernel sizes. For example, further, simulating the initial convolution operation performed by the super-resolution model on the original image based on multiple first lookup tables LUT to obtain the intermediate image may include:

[0098] Based on the multiple first LUTs, simulating multiple initial convolution operations performed by the super-resolution model on each pixel of the original image to obtain multiple simulated convolution operation results, wherein the convolution kernel size of the channel-up convolution operation included in each initial convolution operation is different;

[0099] The multiple simulated convolution operation results of each pixel in the original image are averaged to obtain the value of the pixel in the intermediate image.

[0100] In other words, the super-resolution model can perform multiple initial convolution operations on the original image respectively. The specific exemplary principle of each initial convolution operation can be found in the previous example. The process of each initial convolution operation can be executed independently, and the corresponding LUT tables do not interfere with each other; and the point convolution layer (used to perform point convolution operations) contained in each initial convolution operation can have the same or different weights. Some embodiments provided by the present disclosure actually involve three layers of scenarios, one is the construction and training of the super-resolution model, the second is the generation of LUT, and the third is the application of LUT. The optional architecture in the first layer of scenarios is first introduced below.

[0101] See also Figure 4 For the super-resolution model, the original image can be processed by 3×3 convolution ( Figure 4 Conv n*n in the image actually represents the initial convolution operation composed of the channel-up convolution operation with a convolution kernel size of n*n and the point convolution operation. In this description, this simplified description is also used. Conv n*n is described as "n×n convolution"), 5×5 convolution, and 7×7 convolution, and the simulated convolution operation results corresponding to 3×3 convolution, 5×5 convolution, and 7×7 convolution are obtained respectively (it can also be understood as obtaining the intermediate images corresponding to these convolution operations; of course, the concept of "image" here is only a concept in processing logic, and is not a limitation on the data format and its presentation method). Then, in order to further improve the feature extraction capability of the model, the original image can be concat (also called splicing) with the intermediate images corresponding to 3×3 convolution, 5×5 convolution, and 7×7 convolution, and after the Mean (average) operation, the processing result of this processing stage can be obtained (hereinafter referred to as the first stage). The processing result can be input into the nonlinear activation function to introduce nonlinearity and solve the gradient disappearance problem, for example Figure 4 The Relu (Rectified Linear Unit) function shown in , or the ELU (Exponential Linear Unit) function, etc. This activation function can be used as the end of the first stage or the beginning of the second stage (i.e., another stage of the super-resolution model after the first stage).

[0102] Concat the output of convolution operations with different convolution kernel sizes with the original image, which can be understood as concatenating the feature maps extracted by different convolution kernels with the original input image in the channel dimension. The purpose of this is to increase the richness of feature information. By fusing the features extracted by different scales and different convolution kernels, the diversity and detail information of the image can be better captured (the features extracted by each convolution kernel are different), thereby improving the model's ability to understand and represent images, and helping to improve the model's processing accuracy and effect on images.

[0103] In the above second-layer and third-layer scenarios, the connection operation and the nonlinear activation function do not actually need to be recorded in the LUT, nor do they need to be executed by the device that deploys the LUT. For example, after obtaining multiple simulated convolution operation results through multiple first LUTs corresponding to different convolution kernel sizes, these results are directly averaged (for example, the arithmetic mean, weighted mean, geometric mean, etc.) to obtain the processing result of the first stage, and the result can be directly used for subsequent stage processing.

[0104] Corresponding to the super-resolution model, when the average calculation is performed above, the parameters of the average calculation are not limited to obtaining multiple simulated convolution operation results corresponding to multiple first LUTs of different convolution kernel sizes, and the original image can also be introduced therein, that is, the average calculation of multiple simulated convolution operation results of each pixel in the original image to obtain the value of the pixel in the intermediate image may include:

[0105] The multiple simulated convolution operation results of each pixel in the original image and the value of the pixel in the original image are averaged to obtain the value of the pixel in the intermediate image.

[0106] Based on this, the first LUT can achieve better super-resolution effect and better integrate (simulate) the capabilities of the super-resolution model. Figure 4 In the model architecture shown, according to the rotation average method, the receptive field of the processing stage of the super-resolution model corresponding to the LUT can reach 27 without consuming more LUT space.

[0107] Step S102, executing the operation of the super-resolution model on the intermediate image to obtain a super-resolution image.

[0108] In the process of super-resolution based on LUT, the input and output of the entire super-resolution model can be recorded using LUT, the input and output of each processing stage can be recorded separately, or only the output and output of a part of the processing stage of the super-resolution model can be recorded (and other processing stages and steps can be directly deployed in the device using LUT, similar to the deployment of the addition operation and the average operation in the device using LUT in the previous text). In other words, the execution of the operation of the super-resolution model on the intermediate image can include: based on at least one second LUT, simulating part or all of the operations of the super-resolution model on the intermediate image. In other words, at least one second LUT can be used to record the subsequent operations performed by the super-resolution model on the intermediate image, such as directly recording the input and output when these operations are regarded as a whole.

[0109] For example, after obtaining the intermediate image, the super-resolution model can also perform a first operation, at least one second operation, and a third operation on the intermediate image in sequence to obtain an advanced image (the advanced image here is just a name for convenient distinction, used to indicate a further image obtained after processing the intermediate image), and then the first operation, at least one second operation, and a fourth operation can be performed on the advanced image in sequence to obtain the final super-resolution image. In this case, some LUTs can be used to record the process from the intermediate image to the advanced image (that is, to record their mapping relationship, which is referred to as the second stage in the following text), other LUTs can be used to record the process from the advanced image to the super-resolution image (this process is referred to as the third stage in the following text), and some LUTs can be used to directly record the process from the intermediate image to the super-resolution image.

[0110] Alternatively, super-resolution can also directly perform the first operation, at least one second operation, and the fourth operation on the intermediate image in sequence, and then obtain the final super-resolution image (that is, after the first stage, directly perform the third stage), which is not limited in the present disclosure.

[0111] In the above description, the first operation can be performed by a convolutional neural network layer, a spatial attention network layer, a contrast channel attention network layer, a convolutional neural network layer and a spatial attention network layer deployed in sequence; the second operation can be performed by a nonlinear activation function, a convolutional neural network layer and a spatial attention network layer deployed in sequence; the third operation may include connecting the outputs of the first operation and the at least one second operation, and then performing a convolution operation; the fourth operation may include connecting the outputs of the first operation and the at least one second operation, and then performing a convolution operation and an upsampling operation whose output channel number corresponds to the target super-resolution multiple.

[0112] In other words, it can also be understood that the super-resolution model includes a first network layer, at least one second network layer, a third network layer, and a fourth network layer (of course, these network layers can be arbitrarily selected and matched, and the first network layer, the third network layer, and the fourth network layer can also be multiple), and the first operation is the operation (processing) performed by the first network layer, and the first network layer includes a convolutional neural network layer, a spatial attention network layer, a contrast channel attention network layer, a convolutional neural network layer, and a spatial attention network layer deployed in sequence; the second operation is the operation performed by the second network layer, and the second network layer includes a nonlinear activation function, a convolutional neural network layer, and a spatial attention network layer deployed in sequence; the third operation is the operation performed by the third network layer, and the third network layer includes a connection layer and a convolution layer deployed in sequence, and the connection layer is used to connect the outputs of the first network layer and the at least one second network layer; the fourth operation is the operation performed by the fourth network layer, and the fourth network layer includes the connection layer and the upsampling layer deployed in sequence, and the upsampling layer is used to perform convolution operations and upsampling operations corresponding to the number of output channels and the target super-resolution multiple.

[0113] CCA (Contrast Channel Attention) can be a structure referenced from the lightweight super-resolution IMDN (Information Multi-distillation Network). Compared with the common SENet (Squeeze-and-Excitation Network) channel attention full pooling, CCA calculates the mean and standard deviation of each layer of feature map, and uses the sum of the two as contrast information. This can contain structure and texture information, which is conducive to enhancing image details during image super-resolution. SA (SpatialAttention) captures the dependencies between channels, and the spatial attention network layer can capture the pixel-level relationship in space. SA performs Maxpool (maximum pooling) and AvgPool (average pooling) on ​​the feature N*H*W (channel*height*width) in the channel direction, performs convolution after Concat, and finally passes through the sigmoid activation function (also known as the S-shaped growth curve). Different weights are assigned to each layer of pixels of the spatial feature. The spatial attention network can focus on certain specific areas in the image while ignoring other irrelevant or unimportant areas.

[0114] In some embodiments, the second operation itself is performed by a nonlinear activation function, a convolutional neural network layer, and a spatial attention network layer that are deployed in sequence, but the output of the second operation may not be the direct result obtained by these network layers, but rather a shallow residual of the input of the second operation and the result obtained after the input passes through the nonlinear activation function, the convolutional neural network layer, and the spatial attention network layer.

[0115] See also Figure 5 , where block1 (i.e., module 1) is used to perform the first operation, each block from block2 to block5 is used to perform the second operation, and Concat and ConvS1 constitute the third operation. In block2 to block5, the input is processed by nonlinear activation functions, convolutional neural network layers, and spatial attention network layers, and then the shallow residual between the processing result and the input is determined, and then the residual is used as the input of the next block and the input of the final Concat operation. The shallow residual can be implemented based on the shallow residual network (Shallow Residual Block, SRB) in RFDN (Residual Feature Distillation Network) and similar methods. Compared with ordinary convolution, it can benefit from residual learning without introducing additional parameters, and the effect is excellent.

[0116] See also Figure 6 (The above Figure 5 The description also applies to Figure 6 The processing of the third stage of the super-resolution model can be similar to that of the second stage, but the third operation is replaced by the fourth operation, that is, the result of the first operation (that is, the output of block1) and the results of multiple second operations (that is, the outputs of block2 to block5) are concat, and then the channels are increased through ConvS2 convolution, and finally up-sampling is completed based on Pixel-Shuffle (pixel rearrangement), thereby realizing the super-resolution operation of the original image.

[0117] In an exemplary embodiment, the input at the beginning of the second stage can be the output of the first stage, which can be consistent with the shape (dimension) of the original image. The input first passes through block1, which includes: a layer of convolution (kernellsize1*1, inchannel 1, outchannel 64) to increase the number of channels, and then combines the CCA module with the SA module to improve the feature extraction capability; finally, another layer of convolution (kernelsize 2*2, inchannel 64, outchannel 64) is performed with an additional CCA module. The structures of block2 to block5 are consistent, and the convolutions therein can be implemented based on codes such as (kernelsize 1*1, inchannel 64, outchannel 64). Finally, after the outputs of block1 to block5 are Concat, they are output through the last layer of convolution ConvS1. ConvS1 can be implemented based on (kernelsize 1*1, inchannel 64*5, outchannel 1), and the activation function can be a tanh function.

[0118] The input from the third stage can be the output of the second stage, which is consistent with the shape (dimension) of the original image. The difference from the second stage structure is that the last convolution layer ConvS2 included in the fourth operation can be implemented based on (kernelsize 1*1, inchannel 64*5, outchannel N), where N is the square of the magnification (i.e. the target super-resolution multiple), and the reason why the "64" in the number of input channels has a coefficient of "5" is that Figure 6 The example shown has 5 blocks. After splicing the outputs of these blocks, a feature map with 64*5 channels may be obtained. Finally, Pixelshuffle is used to complete upsampling to obtain the final super-resolution image. The target super-resolution multiple usually refers to how many times the number of pixels of the super-resolution image is the number of pixels of the original image in each direction of the image (usually the horizontal and vertical directions) (for example, the aforementioned process of enlarging a picture with a resolution of 512×512 pixels to a picture with a resolution of 1024×1024 pixels can be regarded as a 2× super-resolution, that is, double super-resolution, and its target super-resolution multiple is 2).

[0119] The above architecture deployment can improve the model complexity and the number of channels as much as possible without affecting the LUT generation and LUT complexity, thereby enhancing the feature extraction capability. It should be understood that in the above descriptions, the specific parameter meanings and the principles for taking the values ​​of the optional parameters (such as the number of channels "64") can be found in the above description of the first stage. In addition, the present disclosure does not limit the specific number of executions of the first operation and the second operation. For example, the first operation can also be executed multiple times in the same stage (for example, there are multiple block1s), and the second operation can also be executed only once (for example, the Concat operation is directly executed after block2). Even the same stage can appear multiple times, for example, after deploying two second stages, the third stage is deployed; at the same time, the number of first operations and the number of second operations included in each second stage and the third stage can also be different, and are not limited to Figure 5 and Figure 6 The form shown.

[0120] In the implementation manner provided by step S101 and step S102, the input and output of the convolution operation contained in the deep learning model (super-resolution model) can be recorded by the LUT, so that the capabilities of the deep learning model can be introduced even if the device does not deploy the convolution layer of the deep learning model and its complex weights. At the same time, since the convolution kernel size of the up-channel convolution operation is larger than the convolution kernel size of the point convolution operation, and the up-channel convolution operation and the point convolution operation are performed linearly, the model has a strong feature-enhancing capability while its results can be split into multiple one-dimensional LUTs; in other words, the LUT table no longer needs to record the convolution results corresponding to various arrangement relationships of all pixels within the receptive field of the up-channel convolution operation, but only needs to record the convolution result corresponding to one pixel, so as to record the training results of the convolution operation of the deep learning model, thereby exponentially reducing the size of the LUT table, and enabling the simulated convolution operation based on the first LUT to achieve effects similar to those of the above-mentioned deep learning model, without occupying a large amount of storage space and retrieval computing power of the device, greatly improving the accuracy of the super-resolution algorithm and the visual effect of the super-resolution image.

[0121] In order to improve the training efficiency of the super-resolution model, reduce the resource usage of the super-resolution model, and improve the generation efficiency of LUT as much as possible, the super-resolution model provided by the present disclosure can be trained based on online distillation training or offline distillation training. Since the super-resolution model used for LUT has a smaller receptive field than other deep learning models (i.e., models that are not used for super-resolution or super-resolution based on LUT), the output of a feature layer in the commonly used teacher model is used to calculate the loss (i.e., loss calculation) of the feature layer with the same shape as the output of the student model. Therefore, optionally, the super-resolution model is also processed as follows: the super-resolution model is distilled as the decoder of the teacher model in the PISR method, wherein the PISR method is a privileged information method for super-resolution, and the encoder of the teacher model in the PISR method includes a downsampling algorithm.

[0122] In other words, the distillation model based on the above distillation training can be a PISR (Privileged Information for Super-Resolution) model (this statement can be understood as equivalent to "distillation based on the PISR method"). Figure 7 , which shows the overall architecture of the PISR model. In this distillation model, the Encoder can be regarded as a degenerate (which can be implemented based on any downsampling algorithm), and the super-resolution model provided in the present disclosure can be trained as the Decoder in the PISR model.

[0123] In an exemplary method, a Teacher model may be trained first, wherein the HR image (High Resolution image) is downsampled by the Encoder to obtain a Compact feature, and the Compact feature is super-resolved by the Decoder to obtain an HR image (High Resolution image). There may be two loss functions in the Teacher model, one of which may be an Imitation loss (which may be understood as determining whether the degradation simulation is similar), and the second may be a common reconstruction loss (reconstruction loss). Next, a Student model may be trained, wherein the LR image (Low Resolution image) is super-resolved to obtain an HR output (High Resolution output), and the loss function based thereon may also be two, one being a reconstruction loss (reconstruction loss) and the other being a distillation loss (distillation loss). The Estimator module can be an MLP (Multi-Layer Perceptron) model. After the corresponding feature layer passes through the Estimator module, μ and b can be obtained (these variables and module names are similar to Figure 7 ), corresponding to the location map and scalemap. The scale map reflects the role of adaptively controlling the weight of the distillation term in the loss function, and the log term is added to avoid trivial solutions. The closer the location map is to the feature map of the Teacher model, the more the Teacher's ability has been learned.

[0124] Based on this, in the three-layer scenarios involved in the above-mentioned present disclosure, the storage space occupied by the method, model, and LUT provided by the present disclosure, the execution efficiency, and the execution effect can all be taken into consideration.

[0125] The second aspect of the present disclosure provides a super-resolution model architecture, for example, the specific architecture of the super-resolution model described in the first aspect, and the model architecture may correspond to the attached Figure 4 To Attachment Figure 6 The content shown. The architecture may include:

[0126] An initial convolution layer, comprising a linearly arranged ascending channel convolution layer and a point convolution layer, wherein the convolution kernel size of the ascending channel convolution layer is larger than the convolution kernel size of the point convolution layer, and each weight of the convolution kernel of the point convolution layer corresponds one-to-one to that of the ascending channel convolution layer;

[0127] Among them, the initial convolution layer is used to generate multiple first LUTs, and the multiple first LUTs are used to simulate the initial convolution operation performed by the super-resolution model on the original image to obtain a super-resolution image. The multiple first LUTs include a one-dimensional LUT corresponding to each position of the convolution kernel of the up-channel convolution operation, and the one-dimensional LUT is used to record the mapping relationship between the pixel input and the point convolution result. The point convolution result is the product of the value of the pixel and the weight of the corresponding position in the convolution kernel of the up-channel convolution operation. The result is obtained by the point convolution operation.

[0128] Furthermore, the architecture may also include:

[0129] A first network layer, at least one second network layer, and a third network layer are sequentially deployed, wherein the first network layer includes a convolutional neural network layer, a spatial attention network layer, a contrast channel attention network layer, a convolutional neural network layer, and a spatial attention network layer, the second network layer includes a nonlinear activation function, a convolutional neural network layer, and a spatial attention network layer, which are sequentially deployed, and the third network layer includes a connection layer and a convolution layer which are sequentially deployed, and the connection layer is used to connect the outputs of the first network layer and the at least one second network layer;

[0130] The first network layer, at least one of the second network layers, and a fourth network layer are deployed in sequence, wherein the fourth network layer includes the connection layer and the upsampling layer deployed in sequence, and the upsampling layer is used to perform convolution operations and upsampling operations whose output channel number corresponds to the target super-resolution multiple.

[0131] For the specific explanation of the concepts in the above model architecture, optional details and corresponding beneficial effects, please refer to the relevant parts of other embodiments of the present disclosure, which will not be repeated here.

[0132] Corresponding to the embodiments of the aforementioned method, the present disclosure also provides embodiments of a device and a terminal to which the device is applied.

[0133] The third aspect of the present disclosure provides an image processing device. Figure 8 , the device comprises:

[0134] A simulation module 801 is used to simulate an initial convolution operation performed by a super-resolution model on an original image based on a plurality of first lookup tables LUT to obtain an intermediate image;

[0135] A super-resolution module 802 is used to perform the operation of the super-resolution model on the intermediate image to obtain a super-resolution image;

[0136] Among them, the initial convolution operation includes a channel-up convolution operation and a point convolution operation performed linearly in sequence, the convolution kernel size of the channel-up convolution operation is larger than the convolution kernel size of the point convolution operation, the point convolution operation corresponds one-to-one to each position of the convolution kernel of the channel-up convolution operation, the multiple first LUTs include a one-dimensional LUT corresponding one-to-one to each position of the convolution kernel of the channel-up convolution operation, the one-dimensional LUT is used to record the mapping relationship between the pixel input and the point convolution result, and the point convolution result is the product of the value of the pixel, the weight of the corresponding position in the convolution kernel of the channel-up convolution operation, and the weight of the convolution kernel of the point convolution operation.

[0137] Optionally, when the simulation module is used to simulate the initial convolution operation performed by the super-resolution model on the original image based on the multiple first lookup tables LUT, it is specifically used to:

[0138] Determine the receptive field pixel within the receptive field corresponding to each pixel in the original image, determine the point convolution result corresponding to the receptive field pixel in the first LUT corresponding to each receptive field pixel of the pixel, and sum the point convolution results corresponding to the receptive field pixels within the receptive field to obtain the simulated convolution operation result corresponding to the pixel, wherein the receptive field corresponding to the pixel is related to the convolution kernel size of the up-channel convolution operation.

[0139] Optionally, when the simulation module is used to simulate the initial convolution operation performed by the super-resolution model on the original image based on the multiple first lookup tables LUT to obtain the intermediate image, it is specifically used to:

[0140] Based on the multiple first LUTs, simulating multiple initial convolution operations performed by the super-resolution model on each pixel of the original image to obtain multiple simulated convolution operation results, wherein the convolution kernel size of the channel-up convolution operation included in each initial convolution operation is different;

[0141] The multiple simulated convolution operation results of each pixel in the original image are averaged to obtain the value of the pixel in the intermediate image.

[0142] Optionally, when the simulation module is used to average the results of multiple simulated convolution operations on each pixel in the original image to obtain the value of the pixel in the intermediate image, it is specifically used to:

[0143] The multiple simulated convolution operation results of each pixel in the original image and the value of the pixel in the original image are averaged to obtain the value of the pixel in the intermediate image.

[0144] Optionally, when the super-resolution module is used to perform the operation of the super-resolution model on the intermediate image, it is specifically used to:

[0145] sequentially performing a first operation, at least one second operation, and a third operation on the intermediate image;

[0146] Among them, the first operation is performed by a convolutional neural network layer, a spatial attention network layer, a contrast channel attention network layer, a convolutional neural network layer and a spatial attention network layer deployed in sequence, the second operation is performed by a nonlinear activation function, a convolutional neural network layer and a spatial attention network layer deployed in sequence, and the third operation includes connecting the outputs of the first operation and at least one of the second operations and then performing a convolution operation.

[0147] Optionally, after the super-resolution module is used to sequentially perform the first operation, at least one second operation, and the third operation on the intermediate image, it is further used to:

[0148] sequentially performing a first operation, at least one second operation, and a fourth operation on the intermediate image;

[0149] The fourth operation includes connecting the outputs of the first operation and the at least one second operation, and then performing a convolution operation and an upsampling operation whose output channel number corresponds to the target super-resolution multiple.

[0150] Optionally, when the super-resolution module is used to perform the operation of the super-resolution model on the intermediate image, it is specifically used to:

[0151] sequentially performing a first operation, at least one second operation, and a fourth operation on the intermediate image;

[0152] Among them, the first operation is performed by a convolutional neural network layer, a spatial attention network layer, a contrast channel attention network layer, a convolutional neural network layer and a spatial attention network layer deployed in sequence, the second operation is performed by a nonlinear activation function, a convolutional neural network layer and a spatial attention network layer deployed in sequence, and the fourth operation includes connecting the outputs of the first operation and at least one of the second operations, and then performing a convolution operation and an upsampling operation whose output channel number corresponds to the target super-resolution multiple.

[0153] Optionally, the output of each second operation includes:

[0154] The shallow residual of the input of the second operation and the result obtained after the input passes through the nonlinear activation function, the convolutional neural network layer and the spatial attention network layer.

[0155] Optionally, when the super-resolution module is used to perform the operation of the super-resolution model on the intermediate image, it is specifically used to:

[0156] Based on at least one second LUT, part or all of the operation of the super-resolution model on the intermediate image is simulated.

[0157] Optionally, the super-resolution model is further processed as follows:

[0158] The super-resolution model is distilled as a decoder of a teacher model in a PISR method, wherein the PISR method is a privileged information method for super-resolution, and the encoder of the teacher model in the PISR method includes a downsampling algorithm.

[0159] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0160] A fourth aspect of the present disclosure provides a computer program product, comprising a computer program / instruction, wherein when the computer program / instruction is executed by a processor, the method described in the first aspect is implemented.

[0161] For the device embodiments and computer program product embodiments, since they basically correspond to the method embodiments, the relevant parts can refer to the partial description of the method embodiments. In addition, the device embodiments described above are merely schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the disclosed solution. Those of ordinary skill in the art can understand and implement it without paying creative labor.

[0162] In a fifth aspect, the image processing apparatus provided by the present disclosure can be applied to a computer device. Fig. 9 , which exemplarily shows a hardware schematic diagram of a computer device. For example, device 900 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0163] Device 900 may include one or more of the following components: a processing component 901 , a memory 902 , a power component 903 , a multimedia component 904 , an audio component 905 , an input / output (I / O) interface 906 , a sensor component 907 , and a communication component 908 .

[0164] The processing component 901 generally controls the overall operation of the device 900, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 901 may include one or more processors 909 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 901 may include one or more modules to facilitate the interaction between the processing component 901 and other components. For example, the processing component 901 may include a multimedia module to facilitate the interaction between the multimedia component 904 and the processing component 901.

[0165] The memory 902 is configured to store various types of data to support operations on the device 900. Examples of such data include instructions for any application or method operating on the device 900, contact data, phone book data, messages, pictures, videos, etc. The memory 902 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0166] The power component 903 provides power to the various components of the device 900. The power component 903 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device 900.

[0167] The multimedia component 904 includes a screen providing an output interface between the device 900 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 904 includes a front camera and / or a rear camera. When the device 900 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0168] The audio component 905 is configured to output and / or input audio signals. For example, the audio component 905 includes a microphone (MIC), and when the device 900 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 902 or sent via the communication component 908. In some embodiments, the audio component 905 also includes a speaker for outputting audio signals.

[0169] I / O interface 906 provides an interface between processing component 901 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0170] The sensor assembly 907 includes one or more sensors for providing various aspects of status assessment for the device 900. For example, the sensor assembly 907 can detect the open / closed state of the device 900, the relative positioning of components, such as the display and keypad of the device 900, and the sensor assembly 907 can also detect the position change of the device 900 or a component of the device 900, the presence or absence of user contact with the device 900, the orientation or acceleration / deceleration of the device 900, and the temperature change of the device 900. The sensor assembly 907 can also include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 907 can also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 907 can also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0171] The communication component 908 is configured to facilitate wired or wireless communication between the device 900 and other devices. The device 900 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, 4G or 5G or a combination thereof. In an exemplary embodiment, the communication component 908 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 908 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0172] In an exemplary embodiment, the device 900 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to execute the image processing method of the above-mentioned computer device.

[0173] In a sixth aspect, the present disclosure, in an exemplary embodiment, further provides a non-transitory computer-readable storage medium including instructions, such as a memory 902 including instructions, and the instructions can be executed by a processor 909 of a device 900 to complete the image processing method of the computer device. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0174] The above describes specific embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0175] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the inventions claimed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not claimed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0176] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

[0177] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the scope of protection of the present disclosure.

Claims

1. An image processing method, characterized in that: The method comprises: Based on a plurality of first lookup tables LUT, simulating an initial convolution operation performed by a super-resolution model on an original image to obtain an intermediate image; Executing an operation of the super-resolution model on the intermediate image to obtain a super-resolution image; Among them, the initial convolution operation includes a channel-up convolution operation and a point convolution operation performed linearly in sequence, the convolution kernel size of the channel-up convolution operation is larger than the convolution kernel size of the point convolution operation, the point convolution operation corresponds one-to-one to each position of the convolution kernel of the channel-up convolution operation, the multiple first LUTs include a one-dimensional LUT corresponding one-to-one to each position of the convolution kernel of the channel-up convolution operation, the one-dimensional LUT is used to record the mapping relationship between the pixel value and the point convolution result, the point convolution result is the product of the pixel value, the weight of the corresponding position in the convolution kernel of the channel-up convolution operation, and the weight of the convolution kernel of the point convolution operation.

2. The image processing method according to claim 1, characterized in that: The method of simulating an initial convolution operation performed by a super-resolution model on an original image based on a plurality of first lookup tables LUTs comprises: Determine the receptive field pixel within the receptive field corresponding to each pixel in the original image, determine the point convolution result corresponding to the receptive field pixel in the first LUT corresponding to each receptive field pixel of the pixel, and sum the point convolution results corresponding to the receptive field pixels within the receptive field to obtain the simulated convolution operation result corresponding to the pixel, wherein the receptive field corresponding to the pixel is related to the convolution kernel size of the up-channel convolution operation.

3. The image processing method according to claim 1, characterized in that: The method of simulating an initial convolution operation performed by a super-resolution model on an original image based on a plurality of first lookup tables LUT to obtain an intermediate image includes: Based on the multiple first LUTs, simulating multiple initial convolution operations performed by the super-resolution model on each pixel of the original image to obtain multiple simulated convolution operation results, wherein the convolution kernel size of the channel-up convolution operation included in each initial convolution operation is different; The multiple simulated convolution operation results of each pixel in the original image are averaged to obtain the value of the pixel in the intermediate image.

4. The image processing method according to claim 3, characterized in that: The averaging of multiple simulated convolution operation results of each pixel in the original image to obtain the value of the pixel in the intermediate image includes: The multiple simulated convolution operation results of each pixel in the original image and the value of the pixel in the original image are averaged to obtain the value of the pixel in the intermediate image.

5. The image processing method according to claim 1, characterized in that: The performing of the operation of the super-resolution model on the intermediate image comprises: sequentially performing a first operation, at least one second operation, and a third operation on the intermediate image; Among them, the first operation is performed by a convolutional neural network layer, a spatial attention network layer, a contrast channel attention network layer, a convolutional neural network layer and a spatial attention network layer deployed in sequence, the second operation is performed by a nonlinear activation function, a convolutional neural network layer and a spatial attention network layer deployed in sequence, and the third operation includes connecting the outputs of the first operation and at least one of the second operations and then performing a convolution operation.

6. The image processing method according to claim 5, characterized in that: After sequentially performing the first operation, at least one second operation, and the third operation on the intermediate image, performing the operation of the super-resolution model on the intermediate image further includes: sequentially performing a first operation, at least one second operation, and a fourth operation on the intermediate image; The fourth operation includes connecting the outputs of the first operation and the at least one second operation, and then performing a convolution operation and an upsampling operation whose output channel number corresponds to the target super-resolution multiple.

7. The image processing method according to claim 1, characterized in that: The performing of the operation of the super-resolution model on the intermediate image comprises: sequentially performing a first operation, at least one second operation, and a fourth operation on the intermediate image; Among them, the first operation is performed by a convolutional neural network layer, a spatial attention network layer, a contrast channel attention network layer, a convolutional neural network layer and a spatial attention network layer deployed in sequence, the second operation is performed by a nonlinear activation function, a convolutional neural network layer and a spatial attention network layer deployed in sequence, and the fourth operation includes connecting the outputs of the first operation and at least one of the second operations, and then performing a convolution operation and an upsampling operation whose output channel number corresponds to the target super-resolution multiple.

8. The image processing method according to any one of claims 5 to 7, characterized in that: The output of each second operation includes: The shallow residual of the input of the second operation and the result obtained after the input passes through the nonlinear activation function, the convolutional neural network layer and the spatial attention network layer.

9. The image processing method according to any one of claims 1 to 7, characterized in that: The performing of the operation of the super-resolution model on the intermediate image comprises: Based on at least one second LUT, part or all of the operation of the super-resolution model on the intermediate image is simulated.

10. The image processing method according to claim 1, characterized in that: The super-resolution model is also processed as follows: The super-resolution model is distilled as a decoder of a teacher model in a PISR method, wherein the PISR method is a privileged information method for super-resolution, and the encoder of the teacher model in the PISR method includes a downsampling algorithm.

11. A super-resolution model architecture, characterized in that: The architecture includes: An initial convolution layer, comprising a linearly arranged ascending channel convolution layer and a point convolution layer, wherein the convolution kernel size of the ascending channel convolution layer is larger than the convolution kernel size of the point convolution layer, and each weight of the convolution kernel of the point convolution layer corresponds one-to-one to that of the ascending channel convolution layer; Among them, the initial convolution layer is used to generate multiple first LUTs, and the multiple first LUTs are used to simulate the initial convolution operation performed by the super-resolution model on the original image to obtain a super-resolution image. The multiple first LUTs include a one-dimensional LUT corresponding to each position of the convolution kernel of the up-channel convolution operation, and the one-dimensional LUT is used to record the mapping relationship between the pixel input and the point convolution result. The point convolution result is the product of the value of the pixel and the weight of the corresponding position in the convolution kernel of the up-channel convolution operation. The result is obtained by the point convolution operation.

12. The super-resolution model architecture according to claim 11, characterized in that The architecture also includes: A first network layer, at least one second network layer, and a third network layer are sequentially deployed, wherein the first network layer includes a convolutional neural network layer, a spatial attention network layer, a contrast channel attention network layer, a convolutional neural network layer, and a spatial attention network layer, the second network layer includes a nonlinear activation function, a convolutional neural network layer, and a spatial attention network layer, which are sequentially deployed, and the third network layer includes a connection layer and a convolution layer which are sequentially deployed, and the connection layer is used to connect the outputs of the first network layer and the at least one second network layer; The first network layer, at least one of the second network layers, and a fourth network layer are deployed in sequence, wherein the fourth network layer includes the connection layer and the upsampling layer deployed in sequence, and the upsampling layer is used to perform convolution operations and upsampling operations whose output channel number corresponds to the target super-resolution multiple.

13. An image processing device, characterized in that: The device comprises: A simulation module, used for simulating an initial convolution operation performed by a super-resolution model on an original image based on a plurality of first lookup tables LUTs to obtain an intermediate image; A super-resolution module, used for performing the operation of the super-resolution model on the intermediate image to obtain a super-resolution image; Among them, the initial convolution operation includes a channel-up convolution operation and a point convolution operation performed linearly in sequence, the convolution kernel size of the channel-up convolution operation is larger than the convolution kernel size of the point convolution operation, the point convolution operation corresponds one-to-one to each position of the convolution kernel of the channel-up convolution operation, the multiple first LUTs include a one-dimensional LUT corresponding one-to-one to each position of the convolution kernel of the channel-up convolution operation, the one-dimensional LUT is used to record the mapping relationship between the pixel input and the point convolution result, and the point convolution result is the product of the value of the pixel, the weight of the corresponding position in the convolution kernel of the channel-up convolution operation, and the weight of the convolution kernel of the point convolution operation.

14. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 10 is implemented.

15. A computer device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 10 is implemented.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images