Image super-resolution method and device based on multi-cascade single look-up table, and storage medium

Abstract

The invention discloses an image super-resolution method and device based on a multi-cascade single look-up table and a storage medium, and relates to the field of image processing, the image super-resolution method comprises an N-cascade look-up table LUT structure and an inference model, and the LUT structure stores a spatial mapping relation of searching a high-resolution HR image by using a low-resolution LR image; in the previous N-1 layers of cascaded LUT structures, the previous layer of LUT structure stores an index value for querying a mapping relation in the next layer of LUT structure, and the Nth layer of LUT structure stores a pixel value of an HR image; when an LR image is input into the reasoning model for reasoning query, the reasoning model performs step-by-step index query based on the N-layer LUT structure, and a corresponding HR pixel value is obtained from the Nth-layer LUT structure. According to the scheme, layer-by-layer searching is carried out in a multi-layer index mode, so that the LUT storage pressure is relieved, the weight of the channel is dynamically adjusted by using the RCAB, the network identification capability is improved, and the SR method indexing efficiency is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing, and in particular to an image super-resolution method, device and storage medium based on multi-stage single lookup tables. Background Art

[0002] With the continuous advancement of imaging science and technology, people have higher requirements for image quality. In practical applications, due to the limitations of imaging equipment and environment, or transmission bandwidth, there is a loss of detailed information in the image, and the resolution of the actual collected image is often low, which will affect the subsequent analysis and understanding of the image. The core goal of image super-resolution (SR) technology is to generate high-resolution (HR) images from low-resolution (LR) images to enhance the details and clarity of the image.

[0003] Related technologies provide a large number of image super-resolution methods, including interpolation-based methods, reconstruction-based methods, and learning-based methods, but their effects are not ideal. Interpolation-based SR methods are widely used because they are simple and fast, but they ignore the complex correlation between pixels, resulting in loss of details in the reconstructed image. Subsequently, sparse coding technology was introduced into the field of super-resolution, reconstructing HR images by learning sparse representations of LR image blocks, but the highly complex optimization operations required to calculate sparse representations have affected their processing speed and efficiency in practical applications to a certain extent. Deep neural networks have made significant progress in the field of SR due to their excellent flexibility and scalability. This technology adaptively learns the deep features of images through training sets, establishes a mapping relationship between high-resolution and low-resolution images, and retains more high-frequency feature information. However, the computational burden of deep networks increases, and they rely on high-performance hardware such as GPUs and NPUs. The high hardware cost and power consumption hinder their widespread application on edge devices.

[0004] Look-up Table (LUT) image super-resolution method has attracted much attention as a novel real-time super-resolution technology. This method pre-establishes the spatial mapping relationship between LR image blocks and HR image blocks through a neural network, and stores these mapping relationships in LUT. The HR image blocks corresponding to the LR image blocks are found by directly accessing the memory of the mobile device. The operation of device accessing memory replaces complex calculations, which significantly improves practicality. However, the storage of LUT is limited by the device memory. As the index dimension increases, the storage space required will increase exponentially. This growth limits the index capacity of LUT, which in turn limits the range of receptive fields that can be covered by the cacheable super-resolution network, and limits the improvement of super-resolution accuracy. Summary of the invention

[0005] The embodiments of the present application provide an image super-resolution method, device and storage medium based on multi-cascade single lookup tables, which solve the problem that the storage capacity of the lookup table image super-resolution limits the super-resolution accuracy.

[0006] On the one hand, the present application provides an image super-resolution method based on a multi-cascade single lookup table, the method is used to include an N-layer cascaded lookup table LUT structure and an inference model, the LUT structure stores a spatial mapping relationship of using a low-resolution LR image to find a high-resolution HR image;

[0007] In the first N-1 layers of cascaded LUT structures, the first layer LUT structure stores the index value for querying the mapping relationship in the second layer LUT structure, and the Nth layer LUT structure stores the pixel value of the HR image; N is a positive integer not less than 3;

[0008] When the LR image is input into the inference model for inference query, the inference model performs a level-by-level index query based on the N-level LUT structure, and obtains the corresponding HR pixel value from the N-th level LUT structure.

[0009] Specifically, the inference model queries the LUT structure based on the following index mechanism:

[0010] V=LUT (N) [LUT (N-1) […LUT (2) [LUT (1) (I LR) ]]]

[0011] V represents the HR pixel value obtained by the index mechanism, N represents the number of LUT structures, and I LR Represents the LR image of the input inference model, and the k-th layer table output value I * k As the k+1th layer LUT (k+1) Index input to the structure; k is a positive integer greater than zero.

[0012] Specifically, in the first N-1 layers of cascaded LUT structure, LUT (k) Structure and LUT (k+1) The index values ​​in each include several indexes, each of which includes a q-dimensional index group Index and a matching index value Value;

[0013] LUT (k) The q Values ​​in the structure constitute the LUT (k+1) An Index in a structure, and so on.

[0014] Specifically, the inference model calculates the LUT structure with 2 4Uniform sampling storage; storage space capacity S of the first N-1 layer LUT structure i It is expressed as:

[0015] S i =n×(2 4 +1) n Byte

[0016] Where i represents the i-th layer LUT structure, and i is a positive integer not greater than N-1, and n represents the dimension of the LUT structure:

[0017] Nth layer LUT structure LUT (N) Storage capacity S N It is expressed as:

[0018] S N =(2 4 +1) n × 2 Byte

[0019] The r here represents the magnification factor from the LR image to the HR image.

[0020] Specifically, the network structure of the inference model includes a cascaded first convolution block, a second convolution block, a multi-scale feature enhancement MFEM module and a sub-pixel upsampling module; the first convolution block and the second convolution block use the small receptive field of the convolution layer to extract low-dimensional features; the MFEM module is used to extract feature information of different scales and extract high-dimensional features; the sub-pixel upsampling module merges low-resolution channels into high-resolution channels to improve image resolution.

[0021] Specifically, the first convolution block and the second convolution block respectively include one layer of 2×2 convolution and five layers of 1×1 convolution, which are used to generate feature maps in the mapping block on the convolution layer. The generated feature mapping relationship is expressed as:

[0022] F1=Conv(I LR )

[0023] F2=Conv(F1)

[0024] F1 and F2 represent the convolution outputs of the first and second convolution blocks respectively, Conv represents the convolution activation operation of the convolution block, and the LUT structure is established according to the mapping relationship between the input and output of the mapping block.

[0025] Specifically, the MFEM module includes a cascaded asymmetric convolution feature fusion block ACFB, a g-layer residual channel attention block RCAB, and a 1×1 convolution layer; a long skip connection is set between the ACFB output and the 1×1 convolution layer input:

[0026] The ACFB performs multi-scale shallow feature extraction and feature fusion splicing on the input feature image; the RCAB is used to identify the difference in feature importance between different channels and capture the high-frequency information of the image; the 1×1 convolution layer is used to reduce the dimension of the output feature map.

[0027] Specifically, the ACFB includes three asymmetric convolution layers, namely, 1×2 horizontal convolution C hor , 2×1 vertical convolution C vert And the 2×2 main diagonal convolution C diag ; Each asymmetric convolution is cascaded with a normalized BN layer and an activation layer; the outputs of the three activation layers are concatenated and fused through the connection layer Concat, which is expressed as follows:

[0028]

[0029] Where Cat[·] represents the concatenation operation along the channel dimension;

[0030] The g-th layer RCAB feature recognition H g It is expressed as follows:

[0031] F g =H g (F g-1 )=H g (H g-1 (…H1(F ACFB )…))

[0032] Among them, H g (·) represents the RCAB function of the g-th layer, F g-1 and F g represents the input and output of the g-th RCAB, and the long skip connection connects the shallow feature F ACFB With the deep feature F g Fusion;

[0033] The sub-pixel upsampling output result S is expressed as follows:

[0034] S=ps[C 1×1 (F ACFB +F g )]

[0035] Among them, C 1×1 represents 1×1 convolution, and ps represents sub-pixel operation.

[0036] On the other hand, the present application provides a computer device, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by the processor to implement the image super-resolution method based on multi-cascade single lookup table described in the above aspect.

[0037] On the other hand, the present application provides a computer-readable storage medium, which stores at least one instruction, at least one program, a code set or an instruction set. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the image super-resolution method based on multi-cascade single lookup table described in the above aspect.

[0038] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

[0039] The cascaded lookup table structure is used to store low-resolution and high-resolution index parameter information level by level, and the image blocks are searched by directly accessing the memory of the mobile device, which significantly improves the access speed.

[0040] A multi-layer index is used to search layer by layer. The index value of each level of LUT is used as an index dimension parameter of the next level, which relieves the LUT storage pressure and increases the space linearly with the expansion of the receptive field.

[0041] RCAB is introduced to dynamically adjust the weights of channels and improve the network's discrimination ability. Multi-layer stacking can improve the performance of the model and optimize feature representation. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 Schematic diagram of the LUT structure of the image super-resolution indexing method based on multi-cascade single lookup table provided in an embodiment of the present application;

[0043] Figure 2 A schematic diagram of the network structure of the inference model provided in an embodiment of the present application is shown;

[0044] Figure 3 A schematic diagram of the network structure of ACFB provided in an embodiment of the present application is shown;

[0045] Figure 4 A comparison of the visual perception quality of images with rich texture and edge features in various datasets is shown;

[0046] Figure 5 A structural block diagram of a computer device provided by an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0047] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0048] The term "multiple" as used herein refers to two or more than two. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.

[0049] In response to the defects or shortcomings of the lookup table image super-resolution method in the related art, the design focus of this application is to linearly increase the LUT size to expand the receptive field while maintaining real-time inference speed to reconstruct richer image details, thereby promoting the real-time application of image super-resolution technology on edge devices.

[0050] Figure 1 It is a schematic diagram of the LUT structure of the image super-resolution indexing method based on a multi-cascade single lookup table provided in an embodiment of the present application, which includes an N-layer cascaded lookup LUT structure and an inference model. When it is necessary to use the LR image to execute the image super-resolution method to query the HR image, the LR image will be input into the inference model. The inference model performs a level-by-level index query based on the N-layer LUT structure to obtain the corresponding HR pixel value from the N-th layer LUT structure.

[0051] The LUT structure stores the spatial mapping relationship data for searching high-resolution HR images using low-resolution LR images. In the first N-1 cascaded LUT structures, the first LUT structure stores the index value for querying the mapping relationship in the next LUT structure, and the Nth LUT structure stores the pixel value of the HR image; N is a positive integer not less than 3. The inference model queries the LUT structure based on the following index mechanism:

[0052] V=LUT (N) [LUT (N-1) […LUT (2) [LUT (1) (I LR )]]]

[0053] V represents the HR pixel value obtained by the index mechanism, N represents the number of LUT structures, and I LR Represents the LR image of the input inference model, and the k-th layer table output value I * k As the k+1th layer LUT (k+1) Index input to the structure; k is a positive integer greater than zero.

[0054] Figure 1Take the three-layer LUT structure as an example. The first two LUTs store index values, and the third LUT stores pixel values. In the actual reasoning process, multi-layer indexing is used to search layer by layer. This method expands the receptive field to 27×27, so that the LUT storage space grows linearly with the expansion of the receptive field.

[0055] Because the pixels of an image are usually 2 8 In order to improve the storage and sampling efficiency, this application uses the inference model to represent the LUT structure in 2 4 Uniform sampling storage, so for the first N-1 layers of LUT structure, its storage space capacity is S j It is expressed as:

[0056] S i =n×(2 4 +1) n Byte

[0057] Here, i represents the i-th layer LUT structure, and i is a positive integer not greater than N-1, and n represents the dimension of the LUT structure.

[0058] For the last layer, that is, the Nth layer LUT structure LUT (N) Storage capacity S N It is expressed as:

[0059] S N =(2 4 +1) n × 2 Byte

[0060] Where r represents the magnification factor from the LR image to the HR image. The storage pressure is relieved by the addition of the index LUT structure. According to the above design, if the image is magnified 4 times, the total storage space of the four-dimensional MFLUT is only 1.912MB, which can greatly reduce the storage space.

[0061] In order to obtain a three-level LUT, this embodiment designs a training network including a two-stage convolution block and a third-stage multi-scale feature enhancement module (MFEM), which is used to learn the complex mapping relationship between low-resolution and high-resolution images, and trains a LUT structure in each stage.

[0062] See also Figure 2 As shown, the network structure of this inference model includes the first convolutional block Con v Block, the second convolution block Con vBlock, multi-scale feature enhancement MFEM module and sub-pixel upsampling module. The first convolution block and the second convolution block are used to extract low-dimensional features using the small receptive field of the convolution layer. The MFEM module is used to extract feature information of different scales and extract high-dimensional features. The multi-scale feature enhancement module can improve the performance of target detection and recognition tasks by fusing information of different scales, making the model more robust and accurate. This module can capture the details and features of the target object at different scales, thereby improving the recognition ability of the model. In addition, the features provide different contextual information at different scales, which helps the model better understand the image content. The sub-pixel upsampling module can improve the resolution of the image through algorithm processing without increasing the number of physical pixels by utilizing the sub-pixel information in the image. This technology is based on the fact that in the imaging process, due to the capacity limitation of the photosensitive element, each pixel only represents the nearby color, and there are countless tiny information between two actual physical pixels. These pixels between two actual physical pixels are called "sub-pixels". The sub-pixel upsampling module calculates and processes these sub-pixel information through software algorithms, thereby improving the display quality and resolution of the image without increasing the hardware cost. This application mainly utilizes a sub-pixel upsampling module to merge low-resolution channels into high-resolution channels to improve image resolution.

[0063] The internal structure of the first convolution block and the second convolution block of the present application is the same, such as Figure 2 As shown in the figure, it contains one layer of 2×2 convolution and five layers of 1×1 convolution, which are used to generate feature maps in the mapping block on the convolution layer. Since the cascaded convolution block structure is limited by a very small receptive field, a too deep network cannot infinitely improve the accuracy. Therefore, only a convolution block consisting of six layers of convolution is used, and the generated feature mapping relationship is expressed as:

[0064] F1=Conv(I LR )

[0065] F2=Conv(F1)

[0066] F1 and F2 represent the convolution outputs of the first and second convolution blocks respectively, Conv represents the convolution activation operation of the convolution block, and except for the last convolution layer, the ReLU activation function is used after each convolution to limit the image value to the range of [0, ∞] to prevent gradient explosion and gradient disappearance problems in the network. The LUT structure is established based on the mapping relationship between the input and output of these mapping blocks.

[0067] In the subsequent stage of the convolution block, as the receptive field increases, it is necessary to extract feature information of different scales and give higher attention to high-frequency information, which requires the use of the MFEM module. In order to obtain the LUT network structure based on the MFEM module (hereinafter referred to as the MFEM LUT network), the training network passes the output value to each level of LUT in turn. This application designs this part of the MFEM module as follows:

[0068] See also Figure 2 The network structure of the MFEM module includes an asymmetric convolutional feature fusion block ACFB, a g-layer residual channel attention block RCAB, and a 1×1 convolutional layer. The output of the convolutional block is connected to ACFB, the output of ACFB is connected to the g-layer RCAB in turn, and finally the output of the g-th layer RCAB is connected to the 1×1 convolutional layer. In order to achieve the fusion output of the deep and shallow features of the module, a long jump connection is set between the first layer RCAB input and the g-th layer RCAB output. In this structure, ACFB performs multi-scale shallow feature extraction and feature fusion splicing on the input feature image. RCAB is used to identify the difference in feature importance between different channels and capture the high-frequency information of the image. The 1×1 convolutional layer is used to reduce the dimension and output the feature map.

[0069] Figure 3 is a schematic diagram of the network structure of ACFB. In this embodiment, ACFB is particularly used for multi-scale shallow feature extraction and fusion splicing to extract features from multiple spatial dimensions. It specifically includes three asymmetric convolution layers, namely, 1×2 horizontal convolution C hor , 2×1 vertical convolution C vert And the 2×2 main diagonal convolution C diag Each asymmetric convolution is cascaded with a normalized BN layer and an activation layer. The outputs of the three activation layers are concatenated and fused through the connection layer Concat, which is expressed as follows:

[0070]

[0071] The Cat[·] represents the concatenation operation along the channel dimension; F2 is the output of the second convolutional block. The three asymmetric convolutions convolve the inputs separately to construct receptive fields of different shapes, which helps to alleviate the problem caused by the mismatch between the receptive field and the uniform region shape, especially for areas close to the object boundary or elongated areas corresponding to fine structures.

[0072] In order to consider the interdependence between channels, this application specifically introduces RCAB to dynamically adjust the weights of channels and improve the network's discrimination ability. RCAB combines residual and channel attention, and the stacking of multiple layers can improve the performance of the model and optimize feature representation.

[0073] The g-th layer RCAB feature recognition H g It is expressed as follows:

[0074] F g =H g (F g-1 )=H g (H g-1 (…H1(F ACFB )…))

[0075] Among them, H g (·) represents the RCAB function of the g-th layer, F g-1 and F g represents the input and output of the g-th RCAB, and the long skip connection connects the shallow feature F ACFB With the deep feature F g Fusion;

[0076] The sub-pixel upsampling output result S is expressed as follows:

[0077] S=ps[C 1×1 (F ACFB +F g )]

[0078] Among them, C 1×1 represents 1×1 convolution, and ps represents sub-pixel operation.

[0079] In summary, the beneficial effects achieved by this application include the following:

[0080] The cascaded lookup table structure is used to store low-resolution and high-resolution index parameter information level by level, and the image blocks are searched by directly accessing the memory of the mobile device, which significantly improves the access speed.

[0081] A multi-layer index is used to search layer by layer. The index value of each level of LUT is used as an index dimension parameter of the next level, which relieves the LUT storage pressure and increases the space linearly with the expansion of the receptive field.

[0082] RCAB is introduced to dynamically adjust the weights of channels and improve the network's discrimination ability. Multi-layer stacking can improve the performance of the model and optimize feature representation.

[0083] Based on the above network structure, the model is trained using the DIV2K dataset, which is widely used in SR tasks, to train the MFLUT network. The DIV2K dataset contains 800 2K resolution training images, covering a variety of content from urban landscapes to natural landscapes. In the experiment, the magnification factor is fixed to ×4, and the model uses the Adam optimizer with a learning rate of 1×10 -4 Perform 2×10 5The training was performed with a batch size of 32 and the mean square error loss function was used as the optimization target. The experimental platform was Windows, and the Nvidia GeForce RTX 4070 was used to train the model. Nvidia CUDA 11.1 and cuDNN 8.0 were used to accelerate the GPU training speed. The network structure was implemented on the PyTorch framework. The cached LUT was spaced at intervals of 2. 4 Uniform sampling is used to reduce storage space, that is, from LUT [0-255] to LUT [0-15]. After locating the coordinates, the final prediction result is obtained by tetrahedral interpolation. Due to the use of the cascade strategy, the LUT results of the first and second stages are quantized into integers for easy indexing, and a perceptual fine-tuning strategy is used to alleviate the information loss caused by interpolation.

[0084] To verify the actual performance of this method, the public benchmark test sets of Set5, Set14, BSD100 and Urban100 were used to evaluate the performance. The Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) were used as objective evaluation criteria. The higher the PSNR, the better the network denoising effect. The higher the SSIM and the closer it is to 1, the closer the denoised image is to the original clean image.

[0085] MFLUT is quantitatively compared with a variety of typical lightweight super-resolution methods, as shown in Table 1, including 3 interpolation-based methods (Nearest, Bilinear, and Bicubic), 4 sparse coding methods (NE+LLE, Zeyde, ANR, and A+), 2 DNN-based methods (FSRCNN, CARN-M), and 4 LUT-based methods (SR-LUT, MuLUT, SPLUT, and RCLUT). All sparse coding-based methods use the same dictionary, FSRCNN is one of the fastest DNN models, and CARN-M is a deep neural network (DNN) model that balances speed and performance.

[0086] Table 1 Comparison results of objective evaluation indicators of different methods under ×4 magnification factor

[0087]

[0088] The quantitative results are shown in Table 1, where the optimal values ​​in the LUT method are marked in bold. Compared with other LUT-based and sparse coding methods, the proposed model has achieved significant performance improvements on multiple datasets, especially on the Set5 dataset, where the PSNR of the LUT method is improved by 0.14dB. Even compared with DNN-based methods, MFLUT has shown strong competitiveness. The MFLUT model not only performs well in performance, but also maintains a similar level of storage efficiency to the high-performance RCLUT, increasing the storage requirement by only 0.399MB, which is a relatively small increase and shows excellent cost performance.

[0089] For lightweight SR methods, inference time is an important factor in model efficiency. This scheme compares MFLUT with other lightweight methods. Considering the convenience of LUT method on edge device deployment, the experiment tests the running time on a mobile phone equipped with Qualcomm Snapdragon 8Gen2 CPU. The official MATLAB implementation of the sparse coding method was tested on a personal computer (PC) with i5-13600KF CPU, and the DNN method was implemented in the CPU version of the PyTorch library. In the experiment, a magnification factor of ×4 was used to generate 1280×720 high-definition images. The results are shown in Table 2. Even if the sparse coding and DNN methods are calculated on a more powerful CPU, the running efficiency of LUT still has a significant advantage thanks to the unique inference mechanism of table lookup. The multi-level single table lookup MFLUT has more advanced running efficiency than the multi-table lookup MuLUT and SPLUT, thanks to the fact that a single LUT does not require averaging operations and fewer lookups, and the inference time is comparable to that of RCLUT, with a difference of less than 4%.

[0090] Table 2 Comparison of reasoning time of different methods

[0091]

[0092] Finally, the visual effects of MFLUT are compared with those of lightweight super-resolution methods Bicubic, SR-LUT, A+, RC-LUT and FSRCNN, which have excellent objective evaluation indicators, to illustrate the advantages of MFLUT in restoring details. Figure 4The images with rich texture and edge features in the Set5, Set14, BSD100 and Urban100 benchmark datasets are shown for comparison of visual perception quality. From the comparison results of Set5 and Set14, it can be seen that in the local areas of butterfly wings and zebra stripes, the output of Bicubic has serious artifact blurring, SR-LUT produces mild artifacts, and the results of A+, RC-LUT and FSRCNN show that the artifacts are improved, but the stripe detail information is still lost, while MFLUT better alleviates the artifacts and retains more dark details of the wings and more subtle stripe information. In the other two examples, it can be observed that this method restores more delicate penguin hair texture and more natural line edges compared to other methods. Therefore, this model has obvious advantages in preserving image detail texture. These visual effects verify the excellent performance of MFLUT in utilizing multi-scale feature fusion.

[0093] The image super-resolution device based on multi-level single lookup table provided in the embodiment of the present application can be applied to the image super-resolution method based on multi-level single lookup table provided in the above embodiment. For relevant details, refer to the above method embodiment. The implementation principle and technical effect are similar and will not be repeated here.

[0094] It should be noted that the image super-resolution device based on multi-level single lookup provided in the embodiment of the present application is only illustrated by the division of the above-mentioned functional modules / functional units. In actual applications, the above-mentioned functions can be assigned to different functional modules / functional units as needed, that is, the internal structure of the image super-resolution device based on multi-level single lookup is divided into different functional modules / functional units to complete all or part of the functions described above. In addition, the implementation method of the image super-resolution method based on multi-level single lookup provided in the above method embodiment and the implementation method of the image super-resolution device based on multi-level single lookup provided in this embodiment belong to the same concept. The specific implementation process of the image super-resolution device based on multi-level single lookup provided in this embodiment is detailed in the above method embodiment, and will not be repeated here.

[0095] Figure 5The block diagram of a computer device provided by an exemplary embodiment of the present application is shown. It is a computer device such as a desktop computer, a laptop computer, a PDA, and a cloud server. The computer device may include, but is not limited to, a processor and a memory. Among them, the processor and the memory may be connected by a bus or in other ways. Among them, the processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, graphics processors (GPU), embedded neural network processors (NPU) or other dedicated deep learning coprocessors, discrete gates or transistor logic devices, discrete hardware components and other chips, or a combination of the above-mentioned various chips.

[0096] The processor may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content that needs to be displayed on the display screen. In some embodiments, the processor may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0097] As a non-transient computer-readable storage medium, the memory can be used to store non-transient software programs, non-transient computer executable programs and modules, such as program instructions / modules corresponding to the methods in the above-mentioned embodiments of the present application. The processor executes various functional applications and data processing of the processor by running the non-transient software programs, instructions and modules stored in the memory, that is, the method in the above-mentioned method implementation is realized. The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0098] In some embodiments, the computer device may further optionally include: a peripheral device interface and at least one peripheral device. The processor, the memory and the peripheral device interface may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit, a display screen and a keyboard.

[0099] The peripheral device interface can be used to connect at least one peripheral device related to I / O (Input / Output) to the processor and the memory. In some embodiments, the processor, the memory, and the peripheral device interface are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor, the memory, and the peripheral device interface can be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0100] The display screen is used to display the UI (User Interface). The UI may include graphics, text, icons, videos and any combination thereof. When the display screen is a touch screen, the display screen also has the ability to collect touch signals on the surface or above the surface of the display screen. The touch signal can be input to the processor as a control signal for processing. At this time, the display screen can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen can be one, arranged on the front panel of the computer device; in other embodiments, the display screen can be at least two, respectively arranged on different surfaces of the computer device or in a folding design; in other embodiments, the display screen can be a flexible display screen, arranged on a curved surface or a folding surface of the computer device. Even, the display screen can be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode) and the like.

[0101] The power supply is used to power various components in the computer device. The power supply can be AC, DC, a disposable battery, or a rechargeable battery. When the power supply includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged through a wired line, and a wireless rechargeable battery is a battery that is charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0102] Those skilled in the art will appreciate that the structure shown in this embodiment does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0103] The embodiment of the present application also discloses a computer-readable storage medium. Specifically, the computer-readable storage medium is used to store a computer program, and when the computer program is executed by the processor, the method in the above-mentioned method implementation is implemented. Those skilled in the art can understand that the implementation of all or part of the process in the above-mentioned implementation method of the present application can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the process of the implementation of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (Hard Disk Drive, abbreviated as: HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0104] This specific embodiment is merely an explanation of the present invention and is not a limitation of the present invention. After reading this specification, those skilled in the art may make non-creative modifications to the present embodiment as needed. However, as long as they are within the scope of the claims of the present invention, they are protected by the patent law.

Claims

1. An image super-resolution indexing method based on multi-level single lookup tables, characterized in that: The method is used for a lookup table LUT structure and an inference model including N layers of cascades, wherein the LUT structure stores a spatial mapping relationship of using a low-resolution LR image to find a high-resolution HR image; In the first N-1 layers of cascaded LUT structures, the first layer LUT structure stores the index value for querying the mapping relationship in the second layer LUT structure, and the Nth layer LUT structure stores the pixel value of the HR image; N is a positive integer not less than 3; When the LR image is input into the inference model for inference query, the inference model performs a level-by-level index query based on the N-level LUT structure, and obtains the corresponding HR pixel value from the N-th level LUT structure.

2. The method according to claim 1, characterized in that: The inference model queries the LUT structure based on the following indexing mechanism: V=LUT (N) [LUT (N-1) […LUT (2) [LUT (1) (I LR )]]] V represents the HR pixel value obtained by the index mechanism, N represents the number of LUT structures, and I LR Represents the LR image of the input inference model, and the k-th layer table output value I * k As the k+1th layer LUT (k+1) Index input to the structure; k is a positive integer greater than zero.

3. The method according to claim 2, characterized in that In the first N-1 layers of cascaded LUT structure, LUT (k) Structure and LUT (k+1) The index values ​​in each include several indexes, each of which includes a q-dimensional index group Index and a matching index value Value; LUT (k) The q Values ​​in the structure constitute the LUT (k+1) An Index in a structure, and so on.

4. The method according to claim 3, characterized in that The inference model uses 2 4 Uniform sampling storage; storage space capacity S of the first N-1 layer LUT structure i It is expressed as: S i =n×(2 4 +1) n Change Where i represents the i-th layer LUT structure, and i is a positive integer not greater than N-1, and n represents the dimension of the LUT structure; Nth layer LUT structure LUT (N) Storage capacity S N It is expressed as: S N =(2 4 +1) n ×r 2 Change The r here represents the magnification factor from the LR image to the HR image.

5. The method according to any one of claims 1 to 4, characterized in that: The network structure of the inference model includes a cascaded first convolution block, a second convolution block, a multi-scale feature enhancement MFEM module and a sub-pixel upsampling module; the first convolution block and the second convolution block use the small receptive field of the convolution layer to extract low-dimensional features; the MFEM module is used to extract feature information of different scales and extract high-dimensional features; the sub-pixel upsampling module merges low-resolution channels into high-resolution channels to improve image resolution.

6. The method according to claim 5, characterized in that The first convolution block and the second convolution block respectively include one layer of 2×2 convolution and five layers of 1×1 convolution, which are used to generate feature maps in the mapping block on the convolution layer. The generated feature mapping relationship is expressed as: F1=Conv(I LR ) F2=Conv(F1) F1 and F2 represent the convolution outputs of the first and second convolution blocks respectively, Conv represents the convolution activation operation of the convolution block, and the LUT structure is established according to the mapping relationship between the input and output of the mapping block.

7. The method according to claim 6, characterized in that The MFEM module comprises a cascaded asymmetric convolution feature fusion block ACFB, a g-layer residual channel attention block RCAB, and a 1×1 convolution layer; a long skip connection is set between the ACFB output and the 1×1 convolution layer input; The ACFB performs multi-scale shallow feature extraction and feature fusion splicing on the input feature image; the RCAB is used to identify the difference in feature importance between different channels and capture the high-frequency information of the image; the 1×1 convolution layer is used to reduce the dimension of the output feature map.

8. The method according to claim 7, characterized in that The ACFB includes three asymmetric convolutional layers, namely 1×2 horizontal convolution C hor , 2×1 vertical convolution C vert And the 2×2 main diagonal convolution C diag ; Each asymmetric convolution is cascaded with a normalized BN layer and an activation layer; the outputs of the three activation layers are concatenated and fused through the connection layer Concat, which is expressed as follows: Where Cat[·] represents the concatenation operation along the channel dimension; The g-th layer RCAB feature recognition H g It is expressed as follows: F g =H g (F g-1 )=H g (H g-1 (…H1(F ACFB )…)) Among them, H g (·) represents the RCAB function of the g-th layer, F g-1 and F g represents the input and output of the g-th RCAB, and the long skip connection connects the shallow feature F ACFB With the deep feature F g Fusion; The sub-pixel upsampling output result S is expressed as follows: S=ps[C 1×1 (F ACFB +F g )] Among them, C 1×1 represents 1×1 convolution, and ps represents sub-pixel operation.

9. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by the processor to implement the image super-resolution method based on a multi-cascade single lookup table as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by the processor to implement the image super-resolution method based on multi-cascade single lookup table as described in any one of claims 1 to 8.

Images