Lookup table compression and image restoration method, and related system, device and medium

Abstract

The application discloses a lookup table compression and image recovery method, and related systems, devices and media, which are corresponding solutions, and in the solutions: without losing the image recovery performance of the lookup table method, a large amount of redundant storage space of the lookup table is removed, excellent lookup table methods can meet strict storage resource requirements of various edge devices, and the lookup table method is more easily deployed on the edge devices; the compression principle is based on the objective fact of a natural image in an image recovery task, that is, an accessed area of the lookup table is concentrated in a local area, so that the compression method can greatly compress the storage space of the lookup table, and can also ensure that the performance of the compressed lookup table is basically unchanged when the compressed lookup table is used for image recovery. Moreover, the compression of the lookup table method used in the image recovery task has a certain universality, and can be more easily deployed on various edge devices with limited storage resources.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a lookup table compression and image restoration method, as well as related systems, devices and media. Background Technology

[0002] Image restoration techniques, including super-resolution, denoising, deblocking, and deblurring, aim to improve image quality by increasing resolution, reducing noise, and removing defects such as fast effects and blurring. In recent years, with the development of deep learning, many methods based on deep neural networks have achieved significant image restoration results. However, these deep neural network-based methods often have a large number of learnable parameters due to their complex neural network structures, leading to heavy computational and storage costs.

[0003] To avoid the computational and storage costs associated with the large number of learnable parameters in deep neural networks, image restoration methods based on lookup table techniques have received widespread attention. A lookup table is a data structure consisting of index-value pairs. It uses the pixel values ​​of all possible input images as indices to retrieve the corresponding values ​​stored in the lookup table as output. This lookup table-based approach aims to quickly achieve nonlinear mapping between pixel values ​​in an image using a "space-for-time" strategy. In image super-resolution tasks, Younghyun Jo and Seon Joo Kim of Yonsei University designed a lookup table method for image super-resolution (Jo Y, Kim S J. Practical single-image super-resolution using look-up table[C] / / Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition. 2021: 691-700.). This method (abbreviated as SR-LUT model) uses local low-resolution image patches composed of spatially adjacent pixels as coordinate indices to retrieve the corresponding high-resolution image patches cached in the lookup table; however, this method is limited by a very small receptive field and cannot achieve good image restoration results. To address the problem of limited receptive field, Li Jiacheng et al. proposed a method using multiple look-up tables in a cascaded manner (Li J, Chen C, Cheng Z, et al. Mulut: Cooperating multiple look-up tables for efficient image super-resolution[C] / / European conference on computer vision. Cham: Springer Nature Switzerland, 2022: 238-256.). Although this method (abbreviated as MuLUT model) expands the receptive field by cascading multiple look-up tables and thus achieves higher image restoration performance, the cascaded multiple look-up tables have a large amount of redundant storage space, which increases storage costs.

[0004] It is evident that existing image restoration methods based on lookup table technology cannot achieve a trade-off between storage size and image restoration performance. High-performance lookup table methods often have higher storage sizes, which makes them difficult to deploy on edge devices with limited storage space.

[0005] In view of this, the present invention is hereby proposed. Summary of the Invention

[0006] The purpose of this invention is to provide a lookup table compression and image restoration method, as well as related systems, devices and media. By removing a large amount of redundancy in the lookup table, the high-performance lookup table method can meet the stringent storage size requirements of different edge devices while achieving an efficient image restoration process.

[0007] The objective of this invention is achieved through the following technical solution:

[0008] A lookup table compression method, comprising:

[0009] Obtain the uncompressed lookup table for image restoration;

[0010] Traverse all indexes of the uncompressed lookup table and determine whether the corresponding lookup table cell satisfies the set diagonal condition based on the index.

[0011] If yes, the lookup table cells that satisfy the diagonal condition are saved to the first new lookup table, which is called the diagonal reindexed lookup table; if no, the lookup table cells that do not satisfy the diagonal condition are sampled at equal intervals, and the values ​​in the sampled lookup table cells are saved to the second new lookup table, which is called the off-diagonal subsampled lookup table.

[0012] An image restoration method, comprising:

[0013] Based on the aforementioned lookup table compression method, the lookup table is compressed to obtain a diagonal reindexed lookup table and an off-diagonal subsampled lookup table.

[0014] When retrieving a lookup table using the input index, it is determined whether the corresponding lookup table cell meets the set diagonal condition based on the input index. If so, the input index is converted into an index of the diagonal reindexed lookup table through index mapping for retrieval. If not, the input index is scaled and then retrieved in the off-diagonal subsampled lookup table.

[0015] Image restoration is performed using the retrieved values.

[0016] A lookup table compression system for implementing the aforementioned lookup table compression method includes:

[0017] The lookup table acquisition unit is used to acquire an uncompressed lookup table for image restoration;

[0018] The diagonal condition judgment unit is used to traverse all indexes of the uncompressed lookup table and determine whether the corresponding lookup table cell meets the set diagonal condition based on the index.

[0019] The first storage unit is used to save the lookup table unit that meets the diagonal condition into the first new lookup table when the set diagonal condition is met. The first new lookup table is called the diagonal reindexed lookup table.

[0020] The second storage unit is used to perform equal-interval sampling on lookup table units that do not meet the set diagonal conditions when the diagonal conditions are not met. The values ​​in the sampled lookup table units are saved to a second new lookup table, which is called a non-diagonal sub-sampling lookup table.

[0021] An image restoration system, comprising:

[0022] A lookup table compression system is used to implement the aforementioned lookup table compression method to obtain a diagonal reindexed lookup table and a non-diagonal subsampled lookup table.

[0023] The image restoration unit is used to determine whether the corresponding lookup table unit meets the set diagonal condition when using the input index to retrieve the lookup table. If yes, the input index is converted into the index of the diagonal reindexed lookup table through index mapping for retrieval. If no, the input index is scaled and then retrieved in the off-diagonal subsampled lookup table. The retrieved value is then used to restore the image.

[0024] A processing device includes: one or more processors; and a memory for storing one or more programs;

[0025] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0026] A readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method.

[0027] As can be seen from the technical solution provided by the present invention, without sacrificing the image restoration performance of the lookup table method, by removing a large amount of redundant storage space of the lookup table, the excellent lookup table method can meet the strict storage resource requirements of various edge devices, making the lookup table method easier to deploy on these edge devices. The compression principle of the present invention is based on the objective fact of natural images in image restoration tasks, namely, the area accessed by the lookup table is concentrated in a local area. Therefore, the compression method of the present invention can greatly compress the storage space of the lookup table, while ensuring that the performance of the compressed lookup table remains basically unchanged when used for image restoration. The present invention has a certain degree of universality for the compression of lookup table methods used in image restoration tasks, which makes it easier to deploy lookup table methods on various edge devices with limited storage resources. Attached Figure Description

[0028] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 A flowchart of a lookup table compression method provided in an embodiment of the present invention;

[0030] Figure 2 This is a schematic diagram of a lookup table compression method provided in an embodiment of the present invention;

[0031] Figure 3 A visual diagram illustrating the access frequency of the lookup table index provided in an embodiment of the present invention;

[0032] Figure 4 A schematic diagram of the diagonal reindexing process provided in an embodiment of the present invention.

[0033] Figure 5 A schematic diagram of the off-diagonal sub-sampling process provided in an embodiment of the present invention.

[0034] Figure 6 Visual comparison in image super-resolution tasks provided in embodiments of the present invention

[0035] Figure 7 Visual comparison of image denoising tasks provided in embodiments of the present invention

[0036] Figure 8 Visual comparison in image deblocking tasks provided in embodiments of the present invention

[0037] Figure 9 Visual comparison for image deblurring tasks provided in embodiments of the present invention

[0038] Figure 10 This is a schematic diagram of a lookup table compression system provided in an embodiment of the present invention;

[0039] Figure 11 This is a schematic diagram of an image restoration system provided in an embodiment of the present invention;

[0040] Figure 12 This is a schematic diagram of a processing device provided in an embodiment of the present invention. Detailed Implementation

[0041] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0042] First, the following explanations are provided for the terms that may be used in this article:

[0043] The term "and / or" means that either or both can be achieved simultaneously. For example, X and / or Y means that it includes both "X" or "Y" as well as the three cases of "X and Y".

[0044] The terms "comprising," "including," "containing," "having," or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.) should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0045] The term "composed of" excludes any technical features not expressly listed. When used in a claim, it closes the claim to exclude all technical features other than those expressly listed, except for associated conventional impurities. If the term appears only in a clause of a claim, it limits the claim to the elements expressly listed in that clause; elements recited in other clauses are not excluded from the overall claim.

[0046] The following provides a detailed description of a lookup table compression and image restoration method, as well as related systems, devices, and media provided by this invention. Contents not described in detail in the embodiments of this invention are prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of this invention, they should be performed according to conventional conditions in the art or conditions recommended by the manufacturer.

[0047] Example 1

[0048] This invention provides a lookup table compression method, called Diagonal-First Compression (DFC). The technical problem it addresses is to ensure that the lookup table method can meet the stringent storage resource requirements of various edge devices without sacrificing image restoration performance. This is achieved by removing a large amount of redundant storage space from the lookup table, making it easier to deploy on these devices. The compression principle of this invention is based on the objective fact of natural images in image restoration tasks: the accessed area of ​​the lookup table is concentrated in a local region. Therefore, the compression method of this invention can significantly compress the storage space of the lookup table while maintaining essentially unchanged performance when the compressed lookup table is used for image restoration. This invention has a certain degree of versatility for compressing lookup table methods used in image restoration tasks, making it easier to deploy lookup table methods on various edge devices with limited storage resources.

[0049] like Figure 1 As shown, the lookup table compression method provided in this embodiment of the invention mainly includes the following steps:

[0050] Step 1: Obtain the uncompressed lookup table for image restoration.

[0051] In this embodiment of the invention, the uncompressed lookup table for image restoration is an n-dimensional (e.g., n=4) lookup table, using natural images... An image patch of a certain size contains n adjacent pixels, which can be used as an n-dimensional index in a lookup table to retrieve the corresponding value from the lookup table.

[0052] Step 2: Traverse all indexes of the uncompressed lookup table and determine whether the corresponding lookup table cell meets the set diagonal condition based on the index. If yes, proceed to step 3; otherwise, proceed to step 4.

[0053] The preferred implementation of this step is as follows: when the coordinate index dimension of the lookup table is When, the index is Each item represents an index of a dimension, with the subscript being the dimension number. When the number of dimensions to be compressed is set to N... c When, determine the index Do they satisfy simultaneously? There are *j* inequality relations, where the *j*-th inequality relation is expressed as:

[0054]

[0055] in, , The set diagonal width, For the first Indexes of each dimension, when the above When all inequalities are true, it indicates that the index is true. The corresponding lookup table cell satisfies the set diagonal condition.

[0056] Taking the first two dimensions as an example, we can treat them as a two-dimensional grid, with the index of the first dimension denoted as... The index of the second dimension is denoted as Because at this time Therefore, only one inequality relationship needs to be satisfied. In this case, this inequality relationship can be used as a diagonal condition, and the two indices can be substituted into the diagonal condition shown in the following equation:

[0057]

[0058] in, The diagonal width is set, and when the above formula is true, it represents the index. The corresponding lookup table cell satisfies the set diagonal condition.

[0059] Step 3: Save the lookup table cells that meet the diagonal condition to the first new lookup table, which is called the diagonal reindexed lookup table.

[0060] When index Create an index when the diagonal condition is met. With index The reindexing relationship between them is represented as:

[0061]

[0062] in, For index and The relative distance between them ;if Then the index will eventually be Reindexed ;if Then the index will eventually be Reindexed .

[0063] Using the previous two dimensions as examples, when the index... When a corresponding lookup table cell satisfies the diagonal condition, it needs to be mapped to a new index. , represented as:

[0064]

[0065] in, For index and The relative distance between them .

[0066] Step 4: Sample the lookup table cells that do not meet the diagonal condition at equal intervals. Save the values ​​in the sampled lookup table cells to a second new lookup table, which is called the non-diagonal subsampled lookup table.

[0067] The above-described solution provided by the embodiments of the present invention can achieve the following beneficial effects: By retaining the frequently accessed elements (i.e., lookup table units that satisfy the diagonal condition) in the uncompressed lookup table and downsampling the frequently accessed elements (i.e., lookup table units that do not satisfy the diagonal condition), the present invention can compress the original lookup table with a large storage space into two lookup tables with a smaller overall storage space, while effectively maintaining the original performance. Using the present invention allows the lookup table method with a large storage space to achieve a trade-off between performance and storage size, facilitating the deployment of lookup tables on edge devices with limited resources.

[0068] To more clearly demonstrate the technical solution and its effects provided by the present invention, the method provided by the embodiments of the present invention will be described in detail below with reference to specific examples.

[0069] Uncompressed look-up tables (LUTs) used for image restoration can be stored as a high-dimensional matrix, with a storage size of [missing information]. It can be calculated as:

[0070]

[0071] in, It is a uniform sampling interval. It is the coordinate index dimension of the lookup table. It is the number of cached values ​​in each lookup table unit. Super resolution, , It refers to the unit of storage size, a byte (Byte). Here, we take a four-dimensional lookup table with an original sampling interval of 4 as an example (i.e., in the formula...). and Existing lookup table methods also employ this approach. Following these methods, the four-dimensional lookup table is formalized here as follows: Its shape is ,in , , and These are the coordinate indices of four dimensions. This refers to the size of each dimension. Here, It can be calculated as .

[0072] The lookup table compression method proposed in this invention is called Diagonal-First Compression (DFC). It compresses the original lookup table to achieve a smaller storage space through two steps: diagonal reindexing and off-diagonal subsampling. Figure 2 This demonstrates the overall flow of the Diagonal-First Compression (DFC) method, using two dimensions of a four-dimensional lookup table ( and Taking this as an example, we can visualize the lookup table as a two-dimensional grid. First, in the diagonal reindexing step, the cached values ​​in the diagonal lookup table cells that satisfy the diagonal conditions are retained and reindexed, resulting in a low-dimensional diagonal reindexed lookup table, denoted as... Then, in the off-diagonal subsampling step, the values ​​cached in the off-diagonal lookup table units are subsampled at larger intervals, with the sampling interval being larger than the uniform sampling interval. Larger, thus resulting in a sparser off-diagonal subsampled lookup table, denoted as .

[0073] The following section will introduce the process from four aspects: principle analysis, diagonal reindexing, off-diagonal subsampling, and theoretical compression ratio.

[0074] 1. Principle analysis.

[0075] This section primarily confirms the diagonal-dominant characteristic of the lookup table. This invention discovers that, essentially, lookup tables used for image restoration employ a "space-for-time" strategy by traversing all possible combinations of input local image patches. This strategy ignores the low-dimensional manifold distribution of natural image data, leading to redundancy in storage space. To verify this argument, this invention reveals the redundancy in lookup tables used for image restoration by observing statistical data on the occurrence of local image patches. First, this invention collects retrieval statistics from the lookup table by statistically analyzing the frequency of occurrence of two spatially adjacent pixel pairs in low-quality input image patches. The frequency of adjacent pixel pairs reflects how often the index of a lookup table cell is used for retrieval. A high frequency means that the cached value in the corresponding lookup table cell is frequently accessed. Figure 3 The system visualizes search statistics. For example... Figure 3 As shown, the numbers in the cells reflect the frequency of occurrence of the lookup table cell indexes. The closer the number is to 1, the higher the frequency; the closer the number is to 0, the lower the frequency. Cells with higher frequencies are mainly distributed on the diagonal. Figure 3In the image, cells with values ​​greater than or equal to 0.1 after rounding to one decimal place are bolded solely to more clearly illustrate this diagonal pattern, without distinguishing between different degrees of diagonal dominance. This invention refers to this observation as the diagonal dominance property of the lookup table. Since the coordinate index of the lookup table is composed of adjacent pixels in low-quality image patches, the diagonal dominance property means that adjacent pixel values ​​in most local image patches are very close, consistent with the distribution of natural image data. Therefore, this invention makes a local smoothness assumption for low-quality image patches and segments diagonal and off-diagonal lookup table cells based on the differences between lookup table cell indices (i.e., adjacent pixels).

[0076] 2. Reindex the diagonal.

[0077] This section primarily focuses on reducing the dimensionality of the lookup table. Based on the diagonal-dominant characteristic, this invention proposes a diagonal re-indexing strategy, utilizing index mapping to map indexes that satisfy diagonal conditions to a new dimension, thereby reducing the dimensionality of the lookup table. For example... Figure 2 As shown, determine the index The conditions for determining whether a corresponding lookup table cell is diagonal are as follows:

[0078]

[0079] in, Defined as diagonal width. For example... Figure 1 As shown, all values ​​cached in the diagonal lookup table cells, if and If the indexes satisfy the above formula, they will all be stored sequentially in the diagonal reindexed lookup table (i.e., ...). In the example of a four-dimensional lookup table, after compressing the first two dimensions, a diagonal reindexed lookup table is obtained. Formalized Its shape is ,in It is a new index generated by counting when enumerating all diagonal lookup table cells (i.e., lookup table cells that satisfy the diagonal condition), where K is the index that satisfies the diagonal condition. The number, in this example, n=4, ,Right now An index that satisfies the diagonal condition. The quantity. Here The calculation formula is: Therefore, by indexing Mapping to index (diagonal reindexing) This compresses the two dimensions of the lookup table into one dimension.

[0080] An index mapper, as a function, builds indexes according to the following rules. and index Reindexing relationships between them:

[0081]

[0082] in, It can be seen as and The relative distance between them is calculated using the following formula: ( ).

[0083] like Figure 4 As shown, taking a four-dimensional lookup table as an example, a schematic diagram of the diagonal re-indexing process is provided. In the uncompressed lookup table... This indicates that the index in the lookup table is If the value stored in a cell is a diagonal lookup table cell, then the value of that cell is saved to the diagonal reindexed lookup table. In the original index Reindexed as After that, the index Used for retrieving three dimensions .

[0084] In practice, this invention can be easily extended to compress more dimensions, for example: a dimension with a total of The lookup table is formalized as If you want to compress this lookup table ( If there are multiple dimensions, then determine the index. When determining whether the diagonal condition is satisfied, first check whether both conditions are met simultaneously. This step involves establishing several inequality relationships, which expands the diagonal condition when compressing two dimensions:

[0085]

[0086] like Simultaneously satisfying the above requirements The inequality conditions, namely If the diagonal condition is met, then create an index. With index The reindexing relationship between the two dimensions is an extension of the reindexing relationship when compressing two dimensions:

[0087]

[0088] in, .if Then the original index can be finally obtained. Reindexed ,if Then the index will eventually be Reindexed .

[0089] 3. Non-diagonal sub-sampling.

[0090] This part mainly involves subsampling the cached values ​​in the off-diagonal lookup table cells at large sampling intervals, because according to... Figure 3 Observations show that the values ​​cached in the off-diagonal lookup table cells are rarely accessed. For example... Figure 5 As shown, by using off-diagonal subsampling with large sampling intervals, this invention reduces the size of each dimension from... Reduce to This yields the off-diagonal subsampling lookup table, i.e. Its shape is . The calculation formula is ,in It is the sampling interval when sampling at equal intervals, relative to In other words It is a large sampling interval, for example, which can be set. . Formalization . The corresponding index can be calculated by scaling the original index:

[0091]

[0092] in, This is for floor function. It should be noted that some values ​​in the diagonal lookup table cells will also be sampled a second time. In this context, we can predict values ​​where the index does not meet the diagonal condition but is near the diagonal boundary.

[0093] 4. Theoretical compression ratio.

[0094] In this embodiment of the invention, a single lookup table is compressed into two lookup tables with smaller storage space, namely, a diagonal reindexed lookup table. Non-diagonal sub-sampling lookup table This reduces the total storage space. The shape of the diagonal reindexed lookup table is... The shape of the off-diagonal sub-sampling lookup table is K is the index that satisfies the diagonal condition. The number of elements, where L is the size of each dimension in the uncompressed lookup table. The coordinate index dimension of the lookup table. The dimension being compressed.

[0095] The storage sizes of the diagonal reindexed lookup table and the off-diagonal subsampled lookup table are as follows:

[0096]

[0097]

[0098] in, To reduce the storage size of the diagonal reindex lookup table, The storage size of the off-diagonal subsampled lookup table; K is the number of cells in the diagonal lookup table.

[0099] The theoretical compression ratio is defined as:

[0100]

[0101] Where CR is the theoretical compression ratio.

[0102] To verify the effectiveness of the lookup table compression method provided in this invention, experiments were conducted on representative image restoration tasks. Two lookup table-based models (SR-LUT and MuLUT) trained on the public dataset DIV2K were used for super-resolution, denoising, and deblocking. These lookup table models were then retrained on the public dataset GoPro training set to enable them to be used for deblurring. In the experiments, the PixelShuffle layer was removed to make these lookup table-based models suitable for denoising, deblocking, and deblurring. The original lookup table-based models were compressed using the diagonal-first compression method (DFC) of this invention to obtain their compressed versions, denoted as +DFC. After compression using DFC, a lookup table-aware fine-tuning strategy was adopted (Li J, Chen C, Cheng Z, et al. Mulut: Cooperating multiple look-up tables for efficient image super-resolution[C] / / Europeanconference on computer vision. Cham: Springer Nature Switzerland, 2022: 238-256.). To maintain a low compression ratio, a default configuration was set for the +DFC version when evaluating performance. , , The evaluation metrics used in this experiment are peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), and storage size.

[0103] The PSNR is calculated as follows:

[0104]

[0105] in, and These are the network's output image and target image, respectively. This represents the maximum dynamic range that the target image can achieve. and These are the height and width of the image, respectively. These are the spectral channels of the image.

[0106] SSIM is calculated as follows:

[0107]

[0108] in, , and All are constants. , For image , Their respective average values , For image , Their respective standard deviations for and The covariance between them.

[0109] 1. Experiments on image super-resolution tasks.

[0110] The diagonal-first compression method of this invention was evaluated on five commonly used super-resolution benchmark datasets (Set5, Set14, BSDS100, Urban100, and Manga109). Degraded images were generated using a bicubic downsampling method. SR-LUT and MuLUT were selected as original versions of the lookup table-based compression model, and their performance was compared with their +DFC versions. Furthermore, classical and deep neural network (DNN) models were used as references. Among them, classic models include Bicubic, the NE + LLE scheme proposed by Hong Chang et al. of Hong Kong University of Science and Technology (ChangH, Yeung DY, Xiong Y. Super-resolution through neighbor embedding[C] / / Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004. IEEE, 2004, 1: II.), the ANR algorithm proposed by Radu Timofte et al. (Timofte R, De Smet V, Van Gool L. Anchored neighborhoodregression for fast example-based super-resolution[C] / / Proceedings of the IEEE international conference on computer vision. 2013: 1920-1927.), and the A+ algorithm (Timofte R, De Smet V, Van Gool L. A+: Adjusted anchored neighborhoodregression for fast super-resolution[C] / / Computer Vision--ACCV 2014: 12thAsian Conference). on Computer Vision, Singapore, Singapore, November 1-5,2014, Revised Selected Papers, Part IV 12. Springer International Publishing, 2015: 111-126.The deep neural network models include the RRDB model proposed by Xintao Wang et al. (Wang X, YuK, Wu S, et al. Esrgan: Enhanced super-resolution generative adversarial networks[C] / / Proceedings of the European conference on computer vision (ECCV) workshops. 2018: 0-0.) and the EDSR model proposed by Bee Lim et al. of Seoul National University (Lim B, Son S, Kim H, et al. Enhanced deep residual networks for single image super-resolution[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 2017: 136-144.). Furthermore, the storage size of DNNs is reported. It is worth noting that, compared to lookup table-based models, DNN models typically require a dedicated computational framework (e.g., the PyTorch library) and incur significant computational overhead. A comparison of different models is shown in Table 1.

[0111] Table 1: Quantitative comparison of image super-resolution tasks at a magnification of 4x

[0112]

[0113] As can be seen, the +DFC version of the lookup table-based model significantly reduces storage space compared to the original version, while the performance degradation in PSNR is negligible. For example, MuLUT +DFC achieves a compression ratio of 10.0%, with only a slight reduction of 0.05dB on the Set5 dataset. Furthermore, the diagonal-first compression method of this invention enables high-level lookup table-based models to achieve better performance within a smaller storage space; for example, MuLUT +DFC outperforms SR-LUT in PSNR with a smaller storage space (0.407MB vs. 1.274MB). Figure 6 The article compares the lookup table-based model and its +DFC version in... Visual quality in super-resolution tasks Figure 6In the diagram, LR represents the given low-resolution image, and GT represents the real target image in the task (as shown in the accompanying figures for subsequent experiments). The parentheses below LR (PSNR / Storage Size) explain the meaning of the numbers before and after the slashes in each method's diagram; that is, the two items before and after the slash represent the PSNR between the output image and the target image of the method, and the storage size of the lookup table for that method, respectively (as shown in the accompanying figures for subsequent experiments). The parentheses after GT indicate which dataset the target image is from, and the text below GT indicates the name of the target image in that dataset. The longitudinal comparison shows that the images generated by the +DFC version are not significantly different from those generated by the original version. In summary, the diagonal-first compression method of this invention maintains visual quality while compressing the storage size of the lookup table-based model.

[0114] 2. Experiments on image denoising tasks.

[0115] The lookup table-based model was evaluated on two benchmark datasets, Set12 and BSD68, used for denoising grayscale images with a noise level of 15. The noisy images were generated using additive white Gaussian noise. Table 2 reports the PSNR and storage size of the lookup table-based model, providing a quantitative comparison. Classical and deep neural network models are introduced as references. Among them, classic models include the BM3D model proposed by Kostadin Dabov et al. (Dabov K, Foi A, Katkovnik V, et al. Image denoising by sparse 3-D transform-domain collaborative filtering[J]. IEEE Transactions on image processing, 2007, 16(8): 2080-2095.), the WNNM model proposed by Shuhang Gu et al. (Gu S, Zhang L, Zuo W, et al. Weighted nuclear norm minimization with application to image denoising[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2014: 2862-2869.), and the TNRD model proposed by Yunjin Chen and Thomas Pock (Chen Y, Pock T. Trainable nonlinear reaction diffusion: A flexible framework for fast and effective image restoration[J]. IEEE transactions on pattern analysis and machine intelligence, 2016, 39(6): ). 1256-1272.); Deep neural network models include the DnCNN model proposed by Kai Zhang et al. (Zhang K, Zuo W, Chen Y, et al. Beyond agaussian denoiser: Residual learning of deep cnn for image denoising[J].IEEE transactions on image processing, 2017, 26(7): 3142-3155.) and FFDNet model (ZhangK, Zuo W, Zhang L. FFDNet: Toward a fast and flexible solution for CNN-basedimage denoising[J]. IEEE Transactions on Image Processing, 2018, 27(9): 4608-4622.) and Jingyun The SwinIR model proposed by Liang et al. (Liang J, Cao J, Sun G, et al. Swinir: Image restoration using swin transformer[C] / / Proceedings of the IEEE / CVF international conference on computer vision. 2021: 1833-1844.). .

[0116] Table 2: Quantitative comparison of image denoising tasks at a noise level of 15

[0117]

[0118] As shown in Table 2, using the diagonal-first compression method of this invention significantly reduces the storage size of the lookup table-based model, while achieving almost no performance degradation on the two benchmark datasets. For example, MuLUT + DFC achieves a compression rate of 10.0% (49.031KB / 489.381KB) on the BSD68 dataset, a decrease of only 0.09dB, but achieves better denoising performance with a smaller storage capacity than the original SR-LUT (49.031KB vs. 81.563KB). Figure 7 The qualitative evaluation yielded conclusions consistent with those obtained in the super-resolution experiment, verifying the versatility of the diagonal-first compression method of this invention. Figure 7 In this context, "Noisy" represents a given noisy image.

[0119] 3. Experiments on image deblocking tasks.

[0120] Table 3 reports a quantitative comparison of PSNR-B based on a lookup table model on two standard test sets (Classic5 and LIVE1) for deblocking JPEG (Joint Group of Image Experts) images with a quality factor of 10, where PSNR-B assesses the blockiness in the images.

[0121] Table 3: Quantitative comparison of JPEG image deblocking tasks with a quality factor of 10

[0122]

[0123] Table 3 also includes classical models and deep neural network models as references. Among them, the classical model adopted is the SA-DCT model proposed by Alessandro Foi et al. (Foi A, Katkovnik V, Egiazarian K. Pointwise shape-adaptive DCT for high-quality denoising and deblocking of grayscale and color images[J]. IEEE transactions on image processing, 2007,16(5): 1395-1411.); the deep neural network models include the SwinIR model and the ARCNN model proposed by Chao Dong et al. (Dong C, Deng Y, Loy CC, et al. Compression artifacts reduction by a deepconvolutional network[C] / / Proceedings of the IEEE international conference on computer vision. 2015: 576-584.). Figure 8 The paper provides a qualitative evaluation, which demonstrates the adaptability of the diagonal-first compression method of the present invention to image deblocking. Figure 8 In this context, JPEG refers to a given JPEG image.

[0124] 4. Experiments on image deblurring tasks.

[0125] Table 4 provides a quantitative comparison based on lookup table models, and also includes two classic models and two DNN models for reference. Classical models include the method proposed by Li Xu et al. (referred to as Xu et al. in Table 4, Xu L, Zheng S, Jia J. Unnatural 10 sparse representation for natural image deblurring[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2013: 1107-1114.) and the method proposed by Tae Hyun Kim and Kyoung Mu Lee (referred to as Kim and Lee in Table 4, Hyun Kim T, Mu Lee K. Segmentation-free dynamicscene deblurring[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2014: 2766-2773.); deep neural network models include the method proposed by Dong Gong et al. (referred to as Gong et al. in Table 4, Gong D, Yang J, Liu L, et al. From motionblur to motion flow: A deep learning solution for removing heterogeneous motion blur[C] / / Proceedings of the (IEEE conference on computer vision and pattern recognition. 2017: 2319-2328.) and the DBGAN model proposed by Kaihao Zhang et al. (Zhang K, Luo W, Zhong Y, et al. Deblurring by realistic blurring[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2020:2737-2746.).

[0126] Table 4: Quantitative Comparison of Image Deblurring Tasks

[0127]

[0128] Table 4 shows the PSNR and SSIM of image deblurring tested on the standard GoPro test set, represented by the two items before and after the slash in the last column of Table 4. Figure 9 The qualitative evaluation provided further validates the versatility of the diagonal-first compression method of this invention. Figure 9 In this context, "Blurry" represents a given blurred image.

[0129] 5. Efficiency evaluation.

[0130] In addition to evaluating performance, an assessment was also conducted to estimate the theoretical energy consumption cost associated with the lookup table method after compression using the diagonal-first compression method of the present invention. Bicubic, RRDB, and EDSR were also used as references. Table 5 lists the costs associated with operating energy consumption by data type.

[0131] Table 5: Energy Costs for Calculations of Different Data Types

[0132]

[0133] Table 5 lists the data types as follows: int8 (8-bit signed integer), int32 (32-bit signed integer), float16 (16-bit floating-point number), and float32 (32-bit floating-point number). Here, Add. and Mul. represent the addition and multiplication operators, respectively, and pJ (i.e., picojoule) in parentheses represents the unit of these two rows of data. The relevant data comes from existing literature (Sze V, Chen YH, Yang TJ, et al. Efficient processing of deep neural networks: A tutorial and survey[J]. Proceedings of the IEEE, 2017, 105(12): 2295-2329.).

[0134] Table 6 lists the efficiency evaluation of the lookup table method after compression using the diagonal-first compression method of this invention. Here, int8, int32, and float32 represent three different data types, Add. and Mul. represent addition and multiplication operations, respectively. The combinations of data types and operators in Table 6 represent the operands produced by the corresponding operators under that data type. The evaluations of operands, energy cost, and peak memory for different data type operators are based on a scaling factor of 4 and a generated image size of [missing information]. The energy cost was measured on an image super-resolution task. The unit of energy cost is pJ. The energy cost of different methods in Table 6 is calculated by weighted sum of the number of operations required by different methods and the corresponding values ​​in Table 5. Table 6 also evaluates the storage size of the libraries that different methods depend on. numpy: 16.5MB means that the dependent library is numpy and the storage size of the library is 16.5MB. torch (CPU): 186.3MB means that the dependent library is the CPU version of the torch library and the storage size of the library is 186.3MB.

[0135] Table 6: Efficiency evaluation of the lookup table method after compression using the diagonal-first compression method of the present invention.

[0136]

[0137] Lookup table-based models perform only integer operations, offering a significant computational efficiency advantage over DNN models. Notably, the diagonal-first compression method of this invention maintains this advantage while reducing storage space. Peak memory was measured on the CPU using the memray library (a memory monitoring tool) in Python. As shown in Table 6, the peak memory usage of the lookup table-based model is lower than that of the DNN model, meaning it requires fewer resources at runtime. Furthermore, the storage size of the dependent libraries listed in Table 6 indicates that the DNN model requires substantial additional storage overhead compared to the lookup table model. In summary, from the perspectives of energy consumption, peak memory, and dependent library size, the lookup table model is more advantageous than DNN when deployed on resource-constrained edge devices. The diagonal-first compression method of this invention effectively preserves the inherent computational efficiency of the original lookup table model, reduces its storage requirements, and makes the lookup table model more practical.

[0138] In the above experiments, the diagonal-first compression method of this invention was quantitatively and qualitatively evaluated on four representative image restoration tasks: image super-resolution, image denoising, image deblocking, and image deblurring. The quantitative evaluation showed that the lookup table method using this invention significantly reduced storage overhead with almost no change in performance. The qualitative evaluation showed that the output of the lookup table method using this invention was visually almost indistinguishable from the original uncompressed lookup table output. Extensive experimental results on these four representative image restoration tasks demonstrate that the diagonal-first compression method of this invention facilitates the deployment of lookup table-based image restoration models on resource-constrained edge devices.

[0139] Example 2

[0140] This invention provides an image restoration method, mainly comprising: compressing a lookup table based on the method provided in Embodiment 1 above to obtain a diagonal reindexed lookup table and a non-diagonal subsampled lookup table; when using the input index to retrieve the lookup table, determining whether the corresponding lookup table unit meets the set diagonal condition based on the input index; if so, converting the input index into the index of the diagonal reindexed lookup table through index mapping (see the index k calculation formula provided in the aforementioned embodiment for details) for retrieval; if not, scaling the input index (see the calculation formula for the corresponding index of the non-diagonal subsampled lookup table unit provided in the aforementioned embodiment for details) for retrieval in the non-diagonal subsampled lookup table; and using the retrieved value for image restoration.

[0141] The image restoration method described above removes a large amount of redundancy in the lookup table, ensuring that the high-performance lookup table method can achieve an efficient image restoration process while meeting the stringent storage size requirements of different edge devices.

[0142] The lookup table compression scheme included in the above image restoration method has been described in detail in the aforementioned Embodiment 1, so it will not be repeated here.

[0143] Through the above description of the embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.), including several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0144] Example 3

[0145] This invention also provides a lookup table compression system, which is mainly used to implement the lookup table compression method provided in the foregoing embodiments, such as... Figure 10 As shown, the system mainly includes:

[0146] The lookup table acquisition unit is used to acquire an uncompressed lookup table for image restoration;

[0147] The diagonal condition judgment unit is used to traverse all indexes of the uncompressed lookup table and determine whether the corresponding lookup table cell meets the set diagonal condition based on the index.

[0148] The first storage unit is used to save the lookup table unit that meets the diagonal condition into the first new lookup table when the set diagonal condition is met. The first new lookup table is called the diagonal reindexed lookup table.

[0149] The second storage unit is used to perform equal-interval sampling on lookup table units that do not meet the set diagonal conditions when the diagonal conditions are not met. The values ​​in the sampled lookup table units are saved to a second new lookup table, which is called a non-diagonal sub-sampling lookup table.

[0150] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0151] Since the specific technical details involved in the above system have been described in detail in the previous embodiments, they will not be repeated here.

[0152] Example 4

[0153] The present invention also provides an image restoration system, which is mainly used to implement the image restoration method provided in the foregoing embodiments, such as... Figure 11 As shown, the system mainly includes:

[0154] The lookup table compression system provided in the foregoing embodiments is used to implement the lookup table compression method of the foregoing embodiments to obtain a diagonal reindexed lookup table and a non-diagonal subsampled lookup table;

[0155] The image restoration unit is used to determine whether the corresponding lookup table unit meets the set diagonal condition when using the input index to retrieve the lookup table. If yes, the input index is converted into the index of the diagonal reindexed lookup table through index mapping for retrieval. If no, the input index is scaled and then retrieved in the off-diagonal subsampled lookup table. The retrieved value is then used to restore the image.

[0156] Example 5

[0157] The present invention also provides a processing device, such as Figure 12 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the lookup table compression method or image restoration method provided in the foregoing embodiments.

[0158] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0159] In this embodiment of the invention, the specific types of the memory, input device, and output device are not limited; for example:

[0160] Input devices can be touchscreens, image acquisition devices, physical buttons, or mice, etc.

[0161] The output device can be a display terminal;

[0162] The memory can be random access memory (RAM) or non-volatile memory, such as disk storage.

[0163] Example 6

[0164] The present invention also provides a readable storage medium storing a computer program that, when executed by a processor, implements the lookup table compression method or image restoration method provided in the foregoing embodiments.

[0165] In this embodiment of the invention, the readable storage medium is a computer-readable storage medium and can be disposed in the aforementioned processing device, for example, as a memory in the processing device. Furthermore, the readable storage medium can also be any medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0166] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for compressing lookup tables, characterized in that, include: Obtain the uncompressed lookup table for image restoration; Iterate through all indices of the uncompressed lookup table, and determine whether the corresponding lookup table cell satisfies the set diagonal condition based on the index, including: when the coordinate index dimension of the lookup table is... When, the index is Each item represents an index of a dimension, with the subscript being the dimension number. When the number of dimensions to be compressed is set to... When, determine the index Do they satisfy simultaneously? There are inequality conditions, where the j-th inequality relation is expressed as: ;in, , The set diagonal width, For the first Indexes of multiple dimensions, when When all inequalities are true, it indicates an index. The corresponding lookup table cell satisfies the set diagonal condition; If yes, the lookup table cells that satisfy the diagonal condition are saved to the first new lookup table, which is called the diagonal reindexed lookup table; if no, the lookup table cells that do not satisfy the diagonal condition are sampled at equal intervals, and the values ​​in the sampled lookup table cells are saved to the second new lookup table, which is called the off-diagonal subsampled lookup table.

2. The lookup table compression method according to claim 1, characterized in that, When index An index is created when the corresponding lookup table cell meets the set diagonal condition. With index The reindexing relationship between them is represented as: ； in, For index and The relative distance between them For index and The relative distance between them; if Then the index will eventually be Reindexed ;if Then the index will eventually be Reindexed .

3. The lookup table compression method according to claim 1, characterized in that, The shape of the diagonal reindexed lookup table is Where K is the index that satisfies the diagonal condition. The number of elements, where L is the size of each dimension in the uncompressed lookup table. The coordinate index dimension of the lookup table. The dimension being compressed.

4. The lookup table compression method according to claim 1, characterized in that, The shape of the off-diagonal sub-sampling lookup table is as follows: ,in, The coordinate index dimension of the lookup table. The size of each dimension in the off-diagonal subsample lookup table is calculated as follows: , This is the sampling interval for equal-interval sampling.

5. An image restoration method, characterized in that, include: Based on the lookup table compression method according to any one of claims 1 to 4, a diagonal reindexed lookup table and an off-diagonal subsampled lookup table are compressed to obtain a diagonal reindexed lookup table and an off-diagonal subsampled lookup table. When retrieving a lookup table using the input index, determine whether the corresponding lookup table cell meets the set diagonal condition based on the input index. If yes, the input index is converted into an index in a diagonal reindexed lookup table through index mapping for retrieval; otherwise, the input index is scaled and then retrieved in a non-diagonal subsampled lookup table. Image restoration is performed using the retrieved values.

6. A lookup table compression system, characterized in that, To implement the lookup table compression method according to any one of claims 1 to 4, comprising: The lookup table acquisition unit is used to acquire an uncompressed lookup table for image restoration; The diagonal condition judgment unit is used to traverse all indexes of the uncompressed lookup table and determine whether the corresponding lookup table cell meets the set diagonal condition based on the index. The first storage unit is used to save the lookup table unit that meets the diagonal condition into the first new lookup table when the set diagonal condition is met. The first new lookup table is called the diagonal reindexed lookup table. The second storage unit is used to perform equal-interval sampling on lookup table units that do not meet the set diagonal conditions when the diagonal conditions are not met. The values ​​in the sampled lookup table units are saved to a second new lookup table, which is called a non-diagonal sub-sampling lookup table.

7. An image restoration system, characterized in that, include: A lookup table compression system is used to implement the lookup table compression method according to any one of claims 1 to 4, and to obtain a diagonal reindexed lookup table and a non-diagonal subsampled lookup table; The image restoration unit is used to determine whether the corresponding lookup table cell meets the set diagonal condition based on the input index when retrieving the lookup table using the input index. If yes, the input index is converted into an index in a diagonal reindexed lookup table through index mapping for retrieval; otherwise, the input index is scaled and retrieved in a non-diagonal subsampled lookup table; the retrieved value is then used for image restoration.

8. A processing apparatus, characterized in that, include: One or more processors; Memory, used to store one or more programs; Wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in any one of claims 1 to 5.

9. A readable storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it implements the method as described in any one of claims 1 to 5.

Images