Multi-layer lookup table image mapping method based on conditional perception and hybrid expert mechanism

Abstract

The present application belongs to the technical field of computer vision and image processing, and aims at the technical problem that the existing technology cannot adaptively adjust the mapping behavior according to the characteristics of different image regions. A multi-layer lookup table image mapping method based on conditional perception and hybrid expert mechanism is proposed. The linear expansion of the receptive field and the diversification of the features are realized through the spatial progressive fusion architecture. The hybrid expert gating mechanism is introduced, and the contribution weights of the basic expert branch and the detail expert branch are dynamically adjusted through the conditional perception module to realize the spatial adaptive mapping. Finally, the diagonal line priority compression algorithm is used to compress the depth of all expert lookup tables, which significantly improves the spatial adaptability and storage efficiency of image mapping while maintaining extremely low inference latency.

Description

Technical Field

[0001] This invention relates to the field of computer vision and image processing technology, specifically to a multi-layer lookup table image mapping method based on conditional awareness and hybrid expert mechanisms, which is particularly suitable for tasks such as image super-resolution reconstruction, denoising, deblocking, and image enhancement. Background Technology

[0002] Image mapping methods based on lookup tables have been widely used in image processing and image reconstruction tasks due to their low computational complexity and high inference efficiency. These methods achieve nonlinear image transformation or feature reconstruction by mapping input pixels or local features using lookup tables.

[0003] On the one hand, existing lookup table-based image mapping methods often employ a uniform mapping function to process different spatial regions. However, in real-world images, flat regions differ significantly from edge or textured regions in terms of gradient distribution, spectral characteristics, and noise sensitivity. Using the same mapping method can easily lead to overfitting in flat regions or insufficient detail representation in complex textured regions. These problems are particularly pronounced when the lookup table dimension is limited and the effective receptive field is small. Furthermore, due to the limitation of the lookup table input dimension, existing lookup table mapping methods typically rely solely on local pixel values ​​for modeling, lacking explicit perception of conditional information such as the complexity of the region structure. This means that the mapping function can only adopt a uniform or approximately uniform mapping behavior when facing different region characteristics, making it difficult to achieve adaptive adjustment.

[0004] On the other hand, in deep neural networks, convolutional layers and fully connected layers perform linear or affine mappings on input features, and activation functions nonlinearly modulate the mapping results, implicitly containing selection or suppression mechanisms for feature responses. In existing lookup table-based modeling methods, the above mapping and activation modulation behaviors are usually merged or approximated as a unified lookup table mapping process, lacking explicit modeling of the mapping modulation and selection mechanisms, thus limiting the flexibility and expressive power of lookup table mapping in different regions.

[0005] Therefore, how to improve the adaptive mapping capability of lookup tables to different image region characteristics while maintaining their efficient reasoning characteristics has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0006] To address the technical problems of existing cascaded lookup tables having huge storage overhead and being unable to adaptively adjust mapping behavior according to the characteristics of different image regions, this invention proposes a multi-layer lookup table image mapping method based on condition awareness and a hybrid expert mechanism.

[0007] The core technical solution of this invention includes the following steps:

[0008] S1. Calculate the roughness information in the target image used to characterize the regional characteristics and generate gated modulation coefficients, wherein the gated modulation coefficients are used to control the expert branch bias;

[0009] S2. Input the target image into the cascaded mapping model, where each layer contains at least a basic expert branch and a detail expert branch set in parallel.

[0010] S3. For each lookup table mapping module in the cascaded mapping model, a diagonal-first compression strategy is used to optimize the storage of the expert lookup table;

[0011] S4. Based on the gated modulation coefficients described in S1, the expert outputs of each branch are weighted and fused to obtain the corresponding output features of each layer;

[0012] S5. Perform cross-layer feature transfer operation on the output features obtained in S4 through channel splitting operation;

[0013] S6. Repeat steps S3 to S5 to the final layer, and fuse all cross-layer retained features with the final layer output to obtain fused features;

[0014] S7. Output the reconstructed final image based on the fusion features.

[0015] Technical effects:

[0016] This invention proposes a multi-layer lookup table image processing scheme that adaptively adjusts based on regional characteristics and possesses efficient storage capabilities. It integrates a framework of "spatial progressive fusion," "hybrid expert gating," and "diagonal-first compression." First, the "spatial progressive fusion" architecture achieves linear expansion of the receptive field and feature diversification. Second, a hybrid expert gating mechanism is introduced, dynamically adjusting the contribution weights of "basic experts" and "detail experts" through a condition-aware module to achieve spatial adaptive mapping. Finally, the "diagonal-first compression" algorithm performs deep compression on all expert lookup tables, utilizing the diagonal-dominant characteristic of natural images to eliminate unnecessary storage redundancy.

[0017] 1. Significantly improved spatial adaptive recovery capability: This invention changes the traditional "one-size-fits-all" mapping mode of lookup tables by explicitly modeling the region complexity (conditional information R). The gating mechanism allows the model to call smoothing strategies to suppress noise in flat regions and enhancement strategies to enhance details in edge texture regions, solving the contradiction between texture loss and visual artifacts commonly found in lookup table mapping, and greatly improving the subjective visual quality of the reconstructed image.

[0018] 2. Achieving large receptive field coverage with extremely low storage overhead: This invention employs a cross-layer feature transfer mechanism, combined with "diagonal-priority compression" technology, to resolve the inherent contradiction between "expanded receptive field" and "increased storage space". Experiments show that this invention can reduce storage overhead by more than 90% without sacrificing performance, enabling high-performance models to reside perfectly in the SoC's high-speed cache (L2 / L3 Cache), greatly reducing DRAM access power consumption and improving inference response speed.

[0019] 3. Significantly enhanced flexibility and diversity of feature representation: Through a multi-channel feature segmentation and preservation (Si) strategy, this invention not only completes the deep transformation of features at each level, but also preserves the original feature information at different scales through cross-layer fusion. This collaborative working method of "multi-path, multi-expert" enables this invention to have representation capabilities comparable to deep convolutional neural networks (CNNs) when handling complex tasks such as super-resolution and denoising, while maintaining the extremely high efficiency of lookup tables. Attached Figure Description

[0020] Figure 1 This is an overall flowchart of an embodiment of the present invention.

[0021] Figure 2 This is a block diagram of the cascaded mapping model architecture of the present invention. Detailed Implementation

[0022] Various embodiments of the present invention will now be clearly and completely described with reference to the accompanying drawings. The embodiments described with reference to the drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0023] like Figure 1 As shown, the specific steps in this embodiment are as follows:

[0024] S1, acquire target image I, which may be a low-resolution, noisy, or low-quality image with compression artifacts; calculate the roughness R in target image I to characterize regional characteristics and generate gated modulation coefficients.

[0025] Specifically, the local maximum value at each pixel location is obtained through a local sliding window. and local minimum The window size is set to 5, the stride to 1, and the padding to 2 to ensure consistent feature map dimensions. The difference between the two is calculated to obtain the continuous roughness. ;

[0026] For each mapping block in the cascaded mapping model, set independent learnable affine parameters a and b, and gated modulation coefficients. ,

[0027] Here, 'a' is initially set to 0.01, and 'b' is initially set to -6, which guides the model to favor the basic expert branch during the initial training phase. This mechanism helps when the target image region is against a stable background (where R is relatively small). The model tends to invoke basic experts to eliminate quantization noise; when the target image region is at the texture edge (R is large). The model tends to call on detail experts to enhance high-frequency details.

[0028] S2. Input the target image I into the L-layer cascaded mapping model. Each layer consists of a base expert branch and a detail expert branch set in parallel. For example... Figure 2 As shown, the cascaded mapping model contains four mapping blocks. In the first three mapping blocks, the number of output feature channels is two each. In the fourth mapping block, the number of output feature channels is one, and the data is directly stored in refine_list. The features (four channels in total) in refine_list are fused spatially through a channel convolutional layer (ChannelConv), which is implemented using a lookup table. The fused features are then upsampled to the target resolution using an upsampling block (UpBlock).

[0029] S3. For the i-th level lookup table mapping module (LUT group), a diagonal-first compression strategy is used to optimize the storage of the expert lookup table.

[0030] Specifically, for each cell in the expert lookup table, the absolute value of the difference between adjacent pixel pairs is calculated. To make a judgment, Usually, 1 to 2 is taken.

[0031] like If the current pixel pair is determined to belong to a high-probability diagonal region, it is classified into the diagonal table. At this point, a one-dimensional index is obtained using the preset index mapping matrix ref2index. To reduce dimensionality, the output is obtained through quadlinear interpolation in the compressed lookup table compress1. , This is the compressed one-dimensional physical memory address index.

[0032] like If it belongs to the off-diagonal region, it is classified into the off-diagonal table and a large step size is used. Perform downsampling ( The output is obtained from the sparse table compress2 processed by the processor.

[0033] This two-way indexing strategy dynamically balances the precision of smooth regions with the storage overhead of off-diagonal regions during the inference phase. This strategy can reduce lookup table storage overhead from megabytes (MB) to kilobytes (KB) while maintaining a nearly constant peak signal-to-noise ratio (PSNR).

[0034] S4. Based on the gated modulation coefficients described in S1, the expert outputs of each branch are weighted and fused to obtain the corresponding output features for each layer. The formula for calculating the output feature Y of the i-th layer is as follows:

[0035] ,

[0036] in, For the gated modulation coefficients, Output value of the basic expert branch. Output values ​​for the detail expert branch.

[0037] By using this hybrid expert value, the model is made to be more inclined to smooth mapping in flat areas and more inclined to high-frequency reconstruction in detailed areas.

[0038] S5. Perform cross-layer feature transfer operation. The output feature Y of each layer obtained in S4 is split through channel segmentation operation. A part of the feature X is used as the index input feature of the next layer lookup table mapping module; a part of the feature S is stored in refine_list as the cross-layer feature information retained in this layer.

[0039] Specifically, a further refinement of the cross-layer feature transfer logic described in S5: the spatial progressive fusion mechanism achieves linear growth of the receptive field through channel splitting.

[0040] The total number of channels for the output feature Y of the i-th layer is C. It is divided into two subsets, feature X and feature S.

[0041] Feature X serves as the index input feature for the next-layer lookup table mapping module. Since each layer of the lookup table further maps the features extracted from the previous layer, its equivalent receptive field (RF) increases linearly with the number of layers L. Feature S bypasses the intermediate layers and is directly passed to the end. This "long skip connection" ensures that the spatial details of the lower layers (such as pixel-level positional information) and the semantic information of the deeper layers are effectively fused in the final stage.

[0042] S6. Repeat S3~S5 until the Lth layer. Through multi-layer cascading, the effective receptive field (RF) of the lookup table is increased from the initial... linear expansion to That's all. Finally, all retained features S are concatenated with the final layer output Y, and the fused features F are obtained through an aggregation lookup table.

[0043] S7. Output the reconstructed final image based on the fusion feature F.

[0044] In one possible implementation, a multi-mode sampling mechanism is employed within expert branches, with each level of expert branch containing three sampling modes:

[0045] 1. Mode s: Using 2 2. Standard neighborhood sampling, with sampling coordinate offset as follows: Used to capture the underlying texture;

[0046] 2. Mode d: Uses dilated neighborhood sampling with an interval of 2, and the sampling coordinate offset is... Used to expand the receptive field;

[0047] 3. Mode y: Asymmetric Y-shaped sampling is used, with a sampling coordinate offset of . This is used to capture directional edge features. The final expert output is a weighted average of the three mode outputs, which significantly enhances the lookup table's ability to represent complex structures.

[0048] In one possible implementation, the training strategy and loss function of the cascaded mapping model are further defined as follows:

[0049] To achieve end-to-end optimization and precisely control the behavior of expert branches, this embodiment employs a comprehensive loss function. Training is defined as follows:

[0050] ,

[0051] Among them, the main reconstruction loss : Used to constrain the fundamental pixel differences between the final image and the ground truth (GT).

[0052] Residual regularization loss in flat regions The core principle lies in using gating states to impose inverse constraints on the output of the detail experts. When the gating module determines that the region is flat, this loss term will strongly suppress the response of the detail experts.

[0053] Gating regularization loss By introducing a roughness threshold With target offset coefficient, guide gate modulation coefficient The distribution is designed to prevent "expert collapse" and ensure precise decoupling of gating decisions from physical region characteristics (smoothness or texture).

[0054] All content not described in detail in this specification belongs to the prior art known to those skilled in the art. Furthermore, for those skilled in the art, there will be changes in specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A multi-layer lookup table image mapping method based on conditional awareness and hybrid expert mechanism, characterized in that, Includes the following steps: S1. Calculate the roughness information in the target image used to characterize the regional characteristics and generate gated modulation coefficients, wherein the gated modulation coefficients are used to control the expert branch bias; S2. Input the target image into the cascaded mapping model, where each layer contains at least a basic expert branch and a detail expert branch set in parallel. S3. For each lookup table mapping module in the cascaded mapping model, a diagonal-first compression strategy is used to optimize the storage of the expert lookup table; S4. Based on the gated modulation coefficients described in S1, the expert outputs of each branch are weighted and fused to obtain the corresponding output feature Y for each layer; S5. Perform cross-layer feature transfer operation on the output feature Y obtained in S4 through channel splitting operation; S6. Repeat steps S3 to S5 to the final layer, and fuse all cross-layer retained features with the final layer output to obtain fused features; S7. Output the reconstructed final image based on the fusion features; Specifically, S1 involves obtaining the local maximum value at each pixel position using a local sliding window. and local minimum Gating modulation coefficient , in, a , b For the independently learnable affine parameters set, roughness This is the roughness threshold; When the target image region is against a stable background, the gate modulation coefficient The model tends to invoke basic experts to eliminate quantization noise; when the target image region is at the texture edge, the gated modulation coefficients... The model tends to call on detail experts to enhance high-frequency details.

2. The multi-layer lookup table image mapping method based on conditional awareness and hybrid expert mechanism according to claim 1, characterized in that, The cascaded mapping model contains four mapping blocks. In the first three mapping blocks, the number of output feature channels is 2 each. In the fourth mapping block, the number of output feature channels is 1, and the feature is directly stored in refine_list. The features in refine_list are fused spatially through channel convolutional layers, which are implemented using lookup tables. The fused features are then restored to the target resolution through upsampling blocks.

3. The multi-layer lookup table image mapping method based on conditional awareness and hybrid expert mechanism according to claim 1, characterized in that, Specifically, S3 involves calculating the absolute value of the difference between adjacent pixel pairs. , ; like If the current pixel pair is determined to belong to the diagonal region of a high-probability distribution, a one-dimensional index is obtained using a preset index mapping matrix. To reduce dimensionality, the output is obtained through quadlinear interpolation in a compressed lookup table. , This is a compressed one-dimensional physical memory address index; like If it belongs to the off-diagonal region, it is classified into the off-diagonal table and a large step size is used. The output is obtained from the sparse table that has undergone downsampling.

4. The multi-layer lookup table image mapping method based on conditional awareness and hybrid expert mechanism according to claim 1, characterized in that, The corresponding output features of each layer ,in, For the gated modulation coefficients, Output value of the basic expert branch. Output values ​​for the detail expert branch.

5. The multi-layer lookup table image mapping method based on conditional awareness and hybrid expert mechanism according to claim 1, characterized in that, Specifically, S5 involves each layer's output feature Y obtained in S4 undergoing a channel splitting operation, with a portion of the feature X serving as the index input feature for the next layer's lookup table mapping module. A portion of the features S are stored in refine_list as cross-layer feature information retained at this level.

6. The multi-layer lookup table image mapping method based on conditional awareness and hybrid expert mechanism according to claim 1, characterized in that, The cascaded mapping model employs a multi-mode sampling mechanism within the expert branch: Mode s: Use 2 2. Standard neighborhood sampling, with sampling coordinate offset as follows: Used to capture the underlying texture; Mode d: Uses dilated neighborhood sampling with an interval of 2, and the sampling coordinate offset is... Used to expand the receptive field; Mode y: Asymmetric Y-shaped sampling is used, with a sampling coordinate offset of . It is used to capture directional edge features.

7. The multi-layer lookup table image mapping method based on conditional awareness and hybrid expert mechanism according to any one of claims 1 to 6, characterized in that, Using a comprehensive loss function Train the cascaded mapping model: ， Among them, the main reconstruction loss Used to constrain the fundamental pixel differences between the final image and the ground truth; flat region residual regularization loss. The gating state is used to apply a reverse constraint to the output of the detail expert; when the gating module determines that the region is flat, this loss term will strongly suppress the response of the detail expert; gating regularization loss. By introducing a roughness threshold With target offset coefficient, guide gate modulation coefficient The distribution of .

Images