Extensible image coding system and method with multiscale semantic and edge prior fusion

Abstract

The present application relates to the technical field of image processing and data compression, and discloses a scalable image coding system and method based on multi-scale semantic and edge prior fusion, comprising the following steps: first, extracting edge feature maps and semantic feature maps from an original image, and compressing the same by using an encoder to generate a feature coding layer code stream for machine vision analysis; second, extracting a latent representation of the original image by using an analysis transformation network, and performing entropy encoding and entropy decoding by using a hyper-prior model and a slice-level auto-regressive network; and finally, in a synthesis transformation network at a decoding end, efficiently integrating the decoded edge and semantic features into a main decoding path of the synthesis transformation network at different resolution levels by using a multi-scale feature fusion network, so as to realize high-quality image reconstruction. By using grouping spatial attention, gate reconstruction and channel fusion to suppress feature redundancy, the present application can significantly improve the perceptual fidelity and structural consistency of the reconstructed image at a low bit rate.

Description

Technical Field

[0001] This invention relates to the field of image processing and data compression technology, and in particular to a scalable image coding system and method that integrates multi-scale semantics and edge priors. Background Technology

[0002] With the booming development of intelligent vision applications such as autonomous driving, intelligent monitoring, and augmented reality, the transmission and analysis of massive amounts of visual data pose a severe challenge to existing encoding technologies. Traditional image coding standards (such as JPEG, JPEG2000, HEVC, and VVC) are primarily designed to maximize compression efficiency for the human visual system (HVS), aiming to maintain good visual quality at low bit rates. However, in machine vision or computer vision tasks, machines are not concerned with pixel-level realism, but rather with the structured semantic information and key features contained in the image. Directly applying traditional HVS-oriented encoders to machine vision tasks leads to suboptimal bit rate allocation: a large number of bits are used to encode texture details that are of little significance to machine analysis, while crucial semantic features may be lost during compression.

[0003] To address the dual needs of human viewing and machine analysis, machine-oriented image coding and scalable coding frameworks have shown great potential. In existing technologies, scalable coding typically divides video streams into feature coding layers and image coding layers with different priorities. However, existing scalable schemes still suffer from the following shortcomings: most schemes primarily focus on temporal, spatial, or quality (PSNR) dimensions, failing to fully explore and utilize image features crucial for machine vision tasks; some methods based on model feature splitting are deeply coupled with specific recognition models, resulting in limited generalization ability; and some methods treat machine tasks and image reconstruction as independent processes, failing to effectively utilize machine features to guide the reconstruction of high-quality images, leading to low reconstruction efficiency. Summary of the Invention

[0004] To address the limitations of existing image coding technologies in simultaneously and efficiently serving the needs of machine vision and visual analysis, and the failure to fully utilize high-level semantic and structural information in scalable coding, this invention proposes a scalable image coding system and method that integrates multi-scale semantics and edge priors. Through hierarchical compression and feature-guided reconstruction, it achieves efficient machine vision analysis at low bit rates and can reconstruct images with high visual quality.

[0005] To achieve the above objectives, the present invention provides a scalable image coding system that fuses multi-scale semantics and edge priors, comprising:

[0006] The feature encoding layer is used to extract and compress feature maps containing semantic and structural information from the input image to generate the basic bitstream; and

[0007] An image coding layer, connected to the feature coding layer, is used to receive the input image and the decoded feature map from the feature coding layer, compress the residual information of the input image or itself, and use the decoded feature map as prior information to reconstruct the image during the decoding process to generate an enhanced bitstream;

[0008] The image coding layer includes a multi-scale feature fusion network, which fuses the decoded feature map with features from the main decoding path at different resolution scales to guide image reconstruction.

[0009] Furthermore, the feature encoding layer includes:

[0010] A feature extraction module is used to extract semantic feature maps and edge feature maps from the input image; and

[0011] The feature map encoder and decoder use the VTM-SCC codec, which includes screen content encoding tools in the VVC video coding standard, to compress the semantic feature map and edge feature map.

[0012] Furthermore, the image coding layer also includes:

[0013] Analyze the transform network used to encode the input image into a latent representation;

[0014] A super-prior encoder-decoder network, connected to the analysis transform network, is used to perform entropy modeling on the latent representation;

[0015] A slice-level autoregressive network, connected to the analysis transformation network and the super-prior encoder-decoder network, is used to slice the latent representation in the channel dimension and perform sequential entropy encoding and decoding based on the context of the decoded slices; and

[0016] A synthetic transformation network, connected to the slice-level autoregressive network and the multi-scale feature fusion network, is used to fuse the decoded quantized latent representation with the multi-scale prior features to reconstruct the output image.

[0017] Furthermore, the multi-scale feature fusion network includes:

[0018] Parallel structure extractors and Sobel information extractors are used to extract multi-scale features from edge feature maps and semantic feature maps, respectively.

[0019] An adaptive fusion module, connected to the structure extractor and the Sobel information extractor, is used to receive and fuse features from the structure extractor and the Sobel information extractor.

[0020] The adaptive fusion module sequentially includes a group spatial attention submodule, a spatial gating reconstruction submodule, and a channel fusion submodule. The group spatial attention submodule groups input features and generates attention weights using parallel convolutional paths and a spatial attention mechanism to weight and enhance the features. The spatial gating reconstruction submodule dynamically generates a gating mask based on the information content of the features, separating the features into information-rich and redundant parts and reconstructing them. The channel fusion submodule adaptively fuses the features after segmenting the feature channels and performing differentiated transformations on the high and low information content parts.

[0021] Furthermore, in the slice-level autoregressive network, an attention-convolution mean estimation module is used. This module integrates the attention mechanism and convolution transformation to estimate the Gaussian distribution parameters of each slice and efficiently reconstruct the latent representation of the image.

[0022] Furthermore, the super-prior encoding / decoding network includes a super-prior analysis network and a super-prior synthesis network. Both the super-prior analysis network and the super-prior synthesis network include shift window convolutional blocks, which are used to segment the input features in the channel dimension and output fusion to simultaneously capture local features and global dependencies.

[0023] Furthermore, the synthetic transformation network is an upsampling network, which fuses prior features of the corresponding scale from the multi-scale feature fusion network in at least two different upsampling stages by adding or concatenating features.

[0024] A scalable image coding method that fuses multi-scale semantics and edge priors, using the aforementioned scalable image coding system that fuses multi-scale semantics and edge priors, includes the following steps:

[0025] S1: Through the feature coding layer, semantic feature maps and edge feature maps are extracted from the input image and compressed using the VTM-SCC encoder to generate and transmit or store the basic bitstream;

[0026] S2: Decompress the base bitstream using the VTM-SCC decoder to obtain the decoded semantic feature map and the decoded edge feature map;

[0027] S3: At the encoding end of the image coding layer, the input image is encoded to generate an enhanced bitstream;

[0028] S4: At the decoding end of the image coding layer, the decoded semantic feature map and decoded edge feature map are transformed into multi-scale prior features through a multi-scale feature fusion network, generating two feature tensors t1 and t2, which are then combined with the feature reconstruction latent representation of the main decoding path. The images are then fused and reconstructed. .

[0029] The scalable image coding method that fuses multi-scale semantics and edge priors includes the following steps in S3 and S4: encoding and decoding the input image.

[0030] The input image x is mapped to a latent representation y by analyzing the transform network;

[0031] All the way through the super-prior analysis network h a Entropy modeling is performed on the latent representation y to obtain the super-prior representation z. The super-prior representation z is then passed through an arithmetic encoder to obtain the image compressed bitstream. Both the super-prior analysis network and the super-prior synthesis network use shifted window convolutional blocks to capture local and global dependencies. At the decoding end, the arithmetic decoder first decodes the compressed bitstream, and then the super-prior synthesis network h... s Then the decoded hyperprior representation Reconstructed into probability distribution parameters (mean μ and standard deviation σ) for estimating the latent representation y.

[0032] Another approach involves inputting the latent representations y, μ, and σ together into SliceNet (a slice-level autoregressive network) to obtain the reconstructed representation. Finally, the edge map and semantic map output from the feature encoding layer are used as prior information to generate two feature tensors t1 and t2 through MFFNet (Multi-Scale Feature Fusion Network). These tensors are then fused into the synthesis transformation network g. s In the middle, g s right The final decoding yields the reconstructed image. .

[0033] A scalable image coding method that integrates multi-scale semantics and edge priors, wherein the multi-scale feature fusion network performs feature fusion in the following steps:

[0034] By using a parallel structure extractor and a Sobel information extractor, features at two different scales are extracted from the edge feature map and the semantic feature map, respectively.

[0035] Structural and semantic features at the same scale are concatenated via channels;

[0036] The stitched features are processed by an adaptive fusion module, which includes feature enhancement through a group spatial attention module, reduction of spatial redundancy through a spatial gating reconstruction submodule, and optimization of channel representation through a channel fusion submodule.

[0037] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0038] 1. The scalable image coding system and method of multi-scale semantic and edge prior fusion of the present invention takes into account the needs of both human and machine vision. The feature coding layer provides compact machine-analyzable features (semantics, edges), which can be applied to other machine vision and image analysis tasks. The image coding layer achieves efficient image compression and obtains visually high-quality reconstructed images. Therefore, the image coding system proposed in this invention achieves scalability for machine vision tasks and image compression.

[0039] 2. The scalable image coding system and method of multi-scale semantic and edge prior fusion of the present invention has high reconstruction quality: The innovative multi-scale feature fusion network can effectively fuse semantic and structural prior information at different resolution levels of the decoder, accurately guide image reconstruction from macro structure to micro details, and significantly improve the visual fidelity of the reconstructed image, especially in terms of edge preservation and texture restoration.

[0040] 3. The scalable image coding system and method of the present invention, which integrates multi-scale semantics and edge priors, has high compression efficiency: by employing a super-prior network with hybrid shifted window convolutional blocks and a slice-level autoregressive network with integrated attention mechanism, it can more accurately capture the local and global correlations of the latent representation, improve the accuracy of probability estimation, and thus effectively reduce the bit rate under the same visual quality.

[0041] 4. The scalable image coding system and method of multi-scale semantic and edge prior fusion of the present invention has strong flexibility and generalization: the feature coding layer uses a standardized tool (VTM-SCC) to compress general feature maps (semantic maps, edge maps), which is not strongly coupled with specific machine task models, thus enhancing the versatility and flexibility of the system. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the overall architecture of the scalable image coding system for multi-scale semantic and edge prior fusion of the present invention. The diagram shows the main components of the system, including a feature coding layer and an image coding layer. The feature coding layer includes a feature extraction module, encoder 1 (VTM-SCC encoder), and decoder 1 (VTM-SCC decoder). The image coding layer contains an analysis transform network (g) based on a variational autoencoder. a ) and synthetic transformation network (g s ), Advanced Prior Codec Network (h a , h s The connections and data interaction processes between encoder 2 (arithmetic encoder) and decoder 2 (arithmetic decoder), slice-level autoregressive network (SliceNet), and multi-scale feature fusion network (MFFNet).

[0043] Figure 2 This is a schematic diagram of the input image and its corresponding feature map provided by the present invention; wherein, (a) is the original input image, (b) is the corresponding edge feature map, and (c) is the corresponding semantic feature map; the diagram intuitively shows the two key prior information extracted and processed by the feature coding layer.

[0044] Figure 3 This is a schematic diagram of the advanced prior codec network and its internal shifted window convolutional blocks in this invention; wherein, (a) is a schematic diagram of the internal structure of the advanced prior codec network, and h in the figure a For the prior analysis network, h s (a) is the advanced prior synthesis network; (b) is the shifted window convolutional block (SCB1); (c) is the shifted window convolutional block (SCB2).

[0045] Figure 4 This is a schematic diagram of the workflow of the SliceNet and its internal modules in this invention; wherein, (a) is a schematic diagram of the workflow of the SliceNet; (b) is a schematic diagram of the workflow of the Hierarchical Attention Latent Feature Optimization Module (SALRM) in the SliceNet; and (c) is a schematic diagram of the workflow of Attention Convolution Mean Estimation (ACME) in the Hierarchical Attention Latent Feature Optimization Module (SALRM). The figure depicts the process of slicing the latent representation y in the channel dimension, sequential entropy encoding and decoding (including encoder 3 (arithmetic encoder) and decoder 3 (arithmetic decoder)), parameter estimation through Attention Convolution Mean Estimation (ACME), and reconstruction of the latent representation.

[0046] Figure 5 This is a schematic diagram of the structure of the Multi-Scale Feature Fusion Network (MFFNet) in this invention; the diagram shows how the parallel structure extractor and Sobel information extractor extract multi-scale features from the edge feature map and semantic feature map, and generate prior features (t1, t2) for optimized reconstruction via the Adaptive Fusion Module (AFM).

[0047] Figure 6 This is a schematic diagram of the Adaptive Fusion Module (AFM) and its internal structure in this invention; wherein, (a) is a detailed internal structure diagram of the Adaptive Fusion Module (AFM), (b) is the cascaded structure and data processing flow of the Group Spatial Attention Submodule (GSA), (c) is the cascaded structure and data processing flow of the Channel Fusion Submodule (CFM), and (d) is the cascaded structure and data processing flow of the Spatial Gated Reconstruction Submodule (SGR).

[0048] Figure 7These are the comparison results of the objective evaluation criteria of depth image structure and texture similarity (dists) and learned perceptual image patch similarity (lpips) used in this invention; wherein, (a) is the result of the objective evaluation criteria of depth image structure and texture similarity (dists); and (b) is the result of the objective evaluation criteria of learned perceptual image patch similarity (lpips).

[0049] Figure 8 This is a comparison result of visual quality evaluation used in this invention. The first row of the figure is the complete image, and the second and third rows are magnified views of two partial screenshots from the first row image. From left to right, the images in each column are, in order, the original image, and the decompressed and reconstructed images using the Ours method (the method of this patent), the TCM method, the CDC method, the qres34m method, and the qarv_base method. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention. (Refer to...) Figure 1 The diagram illustrates the overall architecture of a preferred embodiment of the scalable image coding system based on multi-scale semantic and edge prior fusion of the present invention. The system is generally divided into a Feature Coding Layer and an Image Coding Layer. Depending on the specific application scenario, the two coding layers can be implemented individually or simultaneously, thus achieving flexibility and scalability in image compression.

[0051] Example 1

[0052] Reference Figure 1 and Figure 2As shown, in the Feature Coding Layer, features are first extracted from the original input image to obtain two feature maps: an edge feature map and a semantic feature map. The edge feature map is generated by DexiNed, a dense deep convolutional neural network proposed by Soria in 2023. This model is designed for accurate edge detection and can output detailed contour information. The semantic feature map is generated by UPerNet, a scene-understanding-based semantic feature method proposed by Zhou Bolei in 2019, to obtain pixel-level semantic category information. Then, the edge map and semantic map are encoded and decoded using VTM's SCC (Screen Content Coding) tool (VTM-SCC20.0), corresponding to... Figure 1 The VTM-SCC encoder consists of encoder 1 and decoder 1. The SCC encoding tool is specifically designed for screen content (such as text, lines, etc.), and is better suited for handling images with high contrast, sharp edges, and limited color gamut. In the VTM-SCC encoder, intra-frame coding mode is used, with 1 frame count, 0 frame skips, and a quantization parameter (QP) of 40. The SCC encoding tool is enabled, including Intra-Block Copy (IBC) and Palette Mode. The feature coding layer compresses the feature map at a low bit rate, generating a base layer bitstream. This base layer bitstream can be decoded independently to obtain decoded edge and semantic feature maps. The feature coding layer is designed for machine vision tasks (such as object detection and image segmentation), providing crucial semantic and structural information with extremely low latency.

[0053] Example 2

[0054] Reference Figure 1 As shown, this embodiment uses both a feature coding layer and an image coding layer to compress the image, realizing a scalable image coding system that fuses multi-scale semantics and edge priors. The overall process is summarized as follows:

[0055] 1. Feature Encoding Layer: The feature encoding layer includes a feature extraction module and a feature map encoder. Following the steps in Example 1, in the feature encoding layer, the feature extraction module (composed of the DexiNed model and the UperNet model) first extracts two key feature maps from the original input image: edge feature maps and semantic feature maps. For example... Figure 2 As shown, the edge feature map clearly outlines the contours and structural boundaries of objects in the image, while the semantic feature map provides pixel-level semantic category information. The feature map encoder uses screen content encoding tools from video coding standards to compress the semantic feature map and the edge feature map. In this embodiment, Figure 1Encoder 1 and Decoder 1 in the code are specifically VTM-SCC20.0 codecs. The two decoded feature maps output from Decoder 1 are input into the MFFNet module for image reconstruction and final decoding in the image coding layer.

[0056] 2. Image Coding Layer: The image coding layer uses a variational autoencoder (VAE) framework, employing edge and semantic feature maps decoded from the feature coding layer as prior features to enhance image compression. For example... Figure 1 As shown, the image coding layer includes an analysis transform network g. a Super-prior encoder-decoder network, slice-level autoregressive network (SliceNet), multi-scale feature fusion network (MFFNet), and synthetic transformation network g s The advanced prior codec network includes the advanced prior parsing network h. a and the super-prior synthesis network h s Its core operation is: at the encoding end, analyze the transform network g. a The input image x is transformed into a latent representation y, and the hyperprior analysis network h... a The latent representation y is further compressed to obtain the super-prior representation z, which is then passed through encoder 2 (arithmetic encoder) to obtain the compressed image bitstream. At the decoding end, decoder 2 (arithmetic decoder) first decodes the compressed bitstream, and then the super-prior synthesis network h... s Then the decoded hyperprior representation The parameters (mean μ and standard deviation σ) are reconstructed for estimating the probability distribution of y. Then, y, μ, and σ are input together into SliceNet (a slice-level autoregressive network) to obtain the reconstructed result. Finally, the edge map and semantic map output from the feature encoding layer are used as prior information to generate two feature tensors t1 and t2 through MFFNet (Multi-Scale Feature Fusion Network), which are then input into the synthesis transformation network g. s In the middle, g s right The final decoding yields the reconstructed image. The detailed implementation methods of each core module are as follows:

[0057] Step 1: Analyze the transform network g a Implementation:

[0058] Reference Figure 1As shown, the input image x first passes through a Feature Enhancement Module (FEB), which contains densely connected blocks to enrich the feature representation. Subsequently, it passes through multiple cascaded Residual Blocks (RBs) and Residual Blocks with stride (RBS, Cheng Zhengxue, 2020), each block containing convolution, an activation function (GELU), and downsampling operations, progressively reducing the spatial size and increasing the number of channels. Finally, after a CONV convolutional layer completes the size transformation, a Group Spatial Attention Module (GSA) is also embedded. Figure 6 As shown in the figure, this module dynamically adjusts feature weights by processing convolutional kernels of different sizes in parallel, focuses on important regions, and finally outputs the latent representation y.

[0059] Step 2: Super-prior encoder-decoder network h a and h s Implementation:

[0060] Reference Figure 3 As shown in (a), the advanced prior codec network includes an advanced prior analysis network h. a and the super-prior synthesis network h s The advanced prior analysis network h used in this embodiment a and the super-prior synthesis network h s The innovation of this invention lies in the fact that, compared with the prior knowledge module in the prior knowledge analysis network h, the prior knowledge module is more advanced. a and the super-prior synthesis network h s Both methods introduce Shift-Window Convolutional Blocks (SCBs), including SCB1 and SCB2, which segment the input features along the channel dimension and output a fused result to simultaneously capture local features and global dependencies. (See reference...) Figure 3 As shown in (b) and 3(c), the structural difference between SCB1 and SCB2 is that SCB1 uses a window network module (W-Block), while SCB2 uses a shifted window network module (SW-Block). W-Block and SW-Block are composed of Swing Transformers containing window multi-head self-attention mechanism (W-MSA) and shifted window multi-head self-attention mechanism (SW-MSA), respectively. They were proposed by Liu Ze in 2021 and are both prior art, so they will not be described in detail in this embodiment.

[0061] In this embodiment, the input feature is a three-dimensional tensor. Its dimensions are , representing the number of channels, height, and width, respectively. The implementation steps for SCB1 and SCB2 are as follows:

[0062] First, the input image is processed through a point convolutional layer. Preprocessing is performed to generate adjusted features. :

[0063]

[0064] in Maintain and Same dimensions . It is uniformly divided into two subtensors and The number of channels for each subtensor is :

[0065]

[0066] Next, It is fed into a residual block to extract local features, resulting in :

[0067]

[0068] and It is then fed into the W-Blcok module (located in SCB1) or the SW-Blcok module (located in SCB2) to capture global dependencies and generate... Then, and The data is concatenated along the channel dimension and then fused through another point convolutional layer to obtain intermediate features. :

[0069]

[0070] Final output :

[0071]

[0072] In this embodiment, SCB1 and SCB2 are integrated into the present invention. and In this context, it is used to process prior representations, i.e., side information. Specifically, the input latent representation Side information is generated by the encoder. :

[0073]

[0074] The quantized z is output as a compressed bitstream by encoder 2, and then passed through decoder 2 to obtain the reconstructed data. , go through Mean of the probability distribution of the generative entropy model with standard deviation :

[0075]

[0076] The SCB1 and SCB2 modules combine the local feature extraction capabilities of Convolutional Neural Networks (CNNs) with the global modeling advantages of SwingTransformers, further improving the rate-distortion performance of image compression. They employ an efficient channel segmentation strategy, where the input feature tensor is first divided into two branches, processed separately by the CNN and SwingTransformer. This method effectively reduces the computational burden of each branch, while feature fusion is achieved through point convolution operations, ensuring excellent performance while maintaining high efficiency. Through feature merging and fusion mechanisms, the SCB1 and SCB2 modules ensure full interaction between local details (such as edge textures) and global structure (such as object layout) at different processing stages. This interaction enhances the entropy model's ability to estimate the probability distribution of latent representations.

[0077] Step 3: Implementation of SliceNet:

[0078] Reference Figure 4 As shown in (a), the SliceNet is the core of entropy coding.

[0079] Slicing: From analyzing the transformation network g a The resulting latent representation y is divided into several predefined slices along the channel dimension, denoted as y = y / y. In this embodiment, the latent representation The shape is ,in Indicates batch size. Indicates the number of channels. and They are height and width, respectively, by their channel dimensions. Average score Each subset, each slice The number of channels is This forms a slice set. In this example, i=5. This segmentation strategy constructs an autoregressive mechanism for entropy coding, allowing the encoding process of each slice to rely on previously processed slice information. This autoregressive property effectively captures local and non-local correlations in the latent space by explicitly modeling the dependencies between channels, thus improving the accuracy of the probability distribution.

[0080] Sequential entropy encoding and decoding: In order to make reasonable use of the information from previous slices, parameters are used to encode and decode. Control the traversal of slices; when If <0, use all previously reconstructed slices; otherwise, use only the previous slices. Each slice. This flexible configuration not only reduces computational resource consumption but also avoids the risk of model overfitting due to excessive historical information, while ensuring sufficient contextual support to improve the accuracy of entropy modeling. For each slice First, construct two core support tensors: and These tensors are synthesized by the hyperprior synthesis network h. s The generated global latent space mean and standard deviation This is achieved by stitching together the reconstructed slices along the channel dimension. Next, ,and The input is fed into the Hierarchical Attention Latent Feature Optimization (SALRM) module. Within the SALRM module, to efficiently estimate the parameters of each slice, this framework introduces an Attention Convolution Mean Estimation (ACME) module. For example... Figure 4 As shown in (b) and 4(c), this module combines the SWatten attention mechanism with multiple convolutional layers and the GELU activation function. The SWatten attention mechanism was proposed by Liu Jinming in 2023. For slice i, the parameter vector... Both `std_vector` and `std_vector` are processed by the ACME module to generate accurate mean parameters. and standard deviation parameter As shown in equations (8)-(10), the design of the ACME module fully utilizes the nonlocal modeling capability of the attention mechanism and the local feature extraction capability of the convolutional network, thereby significantly improving the robustness and accuracy of parameter estimation. and Then, it is cropped in the spatial dimension to match the original slice. The dimensions are set and stored in a list for later use, expressed by the formula:

[0081]

[0082]

[0083]

[0084]

[0085] In obtaining and Then, the Gaussian conditional model calculates the current slice based on these parameters. likelihood value And store it in a likelihood list for use in encoder 3 (arithmetic encoder) and decoder 3 (arithmetic decoder) for... The encoding and decoding process. The reconstruction of the slice is shown in Equation (12). The slice is quantized and averaged, and then output by decoder 3 to reconstruct the initial slice. Then, the Latent Residual Prediction (LRP) module, proposed by Minnen in 2020, is employed to further optimize reconstruction quality and reduce quantization error. First, the... and slices The assembly (module C) is then performed, followed by the generation of residuals using the LRP module. The function limits its magnitude to the range [-0.5, 0.5], ultimately yielding an optimized reconstructed slice. :

[0086]

[0087]

[0088] Among them, in equation (12) This indicates a quantization operation, in equation (13) This is a functional representation of LRP. After all slices have been processed, the reconstructed slices are stitched together along the channel dimension to form a complete reconstructed latent representation. ,Right now:

[0089] This process progressively constructs the joint probability distribution of the latent space through autoregression, leveraging the contextual information of previous slices to enhance the encoding efficiency of subsequent slices. Simultaneously, it combines attention mechanisms and residual prediction to significantly improve reconstruction accuracy. SliceNet effectively reduces the computational complexity of entropy models and improves the rate-distortion performance of multi-resolution image compression.

[0090] Step 4: Implementation of the Multi-Scale Feature Fusion Network (MFFNet):

[0091] Reference Figure 5 As shown, MFFNet is the key to connecting the feature coding layer and the image coding layer, thereby improving the reconstruction quality.

[0092] The MFFNet network consists of: a structure extractor, a Sobel information extractor, and an adaptive fusion module (AFM).

[0093] Feature extraction: The decoded edge map and semantic map are respectively input into two structurally symmetrical feature extractors, namely the structure extractor and the Sobel information extractor.

[0094] In this embodiment, the semantic map size is [16, 3, 128, 128], and it is input into the Sobel information extractor. The Sobel information extractor contains two independent branches, each generating two feature maps of different scales through convolutional layers and downsampling convolutions of different stride lengths (stride=4 and stride=8). The Sobel information extractor outputs Sobel features (sobel_feats), a feature dictionary containing two key-value pairs:

[0095] f1: Dimensions are [B, 128, H / 4, W / 4]

[0096] f2: Dimensions are [B, 160, H / 8, W / 8]

[0097] In this embodiment, B=16, H=W=128. Simultaneously, the edge map is input into the structure extractor. The structure extractor employs the exact same network structure and downsampling strategy as the Sobel information extractor, also generating a feature dictionary containing two scales, i.e., edge features. The structure extractor and the Sobel information extractor work in parallel, providing two sets of spatially corresponding multi-scale features for the subsequent fusion step.

[0098] Next, in this embodiment, to fully utilize the complementary information contained in the Sobel features and edge features, the feature vectors from the two extractors are concatenated along the channel dimension. At the H / 4 × W / 4 scale, f1 from the 128-channel Sobel feature is concatenated with f1 from the edge feature with the same number of channels, generating a fused feature map t1 ([B, 256, H / 4, W / 4]) with 256 channels. Similarly, at the H / 8 × W / 8 scale, f2 from the 160-channel Sobel feature is concatenated with f2 from the edge feature with the same number of channels, generating a fused feature map t2 ([B, 320, H / 8, W / 8]) with 320 channels. In this way, the present invention integrates basic and structural information at different resolution levels. These concatenated high-dimensional feature maps can then be fed into subsequent convolutional layers or attention modules for further dimensionality reduction and refinement to generate a more compact and information-rich fused feature representation. Finally, the two tensors output by the AFM module, t1 and t2, are fed into the synthesis transform network. Reconstruction of auxiliary images. The AFM module aims to achieve efficient feature representation through multi-scale feature enhancement and redundancy suppression.

[0099] Reference Figure 6As shown in (a), the Adaptive Fusion Module (AFM) includes the Grouped Spatial Attention (GSA) submodule, the Spatial Gate Reconstructor (SGR) submodule, and the Channel Fusion Module (CFM) submodule.

[0100] like Figure 6 As shown in (a) and (b), the first stage of the Adaptive Fusion Module (AFM) is completed by the Group Spatial Attention Submodule (GSA). Its purpose is to enhance feature representation through multi-scale spatial attention, while avoiding the problems of spatial information loss or excessive computational cost commonly found in traditional attention mechanisms. In the deep feature maps of convolutional neural networks (CNNs), different spatial locations and channels often carry different levels of semantic information, while global pooling or channel dimensionality reduction may ignore local details, resulting in limited feature representation capabilities. The design of the Group Spatial Attention Submodule in this invention is inspired by multi-scale feature extraction and grouped convolution, aiming to capture long- and short-range dependencies through grouped parallel processing and cross-dimensional interaction, while preserving the spatial resolution of the input features.

[0101] Given an input feature map (in For the number of channels, and (Height and width respectively), the group spatial attention submodule first... Divided along the channel dimension Sub-feature groups, i.e. Each group of dimensions is Number of groups Setting it as a hyperparameter (such as 32 or 64) not only reduces the complexity of subsequent calculations (from...) Down to This also allows each sub-feature group to focus on specific semantic patterns, such as edges, textures, or target regions. The grouping strategy borrows from the idea of ​​grouped convolution, but in the group spatial attention submodule, it focuses more on providing a diverse input base for the attention mechanism. The group spatial attention submodule uses two parallel branches to extract multi-scale features. The first branch uses a 1×1 convolutional kernel to capture local cross-channel dependencies. First, global average pooling (AGP) is implemented by combining horizontal and vertical spatial pooling to encode positional information. The AGP module is defined as follows:

[0102]

[0103] in, and These represent spatial statistics in the horizontal and vertical directions, respectively. This operation differs from global pooling; it preserves spatial distribution information and better captures the geometric properties of local features. The pooled features in these two directions are concatenated (module C) and then convolved with a 1×1 convolution. They are then passed through two SGM (Sigmoid) modules and multiplied by the input F(c,h,w) to generate spatial descriptors. To normalize the features, this spatial descriptor is further processed by a GN (Group Normalization) module to obtain the features. The second branch expands the receptive field using 3×3 convolutional kernels to generate multi-scale features. To fuse the outputs of the two branches and generate accurate spatial attention weights, the GSA module introduces a cross-spatial interaction mechanism. First, global statistics are calculated to capture long-range dependencies:

[0104]

[0105]

[0106] Here, SFM represents the softmax module. Subsequently, spatial attention with pixel-level pairwise relationships is generated through matrix multiplication.

[0107]

[0108] in, This is the Sigmoid function (SGM module). This represents spatial attention. Finally, the group spatial attention submodule applies spatial attention to the input features, outputting enhanced features:

[0109]

[0110] Compared to traditional attention mechanisms, the group space attention submodule has the advantage of its multi-scale parallel design and grouping strategy, which not only enhances the semantic richness of features but also maintains computational efficiency through local pooling and matrix operations. This method is particularly suitable for tasks requiring fine-grained spatial information, such as high-resolution image processing or small object detection.

[0111] like Figure 6 As shown in (c) and 6(d), the second stage of the Adaptive Fusion Module (AFM) consists of a Spatial Gated Reconstruction Submodule (SGR) and a Channel Fusion Submodule (CFM), which optimize redundancy in the spatial and channel dimensions, respectively. The features output by the earlier Group Spatial Attention Submodule (GSA) are... Although multi-scale enhancement is performed, significant spatial redundancy may still be present, such as background regions or low-information pixels. This redundancy not only increases computational burden but may also interfere with feature extraction in subsequent tasks. The design goal of SGR is to refine spatial features through gated separation and reconstruction mechanisms, while CFM further reduces channel redundancy and optimizes feature fusion. The collaborative work of these two modules enables the AFM module to achieve efficient feature representation across multiple dimensions. SGR receives features from GSA, and its core idea is to dynamically separate and reconstruct spatial representations using the information distribution of features. First, the input features are normalized using a GN layer. SGR does not directly calculate variance but instead quantifies the spatial information richness of each channel through a trainable scaling factor in the GN layer. Normalized weights are calculated using the normalization weight calculation module (GN.weight / sum(GN.weight)). :

[0112]

[0113] in, The variance representing the spatial dimension. This reflects the information richness of each channel. Unlike mean-based global pooling, variance statistics are better able to capture the dynamic characteristics of spatial variations. Then, and Multiplying them together yields the normalized characteristics. . After passing through the SGM module and the threshold module in sequence, they are divided into two categories:

[0114]

[0115] in, For the Sigmoid function (SGM module), The separation threshold, For the indicator function (threshold module), and These are masks for information-rich and redundant regions, respectively. The generated masks are processed using two multiplication operators. and Applying to the original features to achieve feature separation:

[0116]

[0117] Since directly discarding redundant parts would result in information loss, SGR employs a reconstruction strategy to fuse separated features:

[0118]

[0119] in, and For adjustable parameters (here) 0.5, 0.3), respectively controlling the retention ratio of redundant features and the enhancement magnitude of the main features, This indicates a concatenation operation along the channel dimension. The generated... It is more compact in spatial dimension, preserving key information while effectively reducing the impact of redundant regions. SGR's gating mechanism originates from dynamic network design, but its innovation lies in combining spatial variance and reconstruction strategies, making it more suitable for feature condensation.

[0120] The CFM module receives data from... The features are further optimized through segmentation, transformation, and fusion along the channel dimension. In deep CNNs, channel redundancy is a common problem; many channels may only carry repetitive or inefficient information, leading to wasted parameters. CFM employs a segmentation-transformation-fusion strategy, first... Upon entering the channel sharding module, it is divided into two subsets along the channel dimension:

[0121]

[0122]

[0123] in, , These represent high-information and low-information channel subsets, respectively. Segmentation ratio. This is a compromise that can be adjusted according to task requirements. The high-information branch extracts features through a combination of multiple 3×3 convolutions (group convolutions) and 1×1 convolutions, and the outputs of the two operators are fused through an adder.

[0124]

[0125] in, Use 3×3 convolution kernels (set the number of kernels to 2 or 4). It is a 1×1 convolution kernel. The convolution operation is represented by the group convolution, which reduces computational cost, while the 1×1 convolution enhances channel interaction. The low-information branch is transformed by a lightweight 1×1 convolution and reused with the original path through an addition operator:

[0126]

[0127] in, The kernel is 1×1, and the addition method preserves the structural information of the original features. To achieve efficient fusion, CFM uses the AGP module to calculate global statistics. and :

[0128]

[0129]

[0130] Then, the fusion weights are generated using the SFM module:

[0131]

[0132] Then, the two branches are weighted using a multiplication operator, and the final data slices are combined with the addition to obtain the output features:

[0133]

[0134] This adaptive fusion mechanism ensures a dynamic balance between high and low information branches, reducing channel redundancy and enhancing feature diversity and representativeness. While CFM's design is inspired by channel attention mechanisms, its segmentation and transformation strategies offer significant advantages in efficiency and performance.

[0135] Step 5: Synthesize the transformation network g s Implementation:

[0136] like Figure 1 As shown, g s Decoding begins with the lowest-resolution quantized latent representation ŷ. Before the first upsampling module, ŷ and the lowest-resolution prior features t1 output by MFFNet are fed into the CAT fusion module (element-wise addition). Then, through a series of RB and RBU blocks (Cheng Zhengxue, 2020), the resolution is gradually increased. When an intermediate resolution is reached, it is fused again with the higher-resolution prior features t2 output by MFFNet (element-wise addition). Upsampling continues to the original image resolution. Finally, a CONV convolutional layer completes the resizing, and a FEB module symmetrical to the encoder is used for post-processing to output the final reconstructed image. With g a Similarly, in g s The algorithm also incorporates a GSA module to dynamically adjust feature weights and focus on important regions. This multi-scale fusion mechanism enables the decoder to utilize edge and semantic prior information from the feature encoding layer, significantly improving the reconstruction quality of object boundaries and complex textures.

[0137] Step 6: Dataset Training and Optimization

[0138] For the image coding layer, this embodiment uses the public dataset ImageNet as the training set. ImageNet contains approximately 34K training images, covering 1000 categories, with 30-40 images per category. The test set is the Kodak dataset, which is widely recognized in image compression research. Training on the dataset uses the Adam optimizer with a batch size of 16 and an initial learning rate of 0.0001, which is maintained at 0.00001 from the 200th epoch. The epoch count is set to 250, and the model is trained using an Nvidia RTX3090.

[0139] The loss function consists of two parts: reconstruction loss and bit rate loss. The two are weighted and summed to obtain the final loss value, thereby optimizing the model parameters.

[0140] (1) The reconstruction loss is measured by mean squared error (MSE) of the reconstructed image. With the original image Specifically

[0141]

[0142] in, Calculate the pixel-level mean square error, which reflects the fidelity of the reconstructed image; This is a hyperparameter used to adjust the weight of the reconstruction loss, allowing for flexible adjustment of the emphasis on reconstruction quality under different compression scenarios. This embodiment sets... The value range of is {3,4,5,6,7,8} to meet different bpp (bit rate) compression requirements; the constant 5 in the re_loss formula further amplifies the impact of reconstruction loss, ensuring that the model prioritizes image quality during the optimization process.

[0143] (2) Bit rate loss is used to measure the coding efficiency in the compression process, and it consists of two parts: the reconstructed latent representation. Bit rate loss and the reconstructed prior representation Bit rate loss The calculation is as follows:

[0144]

[0145]

[0146] in and They are respectively and The likelihood probabilities are obtained from the SliceNet module and decoder 2, respectively. Let be the total number of pixels in the image. The bit rate loss, calculated by summing the logarithms of the likelihood probabilities and normalizing, reflects the average code length of the compressed data, with the goal of minimizing the bit rate to improve compression efficiency.

[0147] The final loss function is defined as

[0148]

[0149] By adding the reconstruction loss to the bitrate loss, the trained model can improve the quality of the reconstructed image as much as possible while maintaining a low bitrate. This joint optimization strategy is particularly important in image compression tasks, as it can effectively balance compression ratio and visual fidelity.

[0150] Step 7: Evaluation of Experimental Results

[0151] 1. Objective quality evaluation

[0152] This embodiment employs two evaluation metrics focused on perceived quality: dists (proposed by Ding Keyan, 2020) and lpips (proposed by Zhang Richard, 2018). These two metrics were chosen due to their high consistency with human visual perception. dists is particularly adept at evaluating structural and texture distortion, while lpips primarily measures perceptual similarity. The comparison results of this patented method with methods in the same field on the Kodak test set are as follows: Figure 7 As shown. Regardless of whether it is based on the dists metric (such as the similarity between the depth image structure and texture) Figure 7 (a) or based on the lpips metric (e.g., learning-aware image patch similarity) Figure 7 (b) The method(ours) of this patent achieves the best rate-distortion performance. As can be clearly seen from the figure, the RD curve of the method(ours) of this patent forms the lower envelope of all the comparison methods and is located below the comparison methods. This means that at any given bit rate (bpp), the method(ours) of this patent can generate the reconstructed image with the minimum perceptual distortion. Its performance is significantly better than other advanced methods such as TCM (Liu Jinming, 2023), CDC (Yang Ruihan, 2023), qres34m (Duan Zhihao, 2023) and qarv_base (Duan Zhihao, 2024), and it far surpasses traditional coding standards such as bpg, jpeg, jpeg2000 and jpegxl.

[0153] 2. Subjective quality evaluation

[0154] The decompressed image of the Kodak test set is as follows: Figure 8As shown, the Kodak test results are compared, from left to right: the original image and decompressed images from other image compression methods, including: the method of this patent (ours), TCM method, CDC method, qres34m method, and qarv_base method. It can be seen that the method of this patent (ours) exhibits a significant advantage in reconstructing the fine texture and edges of the image. Methods such as TCM, CDC, qres34m, and qarv_base generally show varying degrees of blurring and loss of detail when processing eyelashes and hat textures. The fundamental reason is that these single-layer compression methods need to consider both the overall structure and local details of the image during encoding. At low bitrates, a trade-off must be made between the two, often prioritizing the smoothness of the overall structure at the expense of high-frequency texture information. Through the MFFNet module proposed in this invention, these structural and semantic priors are gradually integrated into the main decoding path at multiple scales of the decoder, achieving precise guidance and reconstruction from macroscopic structure to microscopic details. Therefore, when methods such as TCM, CDC, qres34m, and qarv_base smooth out details like eyelashes and hat holes due to insufficient bit allocation, the method of this invention can compensate for and enhance these key details during decoding by using the precise contour information provided by the edge map and the object region information provided by the semantic map, thereby reconstructing a clearer and more realistic image.

[0155] Experimental results show that the method of the present invention achieves a good balance between compression efficiency and image quality fidelity, and outperforms other benchmark methods in both objective quality evaluation and subjective quality evaluation.

[0156] The embodiments and implementation process of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions, and variations can be made to these embodiments, including components, without departing from the principles and spirit of the present invention, and these variations still fall within the protection scope of the present invention.

Claims

1. A scalable image coding system that fuses multi-scale semantics and edge priors, characterized in that, include The feature encoding layer is used to extract and compress feature maps containing semantic and structural information from the input image to generate the basic bitstream. as well as An image coding layer, connected to the feature coding layer, is used to receive the input image and the decoded feature map from the feature coding layer, compress the residual information of the input image or itself, and use the decoded feature map as prior information to reconstruct the image during the decoding process to generate an enhanced bitstream; The image coding layer includes a multi-scale feature fusion network, which fuses the decoded feature map with the features of the main decoding path at different resolution scales to guide image reconstruction. The image coding layer further includes: Analyze the transform network used to encode the input image into a latent representation; A super-prior encoder-decoder network, connected to the analysis transform network, is used to perform entropy modeling on the latent representation; A slice-level autoregressive network, connected to the analytical transformation network and the super-prior encoder-decoder network, is used to slice the latent representation in the channel dimension and reconstruct the latent representation by performing sequential entropy encoding and decoding based on the context of the decoded slices; and A synthetic transformation network, connected to the slice-level autoregressive network and the multi-scale feature fusion network, is used to fuse the decoded quantized latent representation with the multi-scale prior features to reconstruct the output image. The multi-scale feature fusion network includes: Parallel structure extractors and Sobel information extractors are used to extract multi-scale features from edge feature maps and semantic feature maps, respectively. An adaptive fusion module, connected to the structure extractor and the Sobel information extractor, is used to receive and fuse features from the structure extractor and the Sobel information extractor. The adaptive fusion module includes, in sequence, a group spatial attention submodule, a spatial gating reconstruction submodule, and a channel fusion submodule; The group spatial attention submodule enhances the features by grouping the input features and generating attention weights using parallel convolutional paths and spatial attention mechanisms. The spatial gating reconstruction submodule dynamically generates gating masks based on the information content of the features, separates the features into information-rich parts and redundant parts, and reconstructs them. The channel fusion submodule adaptively fuses the feature channels after segmenting them and performing differential transformations on the high and low information content parts.

2. The scalable image coding system based on multi-scale semantic and edge prior fusion according to claim 1, characterized in that: The feature coding layer includes: A feature extraction module is used to extract semantic feature maps and edge feature maps from the input image; and The feature map encoder uses screen content encoding tools from video coding standards to compress the semantic feature map and edge feature map.

3. The scalable image coding system based on multi-scale semantic and edge prior fusion according to claim 1, characterized in that: In the slice-level autoregressive network, an attention-convolution mean estimation module is used. This module integrates the attention mechanism and convolution transformation to estimate the Gaussian distribution parameters of each slice.

4. The scalable image coding system based on multi-scale semantic and edge prior fusion according to claim 1, characterized in that: The super-prior encoding / decoding network includes a super-prior analysis network and a super-prior synthesis network. Both the super-prior analysis network and the super-prior synthesis network include shift window convolutional blocks, which are used to segment the input features in the channel dimension and output fusion to capture local features and global dependencies simultaneously. The synthetic transformation network is an upsampling network, which fuses prior features from the corresponding scale of the multi-scale feature fusion network in at least two different upsampling stages by adding or concatenating features.

5. A scalable image coding method for multi-scale semantic and edge prior fusion, using the scalable image coding system for multi-scale semantic and edge prior fusion as described in any one of claims 1-4, characterized in that, Includes the following steps: S1: Using the VTM-SCC encoder as a screen content encoding tool, semantic feature maps and edge feature maps are extracted from the input image and compressed by the VTM-SCC encoder through the feature encoding layer, generating and transmitting or storing the basic bitstream; S2: Decompress the base bitstream using the VTM-SCC decoder to obtain the decoded semantic feature map and the decoded edge feature map; S3: At the encoding end of the image coding layer, the input image is encoded to generate an enhanced bitstream; S4: At the decoding end of the image coding layer, the decoded semantic feature map and decoded edge feature map are transformed into multi-scale prior features through a multi-scale feature fusion network, and then fused with the reconstruction latent representation of the main decoding path to reconstruct the image.

6. The scalable image coding method based on multi-scale semantic and edge prior fusion as described in claim 5, characterized in that, Steps S3 and S4, the steps of encoding and decoding the input image, include: The input image is mapped to a latent representation by analyzing the transform network; The latent representation is entropy-modeled through a super-prior analysis network to obtain a super-prior representation. The super-prior representation is then passed through an arithmetic encoder to obtain an image compressed bitstream. Both the super-prior analysis network and the super-prior synthesis network use shifted window convolutional blocks to capture local and global dependencies. At the decoding end, the arithmetic decoder first decodes the compressed bitstream, and then the super-prior synthesis network reconstructs the decoded super-prior representation into probability distribution parameters for estimating the latent representation. The probability distribution parameters include the mean and standard deviation. Another approach inputs the latent representation, mean, and standard deviation into a slice-level autoregressive network to obtain the reconstructed latent representation. Finally, the edge map and semantic map output from the feature encoding layer are used as prior information to generate two feature tensors through a multi-scale feature fusion network. These tensors are then fused into a synthetic transformation network, and the reconstructed image is obtained after final decoding.

7. The scalable image coding method based on multi-scale semantic and edge prior fusion as described in claim 6, characterized in that, The steps for feature fusion performed by the multi-scale feature fusion network include: By using a parallel structure extractor and a Sobel information extractor, features at two different scales are extracted from the edge feature map and the semantic feature map, respectively. Structural and semantic features at the same scale are concatenated via channels; The stitched features are processed by an adaptive fusion module, which includes feature enhancement through a group spatial attention module, reduction of spatial redundancy through a spatial gating reconstruction submodule, and optimization of channel representation through a channel fusion submodule.

Images