Learning-based image compression method and system based on hybrid decoding expert cooperation

Abstract

The application provides a learning type image compression method and system based on hybrid decoding expert cooperation, relates to the technical field of digital image processing and data compression, and comprises double-path parallel encoding, double-expert main freezing, expert-specific enhancement, cross-expert gate modulation and double-anchored reconstruction output. Scalar quantization and vector quantization tokenization double-path parallel encoding are used at the encoding end to generate two bit streams; at the decoding end, a pre-trained double-expert main is completely frozen, and only the expert-specific enhancement module and the cross-expert gate modulation module are trained, cross-path information is accurately injected in a spatial self-adaptive residual gate modulation manner under the premise of maintaining expert-specific characteristics, and finally, high-fidelity reconstructed images and high-perception reconstructed images are simultaneously output. The application can solve the technical defects of expert feature drift, single-path performance bottleneck and cross-path fusion inefficiency caused by end-to-end coupling training in the prior art, and has the advantages of stable training, light weight, high performance, double anchoring and consideration of fidelity and perception.

Description

Technical Field

[0001] This invention relates to the field of digital image processing and data compression technology, and more specifically, to a learning-based image compression method and system based on hybrid decoding expert collaboration. Background Technology

[0002] With the development of mobile internet and cloud storage, the scale of image data continues to grow, making the demand for efficient image compression technology increasingly urgent. Although traditional image coding standards have mature engineering systems, they often suffer from problems such as loss of texture details and excessive smoothing under extremely low bitrate conditions.

[0003] Learning-based image compression typically employs continuous latent variables combined with scalar quantization and entropy models for end-to-end optimization, achieving high distortion performance at medium to high bitrates. However, it is prone to oversmoothing and texture loss at low bitrates. On the other hand, perceptual compression methods based on vector quantization and discrete codebook tokenization can better preserve semantic and visual realism at ultra-low bitrates, but their discretization mechanism often loses fine-grained fidelity information, resulting in insufficient structural detail and limited fidelity scalability with increasing bitrate, making it difficult to cover wide bitrate scenarios.

[0004] To compensate for the shortcomings of the single latent variable paradigm, an intuitive approach is to fuse or couple the SQ (Scalar Quantization) and VQ (Vector Quantization) branches end-to-end during training. However, this type of coupling design often leads to increased training complexity, unstable optimization, and multi-objective gradient interference. Furthermore, if the overhead control of the dual-branch bitstream and the cross-branch information interaction method are not designed properly, it can easily destroy the "expert characteristics" of each decoder, making it impossible to achieve an ideal balance between fidelity and perception.

[0005] Therefore, how to achieve dual-path expert collaboration and cross-path information adaptive enhancement without destroying the pre-trained expert model, without end-to-end coupled training, and without introducing additional computational overhead has become a key technical problem that urgently needs to be solved in the field of deep learning image compression. Summary of the Invention

[0006] Given that existing learning-based image compression technologies struggle to simultaneously achieve both reconstruction fidelity and perceptual quality across different bitrate ranges, the purpose of this invention is to provide a learning-based image compression method and system based on hybrid decoding expert collaboration. By performing pluggable gated residual modulation on two pre-trained decoding experts at the decoding end, selective injection of cross-expert complementary information is achieved while keeping the expert backbone parameters frozen, thereby obtaining a better fidelity-perceptual compromise under a unified dual-stream bitstream overhead.

[0007] On one hand, the present invention provides a learning-based image compression method based on hybrid decoding expert collaboration, comprising: At the encoding end, the input image is subjected to scalar quantization compression encoding of the first branch and vector quantization tokenization compression encoding of the second branch to generate the first bit stream and the second bit stream. The first bitstream and the second bitstream are decoded at the decoding end to obtain the first potential representation and the second potential representation; With the backbone parameters of the first and second decoders at the decoding end kept frozen, the first latent representation and the second latent representation are subjected to multi-level decoding processing by the first decoder and the second decoder, respectively. Among them, a trainable expert-specific enhancement module is inserted at at least one decoding level of the first decoder and the second decoder to generate enhanced expert features based on expert stream features and modulation stream features of the same branch. In addition, a trainable cross-expert gated modulation module is inserted at at least one decoding level of the first decoder and the second decoder to predict spatial adaptive gated weights based on the modulation stream features of the current branch and the enhanced expert features of another branch, and injects the cross-branch complementary features into the modulation stream features of the current branch in a gated residual manner to obtain the modulated features. Based on the modulated features, a high-fidelity reconstructed image anchored to the first decoder and a perceptually reconstructed image anchored to the second decoder are output.

[0008] Alternatively, the step of performing scalar quantization compression encoding (first branch) and vector quantization tokenization compression encoding (second branch) on the input image at the encoding end includes: The first branch encoder extracts the continuous latent representation of the input image and performs scalar quantization compression on the continuous latent representation to obtain the first quantized latent representation; and the second branch encoder extracts the feature representation of the input image, maps the feature representation to the embedding space of the pre-trained codebook or equivalently finds the quantized embedding by the codebook index, and performs vector quantization tokenization to obtain the second quantized latent representation. The first quantized latent representation and the side information generated based on the super-prior model are entropy encoded to generate the first bit stream; and the second quantized latent representation is entropy encoded to generate the second bit stream.

[0009] In addition, an alternative approach is to further include, before entropy encoding the first quantized latent representation and the generated edge information based on the super-prior model, using the second quantized latent representation of the second branch as conditional information to conditionally modulate the super-prior edge information of the first branch.

[0010] Alternatively, the expert-specific enhancement module may concatenate the expert stream features and modulation stream features at the channel dimension and then input them into the enhancement network to output enhanced expert features.

[0011] Alternatively, the cross-expert gating modulation module may employ a Sigmoid constraint to set the gating weights to [0,1], and the gating weights may be a dense gating graph consistent with the feature space resolution.

[0012] Meanwhile, the present invention also provides a learning-based image compression system based on hybrid decoding expert collaboration, applied to electronic devices, comprising: The dual-stream coding unit is used to perform scalar quantization compression coding of the first branch and vector quantization tokenization compression coding of the input image at the coding end to generate a first bit stream and a second bit stream. A primary decoding unit is used to decode the first bitstream and the second bitstream at the decoding end to obtain a first latent representation and a second latent representation, respectively. A multi-level decoding unit is used to perform multi-level decoding processing on the first latent representation and the second latent representation respectively through the first decoder and the second decoder, while keeping the backbone parameters of the first decoder and the second decoder frozen. Specifically, a trainable expert-specific enhancement module is inserted at at least one decoding level of the first decoder and the second decoder to generate enhanced expert features based on expert stream features and modulation stream features of the same branch. Furthermore, a trainable cross-expert-gated modulation module is inserted at at least one decoding level of the first decoder and the second decoder to predict spatial adaptive gating weights based on the modulation stream features of the current branch and the enhanced expert features of another branch, and injects cross-branch complementary features into the modulation stream features of the current branch in a gated residual manner to obtain modulated features. The image reconstruction unit is used to output a high-fidelity reconstructed image anchored to the first decoder and a perceptually reconstructed image anchored to the second decoder, based on the modulated features.

[0013] In another aspect, the present invention also provides an electronic device, the electronic device including a memory, a processor, and a learning-based image compression program based on hybrid decoding expert collaboration stored in the memory and executable on the processor, wherein the learning-based image compression program based on hybrid decoding expert collaboration implements the learning-based image compression method based on hybrid decoding expert collaboration as described above when executed by the processor.

[0014] The learning-based image compression method and system based on hybrid decoding expert collaboration according to the present invention has the following advantages compared with the prior art: 1. Completely freeze the pre-trained expert backbone to completely avoid problems such as expert feature drift, model performance degradation, and training instability, resulting in a highly stable training process and fast convergence; 2. Only the lightweight enhancement module is trained, resulting in low computational overhead, few parameters, and easy deployment, achieving lightweight and high-performance image compression; 3. Expert-specific enhancement modules effectively maintain the unique characteristics of experts for each path, without disrupting, interfering with, or coupling them; 4. The cross-expert gating modulation module achieves cross-path information enhancement with spatial adaptation, local precision, and residual injection, resulting in significant collaborative effects; 5. Dual-anchored output simultaneously achieves high fidelity and high-perception visual effects, making it suitable for a wide range of scenarios; 6. The overall framework is modular, highly interpretable, has a clear training process, and outstanding performance, making it extremely valuable for engineering applications.

[0015] To achieve the foregoing and related objectives, one or more aspects of the invention include the features which will be described in detail below and specifically pointed out in the claims. The following description and accompanying drawings illustrate certain exemplary aspects of the invention. However, these aspects indicate only a few of the various ways in which the principles of the invention can be used. Furthermore, the invention is intended to include all such aspects and their equivalents. Attached Figure Description

[0016] Other objects and results of the invention will become more apparent and readily understood with reference to the following description taken in conjunction with the accompanying drawings and the contents of the claims, and with a more complete understanding of the invention. In the drawings: Figure 1 This is a schematic diagram illustrating the overall principle of the learning-based image compression method based on hybrid decoding expert collaboration according to an embodiment of the present invention. Figure 2 This is a schematic diagram of a hybrid decoding expert collaborative framework for the decoding end according to an embodiment of the present invention; Figure 3 This is a flowchart illustrating a learning-based image compression method based on hybrid decoding expert collaboration according to an embodiment of the present invention. Figure 4 This is a schematic diagram illustrating the subjective reconstruction effect of fidelity anchoring according to an embodiment of the present invention; Figure 5 The subjective reconstruction effect of perception anchoring according to embodiments of the present invention; Figure 6 This is a schematic diagram of the logical structure of a learning-based image compression system based on hybrid decoding expert collaboration according to an embodiment of the present invention; Figure 7 This is a schematic diagram of the logic structure of an electronic device according to an embodiment of the present invention.

[0017] In all the accompanying drawings, the same reference numerals indicate similar or corresponding features or functions. Detailed Implementation

[0018] In the following description, numerous specific details are set forth for illustrative purposes and to provide a thorough understanding of one or more embodiments. However, it will be apparent that these embodiments may also be implemented without these specific details. In other instances, well-known structures and devices are shown in block diagram form for ease of description of one or more embodiments.

[0019] In existing learning-based image compression techniques, distortion-guided codecs based on scalar quantization are prone to over-smoothing and texture loss at low bitrates, while perception-guided codecs based on vector quantization and tokenization can improve visual realism at ultra-low bitrates, but usually suffer from insufficient structural fidelity and limited effective bitrate coverage. Furthermore, existing dual-latent variable / dual-branch fusion methods often require end-to-end retraining or deep coupling, which can easily introduce optimization instability and damage the expertise of each branch. To address these issues, this invention proposes a learning-based image compression method and system based on hybrid decoding expert collaboration. The scalar quantization branch decoder and the vector quantization branch decoder are treated as frozen "fidelity experts" and "perceptual experts," respectively. A trainable expert-specific enhancement module and a cross-expert gating modulation module are inserted at the decoding end to selectively inject complementary cross-expert information under a unified dual-stream bitstream overhead, thereby generating fidelity-anchored reconstructed images and perception-anchored reconstructed images.

[0020] To better illustrate the technical solution of the present invention, some of the technical terms involved in the present invention will be briefly explained below.

[0021] Scalar Quantization Branch (SQ Branch) / First Branch: This refers to a learning-based image compression branch that uses continuous latent representation and scalar quantization, which can provide strong structural fidelity and rate-distortion scalability at medium to high bit rates.

[0022] Vector Quantization Tokenization Branch (VQ Tokenization Branch) / Second Branch: Refers to the compression branch that uses discrete codebook tokens (indexes) to better maintain semantic and visual realism at ultra-low bitrates.

[0023] Dual-stream bitstream: refers to the first bitstream generated by the first branch and the second bitstream generated by the second branch; in this embodiment of the invention, the total bit rate (bpp) is calculated based on the combined overhead of the two.

[0024] Freezing the expert decoder: This refers to keeping the backbone network parameters of the first and second decoders frozen during the training phase, and only training the plug-and-play modules to avoid instability and loss of expert capabilities caused by end-to-end retraining.

[0025] Expert-Specific Enhancement Module (ESE): This refers to a module that enhances the features of the expert in the current branch and maintains their specificity before cross-branch interaction. Its output serves as a "stable expert reference" for gating modulation of the opposite branch.

[0026] Cross-expert-gated modulation module (CEM): refers to a module that predicts spatial adaptive gating weights based on the current branch modulation stream features and the enhanced expert features of another branch, and injects complementary features in the form of gating residuals.

[0027] PlugMoDE-F and PlugMoDE-P: Two anchoring output modes under a unified dual-stream bitstream; PlugMoDE-F is a fidelity expert anchor (improving the perceived quality of SQ), and PlugMoDE-P is a perception expert anchor (restoring the structural fidelity of VQ).

[0028] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0029] The learning-based image compression method based on hybrid decoding expert collaboration provided by this invention is mainly applied to the decoding end and outputs fidelity anchored reconstruction results and perceptual anchored reconstruction results under a unified dual-stream bitstream overhead. Figure 1 and Figure 2 The overall principle of the learning-based image compression method based on hybrid decoding expert collaboration according to embodiments of the present invention and the hybrid decoding expert collaboration framework at the decoding end are shown respectively. Figure 3 The flowchart of a learning-based image compression method based on hybrid decoding expert collaboration according to an embodiment of the present invention is shown.

[0030] like Figures 1 to 3 As shown in the figure, the learning-based image compression method based on hybrid decoding expert collaboration provided in this embodiment mainly includes the following steps: S100: At the encoding end, the input image is subjected to scalar quantization compression encoding of the first branch and vector quantization tokenization compression encoding of the second branch to generate a first bitstream and a second bitstream. This step is a dual-stream encoding step, which forms a compact dual-stream bitstream containing the first bitstream and the second bitstream by performing dual-branch pre-compression encoding on the input image; the bitrate statistics are measured by the total overhead of the first bitstream and the second bitstream.

[0031] Specifically, as an example, in the scalar quantization compression coding process of the first branch, the first quantization latent representation and side information are generated through convolutional downsampling, the main encoder, and the super-prior encoder; in the vector quantization tokenization compression coding process of the second branch, the second quantization latent representation is generated through tokenization mapping, codebook lookup, and the quantization encoder. The encoding end performs arithmetic coding in two separate paths to generate the first bitstream and the second bitstream.

[0032] S200: At the decoding end, the first bitstream and the second bitstream are decoded respectively to obtain the first latent representation and the second latent representation. This step is the primary decoding step, the purpose of which is to decode the two latent representation data in the dual-stream bitstream.

[0033] S300: With the backbone parameters of the first decoder and the second decoder at the decoding end kept frozen, the first latent representation and the second latent representation are subjected to multi-level decoding processing by the first decoder and the second decoder, respectively.

[0034] This step is a multi-level decoding step. The purpose of keeping the backbone parameters frozen is to maintain the backbone parameters of the pre-trained scalar quantization experts and vector quantization experts unchanged (including encoder weights, quantizer weights, decoder backbone weights, upsampling module weights, etc., which do not participate in any gradient updates), and only train the lightweight enhancement module. Specifically, at least one decoding level in the first and second decoders inserts a trainable expert-specific enhancement module to generate enhanced expert features based on expert flow features and modulation flow features of the same branch. This enhances the expert flow features specifically in each decoding unit, preserving path expert characteristics. Furthermore, at least one decoding level in the first and second decoders also inserts a trainable cross-expert-gated modulation module to predict spatially adaptive gating weights based on the modulation flow features of the current branch and the enhanced expert features of another branch. It then injects cross-branch complementary features into the modulation flow features of the current branch using a gated residual method to obtain the modulated features. The cross-expert-gated modulation module can inject cross-path complementary information into the current path using a residual gating method with spatially adaptive gating weights, achieving non-interference, non-destructive, and decoupled collaborative enhancement.

[0035] In this step, the first latent representation and the second latent representation are input into a pre-trained dual decoding expert whose backbone parameters are frozen but which is embedded with a trainable decoder plug-in module, in order to perform decoder-side collaborative feature enhancement based on expert-specific enhancement and cross-expert gating modulation.

[0036] S400: Based on the modulated features, output a fidelity reconstructed image anchored to the first decoder and a perceptual reconstructed image anchored to the second decoder.

[0037] This step is the image reconstruction step. Based on the decoded features enhanced by the collaborative features at the decoding end (i.e., the modulated features), it outputs two types of expert-anchored reconstructed images: a fidelity-anchored reconstructed image (PlugMoDE-F) and a perceptually-anchored reconstructed image (PlugMoDE-P), thus balancing both objective metrics and subjective visual effects. The fidelity-anchored reconstructed image is jointly supervised and trained using mean squared error loss and perceptual loss. The perceptually-anchored reconstructed image is trained using pixel reconstruction loss, perceptual loss, and token consistency loss (the reconstructed image is input into a frozen second-branch encoder, causing its token sequence / quantization embedding to be consistent with the original). Figure 1 (To) conduct joint supervision training.

[0038] The "cooperative enhancement" in this invention specifically refers to the process of "enhancing expert representations—gated residual injection" on the intermediate decoding features of the two branches by inserting trainable expert-specific enhancement (ESE) modules and cross-expert modulation (CEM) modules into multi-level decoding layers while keeping the backbone parameters of the pre-trained scalar quantization (SQ) decoder and vector quantization (VQ) decoder frozen. This enhances the decoding features affected by quantization and codebook discretization degradation into higher-quality decoding features and outputs reconstructed images anchored by two types of experts. This cooperative enhancement process includes at least the following three key aspects: First, preserve expert characteristics: enhance the features of each branch expert flow before cross-branch interaction through ESE, making them stable branch-specific references, so that the inductive bias and expertise of pre-trained experts are not destroyed when cross-branch complementary injection (e.g., the scalability of SQ experts to distortion-preservation, and the ability of VQ experts to preserve semantic and perceptual realism).

[0039] Second, selective complementary injection: spatial adaptive gating weights consistent with the feature space resolution are predicted by CEM, and complementary cues from another branch are injected into the current branch modulation stream features by gating residuals, so that "complementary cues are only used where they are needed and no interference is needed where they are not needed", thereby avoiding the forced routing and expert interference caused by traditional softmax competitive gating.

[0040] Third, dual-anchored output: Under the unified dual-stream bit overhead, the co-enhanced modulation stream outputs a fidelity reconstructed image anchored by the SQ decoder (PlugMoDE-F) and a perceptual reconstructed image anchored by the VQ decoder (PlugMoDE-P). The two modes share the same set of pre-trained experts and the same set of MoDE plugin structures, and only the differences in branch training objectives form different cross-expert modulation behaviors.

[0041] Furthermore, in the process of forming a dual-stream bitstream, to reduce the additional overhead caused by the simultaneous transmission of the first and second bitstreams, in a specific embodiment of the present invention, an optional scheme of token-conditionalized super-prior side information can be adopted: without modifying the pre-trained SQ coding backbone, the quantization embedding of the VQ branch is used as a conditional signal to modulate the super-prior side information of the SQ branch, so that the super-prior side information can more compactly express global semantic cues, thereby reducing dual-stream redundancy under a unified code rate statistical caliber (total overhead of the first and second bitstreams). Specifically, the side information can be obtained by first quantizing and decoding the super-latent representation using the SQ branch, then concatenating the side information with the VQ quantization embedding, and conditionally modulating the side information through a scale / offset form feature transformation.

[0042] It should be noted that the focus of this invention is on the decoding end, especially the hybrid decoding expert collaborative modulation part of the decoding end. However, encoding and decoding are two parts of a compression scheme. Therefore, for the sake of completeness of the technical solution demonstration and convenience of description, the implementation scheme of this invention will be described from both encoding and decoding aspects in the following description of the embodiments. However, it should be noted that the exemplary description of the encoding scheme in the following embodiments does not mean that the possible implementation schemes of this invention are limited.

[0043] The following will describe in detail each step of the learning-based image compression method based on hybrid decoding expert collaboration at the decoding end of the present invention with reference to more specific embodiments.

[0044] In this embodiment, step S100 is the process of performing dual-branch pre-compression coding on the input image to form a compact dual-stream bitstream containing a first bitstream and a second bitstream. The first branch is a scalar quantization compression branch of continuous latent representation, and the second branch is a vector quantization compression branch of discrete codebook tokenization; the bitrate is measured by the total overhead of the first bitstream and the second bitstream.

[0045] Specifically, given an input image XThe system extracts latent representations from the input image using a first-branch encoder and a second-branch encoder, respectively, and performs corresponding quantization operations to obtain quantized latent representations: the first-branch encoder extracts continuous latent representations from the input image and performs scalar quantization compression on these extracted continuous latent representations to obtain the first quantized latent representation (also known as the first-branch super-latent representation); the second-branch encoder extracts feature representations from the input image, maps these feature representations to the embedding space of the pre-trained codebook, or equivalently obtains quantized embeddings by looking up the codebook index, and performs vector quantization tokenization to obtain the second quantized latent representation (also known as the second-branch super-latent representation). After obtaining the two latent representations to be transmitted, the first quantized latent representation and the side information generated based on the advanced prior model are entropy encoded to generate the first bitstream, and the second quantized latent representation is entropy encoded to generate the second bitstream.

[0046] Furthermore, in order to reduce the overhead of dual-stream transmission, in an optional implementation, before entropy encoding the first quantized latent representation and the side information generated based on the super-prior model, the super-prior side information of the first branch can be conditionally modulated by the quantization embedding of the second branch. That is, the second quantized latent representation of the second branch is used as conditional information, and the super-prior side information of the first branch is conditionally modulated by scale and shift.

[0047] Specifically, as an example, the first branch hyper-latent representation is first quantized and entropy encoded to obtain side information bits, then decoded to obtain side information features, and concatenated with the second quantized latent representation. The side information is then modulated using a scaling / offset method. Based on this, the entropy encoding of the first branch main latent representation and the second branch latent representation is completed, thereby forming a compact dual-stream bit stream.

[0048] Step S200 involves decoding the first bitstream at the decoding end to obtain the first latent representation and decoding the second bitstream to obtain the second latent representation. Specifically, during the scalar quantization branch decoding process, entropy decoding is first performed on the first bitstream to restore the scalar quantized master latent quantization value, and simultaneously decoding the super-prior edge information bitstream of the scalar quantization branch; then, entropy decoding is performed on the super-prior edge information bitstream to obtain the super-latent quantization value, which is then used by the super decoder to recover the distribution parameters (mean, variance, etc.) required by the entropy model; finally, using the entropy model distribution parameters as an aid, inverse scalar quantization transform is performed on the master latent quantization value to remove quantization noise and reconstruct it, ultimately obtaining the first latent representation (continuous, high-fidelity-oriented latent feature). This first latent representation is a continuous numerical feature, preserving strong structural fidelity and fine-grained pixel information, and serves as the input to the fidelity decoder. Similarly, in the vector quantization branch decoding process, entropy decoding is first performed on the second bitstream to directly reconstruct the discrete codebook index sequence (i.e., the token sequence) of vector quantization. Then, using the decoded codebook index, a lookup operation is performed in the pre-trained fixed codebook to map the index to the corresponding codebook vector. Finally, the codebook vector is rearranged in spatial dimensions to form a second latent representation (discrete, perception-oriented quantization embedding) aligned with the spatial resolution of the first latent representation. This second latent representation is a discrete codebook embedding feature that retains strong semantic and visual perception information and serves as the input to the perceptual decoder.

[0049] Step S300 involves performing multi-level decoding processing with frozen backbone parameters on the two decoded latent representations. The first latent representation and the second latent representation obtained from the initial decoding in step S200 are respectively input into the pre-trained first decoder and second decoder for multi-level decoding, wherein the backbone parameters of the first decoder and the second decoder are kept frozen.

[0050] Specifically, before decoding begins, the intermediate features of each decoder are split into two parallel feature streams: an expert stream and a modulation stream. The quantized latent representations of the corresponding branches are used as the initialization inputs for the two feature streams. The expert stream is used to generate stable expert reference representations, and the modulation stream is used to receive cross-expert complementary injections and pass them to subsequent decoding and upsampling layers.

[0051] The first and second latent representations are input into a pre-trained dual-decoding expert with frozen backbone parameters but embedded with a trainable decoder plug-in module to perform a decoder-end collaborative feature enhancement process based on expert-specific enhancement and cross-expert gating modulation.

[0052] In one example implementation, each decoder is divided into N resolution levels, each level containing a decoded block and an upsampled block; a MoDE plugin module (ESE+CEM) is inserted between the decoded block and the upsampled block in at least one level, and the intermediate features of each level are processed as follows: Before decoding begins, the intermediate features of each decoder are split into two parallel feature streams: an expert stream and a modulation stream. Both the expert stream and modulation stream features are then initialized using the same quantized latent representation. The initial values ​​of the expert stream and modulation stream features are completely different, and their subsequent roles differ. The expert stream features are used to generate stable, invariant expert reference representations, maintaining the native expertise of that branch of the decoder throughout (for fidelity and perception). The modulation stream features are used to receive cross-expert complementary information injections, which are passed layer by layer to subsequent decoding levels, ultimately outputting the reconstructed image.

[0053] More specifically, before performing multi-level decoding processing on the first latent representation and the second latent representation, the first latent representation is used as the initial value of the expert stream feature and the modulation stream feature of the first decoder, respectively; and the second latent representation is used as the initial value of the expert stream feature and the modulation stream feature of the second decoder; wherein, the expert stream feature is used to maintain the expert characteristics of the corresponding branch decoder and form a stable branch reference representation; the modulation stream feature is used to receive the injection of cross-branch complementary features and pass them layer by layer to the subsequent decoding levels.

[0054] During decoding, the ESE (Expert Stream Separator) first concatenates the expert stream features and modulation stream features along the channel dimension and inputs them into the enhancement network to obtain enhanced expert features. Then, the CEM (Constant Stream Equalizer) generates spatially adaptive gated weights and a modulation signal (the modulation signal being the enhanced expert features of another branch) based on the modulation stream features of the current branch and the enhanced expert features of another branch. This modulation signal is then injected into the modulation stream features of the current branch using a gated residual method to obtain the modulated features, which then proceed to subsequent upsampling and the next level of decoding. Low-resolution layers are responsible for semantic and global structure modulation, while high-resolution layers are responsible for detail and texture modulation, with enhancement occurring layer by layer until a full-resolution reconstructed image is output.

[0055] In other words, the multi-level decoding process of this invention is divided into multiple consecutive levels according to the feature resolution from low to high. Each level sequentially performs decoding, expert-specific enhancement, cross-expert gating modulation, and upsampling operations. Cross-expert gating modulation exhibits an injection intensity distribution related to the target bitrate at different levels, and this distribution shows a strong correlation with the target bitrate at different resolution levels. The injection intensity can be represented by the magnitude of the spatial adaptive gating weights, which represent the complementary influence of the expert features of another branch on the current branch: the closer the weight is to 1, the stronger the injection; the closer it is to 0, the weaker the injection. Among them, at low target bitrates (ultra-low bitrate / low bitrate), the overall injection intensity is significantly higher, and the cross-expert complementary effect is stronger. Furthermore, the injection intensity is highest at low-resolution levels, prioritizing the correction of structural and semantic distortions; the injection intensity is moderate at high-resolution levels, supplementing texture and detail perception information. At medium to high target bitrates, the overall injection intensity is lower, mainly to preserve the original expert capabilities. The injection intensity is weak at low-resolution levels, and the structure is already guaranteed by the fidelity branch; the injection intensity is slightly increased at high-resolution levels, performing fine-tuning of perception. The injection intensity within the same level exhibits a spatially adaptive distribution: weak injection in smooth regions and strong injection in texture / edge / semantic key regions, which, together with the bitrate, determine the final gating graph.

[0056] The dense gated weights are constrained to [0,1] by the Sigmoid function of the cross-expert gated modulation module, and their spatial resolution is consistent with the feature spatial resolution, thereby achieving pixel / region-level adaptive complementary injection. The gated residual method satisfies "modulated features = original modulated stream features + gated weights ⊙ modulated signal", ensuring that cross-expert injection is a progressive and controllable enhancement rather than a hard replacement of the original features.

[0057] Step S400 is the process of outputting two expert-anchored reconstructed images based on the enhanced decoding features, including a fidelity-anchored reconstructed image (PlugMoDE-F) and a perceptual-anchored reconstructed image (PlugMoDE-P).

[0058] Specifically, after N After multi-level collaborative enhancement, the modulation stream outputs two final reconstructions: the output anchored by the SQ decoder enhances the subjective realism at low bit rates by selectively injecting semantic / texture cues from perceptual experts; the output anchored by the VQ decoder enhances the scalability of structural fidelity and distortion metrics by selectively injecting structural / edge cues from fidelity experts, thereby achieving a balance between fidelity and perceptual quality over a wide bit rate range under a unified dual-stream bit rate overhead.

[0059] In some embodiments, to enable the hybrid decoding expert collaboration module at the decoding end to selectively and complementaryly inject without compromising the original capabilities of each expert, this invention employs a training strategy during the training phase that freezes the expert backbone and optimizes only the parameters of the plug-in modules. Specifically, the first decoding expert and the second decoding expert are pre-trained decoder networks, and their encoder, quantizer, entropy model, and decoder backbone parameters are kept frozen; only the parameters of the expert-specific enhancement modules and cross-expert gating modulation modules inserted into each resolution level are updated.

[0060] Furthermore, during the training phase, two sets of pluggable module parameters can be trained separately for the two anchoring modes. The two anchoring modes include: a fidelity reconstruction mode anchored to the first decoding expert (denoted as mode). ) and the perception reconstruction model anchored by the second decoding expert (denoted as the model) In any mode ( Under these conditions, training sample images The forward process may include the following steps: 1) Input image processed by the frozen encoding and quantization module Pre-compression is performed to obtain a quantized latent representation; this quantized latent representation is then used as the initialization input for the modulation stream and the expert stream in the current mode, respectively. 2) Input the modulation flow features and expert flow features into the first... resolution levels Freeze the decoded block to obtain intermediate features at the same level; 3) Input the modulation stream features and expert stream features of the same level into ESE to obtain enhanced expert features; then input the modulation stream features of the current branch and the enhanced expert features of another branch into CEM to predict dense gate weights and modulation residual signals, and inject and update the modulation stream features in the gated residual manner to obtain the modulated features; 4) The enhanced expert features and the modulated features are respectively sent to the frozen upsampling module to enter the next level, until all levels of decoding are completed; 5) The frozen reconstruction head outputs the reconstructed image to the expert stream and the modulation stream respectively, where the expert stream output is the auxiliary reconstructed image and the modulation stream output is the final reconstructed image.

[0061] The training objective of ESE is to maintain the original task attributes of the expert flow and avoid expert ability drift caused by cross-expert interaction; the training objective of CEM is to selectively introduce complementary cues from another expert through gating while maintaining the stability of the expert flow, thereby forming a final reconstructed image oriented towards the goal trade-off.

[0062] In some embodiments, the training objective function is composed of two types of losses: expert enhancement loss and selective complementarity loss.

[0063] Expert enhancement loss is used to constrain the auxiliary reconstructed image obtained by expert stream decoding, ensuring that it conforms to the original objectives of each expert, thus providing a stable reference for subsequent cross-expert complementarity. Specifically, for the pattern... Let the auxiliary reconstructed image generated by the expert stream be Then you can set: For the fidelity anchoring mode Constraints are imposed using a distortion metric (e.g., mean squared error, MSE). With input image Differences; Perception anchoring mode Constraints are imposed using a perceptual metric (e.g., perceptual loss LPIPS based on feature distance). With input image Differences in perception.

[0064] Selective complementation loss is used to constrain the final reconstructed image output from the modulated stream, ensuring it achieves the desired fidelity / perceptual tradeoff in the target mode, and driving the CEM to learn a gating strategy of "complementing only where complementation is needed." Specifically, let the final reconstructed image generated by the modulated stream be... Then you can set: For the fidelity anchoring mode : A perceptual term is superimposed on the distortion term, and an adversarial term can be optionally superimposed to enhance the realism of the texture; Perception anchoring mode : In pixel domain differences (e.g. Based on the perception term, a "coding consistency / token consistency" constraint is introduced: the input image... With the final reconstructed image The data are fed into the frozen second branch encoder to constrain the differences between its encoded features or discrete representations, in order to encourage semantic and structural consistency during perceptual reconstruction; and optionally, adversarial terms are superimposed to enhance the generated details.

[0065] In some embodiments, the overall training objective can be expressed as:

[0066] in, To enhance the experts' understanding of the losses, This is a selective complementary loss. The weighting coefficients for each loss term can be set according to the dataset and the target bitrate range, and the values ​​of these weighting coefficients do not constitute a limitation on the present invention.

[0067] In some embodiments, the training process for the plug-in / plug-out module may include the following iterative steps: S500: Initialization Mode The corresponding ESE and CEM parameters; S510: Take a training sample image X from the training dataset; S520: Under the frozen coding, quantization and entropy model, X is pre-compressed to obtain the quantization latent representation, and the modulation stream latent representation and expert stream latent representation are initialized; S530: Yes arrive Each resolution level is executed sequentially: intermediate features are obtained by freezing the decoding block; enhanced expert features are obtained by ESE; gate weights and modulation residuals are obtained by CEM and the modulation stream features are updated; and the input of the next level is obtained by freezing the upsampling module. S540: Outputs auxiliary and final reconstructed images separately by freezing the reconstruction head, and calculates the overall loss. And only update the parameters of ESE and CEM; S550: Repeat S510 to S540 until the stopping condition is met to obtain the parameters of the plug-in module after training is completed.

[0068] As can be seen from the above embodiments, the present invention generates a first bitstream by performing scalar quantization compression encoding on the input image at the transmitting end to generate a second bitstream by performing vector quantization tokenization encoding; at the receiving end, the first bitstream and the second bitstream are decoded to obtain a fidelity latent representation and a perceptual latent representation, and while keeping the parameters of the pre-trained fidelity decoder and perceptual decoder frozen, a trainable expert-specific enhancement module and a cross-expert gating modulation module are inserted into the decoding layer: the expert-specific enhancement module is used to enhance and maintain the specificity of each expert feature before cross-branch interaction, and the cross-expert gating modulation module is used to predict spatial adaptive gating weights and inject complementary features in a gating residual manner, thereby generating a fidelity anchored reconstructed image and a perceptual anchored reconstructed image (e.g., Figure 4 and Figure 5 The subjective reconstruction effects of fidelity anchoring and perceptual anchoring are shown respectively, achieving a balance between fidelity and perceptual quality over a wide bitrate range. This method can improve the perceptual quality of low bitrate networks or restore the structural fidelity of ultra-low bitrate networks without end-to-end retraining of the compressed backbone network, making it suitable for image communication and storage scenarios.

[0069] The above describes a learning-based image compression method based on hybrid decoding expert collaboration according to an embodiment of the present invention. This method is a deep learning image compression scheme based on frozen dual-expert collaboration, aiming to address the technical shortcomings of existing technologies, such as expert feature drift caused by end-to-end coupled training, single-path performance bottlenecks, and inefficient cross-path information fusion. By freezing the backbone of the pre-trained expert model and introducing expert-specific enhancement modules and cross-expert gating modulation modules only in multi-level decoding units, dual-path expert collaboration and adaptive cross-path information enhancement are achieved. The final output is a high-fidelity reconstructed image anchored by scalar quantization experts and a high-perception reconstructed image anchored by vector quantization experts. This learning-based image compression method based on hybrid decoding expert collaboration can be implemented in software, hardware, or a combination of both.

[0070] Corresponding to the above-mentioned learning-based image compression method based on hybrid decoding expert collaboration, the present invention also provides a learning-based image compression system based on hybrid decoding expert collaboration.

[0071] Figure 6 The framework structure of a learning-based image compression system 600 based on hybrid decoding expert collaboration according to an embodiment of the present invention is shown. Figure 6 As shown, the learning-based image compression system based on hybrid decoding expert collaboration provided by the present invention generates video using the learning-based image compression method based on hybrid decoding expert collaboration as described above. According to the functions implemented, the learning-based image compression system 600 based on hybrid decoding expert collaboration mainly includes a dual-stream coding unit 610, a primary decoding unit 620, a multi-level decoding unit 630, and an image reconstruction unit 640.

[0072] The unit of this invention can also be called a module, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and are stored in the memory of the electronic device.

[0073] Furthermore, in one possible implementation of the present invention, the functions of each module / unit are as follows: The dual-stream coding unit 610 is used to perform scalar quantization compression coding of the first branch and vector quantization tokenization compression coding of the input image at the coding end to generate a first bit stream and a second bit stream. The primary decoding unit 620 is used to decode the first bitstream and the second bitstream at the decoding end to obtain the first potential representation and the second potential representation, respectively. The multi-level decoding unit 630 is used to perform multi-level decoding processing on the first latent representation and the second latent representation respectively through the first decoder and the second decoder, while keeping the backbone parameters of the first decoder and the second decoder at the decoding end frozen; wherein, a trainable expert-specific enhancement module is inserted at at least one decoding level of the first decoder and the second decoder to generate enhanced expert features based on expert stream features and modulation stream features of the same branch, and a trainable cross-expert-gated modulation module is inserted at at least one decoding level of the first decoder and the second decoder to predict spatial adaptive gating weights based on the modulation stream features of the current branch and the enhanced expert features of another branch, and injects cross-branch complementary features into the modulation stream features of the current branch in a gated residual manner to obtain modulated features; The image reconstruction unit 640 is used to output a high-fidelity reconstructed image anchored to the first decoder and a perceptually reconstructed image anchored to the second decoder based on the modulated features.

[0074] In addition, in one specific embodiment of the present invention, the multi-level decoding unit 630 may further include: The expert decoding unit 631 is used to input the first quantized latent representation into the first decoding expert and the second quantized latent representation into the second decoding expert, and to form two parallel feature streams, an expert stream and a modulation stream, within each decoding expert; The expert-specific enhancement unit 632 is used to enhance the expert stream features and modulation stream features of the same branch at each resolution level and output enhanced expert features to maintain the stability of the expert capability of that branch. The cross-expert gated modulation unit 633 is used to predict the spatial adaptive gate weight and modulation residual signal based on the modulation stream characteristics of the current branch and the enhanced expert characteristics of another branch, and update the modulation stream characteristics of the current branch in a gated residual manner, thereby realizing cross-expert selective complementary injection.

[0075] The aforementioned learning-based image compression system based on hybrid decoding expert collaboration is an implementation method corresponding to the aforementioned learning-based image compression method based on hybrid decoding expert collaboration. Its specific execution steps can be referred to in the specific embodiments of the aforementioned learning-based image compression method based on hybrid decoding expert collaboration, and will not be described in detail here.

[0076] like Figure 7 As shown, the present invention also provides an electronic device 1 for implementing a learning-based image compression method based on hybrid decoding expert collaboration.

[0077] The electronic device 1 may include a processor 10, a memory 11, and a bus. It may also include a computer program stored in the memory 11 and executable on the processor 10, such as a learning-based image compression program 12 based on hybrid decoding expert collaboration. The memory 11 may include both internal storage units of the aforementioned learning-based image compression system based on hybrid decoding expert collaboration and external storage devices. The memory 11 can be used not only to store application software and various types of data, such as the code of the learning-based image compression program based on hybrid decoding expert collaboration, but also to temporarily store data that has been output or will be output.

[0078] The memory 11 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory, magnetic memory, disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device 1, such as a portable hard drive. In other embodiments, the memory 11 can be an external storage device of the electronic device 1, such as a plug-in portable hard drive, smart memory card, secure digital card, flash memory card, etc., equipped on the electronic device 1. Furthermore, the memory 11 can include both internal and external storage units of the electronic device 1. The memory 11 can be used not only to store application software and various types of data installed on the electronic device 1, such as the code of a learning-based image compression program based on hybrid decoding expert collaboration, but also to temporarily store data that has been output or will be output.

[0079] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units, microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control core of the electronic device, connecting various components of the entire electronic device via various interfaces and lines. It executes programs or modules stored in the memory 11 (e.g., a learning-based image compression program based on hybrid decoding expert collaboration) and calls data stored in the memory 11 to perform various functions of the electronic device 1 and process data.

[0080] The bus can be a peripheral component interconnection standard bus or an extended industry standard structure bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 11 and at least one processor 10, etc.

[0081] Figure 7 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 7The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0082] For example, although not shown, the electronic device 1 may also include a power supply for powering the various components. This power supply can be logically connected to the at least one processor 10 via a power management system, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power sources, a recharging system, a power fault detection circuit, a power converter or inverter, a power status indicator, or any other components. The electronic device 1 may also include various sensors, a Bluetooth module, a Wi-Fi module, etc., which will not be elaborated further here.

[0083] Furthermore, the electronic device 1 may also include a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface, typically used to establish communication connections between the electronic device 1 and other electronic devices. The user interface may be a display, an input unit, a standard wired interface, or a wireless interface. The display, which may also be appropriately referred to as a screen or display unit, is used to display information processed in the electronic device 1 and to display a visual user interface.

[0084] The learning-based image compression program 12 based on hybrid decoding expert collaboration stored in the memory 11 of the electronic device 1 is a combination of multiple instructions. When run in the processor 10, it can implement the learning-based image compression method based on hybrid decoding expert collaboration as described above.

[0085] Specifically, the processor 10's implementation method for the above instructions can be found in [reference needed]. Figure 2 The descriptions of the relevant steps in the corresponding embodiments are not repeated here. Furthermore, if the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable medium may include: any entity or system capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory.

[0086] This invention also provides a computer-readable storage medium, which may be non-volatile or volatile, wherein a computer program is stored therein. When the computer program is executed by a processor, it implements the learning-based image compression method based on hybrid decoding expert collaboration as described above.

[0087] Specifically, the specific implementation method of the computer program when executed by the processor can be referred to the description of the relevant steps in the embodiment of the learning-based image compression method based on hybrid decoding expert collaboration, and will not be repeated here.

[0088] In the several embodiments provided by this invention, it should be understood that the disclosed devices, systems, and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0089] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0090] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in a combination of hardware and software functional modules. Moreover, the word "comprising" obviously does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or systems stated in the system claims can also be implemented by a single unit or system through software or hardware.

[0091] The learning-based image compression method and system based on hybrid decoding expert collaboration according to the present invention have been described above by way of example with reference to the accompanying drawings. However, those skilled in the art should understand that various modifications can be made to the learning-based image compression method and system based on hybrid decoding expert collaboration proposed in the present invention without departing from the scope of the invention. Therefore, the scope of protection of the present invention should be determined by the content of the appended claims.

Claims

1. A learning-based image compression method based on hybrid decoding expert collaboration, characterized in that, include: At the encoding end, the input image is subjected to scalar quantization compression encoding of the first branch and vector quantization tokenization compression encoding of the second branch to generate the first bit stream and the second bit stream. The first bitstream and the second bitstream are decoded at the decoding end to obtain the first potential representation and the second potential representation; With the backbone parameters of the first and second decoders at the decoding end kept frozen, the first latent representation and the second latent representation are subjected to multi-level decoding processing by the first decoder and the second decoder, respectively. Among them, a trainable expert-specific enhancement module is inserted at at least one decoding level of the first decoder and the second decoder to generate enhanced expert features based on expert stream features and modulation stream features of the same branch. In addition, a trainable cross-expert gated modulation module is inserted at at least one decoding level of the first decoder and the second decoder to predict spatial adaptive gated weights based on the modulation stream features of the current branch and the enhanced expert features of another branch, and injects the cross-branch complementary features into the modulation stream features of the current branch in a gated residual manner to obtain the modulated features. Based on the modulated features, a high-fidelity reconstructed image anchored to the first decoder and a perceptually reconstructed image anchored to the second decoder are output.

2. The learning-based image compression method based on hybrid decoding expert collaboration as described in claim 1, characterized in that, The step of performing scalar quantization compression encoding (first branch) and vector quantization tokenization compression encoding (second branch) on the input image at the encoding end includes: The first branch encoder extracts the continuous latent representation of the input image and performs scalar quantization compression on the continuous latent representation to obtain the first quantized latent representation; and the second branch encoder extracts the feature representation of the input image, maps the feature representation to the embedding space of the pre-trained codebook or equivalently finds the quantized embedding by the codebook index, and performs vector quantization tokenization to obtain the second quantized latent representation. The first quantized latent representation and the side information generated based on the super-prior model are entropy encoded to generate the first bit stream; and the second quantized latent representation is entropy encoded to generate the second bit stream.

3. The learning-based image compression method based on hybrid decoding expert collaboration as described in claim 2, characterized in that, Before entropy encoding the first quantized latent representation and the generated edge information based on the super-prior model, the method further includes: The second quantized latent representation of the second branch is used as conditional information to conditionally modulate the super-prior edge information of the first branch.

4. The learning-based image compression method based on hybrid decoding expert collaboration as described in any one of claims 1 to 3, characterized in that, The expert-specific enhancement module concatenates the expert stream features and the modulation stream features at the channel dimension and inputs them into the enhancement network to output enhanced expert features.

5. The learning-based image compression method based on hybrid decoding expert collaboration as described in claim 4, characterized in that, The cross-expert gating modulation module uses Sigmoid constraints to set the gating weights to [0,1], and the gating weights are a dense gating graph consistent with the feature space resolution.

6. The learning-based image compression method based on hybrid decoding expert collaboration as described in claim 5, characterized in that, The multi-level decoding process includes multiple resolution levels, and the cross-expert gating modulation presents an injection intensity distribution related to the target bit rate at different levels.

7. The learning-based image compression method based on hybrid decoding expert collaboration as described in claim 6, characterized in that, The gated residual method satisfies: The modulated characteristics = the original modulated stream characteristics + gating weights ⊙ modulated signal.

8. The learning-based image compression method based on hybrid decoding expert collaboration as described in claim 7, characterized in that, The fidelity-reconstructed image is trained under joint supervision using mean squared error loss and perceptual loss, while the perceptually reconstructed image is trained under joint supervision using pixel reconstruction loss, perceptual loss, and token consistency loss.

9. A learning-based image compression system based on hybrid decoding expert collaboration, applied to electronic devices, characterized in that, include: The dual-stream coding unit is used to perform scalar quantization compression coding of the first branch and vector quantization tokenization compression coding of the input image at the coding end to generate a first bit stream and a second bit stream. A primary decoding unit is used to decode the first bitstream and the second bitstream at the decoding end to obtain a first latent representation and a second latent representation, respectively. A multi-level decoding unit is used to perform multi-level decoding processing on the first latent representation and the second latent representation respectively through the first decoder and the second decoder, while keeping the backbone parameters of the first decoder and the second decoder frozen. Specifically, a trainable expert-specific enhancement module is inserted at at least one decoding level of the first decoder and the second decoder to generate enhanced expert features based on expert stream features and modulation stream features of the same branch. Furthermore, a trainable cross-expert-gated modulation module is inserted at at least one decoding level of the first decoder and the second decoder to predict spatial adaptive gating weights based on the modulation stream features of the current branch and the enhanced expert features of another branch, and injects cross-branch complementary features into the modulation stream features of the current branch in a gated residual manner to obtain modulated features. The image reconstruction unit is used to output a high-fidelity reconstructed image anchored to the first decoder and a perceptually reconstructed image anchored to the second decoder, based on the modulated features.

10. An electronic device, characterized in that, The electronic device includes a memory, a processor, and a learning-based image compression program based on hybrid decoding expert collaboration stored in the memory and executable on the processor. When executed by the processor, the learning-based image compression program based on hybrid decoding expert collaboration implements the learning-based image compression method based on hybrid decoding expert collaboration as described in any one of claims 1 to 8.

Images