Learning type image compression method and system based on stage module distillation

Abstract

The invention provides a learning type image compression method and system based on stage module distillation, and the method comprises the steps: dividing a coding network and a decoding network of a teacher model into n stages according to feature resolution change nodes, and n is an integer greater than 1; on the basis of the divided teacher model, generating a student model which keeps a stage corresponding relationship with the teacher model, and pruning; all parameters of the teacher model are frozen, only the ith stage of training the student model is distilled, i is an integer larger than or equal to 1 and smaller than or equal to n, and distribution of output latent variables of the ith stage approaches latent variable distribution of the corresponding stage of the teacher model; after completion, freezing the stage and turning to the next stage until all stages are trained; and encoding a to-be-compressed image by using the trained student model to obtain a code stream and reconstruct the image. According to the method and the device, the parameter quantity, the FLOPs (floating point operation times) and the reasoning time delay can be synchronously reduced, and the real-time application requirement is met.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image data compression, in particular, to a learning-based image compression method and system based on stage module distillation. BACKGROUND

[0002] With the wide success of neural networks in visual tasks, learned image compression (LIC) gradually replaces traditional codecs such as JPEG, HEVC / VVC, and becomes the focus of academia and industry. The typical LIC framework adopts an end-to-end learnable "transform-quantization-entropy coding" process: the encoder maps the input image to the latent variable space, the quantizer discretizes the latent variable, and then the rate estimation is completed through the entropy model, and the decoder performs reverse transformation to complete reconstruction. Compared with the hand-designed discrete cosine transform (DCT) or variable length coding (VLC), LIC can obtain higher peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) at the same code rate by jointly optimizing the rate-distortion (R-D) objective.

[0003] However, the existing LIC model also faces three bottlenecks:

[0004] Large model and high computational complexity: Early CNN-based end-to-end proposals (such as Ballé et al. 2018) contain tens of millions of parameters; to improve the modeling ability of long-distance dependence, in recent years, a large number of works introduce self-attention / Transformer structure (such as TCM, FTIC, WeConvene), and the model size further expands to the order of 100M parameters, and the floating point operation (FLOPs) is as high as hundreds of T, and the decoding delay is difficult to meet the real-time demand in the high-resolution scenario.

[0005] End-to-end distillation / pruning is unstable: Most existing compression network pruning methods directly apply sparsification regularization to convolution channels or attention heads, and then distill the whole lightweight student network. Due to the lack of fine alignment coupled with the encoding process, the latent variable distribution often drifts, leading to a sharp increase in code rate or image reconstruction distortion; the whole distillation also brings the problems of unstable training and slow convergence.

[0006] Rate distortion and complexity are difficult to balance: When pursuing lower code rate, researchers tend to stack deeper networks and increase the complexity of context models to improve RD performance, but this often comes at the cost of significantly sacrificing parameters / delay. Industrial deployment scenarios (mobile end, edge device) urgently need a lightweight solution that can significantly reduce parameter amount, FLOPs, and even memory usage while maintaining or even improving RD performance.

[0007] In-depth, the root of the above bottleneck lies in the insufficient exploration of the distillation framework, mainly the mismatch of distillation signals caused by the large structural difference between student model and teacher model; the hierarchical strategy does not strictly correspond to the coding stage, making it difficult to ensure that the latent variables are aligned step by step. Of course, it also includes not fully utilizing the gradient of the rate-distortion joint target, and the distillation process lacks end-to-end optimization closed loop for compression tasks.

[0008] After searching, the Chinese patent with application number 202311620315.1 discloses a self-distillation model compression method, which constructs a teacher model that needs model compression, divides its basic structure into several residual blocks, and constructs a student model accordingly; the result knowledge is enhanced through knowledge distillation of each model, and the process knowledge is enhanced through attention map mapping of the feature maps of the teacher and student models; according to the enhanced results of effective knowledge and process knowledge, the loss function of the teacher and each student model is calculated; the student model with the closest classification effect to the teacher model is extracted according to the loss function, and is used as the compression model. This method constructs teacher and student models based on ResNet, and optimizes by knowledge distillation, but its computational complexity is high, and the knowledge enhancement and loss calculation involve a large number of operations, which is time-consuming and dependent on hardware. SUMMARY

[0009] In view of the defects in the prior art, the purpose of the present application is to provide a learning-based image compression method and system based on stage module distillation.

[0010] The first aspect of the present application provides a learning-based image compression method based on stage module distillation, comprising:

[0011] The encoding network and the decoding network of the teacher model are divided into n stages according to the feature resolution change nodes, and n is an integer greater than 1;

[0012] Based on the divided teacher model, the student model is generated and pruned while maintaining a stage corresponding relationship with the teacher model;

[0013] Freeze all parameters of the teacher model, freeze all parameters outside the i-th stage of the student model, only distill and train the i-th stage, 1≤i≤n, so that the distribution of the output latent variable of the i-th stage of the student model approximates the latent variable distribution of the corresponding stage of the teacher model; after completion, freeze the i-th stage of the student model and go to the i+1 stage, until all stages are trained;

[0014] Using the trained student model to encode the image to be compressed, obtaining the code stream and reconstructing the image.

[0015] Optionally, the encoding network and the decoding network of the teacher model are divided into n stages according to the feature resolution change nodes, and n is an integer greater than 1, comprising:

[0016] Traverse all layers of the encoding network, record all down-sampling nodes n1 that cause the feature resolution to decrease;

[0017] Traverse all layers of the decoding network, record all up-sampling nodes n2 that cause the feature resolution to increase;

[0018] Divide the teacher model into n stages at the down-sampling nodes and the up-sampling nodes, n=n1+n2.

[0019] Optionally, the generating the student model and pruning the student model based on the divided teacher model comprises:

[0020] Divide the student model into corresponding stages according to the stages of the teacher model;

[0021] Prune each stage by deleting channels, reducing the number of nonlinear transformation blocks, and expanding the feedforward factor.

[0022] Optionally, the approximating the distribution of the i-th stage output latent variable to the latent variable distribution of the corresponding stage of the teacher model comprises:

[0023] Using RD Loss based on KL divergence as stage distillation loss to constrain the student latent variable distribution to converge to the teacher distribution.

[0024] Optionally, it further comprises: introducing an implicit supervision signal based on the rate-distortion joint target in the process of each stage distillation training to adaptively adjust the latent variable energy compression and the code rate regularization.

[0025] Optionally, the using the trained student model to encode the image to be compressed comprises:

[0026] Obtaining an image to be compressed as an input image;

[0027] The encoding network performs feature compression and transformation on the input image, extracts key information through the pruned lightweight structure, and outputs a quantized compressed representation;

[0028] The decoding network receives the compressed representation output by the encoding network, performs feature recovery and reconstruction through the corresponding lightweight inverse transformation structure, and finally outputs a reconstructed image.

[0029] The second aspect of the present application provides a learning-based image compression system based on stage module distillation, comprising:

[0030] Stage division module: divide the encoding network and the decoding network of the teacher model into n stages according to the feature resolution change nodes, n is an integer greater than 1;

[0031] Pruning construction module: based on the divided teacher model, generating the student model that maintains the stage correspondence relationship with the teacher model and pruning it;

[0032] Stage distillation module: freeze all parameters of the teacher model, freeze all parameters of the student model except the i-th stage, and only distill and train its i-th stage, 1≤i≤n, so that the distribution of the output latent variables of the i-th stage of the student model approaches the distribution of the latent variables of the corresponding stage of the teacher model; application module: use the trained student model to encode the compressed image, obtain the code stream and reconstruct the image.

[0033] Optionally, an end-to-end supervision module is also included, which introduces an implicit supervision signal based on the rate-distortion joint objective during the distillation training process at each stage to adaptively adjust the latent variable energy compression and rate regularization.

[0034] According to a third aspect of the present application, a terminal is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can be used to execute any one of the methods described, or to execute any one of the systems described, when executing the program.

[0035] In a fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, it can be used to execute any of the methods described, or to run any of the systems described.

[0036] The learning-based image compression method based on staged module distillation provided in this application precisely divides the stages at nodes where feature resolution changes, and stabilizes the distillation process through stage-by-stage training and freezing. Ultimately, this method achieves a simultaneous reduction in parameter count, FLOPs (floating-point operations), and inference latency, meeting the requirements of real-time applications. Other technical effects brought about by the additional features will be further elaborated in the corresponding embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Other features, objects and advantages of the present application will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0038] Figure 1 4 is a flowchart of a learning-based image compression method based on stage module distillation according to an exemplary embodiment;

[0039] Figure 2 FIG4 is a schematic diagram showing the overall performance of a learning-based image compression method based on stage module distillation according to an exemplary embodiment;

[0040] Figure 3 A teacher-student model stage division and distillation execution framework according to an exemplary embodiment is shown;

[0041] Figure 4 Performance comparison of the phase module distillation-based learning image compression method according to an exemplary embodiment of the present application with existing advanced methods;

[0042] Figure 5 Distillation effect display of the phase module distillation-based learning image compression method according to an exemplary embodiment of the present application;

[0043] Figure 6 Structure diagram of the phase module distillation-based learning image compression system according to an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0044] The present application will be described in detail below with specific embodiments. The following embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any form. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. The parts not described in detail in the following embodiments can be implemented by using existing technologies.

[0045] In the current lightweight scheme, the structure of the student model and the teacher model is significantly different, which causes the distillation signal to be mismatched; the hierarchical strategy and the coding stage do not form a strict corresponding relationship, so it is difficult to ensure that the latent variables are aligned step by step. Based on the above problems, the present application provides a phase module distillation-based learning image compression method to solve the above problems.

[0046] Referring to Figure 1 and Figure 2 , a phase module distillation-based learning image compression method includes the following steps:

[0047] S100, the coding network and the decoding network of the teacher model are divided into n stages according to the feature resolution change nodes, n is an integer greater than 1;

[0048] S200, based on the teacher model divided in S100, a student model maintaining a stage corresponding relationship with the teacher model is generated and pruned;

[0049] S300, freeze all parameters of the teacher model, freeze all parameters outside the i-th stage of the student model, only distill the i-th stage of the student model, i is an integer greater than or equal to 1 and less than or equal to n, so that the distribution of the latent variables output by the i-th stage approximates the distribution of the latent variables of the corresponding stage of the teacher model; after completion, freeze the stage and go to the next stage, until all stages are trained;

[0050] S400, encoding the image to be compressed using the student model trained in S300, obtaining a code stream and reconstructing the image.

[0051] The above-mentioned embodiment of the present application achieves a simultaneous reduction in the number of parameters, FLOPs (floating-point operations) and inference latency of the obtained student model by accurately dividing the stages at the nodes where the feature resolution changes, and stabilizing the distillation process by gradually training and freezing the stages, thereby meeting the needs of real-time applications.

[0052] In order to make the knowledge transfer and learning from the teacher model to the student model simpler and more direct, the teacher model with a complex framework can be decomposed into multi-stage tasks. In some specific implementations of the present application, step S100 divides the encoding network and decoding network of the teacher model into n stages according to the feature resolution change nodes, where n is an integer greater than 1, and the following steps can be used:

[0053] S101, traverse all layers of the encoding network and record all downsampling nodes n1 that cause feature resolution reduction;

[0054] S102, traverse all layers of the decoding network and record all upsampling nodes n2 that result in increased feature resolution;

[0055] S103, at the downsampling node and the upsampling node, divide the teacher model into n stages, n=n1+n2.

[0056] In the above embodiments of the present application, each stage of the student model imitates the knowledge of the corresponding stage of the teacher model, thereby reducing the difficulty of adaptation in the distillation process and effectively improving the overall performance of the model.

[0057] In order to ensure that the student model is isomorphic to the teacher model and is more lightweight, in one embodiment of the present application, in step S200, based on the divided teacher model, a student model that maintains a stage correspondence with the teacher model is generated and pruned. The following steps can be used:

[0058] S201, corresponding to the stages of the teacher model, divides the student model into corresponding stages;

[0059] For example, Figure 3 As shown in Figure 2, the teacher encoder and decoder are split at three upsampling / downsampling nodes, resulting in 6 stages T1–T6. The student model S1–S6 is also divided into 6 stages by retaining these node positions.

[0060] S202, pruning is performed at each stage by deleting channels, reducing the number of nonlinear transformation blocks and the feedforward expansion factor.

[0061] Specifically, pruning is an important model compression and optimization technique in deep learning, and the core purpose is to remove the "redundant" parameters, neurons or structural components in the model, reduce the parameter quantity, calculation quantity and storage requirement of the model, and improve the running efficiency under the premise of ensuring that the performance of the model remains basically unchanged.

[0062] "Channel" refers to the dimension (or "depth") of the feature map in the neural network. Deleting channels means removing redundant dimensions in the feature map (or the output dimension of the entire convolution kernel / linear layer), reducing the calculation quantity and parameter storage of the model in the feature extraction process.

[0063] "Non-linear transformation block" refers to a repeated structural unit (for example, "attention block + feed-forward block" in Transformer, "residual block" in ResNet) in the model composed of a nonlinear activation function (such as ReLU, GELU) and a linear transformation (such as convolution, full connection). Reducing the number of blocks means reducing the number of such repeated units, reducing the overall depth and calculation quantity of the model.

[0064] "Feed-forward expansion factor" refers to the ratio of the intermediate dimension to the input dimension of the feed-forward neural network (FFN) in models such as Transformer. Reducing the expansion factor means reducing this ratio and reducing the intermediate dimension of the feed-forward layer, thereby reducing the calculation quantity and parameter quantity of the feed-forward layer.

[0065] Example: Pruning the student model in the above example, the number of channels in the first stage of the encoder and decoder is reduced by 33% compared to the teacher model, and the number of non-linear transformation blocks in the second and third stages is reduced by 50% each. By reducing the number of channels and blocks, the structural consistency is ensured and the capability gap is narrowed.

[0066] It should be noted that the reduction ratio of the number of channels and the adjustment ratio of the feed-forward expansion factor are empirical hyperparameters. In actual application, these hyperparameters can be set to different values at different stages of the model, and the optimal values also differ for different structures of the teacher model, so they need to be adjusted according to the specific scene.

[0067] The above embodiments of the present application reduce the difficulty of single-step learning by decomposing knowledge levels in stages, allowing the student model to efficiently absorb joint knowledge from the teacher model in stages, and avoiding ineffective distillation caused by the concentration of knowledge complexity. By pruning, the architecture skeleton of the student model is aligned with that of the teacher model, allowing the student model to have the basic conditions to imitate the teacher model and avoiding "imitation ability loss" caused by a large structural gap.

[0068] In some embodiments of the present application, step S300 causes the latent variable distribution output by the i-th stage to approach the latent variable distribution of the corresponding stage of the teacher model, and the distillation loss thereof is composed of two parts: the constraint of the latent variable distribution by the KL divergence and the constraint of the reconstruction error by the MSE.

[0069] The above embodiments of the present application achieve "double capture" of the knowledge of the teacher model by simultaneously constraining the latent variable distribution (KL divergence) and the reconstruction error (MSE). Both the probability distribution pattern of the intermediate features in the teacher model (deep rule) and the numerical consistency of the output results (surface performance) are learned by the student model, avoiding incomplete knowledge learning caused by single constraint.

[0070] The above embodiments of the present application design a dedicated distillation loss for the i-th stage, which can force the student model to form an accurate match with the teacher model at the corresponding stage. This "point-to-point" constraint method avoids distortion or deviation of knowledge in cross-stage transmission, making the imitation effect of each stage more reliable.

[0071] The above embodiments of the present application reduce the number of parameters of the student model after pruning and other compression operations. Through this double-constrained distillation method, the key features and output capabilities of the teacher model can be maximally retained, maintaining or even improving the performance of the model while reducing the complexity.

[0072] In order to maintain the latent variable distribution and the RD performance (Rate-Distortion performance, an index for measuring compression effect) of the student model while reducing the complexity, in some embodiments of the present application, an end-to-end supervision method is introduced. In the distillation training process of each stage, an implicit supervision signal based on the rate-distortion joint target is introduced to adaptively adjust the latent variable energy compression and the code rate regularization.

[0073] Specifically, in the stage training process, the global is calculated simultaneously

[0074] L RD =R+λD

[0075] LRD(Rate-Distortion Loss) wherein R is the code rate estimated by the entropy model, D is the pixel MSE or MS-SSIM distortion, and λ is set according to the target compression rate. This strategy causes the student model to adaptively compress energy and regularize the code rate without explicit channel alignment.

[0076] The above embodiments of the present application realize multiple technical gains by introducing end-to-end supervision and a rate-distortion joint objective: on the one hand, the entropy model is used to estimate the code rate, and the pixel MSE or MS-SSIM is used to measure the distortion, to construct a Rate-Distortion Loss (LRD), so that the student model can adaptively balance energy compression and code rate regularization in the stage distillation training, without forced channel alignment, thereby reducing the model structure adaptation cost; on the other hand, based on the implicit supervision signal of the joint objective, the student model is driven to accurately maintain the latent variable distribution characteristics while reducing the complexity, to guarantee the Rate-Distortion (RD) performance, so as to achieve a better synergy between model lightweight and compression effect, and to provide a technical path that takes into account efficiency and accuracy for model deployment in resource-constrained scenarios.

[0077] In some specific embodiments of the present application, S400, the student model trained in S300 is used to encode the image to be compressed, to obtain a code stream and reconstruct an image, which can adopt the following steps:

[0078] S401, an image to be compressed is obtained as an input image;

[0079] S402, the encoding network performs feature compression and transformation on the input image, extracts key information through the pruned lightweight structure, and outputs a quantized compressed representation;

[0080] S403, the decoding network receives the compressed representation output by the encoding network, performs feature recovery and reconstruction through the corresponding lightweight inverse transformation structure, and finally outputs a reconstructed image.

[0081] The above embodiments of the present application realize efficient encoding and decoding through lightweight pruning, and at the same time, retain the core compression capability with the structural consistency of the teacher model, so that finally, on the basis of reducing resource consumption and improving processing efficiency, high-quality reconstructed images can still be output, which has strong practicality and wide deployment potential.

[0082] Figure 2 Fig. (a) is a radar chart, and from the outermost circle to the center, the smaller / better the value is. As can be seen from the chart, the method of the present application is obviously better than other methods in terms of parameter quantity, floating point operations per second, decoding delay, throughput, BD-rate, etc. Fig. (b) is a scatter plot, the horizontal axis (FLOPS(G)) represents the floating point operation amount (unit: 10^9 times) of model inference, and the larger the value is, the higher the computational complexity is; the vertical axis (BD-Rate over VTM-21.0 (%)) represents the BD-rate based on VTM 21.0 (a reference model of video coding standard), and the smaller the value is, the higher the compression rate relative to VTM-21.0 (the better the performance is).

[0083] Figure (b) is a scatter plot (compression rate vs. computational amount)

[0084] Horizontal axis (FLOPS(G)): floating-point operation amount of model inference (unit: 109times), the larger the value, the higher the computational complexity (the more computing power is consumed).

[0085] Vertical axis (BD-Rate over VTM-21.0(%)): BD-rate comparison based on VTM-21.0 (a reference model of a video coding standard), the smaller the value, the higher the compression rate relative to VTM-21.0 (the better the performance). Each point represents a model, and the color / shape distinguishes the model type (for example, S2CFormer (Teacher) is a teacher model, and KDIC wKD and KDIC w / o KD are derived student models thereof).

[0086] The red arrow shows the advantage of “KDIC w KD” over other models, indicating that it has both “lower computational amount (smaller FLOPS)” and “lower BD-rate (higher compression rate)”, that is, it achieves the breakthrough of “lighter weight + higher compression quality”.

[0087] Figure 4 For the performance comparison of the learning-based image compression method based on stage module distillation of the present application shown according to an exemplary embodiment and the existing advanced method, it can be seen that the new model (such as KDIC) has higher compression efficiency: better reconstruction quality at the same code rate, or lower code rate at the same quality, verifying the value of lightweight technologies such as pruning and distillation.

[0088] Figure 5 For the distillation effect display of the learning-based image compression method based on stage module distillation according to an exemplary embodiment, it can be seen that the student model (KDIC) is better than / matches the teacher model (SSC-Hybrid): it is shown that through model compression technologies (such as pruning and knowledge distillation), the student model can still inherit or even surpass the compression performance of the teacher model while reducing the complexity.

[0089] Based on the same technical concept, in some specific embodiments of the present application, a learning-based image compression system 100 based on stage module distillation is provided, comprising:

[0090] The stage division module 110 divides the encoding network and the decoding network of the teacher model into n stages according to the feature resolution change nodes, and n is an integer greater than 1;

[0091] The pruning construction module 120 generates a student model corresponding to the teacher model in terms of stages based on the divided teacher model and performs pruning;

[0092] the i-th stage of the student model, i being an integer greater than or equal to 1 and less than or equal to n, so that the distribution of the latent variable output by the i-th stage approximates the latent variable distribution of the corresponding stage of the teacher model; after completion, the stage is frozen and the next stage is turned to, until all stages are trained;

[0093] the application module 140: using the trained student model to encode the image to be compressed, obtaining a code stream and reconstructing the image.

[0094] The above modules / units in the examples of the present application can refer to the implementation techniques of the corresponding steps of the learning-based image compression method based on stage module distillation in the above embodiments, which will not be described here.

[0095] Based on the same technical concept, in some embodiments of the present application, a terminal is provided, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor can be used to execute any learning-based image compression method based on stage module distillation or run any learning-based image compression system based on stage module distillation when executing the program.

[0096] Based on the same technical concept, in some embodiments of the present application, a computer readable storage medium is also provided, which stores a computer program, and the program can be used to execute any learning-based image compression method based on stage module distillation or run any learning-based image compression system based on stage module distillation when executed by a processor.

[0097] Optionally, the memory is used to store programs; the memory may include volatile memory (English: volatile memory), such as random-access memory (English: random-access memory, abbreviated: RAM), such as static random-access memory (English: static random-access memory, abbreviated: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated: DDR SDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory is used to store computer programs (such as applications, functional modules, etc. that implement the above-mentioned methods), computer instructions, etc., and the above-mentioned computer programs, computer instructions, etc. can be partitioned and stored in one or more memories. In addition, the above-mentioned computer programs, computer instructions, data, etc. can be called by the processor.

[0098] The aforementioned computer programs, computer instructions, etc. may be partitioned and stored in one or more memories, and the aforementioned computer programs, computer instructions, data, etc. may be called by a processor.

[0099] The processor is configured to execute the computer program stored in the memory to implement the various steps of the method involved in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0100] The processor and memory can be independent structures or integrated structures. When the processor and memory are independent structures, the memory and processor can be coupled via a bus.

[0101] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0102] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 one or more flow or blocks

[0103] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 one or more flow or blocks

[0104] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 one or more flow or blocks

[0105] The above describes some specific embodiments of the present application. It needs to be understood that the present application is not limited to the above specific embodiments, and those skilled in the art can make various modifications or changes within the scope of the claims, which does not affect the essential content of the present application. The above preferred features can be used in combination as long as they are not in conflict with each other.

Claims

1. A learning-based image compression method based on stage-wise modular distillation, characterized by, The application comprises the following steps: Divide the encoding network and the decoding network of the teacher model into n stages according to the feature resolution change nodes, wherein n is an integer greater than 1; Based on the divided teacher model, generate a student model maintaining the stage corresponding relationship with the teacher model and perform pruning; Freeze all parameters of the teacher model, freeze all parameters outside the i-th stage of the student model, only distill train the i-th stage, 1≤i≤n, so that the output latent variable distribution of the i-th stage of the student model approximates the latent variable distribution of the corresponding stage of the teacher model; after completion, freeze the i-th stage of the student model and go to the i+1 stage, until all stages are trained; Use the trained student model to encode the image to be compressed to obtain a code stream and reconstruct the image.

2. The learning-based image compression method based on stage-wise modular distillation according to claim 1, wherein, The teacher model is divided into n stages according to the feature resolution change nodes, wherein n is an integer greater than 1, comprising: Traverse all layers of the encoding network and record all down-sampling nodes n1 that cause the feature resolution to decrease; Traverse all layers of the decoding network and record all up-sampling nodes n2 that cause the feature resolution to increase; Divide the teacher model into n stages at the down-sampling nodes and the up-sampling nodes, n=n1+n2.

3. The learning-based image compression method based on stage-wise modular distillation according to claim 1, wherein, Based on the divided teacher model, the student model maintaining the stage corresponding relationship with the teacher model is generated and pruned, comprising: Divide the student model into corresponding stages according to the stages of the teacher model; Prune each stage by deleting channels, reducing the number of nonlinear transformation blocks, and expanding the feedforward factor.

4. The learning-based image compression method based on stage-wise modular distillation according to claim 1, wherein, The distribution of the output latent variable of the i-th stage approximates the latent variable distribution of the corresponding stage of the teacher model, comprising: Use RD Loss based on KL divergence as the stage distillation loss to constrain the student latent variable distribution to converge to the teacher distribution.

5. The learning-based image compression method based on stage-wise modular distillation according to claim 1, wherein, Further comprising: During the distillation training process of each stage, introduce an implicit supervision signal based on the rate-distortion joint target to adaptively adjust the latent variable energy compression and the code rate regularization.

6. The learning-based image compression method based on stage-wise modular distillation according to claim 1, wherein, The student model trained is used to encode the image to be compressed to obtain a code stream and reconstruct the image, comprising: Obtain an image to be compressed as an input image; The encoding network performs feature compression and transformation on the input image, extracts key information through the pruned lightweight structure, and outputs a quantized compressed representation; The decoding network receives the compressed representation output by the encoding network, performs feature recovery and reconstruction through the corresponding lightweight inverse transformation structure, and finally outputs a reconstructed image.

7. A learning-based image compression system based on stage-wise modular distillation, characterized by, The application comprises the following steps: Stage division module: divide the encoding network and the decoding network of the teacher model into n stages according to the feature resolution change nodes, wherein n is an integer greater than 1; Pruning and construction module: based on the divided teacher model, generate a student model maintaining the stage corresponding relationship with the teacher model and perform pruning; Phase distillation module: freeze all parameters of the teacher model, freeze all parameters of the student model except the i-th phase, only distill the i-th phase, 1≤i≤n, to make the output latent variable distribution of the i-th phase of the student model approach the latent variable distribution of the corresponding phase of the teacher model; Application module: using the trained student model to encode the image to be compressed, obtaining the code stream and reconstructing the image.

8. The learning-based image compression system based on stage-wise modular distillation of claim 7, wherein, Also includes an end-to-end supervision module, during the distillation training process of each stage, an implicit supervision signal based on the rate-distortion joint target is introduced to adaptively adjust the latent variable energy compression and the code rate regularization.

9. A terminal comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program can be used to execute the method of any one of claims 1-6, or, run the system of any one of claims 7-8.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor can be used to execute the method of any one of claims 1-6, or, run the system of any one of claims 7-8. The program is executed by the processor can be used to execute the method of any one of claims 1-6, or, run the system of any one of claims 7-8.

Images