A multi-task image encoding method and device based on multi-branch aggregation

Abstract

The application provides a multi-task image coding method and device based on multi-branch aggregation, which comprises the following steps: step 1, training a basic image coding model, a multi-branch aggregation module and an importance predictor; step 2, sending features into corresponding importance predictors, and calculating output binary masks according to the importance of different features to different tasks; step 3, splitting the features into two groups of sub-features, namely main path features and bypass features; step 4, selecting corresponding bypasses according to specified tasks, and sending the main path features and the bypass features into the main path and the selected bypasses respectively; and step 5, recombining the two groups of sub-features calculated by the main path and the bypass according to the positions and sizes of the original features, and completing decoding to output a complete image. The method of the application can greatly reduce the optimization difficulty and training cost without optimizing a separate specific task model.

Description

Technical Field

[0001] This invention relates to the field of image coding, and in particular to a multi-task image coding method and apparatus based on multi-branch aggregation. Background Technology

[0002] In recent years, research on multi-task coding to meet the needs of human perception and machine vision has made some progress. To achieve optimal performance across various tasks, an intuitive and simple solution is to optimize different coding models for different tasks and employ multiple bitstreams (independent or scalable). However, this approach leads to significant parameter overhead and redundant bitstream storage. Therefore, some research has attempted to enhance multi-task compression efficiency by extracting a unified, compact representation from images. Typically, these methods rely on a general encoder for feature extraction and deploy different decoding models to support the corresponding visual tasks. However, these methods still require the use of multiple task-specific decoders, resulting in low parameter efficiency. Recent research has begun to explore the design of unified compression models for multi-task applications, where the encoder generates a unified representation of the image, while the decoder focuses on task-oriented reconstruction. Common techniques include conditional generation and residual prediction, which enable user-controllable tuning of the decoded image through task-driven guidance. Although these methods approach the performance of multi-decoder models, they only focus on human visual needs, such as perceptual quality-oriented and fidelity-oriented reconstruction. More importantly, in the context of multi-task learning, unified compression models for multi-task learning also face the challenge of multi-objective optimization, and their performance struggles to match the results of independent optimization for each task. The key to this problem lies in the indiscriminate treatment of unified representations across different tasks in existing research, which necessitates finding accurate Pareto fronts across tasks. This typically requires the use of many complex techniques for optimization. Failure to find an optimal solution can lead to a significant performance drop, and the difficulty of the optimization search increases dramatically with the number of tasks. To date, few methods consider the correlation of features between tasks to ensure efficient multi-task collaboration, while simultaneously considering the specificity of different tasks to optimize features for efficient multi-task expansion. Summary of the Invention

[0003] Objective: This invention addresses the shortcomings of existing technologies by providing a multi-task image coding method based on multi-branch aggregation. In the existing MetaFormer backbone network, the original single-path MLP (Multi-Layer Perceptron) module is replaced with a multi-branch aggregation module, composed of multiple MLPs. Each MLP branch is optimized for a specific target task, and an additional predictor is designed to assign features to each MLP, enabling transfer between multiple targets. Experiments demonstrate that this scheme achieves state-of-the-art performance in multi-task coding scenarios such as high-fidelity, high-realism, classification, and semantic segmentation, and can support smooth transfer between various reconstruction tasks by adjusting the feature aggregation ratio.

[0004] The method of the present invention includes the following steps:

[0005] Step 1, Training Phase: To address perceptual loss and visual task loss, the correlation of semantic features between machine vision tasks is utilized to train the basic image coding model, multi-branch aggregation module, and importance predictor.

[0006] Step 2, Testing Phase: Based on the specified task and feature aggregation ratio, the features are fed into the corresponding importance predictor. The predictor calculates and outputs a binary mask based on the importance of different features to different tasks.

[0007] Step 3: The multi-branch aggregation module splits the features into two groups of sub-features based on the binary mask, namely the main path features and the side path features.

[0008] Step 4: Based on the specified task, select the corresponding bypass and send the main path features and bypass features to the main path and the selected bypass, respectively.

[0009] Step 5: Recombine the two sets of sub-features calculated for the main path and the bypass path according to the position and size of the original features, and complete the subsequent decoding to output a complete image.

[0010] Step 1 includes: considering the correlation of semantic features among machine vision tasks, optimizing the basic image coding model and its main path using an image coding training method optimized for perceptual quality loss, so that the basic image coding model can extract rich semantic information from the original image during the encoding process, realize feature reuse among multiple tasks, and reconstruct an image with excellent perceptual quality during decoding; after obtaining the perceptual quality optimized basic image coding model, adding side paths for specific tasks alongside the main path to form a multi-branch aggregation module, and fine-tuning some parameters of the loss function for specific tasks to obtain a multi-task image coding model that supports multi-task target decoding.

[0011] In step 1, the multi-task image coding model includes one basic image coding model and six multi-branch aggregation modules. The basic image coding model is an existing learning-based image coding model such as TinyLIC, which includes one encoder, one decoder, and one entropy model. The encoder encodes the image into a binary bitstream, the entropy model predicts the entropy of the binary bitstream, and the decoder decodes the binary bitstream into an image. The multi-task image coding model is built based on the basic image coding model. The encoder includes three multi-branch aggregation modules, and the decoder includes three multi-branch aggregation modules. The multi-branch aggregation modules enable the encoder and decoder to support multi-task encoding and decoding.

[0012] Each multi-branch aggregation module contains one main path, three side paths, and three importance predictors. The main path supports high-fidelity image reconstruction, while the three side paths support low-distortion image reconstruction, classification image reconstruction, and semantic segmentation image reconstruction, respectively. The importance predictors are used to assign semantic features to the main path and side paths.

[0013] Both the main path and the branch path contain two fully connected layers and one activation layer, while the importance predictor contains three fully connected layers and two activation layers. The main path, the bypass path, and the importance predictor are denoted as φ. main φ side φ pred .

[0014] In step 1, the training phase includes:

[0015] Step 1-1, Stage 1: Using Variational Auto-Encoder (VAE) and Generative Adversarial Network (GAN), combined with perceptual quality loss, rate-distortion optimization is performed on the basic image coding model and its main path to obtain a basic image coding model for multi-task optimization.

[0016] Step 1-2, Stage 2: Add bypasses according to the preset tasks to form a multi-branch aggregation module, and use the loss function of each task to optimize the corresponding bypasses and importance predictors respectively.

[0017] Step 1-1 includes: during the computation of each training batch Epoch, optimizing the base image coding model according to the following loss function:

[0018]

[0019] Among them, the proportional loss function Used to optimize the importance predictor and generator loss function. The generator G(·) and discriminator loss function are used to optimize Generative Adversarial Networks (GANs). Discriminator D(·) used to optimize Generative Adversarial Networks (GANs), image coding loss function The encoder E(·), generator G(·), and entropy model P(·) are used to optimize the basic image coding model; x, y、 These are the input image, the output reconstructed image, the latent space features of the variational autoencoder (VAE) before quantization, and the latent space features of the variational autoencoder (VAE) after quantization. This indicates that when x follows a probability distribution p x Expectations at that time; Indicates when Follows probability distribution p y The expected value at time; S represents the number of stages in the basic image coding model that use multi-branch aggregation, M (s) The mask M and H represent the importance predictor generated in the s-th stage of the multi-task image coding model. (s) and W (s) represents the spatial height and width of the s-th stage of the multi-task image coding model, respectively, and ρ is the feature aggregation ratio; Entropy model pairs in image coding models Predicted bitrate size Representing x and The mean square error between them; It is Learned Patch Image Perceptual Similarity (LPIPS); the bitrate loss weights for encoding quality q. Generator loss weight λ G Perceived quality loss weight λ perc , proportional loss weight λ ratio The hyperparameters used to adjust the loss weights.

[0020] Steps 1-2 include: during the calculation of each training batch Epoch, optimizing the multi-branch aggregation module according to the following optimization objectives:

[0021]

[0022] Among them, the bypass parameter set And importance predictor parameter set These are bypass φ side Optimal parameters and importance predictor φ pred The optimal parameters, the task loss function This is the loss function for the machine vision task selected in the current training batch, with task loss weights λ. task The hyperparameters used to adjust the loss weights.

[0023] In step 1, the loss function for the specific task undergoes partial parameter fine-tuning, specifically including: in step 1-1, all model parameters are updated during training; in step 1-2, only φ is adjusted. side and φ pred The parameters are updated during training, while the other parameters remain fixed.

[0024] Step 2 includes: First, the user selects the corresponding importance predictor based on the task the decoded image needs to support, and sets the feature aggregation ratio ρ according to the required accuracy for the task. Then, the optimized importance predictor generates the optimal binary mask M suitable for the current task and accuracy setting based on the importance of different features to different tasks. The importance predictor uses a non-parametric partial averaging layer. To fuse multi-scale information; given the intermediate feature tensor u of the importance predictor with spatial dimensions H×W×C′ in the real domain, a partial averaging layer Global average pooling is performed on the latter half of the intermediate features, and then the data is expanded back to the feature space size as global information, while the first half of the features remains unchanged as local information. The process can be formally expressed as follows:

[0025]

[0026] Where u(h,w,c) and u(i,j,c) represent the real values ​​at spatial coordinates (h,w,c) and (i,j,c) in tensor u, respectively.

[0027] During training, since directly binarizing the importance scores based on the feature aggregation ratio ρ to obtain a binary mask is non-differentiable, a set of learnable bias parameters b is first used to bias the output u′ of the last fully connected layer in the importance predictor, thus obtaining the intermediate variable corresponding to each discrete ρ. Then, Gumbel noise is introduced during the binarization process with a threshold τ = 0.5, i.e., the Gumbel-Sigmoid technique is used to soften the sampling process, as shown in the following formula:

[0028] M = Gumbel - Sigmoid(u′+b,τ)

[0029] After training, the importance predictor can binarize the score map according to the feature aggregation ratio ρ, and the bias layer will be discarded.

[0030] In step 3, the multi-branch aggregation module will divide the features sent into the multi-branch aggregation module into two parts according to the optimal binary mask M generated in step 2. The features corresponding to 1 in the optimal binary mask M are the main road features, and the features corresponding to 0 are the bypass features. There is no overlap between the main road features and the bypass features.

[0031] In step 4, based on the tasks that the decoded image needs to support, a bypass φ optimized for the task is selected in the multi-branch aggregation module. side The main road features and bypass features extracted in step 3 are then fed into the main road φ. main and bypass φ side Main road φ main and various bypass φ side The complexity can be designed differently according to different task requirements, so that the two sets of sub-features can be computed independently and in parallel in different branches of the multilayer perceptron.

[0032] In step 5, after the main path features and bypass features are calculated in the multi-branch aggregation module, they will be re-aggregated into new features with the same size and position as the original features based on the spatial size and feature position before splitting. The new features output by the multi-branch aggregation module will be sent to the subsequent multi-branch aggregation module, and steps 2 to 5 will be repeated until the decoding process is completed and a complete image is output.

[0033] The present invention also provides a multi-task image coding device based on multi-branch aggregation, including a multi-branch aggregation module, an importance predictor, and a corresponding training strategy;

[0034] The multi-branch aggregation module training strategy is used to perform the following steps:

[0035] Step 1, Training Phase: To address the perceptual quality loss and visual task loss, the correlation of semantic features between machine vision tasks such as image reconstruction, image classification, and image segmentation is used to train the basic image coding model, multi-branch aggregation module, and importance predictor to form a multi-task image coding model.

[0036] The importance predictor is used to perform the following steps:

[0037] Step 2, Testing Phase: Based on the specified task and feature aggregation ratio, the features are fed into the corresponding importance predictor. The predictor calculates and outputs a binary mask based on the importance of different features to different tasks.

[0038] The multi-branch aggregation module is used to perform the following steps:

[0039] Step 3: The multi-branch aggregation module splits the features into two groups of sub-features based on the binary mask, namely the main path features and the side path features.

[0040] Step 4: Based on the specified task, select the corresponding bypass and send the main path features and bypass features to the main path and the selected bypass, respectively.

[0041] Step 5: Recombine the two sets of sub-features calculated for the main path and the bypass path according to the position and size of the original features, and complete the subsequent decoding to output a complete image.

[0042] Beneficial Effects: Considering that features in image coding models have different importance for different tasks, this invention can perform non-uniform feature processing. By using the importance predictor proposed in this invention, the model can leverage the importance of different features in different tasks, reuse task-general semantic features, and allocate features in the main and side paths according to the required task and accuracy. This enables the model to support seamless multi-task conversion in an integrated manner. Simultaneously, utilizing the feature correlation between tasks, this invention proposes a two-stage optimization strategy with partial parameter fine-tuning capabilities to overcome the multi-objective optimization challenge in multi-task coding. This allows the multi-branch aggregation module to support new tasks by adding side paths and corresponding importance predictors and using partial parameter optimization, without needing to optimize separate task-specific models, greatly reducing optimization difficulty and training costs. Attached Figure Description

[0043] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0044] Figure 1 This is a flowchart of the overall method of the present invention.

[0045] Figure 2 This is a diagram of a TinyLIC system that uses a multi-branch aggregation module.

[0046] Figure 3a This is a schematic diagram of a single branch block.

[0047] Figure 3b This is a schematic diagram of multiple branch blocks.

[0048] Figure 3c This is a schematic diagram of a multi-branch aggregation module.

[0049] Figure 3d This is a schematic diagram of the importance prediction module.

[0050] Figure 4a , Figure 4b , Figure 4c , Figure 4d , Figure 4e , Figure 4f These are the quantitative test results of this invention on multiple datasets.

[0051] Figure 5 This is the visualization effect of the present invention on the multi-realism image reconstruction task.

[0052] Figure 6This is the visualization effect of the present invention on the image classification task.

[0053] Figure 7 This is a visualization of the semantic segmentation task of this invention. Detailed Implementation

[0054] like Figure 1 As shown, this embodiment of the invention provides a multi-task image coding method based on multi-branch aggregation, including the following steps:

[0055] Step 1, Training Phase: To address the perceptual quality loss and visual task loss, the correlation of semantic features between machine vision tasks such as image reconstruction, image classification, and image segmentation is used to train the basic image coding model, multi-branch aggregation module, and importance predictor to form a multi-task image coding model.

[0056] Step 2, Testing Phase: Based on the specified task and feature aggregation ratio, the features are fed into the corresponding importance predictor. The predictor calculates and outputs a binary mask based on the importance of different features to different tasks.

[0057] Step 3: The multi-branch aggregation module splits the features into two groups of sub-features based on the binary mask, namely the main path features and the side path features.

[0058] Step 4: Based on the specified task, select the corresponding bypass and send the main path features and bypass features to the main path and the selected bypass, respectively.

[0059] Step 5: Recombine the two sets of sub-features calculated for the main path and the bypass path according to the position and size of the original features, and complete the subsequent decoding to output a complete image.

[0060] Step 1 includes: considering the correlation of semantic features among machine vision tasks, optimizing the basic image coding model and its main path using an image coding training method optimized for perceptual quality loss, so that the basic image coding model can extract rich semantic information from the original image during the encoding process, realize feature reuse among multiple tasks, and reconstruct an image with excellent perceptual quality during decoding; after obtaining the perceptual quality optimized basic image coding model, adding side paths for specific tasks alongside the main path to form a multi-branch aggregation module, and fine-tuning some parameters of the loss function for specific tasks to obtain a multi-task image coding model that supports multi-task target decoding.

[0061] The multi-task image coding model comprises one basic image coding model and six multi-branch aggregation modules. The basic image coding model is an existing learning-based image coding model such as TinyLIC, which includes one encoder, one decoder, and one entropy model. Figure 2As shown in the diagram. The encoder contains three multi-branch aggregation modules, and the decoder also contains three multi-branch aggregation modules. The encoder's function is to encode the image into a binary bitstream, the entropy model's function is to predict the entropy of the binary bitstream, the decoder's function is to decode the binary bitstream into an image, and the multi-branch aggregation modules enable the encoder and decoder to support multi-task encoding and decoding. Each multi-branch aggregation module contains one main path, three side paths, and three importance predictors, such as... Figure 3c As shown, the main path supports high-fidelity image reconstruction, while the three side paths support low-distortion image reconstruction, classification image reconstruction, and semantic segmentation image reconstruction, respectively. An importance predictor is used to assign semantic features to the main path and side paths. Both the main path and the side paths contain two fully connected layers and one activation layer, while the importance predictor consists of three fully connected layers and two activation layers. The main path, side paths, and importance predictor are denoted as φ. main φ side φ pred .

[0062] In the training phase of step 1, to ensure the stability of the training process, this invention is divided into two training phases. First, the basic image coding model and its main path are optimized to address perceptual quality loss. Then, by adding bypasses alongside the main path for different tasks to form a multi-branch aggregation module, the corresponding bypasses and importance predictors are optimized for different tasks. Specifically, this includes:

[0063] Step 1-1: Using Variational Auto-Encoder (VAE) and Generative Adversarial Network (GAN), combined with perceptual quality loss, rate-distortion optimization is performed on the basic image coding model and its main path to obtain a basic image coding model for multi-task optimization.

[0064] Steps 1-2: Add bypasses according to the preset tasks to form a multi-branch aggregation module, and use the loss function of each task to optimize the corresponding bypasses and importance predictors respectively.

[0065] For step 1-1, during the calculation of each training batch Epoch, the base image coding model is optimized according to the following loss function:

[0066]

[0067] Among them, the proportional loss function Used to optimize the importance predictor and generator loss function. The generator G(·) and discriminator loss function are used to optimize Generative Adversarial Networks (GANs). Discriminator D(·) used to optimize Generative Adversarial Networks (GANs), image coding loss function The encoder E(·), generator G(·), and entropy model P(·) are used to optimize the basic image coding model; x, y、 These are the input image, the output reconstructed image, the latent space features of the variational autoencoder (VAE) before quantization, and the latent space features of the variational autoencoder (VAE) after quantization. and Indicates when x and Follows probability distribution p x and p y The expected value at time; S represents the number of stages in the basic image coding model that use multi-branch aggregation, M (s) The mask M and H represent the importance predictor generated in the s-th stage of the multi-task image coding model. (s) and W (s) represents the spatial height and width of the s-th stage of the multi-task image coding model, respectively, and ρ is the feature aggregation ratio; Entropy model pairs in image coding models Predicted bitrate size Representing x and The mean square error between them; It is Learned Patch Image Perceptual Similarity (LPIPS); the bitrate loss weights for encoding quality q. Generator loss weight λ G Perceived quality loss weight λ perc , proportional loss weight λ ratio The hyperparameters used to adjust the loss weights.

[0068] For steps 1-2, during the calculation of each training batch Epoch, the multi-branch aggregation module is optimized according to the following optimization objectives:

[0069]

[0070] Among them, the bypass parameter set And importance predictor parameter set These are bypass φ side and importance predictor φ pred The optimal parameters, the task loss function This is the loss function for the machine vision task selected in the current training batch, with task loss weights λ. task The hyperparameters used to adjust the loss weights.

[0071] In step 1-1, all model parameters are updated during training, while in step 1-2, only φ is updated. side and φpred The parameters are updated during training, while the other parameters remain fixed.

[0072] Step 2 includes: First, manually selecting the corresponding importance predictor based on the task the decoded image needs to support, and manually setting the feature aggregation ratio ρ according to the required accuracy of the supported task. Then, the optimized importance predictor can generate the optimal binary mask M suitable for the current task and accuracy setting based on the importance of different features to different tasks. The importance predictor consists of multiple cascaded fully connected layers and activation layers, such as... Figure 3d As shown, the input image coding model contains features, and the output is a corresponding binary mask M. In addition to only three parameterized fully connected layers, this invention also uses a non-parametric partially averaging layer. To fuse multi-scale information. Given an importance predictor intermediate feature tensor u of spatial size H×W×C′ in the real domain, a partial averaging layer... Global average pooling is performed on the latter half of the intermediate features, and then the data is expanded back to the feature space size as global information, while the first half of the features remains unchanged as local information. The process can be formally expressed as follows:

[0073]

[0074] Where u(h,w,c) and u(i,j,c) represent the real values ​​at spatial coordinates (h,w,c) and (i,j,c) in tensor u, respectively.

[0075] During training, since directly binarizing the importance scores based on the feature aggregation ratio ρ to obtain a binary mask is non-differentiable, a set of learnable bias parameters b is first used to bias the output u′ of the last fully connected layer in the importance predictor, thus obtaining the intermediate variable corresponding to each discrete ρ. Then, Gumbel noise is introduced during the binarization process with a threshold τ = 0.5, i.e., the Gumbel-Sigmoid technique is used to soften the sampling process, as shown in the following formula:

[0076] M = Gumbel - Sigmoid(u′+b,τ)

[0077] After training, the importance predictor can binarize the score map according to the feature aggregation ratio ρ. Figure 3d The bias layer of the medium importance predictor will be discarded.

[0078] In step 3, the multi-branch aggregation module will divide the features sent into the multi-branch aggregation module into two parts according to the optimal binary mask M generated in step 2. The features corresponding to 1 in the optimal binary mask M are the main road features, and the features corresponding to 0 are the bypass features. There is no overlap between the main road features and the bypass features.

[0079] In step 4, based on the task that the decoded image needs to support, a bypass φ optimized for that task is selected in the multi-branch aggregation module. side The main road features and bypass features extracted in step 3 are then fed into the main road φ. main and bypass φ side Main road φ main and various bypass φ side The complexity can be designed differently depending on the task requirements, so that the two sets of sub-features can be computed independently and in parallel in different branches of the multilayer perceptron.

[0080] In step 5, after the two sets of sub-features are calculated in the multi-branch aggregation module, they are re-aggregated into new features that are the same size and position as the original features, based on their spatial dimensions and feature positions before splitting. The new features output by the multi-branch aggregation module are then fed into subsequent multi-branch aggregation modules, repeating steps 2 to 5 until the decoding process is completed and a complete image is output.

[0081] The present invention also provides a multi-task image coding device based on multi-branch aggregation, including a multi-branch aggregation module, an importance predictor, and a corresponding training strategy;

[0082] The multi-branch aggregation module training strategy is used to perform the following steps:

[0083] Step 1, Training Phase: To address the perceptual quality loss and visual task loss, the correlation of semantic features between machine vision tasks such as image reconstruction, image classification, and image segmentation is used to train the basic image coding model, multi-branch aggregation module, and importance predictor to form a multi-task image coding model.

[0084] The importance predictor is used to perform the following steps:

[0085] Step 2, Testing Phase: Based on the specified task and feature aggregation ratio, the features are fed into the corresponding importance predictor. The predictor calculates and outputs a binary mask based on the importance of different features to different tasks.

[0086] The multi-branch aggregation module is used to perform the following steps:

[0087] Step 3: The multi-branch aggregation module splits the features into two groups of sub-features based on the binary mask, namely the main path features and the side path features.

[0088] Step 4: Based on the specified task, select the corresponding bypass and send the main path features and bypass features to the main path and the selected bypass, respectively.

[0089] Step 5: Recombine the two sets of sub-features calculated for the main path and the bypass path according to the position and size of the original features, and complete the subsequent decoding to output a complete image.

[0090] In a specific embodiment of the present invention, a multi-task image coding method based on multi-branch aggregation is provided, comprising:

[0091] (1) As Figure 2 , Figure 3a and Figure 3b As shown, a scaling factor (SF) is added to TinyLIC to support variable bitrate, and a multi-branch aggregation module is added to the first, second and third stages of the codec, including a perceptual optimization main path, a mean square error optimization side path, a classification optimization side path and a semantic segmentation optimization side path, which respectively support high realism, low distortion, classification and semantic segmentation tasks. The remaining components are consistent with the original TinyLIC.

[0092] During the training process in step 1-1, the perceptual quality loss is addressed. The image coding model is optimized to obtain a base image coding model for subsequent multi-task extensions. During the training process in steps 1-2, some parameters of the model are fine-tuned, with only φ being updated. side and φ pred The parameters are set as follows, while the remaining parameters remain unchanged. In this embodiment, a comprehensive dataset including Flicker2W, DIV2K, and CLIC training sets is used for optimization of perceptual quality and mean squared error; the ImageNet-1K dataset is used for optimization of classification tasks; and the ADE20K dataset is used for optimization of semantic segmentation tasks. The hyperparameters are set as follows: λ G =2.56, λ perc =4.26, λ task =1,λ ratio =10. The bitrate point q is randomly selected from integers from 1 to 8, and the aggregation ratio ρ is randomly sampled from the real number interval [0,1]. The number of iterations in step 1-1 is set to 3M, with the first 1.5M excluding... The latter half contains The number of iterations in steps 1-2 is set to 500K. The initial learning rate is set to 10 in each training step. -4 It decays to 10 in the last 25% of iterations. -5 The training images were randomly cropped to 256×256×3 and randomly horizontally flipped. ConvNeXt-Tiny was used as the classification model when training the classification bypass, and PSPNet50 was used as the segmentation model when training the semantic model.

[0093] (2) In practical applications, the model trained in step (1) requires specifying a task, manually selecting the corresponding importance predictor, and manually setting the aggregation ratio ρ according to the required task accuracy. Then, the optimized importance predictor can generate the optimal binary mask M suitable for the current task and accuracy setting based on the importance of different features to different tasks, such as... Figure 3d As shown.

[0094] (3) The multi-branch aggregation module will divide the features sent to the module into two parts according to the optimal binary mask M generated in step (2). In M, the features corresponding to 1 are the main path features, and the features corresponding to 0 are the bypass features. There is no intersection between the main path features and the bypass features, such as... Figure 3c As shown.

[0095] (4) Select the bypass φ optimized for the task in the multi-branch aggregation module according to the task that the decoded image needs to support. side The main road features and bypass features separated in step (3) are then sent to the main road φ respectively. main and bypass φ side Main road φ main and various bypass φ side The complexity can be designed differently depending on the task requirements, allowing for independent and parallel multilayer perceptron computation of the two sets of sub-features in different branches, such as... Figure 3c As shown.

[0096] (5) After the two sets of sub-features are calculated in the multi-branch aggregation module, they will be re-aggregated into new features that are the same size and position as the original features, based on the spatial size and feature positions before splitting, such as... Figure 3c As shown. The new features output by the multi-branch aggregation module are fed into subsequent multi-branch aggregation modules, and steps 2 to 5 are repeated until the decoding process is completed and a complete image is output.

[0097] The performance of this embodiment is evident. Figure 4a , Figure 4b , Figure 4c , Figure 4d , Figure 4e , Figure 4f , Figure 5 , Figure 6 and Figure 7Among them, Bit-rate is the bitrate, measured in bpp (bits per pixel). PSNR, LPIPS, FID, Top-1Acc, and mIoU represent Peak Signal-to-Noise Ratio, Learnable Patch Perceptual Similarity, Fréchet Inception Distance, Top-1 Accuracy, and Mean Intersection over Union (MIoU), respectively. CRDR, DIRAC, MRIC, HiFiC, TransTIC, SFT, and VTM are currently the most advanced baseline contrast methods. By adding this invention to the TinyLIC image coding model, the model can support smooth transfer between reconstruction targets in multiple tasks by adjusting the feature aggregation ratio. Final tests show that on the CLIC dataset, at a bitrate of 0.0789 bpp, the PSNR (high-quality metric) ranges from 29.49 to 30.83 dB, and the FID (low-quality metric) ranges from 4.52 to 45.76; on the Kodak dataset, at a bitrate of 0.1121 bpp, the PSNR ranges from 26.61 to 28.10 dB, and the LPIPS (low-quality metric) ranges from 0.1084 to 0.2758; on the ImageNet dataset, at a bitrate of 0.1521 bpp, the classification accuracy (high-quality metric) ranges from 60.59 to 76.77%; and on the ADE20K dataset, at a bitrate of 0.0948 bpp, the mIoU (high-quality metric) ranges from 28.34 to 37.41%.

[0098] This application also provides a computer program product, including computer instructions stored in a computer-readable storage medium. When a processor of an electronic device reads the computer instructions from the computer-readable storage medium, the processor executes the computer instructions, causing the electronic device to perform the steps of the method.

[0099] This invention provides a multi-task image coding method and apparatus based on multi-branch aggregation. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A multi-task image coding method based on multi-branch aggregation, characterized in that, Includes the following steps: Step 1, Training Phase: To address perceptual loss and visual task loss, the correlation of semantic features between machine vision tasks is utilized to train the basic image coding model, multi-branch aggregation module, and importance predictor. Step 2, Testing Phase: Based on the specified task and feature aggregation ratio, the features are fed into the corresponding importance predictor. The predictor calculates and outputs a binary mask based on the importance of different features to different tasks. Step 2 includes: First, the user selects the corresponding importance predictor based on the task the decoded image needs to support, and sets the feature aggregation ratio ρ according to the required accuracy for the task. Then, the optimized importance predictor generates the optimal binary mask M suitable for the current task and accuracy setting based on the importance of different features to different tasks. The importance predictor uses a non-parametric partial averaging layer. To fuse multi-scale information; given the intermediate feature tensor u of the importance predictor with spatial dimensions H×W×C′ in the real domain, a partial averaging layer Global average pooling is performed on the latter half of the intermediate features, and then the data is expanded back to the feature space size as global information, while the first half of the features remains unchanged as local information. The process can be formally expressed as follows: Where u(h,w,c) and u(i,j,c) represent the real values ​​at spatial coordinates (h,w,c) and (i,j,c) in tensor u, respectively. During training, a set of learnable bias parameters b are first used to bias the output u′ of the last fully connected layer in the importance predictor, thus obtaining the intermediate variables corresponding to each discrete ρ. Then, Gumbel noise is introduced during the binarization of the threshold τ, i.e., the Gumbel-Sigmoid technique is used to soften the sampling process, as shown in the following formula: M = Gumbel - Sigmoid(u′+b,τ) After training, the importance predictor can binarize the score map according to the feature aggregation ratio ρ, and the bias layer will be discarded. Step 3: The multi-branch aggregation module splits the features into two groups of sub-features based on the binary mask, namely the main path features and the side path features. Step 4: Based on the specified task, select the corresponding bypass and send the main path features and bypass features to the main path and the selected bypass, respectively. Step 5: Recombine the two sets of sub-features calculated for the main path and the bypass path according to the position and size of the original features, and complete the subsequent decoding to output a complete image.

2. The method according to claim 1, characterized in that, Step 1 includes: considering the correlation of semantic features among machine vision tasks, optimizing the basic image coding model and its main path using an image coding training method optimized for perceptual quality loss, so that the basic image coding model can extract rich semantic information from the original image during the encoding process, realize feature reuse among multiple tasks, and reconstruct an image with excellent perceptual quality during decoding; after obtaining the perceptual quality optimized basic image coding model, adding side paths for specific tasks alongside the main path to form a multi-branch aggregation module, and fine-tuning some parameters of the loss function for specific tasks to obtain a multi-task image coding model that supports multi-task target decoding.

3. The method according to claim 2, characterized in that, In step 1, the multi-task image coding model includes one basic image coding model and six multi-branch aggregation modules. The basic image coding model includes one encoder, one decoder, and one entropy model. The encoder encodes the image into a binary bitstream, the entropy model predicts the entropy of the binary bitstream, and the decoder decodes the binary bitstream into an image. The multi-task image coding model is built based on the basic image coding model. The encoder includes three multi-branch aggregation modules, and the decoder includes three multi-branch aggregation modules. The multi-branch aggregation modules enable the encoder and decoder to support multi-task encoding and decoding. Each multi-branch aggregation module contains one main path, three side paths, and three importance predictors. The main path supports high-fidelity image reconstruction, while the three side paths support low-distortion image reconstruction, classification image reconstruction, and semantic segmentation image reconstruction, respectively. The importance predictors are used to assign semantic features to the main path and side paths. Both the main path and the branch path contain two fully connected layers and one activation layer, while the importance predictor contains three fully connected layers and two activation layers. The main path, the bypass path, and the importance predictor are denoted as φ. main φ side φ pred .

4. The method according to claim 3, characterized in that, In step 1, the training phase includes: Step 1-1, Stage 1: Using variational autoencoders and generative adversarial networks, combined with perceptual quality loss, rate-distortion optimization is performed on the basic image coding model and its main path to obtain a basic image coding model for multi-task optimization. Step 1-2, Stage 2: Add bypasses according to the preset tasks to form a multi-branch aggregation module, and use the loss function of each task to optimize the corresponding bypasses and importance predictors respectively.

5. The method according to claim 4, characterized in that, Step 1-1 includes: during the computation of each training batch Epoch, optimizing the base image coding model according to the following loss function: Among them, the proportional loss function Used to optimize the importance predictor and generator loss function. The generator G(·) and discriminator loss function are used to optimize the generative adversarial network. Discriminator D(·) used to optimize generative adversarial networks, image coding loss function The encoder E(·), generator G(·), and entropy model P(·) are used to optimize the basic image coding model; x, y、 These are the input image, the output reconstructed image, the latent space features of the variational autoencoder (VAE) before quantization, and the latent space features of the variational autoencoder (VAE) after quantization. This indicates that when x follows a probability distribution p x Expectations at that time; Indicates when Follows probability distribution p y The expected value at time; S represents the number of stages in the basic image coding model that use multi-branch aggregation, M (s) The mask M and H represent the importance predictor generated in the s-th stage of the multi-task image coding model. (s) and W (s) represents the spatial height and width of the s-th stage of the multi-task image coding model, respectively, and ρ is the feature aggregation ratio; Entropy model pairs in image coding models Predicted bitrate size Representing x and The mean square error between them; It is a learnable image patch-aware similarity; the bitrate loss weights are used when the encoding quality is q. Generator loss weight λ G Perceived quality loss weight λ perc , proportional loss weight λ ratio The hyperparameters used to adjust the loss weights.

6. The method according to claim 5, characterized in that, Steps 1-2 include: during the calculation of each training batch Epoch, optimizing the multi-branch aggregation module according to the following optimization objectives: Among them, the bypass parameter set And importance predictor parameter set These are bypass φ side Optimal parameters and importance predictor φ pred The optimal parameters, the task loss function This is the loss function for the machine vision task selected in the current training batch, with task loss weights λ. task The hyperparameters used to adjust the loss weights.

7. The method according to claim 6, characterized in that, In step 1, the loss function for the specific task undergoes partial parameter fine-tuning, specifically including: in step 1-1, all model parameters are updated during training; in step 1-2, only φ is adjusted. side and φ pred The parameters are updated during training.

8. The method according to claim 7, characterized in that, In step 3, the multi-branch aggregation module will divide the features sent into the multi-branch aggregation module into two parts according to the optimal binary mask M generated in step 2. The features corresponding to 1 in the optimal binary mask M are the main road features, and the features corresponding to 0 are the bypass features. There is no overlap between the main road features and the bypass features. In step 4, based on the tasks that the decoded image needs to support, a bypass φ optimized for the task is selected in the multi-branch aggregation module. side The main road features and bypass features extracted in step 3 are then fed into the main road φ. main and bypass φ side ; In step 5, after the main path features and bypass features are calculated in the multi-branch aggregation module, they will be re-aggregated into new features with the same size and position as the original features based on the spatial size and feature position before splitting. The new features output by the multi-branch aggregation module will be sent to the subsequent multi-branch aggregation module, and steps 2 to 5 will be repeated until the decoding process is completed and a complete image is output.

9. A multi-task image coding apparatus based on multi-branch aggregation implemented according to any one of claims 1 to 8, characterized in that, This includes a multi-branch aggregation module, an importance predictor, and a corresponding training strategy; The multi-branch aggregation module training strategy is used to perform the following steps: Step 1, Training Phase: To address the perceptual quality loss and visual task loss, the correlation of semantic features between machine vision tasks such as image reconstruction, image classification, and image segmentation is used to train the basic image coding model, multi-branch aggregation module, and importance predictor to form a multi-task image coding model. The importance predictor is used to perform the following steps: Step 2, Testing Phase: Based on the specified task and feature aggregation ratio, the features are fed into the corresponding importance predictor. The predictor calculates and outputs a binary mask based on the importance of different features to different tasks. The multi-branch aggregation module is used to perform the following steps: Step 3: The multi-branch aggregation module splits the features into two groups of sub-features based on the binary mask, namely the main path features and the side path features. Step 4: Based on the specified task, select the corresponding bypass and send the main path features and bypass features to the main path and the selected bypass, respectively. Step 5: Recombine the two sets of sub-features calculated for the main path and the bypass path according to the position and size of the original features, and complete the subsequent decoding to output a complete image.

Images