Feature disassembling and compressing method and device, equipment and storage medium

Abstract

The invention relates to the technical field of machine learning, and discloses a feature disassembly and compression method and device, equipment and a storage medium, and the method comprises the steps: obtaining an initial feature of to-be-predicted image data, determining an expert activation weight corresponding to a downstream prediction task based on a hybrid expert network, the initial features are processed through a plurality of expert networks in the hybrid expert network to generate disassembled features, the plurality of expert networks are networks constrained through a low-rank matrix, the disassembled features are quantized to generate quantized features, entropy coding is carried out on the quantized features through a hyper-prior network, and compressed features are obtained. According to the method, the expert activation weight is dynamically adjusted along with task requirements, different task scenes are adapted, initial features are disassembled based on a plurality of expert networks, task irrelevant information is explicitly stripped, task key features are reserved, redundant data volume is reduced, quantized coding is performed on disassembled feature input, code stream self-adaption is realized, and bandwidth occupation is reduced.

Description

Technical Field

[0001] This application relates to the field of machine learning technology, and in particular to a feature decomposition and compression method, apparatus, device and storage medium. Background Technology

[0002] With the popularization of the Internet of Things (IoT), the synergy between edge computing and cloud computing has become a core direction for the evolution of intelligent systems. In the edge-cloud collaborative architecture, edge devices reduce the load on the cloud through local preprocessing, while the cloud utilizes its strong computing power to optimize models and distribute updates.

[0003] However, in multi-task scenarios, existing technologies face significant challenges: on the one hand, different downstream prediction tasks have different feature requirements, and the transmission of general features can easily lead to bandwidth redundancy; on the other hand, traditional compression methods are mostly designed for single tasks and lack end-side adaptive reconstruction capabilities. Summary of the Invention

[0004] The main objective of this application is to provide a feature decomposition and compression method, apparatus, device, and storage medium, which aims to solve the technical problem that existing technologies are unable to efficiently decompose and compress features to reduce bandwidth usage and adapt to different task requirements.

[0005] To achieve the above objectives, this application proposes a feature decomposition and compression method, the method comprising:

[0006] Obtain initial features of the image data to be predicted, wherein the image data to be predicted is associated with at least one downstream prediction task;

[0007] Determine the expert activation weights for downstream prediction tasks based on a hybrid expert network;

[0008] The initial features are processed by multiple expert networks in the hybrid expert network to generate decomposed features, wherein the multiple expert networks are networks constrained by a low-rank matrix;

[0009] The decomposed features are quantized to generate quantized features, and then entropy-encoded using a super-prior network to obtain compressed features.

[0010] In one embodiment, the hybrid expert network includes a routing module, the routing module including at least one convolutional layer, and the step of determining the expert activation weights corresponding to the downstream prediction task based on the hybrid expert network includes:

[0011] The initial features are processed spatially using convolutional layers to obtain intermediate features.

[0012] The intermediate features are subjected to layer normalization to generate normalized features;

[0013] The normalized features are nonlinearly transformed by an activation function to obtain the expert activation weights for the downstream prediction task.

[0014] In one embodiment, the hybrid expert network further includes a feature extraction module, which comprises multiple expert networks, each consisting of an autoencoder. The step of processing the initial features through the multiple expert networks in the hybrid expert network to generate decomposed features includes:

[0015] The first convolutional layer of the autoencoder maps the initial features in the first feature space to the second feature space to obtain low-dimensional features, where the dimension of the second feature space is much smaller than the dimension of the first feature space.

[0016] The low-dimensional features are mapped back to the first feature space through the second convolutional layer of the autoencoder to generate expert feature components.

[0017] The feature components are weighted and combined based on the expert activation weights to obtain the decomposed features.

[0018] In one embodiment, the step of weighting and combining the feature components based on the expert activation weights to obtain the decomposed features includes:

[0019] When there are multiple downstream prediction tasks, the average value of the expert activation weights is calculated, and the average value is used as the final expert activation weights.

[0020] The weighted feature components are determined by multiplying the feature components and their corresponding expert activation weights element by element.

[0021] Summing is performed on all weighted feature components to obtain the decomposed features.

[0022] In one embodiment, after the steps of quantizing the decomposed features to generate quantized features and entropy encoding the quantized features using a super-prior network to obtain compressed features, the method further includes:

[0023] The compressed features are input into the corresponding task decoder to obtain the prediction results of the downstream prediction task;

[0024] Based on the prediction results, the loss value of the downstream prediction task is determined by a loss function;

[0025] Determine the bitstream loss value of the compressed features;

[0026] The prediction result is jointly optimized based on the loss value and the bitstream loss value to obtain the optimized prediction result.

[0027] In one embodiment, the step of determining the bitstream loss value of the compressed features includes:

[0028] The compressed features are quantized to obtain quantized features, and the probability distribution of the quantized features is determined.

[0029] The difference between the probability distribution and the Gaussian distribution is represented by the negative log-likelihood function, and this difference is used as the bitstream loss value of the compressed features.

[0030] In one embodiment, prior to the step of obtaining the initial features of the image data to be predicted, the method further includes:

[0031] Obtain a labeled image dataset containing multiple downstream prediction tasks;

[0032] Based on the labeled image dataset, a large visual model is constructed using the visual base model trained in historical downstream prediction tasks.

[0033] Accordingly, the step of obtaining the initial features of the image data to be predicted includes:

[0034] The image data to be predicted is input into the large visual model to obtain initial features.

[0035] Furthermore, to achieve the above objectives, this application also proposes a feature disassembly and compression device, which includes:

[0036] A general feature acquisition module is used to acquire initial features of the image data to be predicted, wherein the image data to be predicted is associated with at least one downstream prediction task;

[0037] The expert weight determination module is used to determine the expert activation weights corresponding to downstream prediction tasks based on a hybrid expert network.

[0038] The feature decomposition module is used to process the initial features through multiple expert networks in the hybrid expert network to generate decomposed features, wherein the multiple expert networks are networks constrained by a low-rank matrix;

[0039] The feature compression module is used to quantize the decomposed features to generate quantized features, and then entropy-encode the quantized features through a super-prior network to obtain compressed features.

[0040] In addition, to achieve the above objectives, this application also proposes a feature disassembly and compression device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the feature disassembly and compression method as described above.

[0041] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the feature decomposition and compression method described above.

[0042] The technical solution proposed in this application includes: acquiring initial features of image data to be predicted, wherein the image data to be predicted is associated with at least one downstream prediction task; determining expert activation weights corresponding to the downstream prediction task based on a hybrid expert network; processing the initial features through multiple expert networks in the hybrid expert network to generate decomposed features, wherein the multiple expert networks are networks constrained by a low-rank matrix; quantizing the decomposed features to generate quantized features; and entropy encoding the quantized features through a super-prior network to obtain compressed features. This application generates expert activation weights corresponding to the downstream prediction task through a hybrid expert network, and decomposes the initial features based on multiple expert networks, explicitly removing task-irrelevant information, retaining key task features, and reducing redundant data; quantizing and encoding the input of the decomposed features to achieve adaptive bitstream and reduce bandwidth usage; and dynamically adjusting the expert weight combination according to task requirements to adapt to different task scenarios. Attached Figure Description

[0043] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0044] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 A flowchart illustrating an embodiment of the feature disassembly and compression method of this application;

[0046] Figure 2 A flowchart illustrating Embodiment 2 of the feature disassembly and compression method of this application;

[0047] Figure 3 A flowchart illustrating Embodiment 3 of the feature disassembly and compression method of this application;

[0048] Figure 4 This is a schematic diagram of the training process for the feature decomposition and compression method of this application;

[0049] Figure 5 This is a schematic diagram of the reasoning process for the feature decomposition and compression method of this application;

[0050] Figure 6 This is a schematic diagram of the modular structure of the feature disassembly and compression device in an embodiment of this application;

[0051] Figure 7 This is a schematic diagram of the device structure of the hardware operating environment involved in the feature disassembly and compression method in the embodiments of this application.

[0052] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0053] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0054] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0055] In multi-task scenarios, existing technologies face significant challenges: on the one hand, different downstream prediction tasks have different feature requirements, and the transmission of general features can easily lead to bandwidth redundancy; on the other hand, traditional compression methods are mostly designed for single tasks and lack end-side adaptive reconstruction capabilities.

[0056] Therefore, in order to overcome the above-mentioned defects, this application provides a solution that generates expert activation weights corresponding to downstream prediction tasks through a hybrid expert network, and decomposes the initial features based on multiple expert networks, explicitly removing task-irrelevant information, retaining key task features, and reducing redundant data; the decomposed feature input is quantized and encoded to achieve adaptive bitstream and reduce bandwidth usage; the expert weight combination is dynamically adjusted according to task requirements to adapt to different task scenarios.

[0057] It should be noted that the executing entity of each embodiment of this application can be a computing service system with data processing, network communication, and program execution functions, such as an electronic system or a feature decomposition and compression system capable of realizing the above functions. The following description uses a feature decomposition and compression system (hereinafter referred to as "the system") as an example to illustrate the following embodiments.

[0058] Based on this, embodiments of this application provide a feature decomposition and compression method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the feature disassembly and compression method of this application.

[0059] In this embodiment, the feature decomposition and compression method includes steps S10 to S40:

[0060] Step S10: Obtain the initial features of the image data to be predicted, wherein the image data to be predicted is associated with at least one downstream prediction task.

[0061] With the development of artificial intelligence and computer vision technologies, large-scale pre-trained visual models, leveraging the rich representations learned from massive amounts of data, have significantly improved performance in downstream prediction tasks such as image recognition and semantic segmentation. However, the increase in model size leads to a surge in computing resource requirements and deployment costs, making it difficult to deploy on resource-constrained edge devices (such as smartphones and IoT devices). Meanwhile, terminal application scenarios are highly diverse, ranging from intelligent monitoring to industrial inspection, with significantly different requirements for model processing capabilities and resource allocation, making it difficult to meet customized needs by directly deploying existing models.

[0062] Against this backdrop, edge-cloud collaboration has become a key solution: large models in the cloud handle computationally intensive tasks, while edge devices deploy lightweight task decoding networks, enabling customized processing through cloud-based feature transmission. However, in multi-task scenarios, the transmission of feature data from the cloud to edge devices faces bandwidth and latency bottlenecks, necessitating efficient semantic decomposition and compression technologies. This involves extracting task-related semantic information from high-dimensional features and encoding it in a compact form to reduce transmission bandwidth. Effective compression not only reduces network load but also adaptively eliminates redundant information, such as removing irrelevant features for specific tasks, thereby improving transmission efficiency and task adaptability.

[0063] To address the aforementioned technical issues, this application proposes a feature decomposition and compression method based on a hybrid expert network comprising multiple low-rank expert networks. This method improves upon existing hybrid expert techniques and adapts them to the task framework of feature decomposition and compression. Specifically, it proposes multiple linear low-rank expert networks, which are decomposed and recombined by different experts to obtain the final features. The output of each expert is constrained in a low-rank space, limiting the information content of that part. The final output features only contain task-specific information required by the user, resulting in a smaller bitrate after compression while maintaining performance for the specific task.

[0064] In this step, it should be noted that the image data to be predicted refers to the original image that needs to be used for task inference; the downstream prediction task refers to a specific task completed based on image features, such as semantic segmentation, human body parsing, saliency detection, and surface normal prediction tasks.

[0065] It should be understood that pre-trained large-scale visual models can be used to extract features from the input image data to be predicted, obtaining initial features. During the input stage, it is necessary to ensure that the image data has been labeled with at least one downstream prediction task label for subsequent task adaptation. For example, if the image needs to simultaneously support semantic segmentation and surface normal prediction, then the spatial and semantic information related to these two tasks must be preserved during feature extraction. The extracted initial features can be multi-scale (such as the output of a feature pyramid network structure) or a single high-dimensional tensor (such as a feature vector after global average pooling).

[0066] As one implementation, before step S10 above in this embodiment, the method may further include: obtaining a labeled image dataset containing multiple downstream prediction tasks; and constructing a large visual model based on the labeled image dataset using the visual base model trained in historical downstream prediction tasks.

[0067] Accordingly, the step of obtaining the initial features of the image data to be predicted includes:

[0068] The image data to be predicted is input into the large visual model to obtain initial features.

[0069] In the specific implementation, before obtaining the initial features of the image data to be predicted, it is necessary to collect an image dataset containing annotations for multiple downstream prediction tasks. This dataset needs to cover annotation information for different task types (such as pixel-level labels for semantic segmentation, geometric annotations for surface normal prediction, etc.) to provide multi-task supervision signals for model training. Secondly, based on this labeled dataset, transfer learning is performed on a visual base model pre-trained in historical downstream prediction tasks (such as ViT (Vision Transformer)-Base or residual networks pre-trained on a large visualization database) to construct a large visual model adapted to multiple tasks. Specifically, in the initial feature acquisition step, the image data to be predicted is input into the constructed large visual model. This model extracts high-dimensional semantic features of the image through a multi-layer Transformer encoder or convolutional structure to generate initial feature vectors suitable for subsequent hybrid expert network processing.

[0070] Step S20: Determine the expert activation weights corresponding to the downstream prediction task based on the hybrid expert network.

[0071] It should be noted that a Mixture of Experts (MoE) is a model composed of multiple sub-networks (experts) that dynamically allocates tasks to different experts through a routing mechanism; the expert activation weight refers to the degree of contribution of each expert to the current task.

[0072] When given general visual features, MoE calculates the contributions of different expert networks to the current task and generates a weight vector. For example, if the task combination is semantic segmentation and surface normal prediction, the routing module calculates the weights for each task separately and then averages them to obtain the fused weights, which are used for subsequent feature decomposition.

[0073] In one implementation, the hybrid expert network includes a routing module, which includes at least one convolutional layer. In this embodiment, step S20 may include: processing the initial features spatially using the convolutional layer to obtain intermediate features; performing layer normalization on the intermediate features to generate normalized features; and performing a nonlinear transformation on the normalized features using an activation function to obtain expert activation weights for the downstream prediction task.

[0074] In its implementation, the routing module dynamically allocates expert activation weights through a hierarchical processing mechanism: First, the initial features undergo spatial dimension processing via convolutional layers, utilizing local receptive fields to capture spatial correlations and generate intermediate features containing multi-scale contextual information. Then, layer normalization is applied to these intermediate features, independently normalizing each sample's feature channels to eliminate the impact of input scale differences on subsequent activations, resulting in normally distributed features with stable numerical distributions. Finally, a nonlinear transformation is applied to the normalized features using an activation function (such as ReLU), mapping the feature values ​​to non-negative weights. These weights directly reflect the contribution of different expert networks to the current downstream prediction task; for example, semantic segmentation tasks rely more on spatial structure experts, while surface normal prediction tasks rely more on geometric feature experts. This entire process, through a cascaded structure of convolution-layer normalization-activation, ensures that the routing module can dynamically generate expert activation weights that adapt to the task requirements based on the input features.

[0075] Step S30: The initial features are processed by multiple expert networks in the hybrid expert network to generate decomposed features, wherein the multiple expert networks are networks constrained by a low-rank matrix.

[0076] It should be noted that low-rank matrix constraints refer to reducing parameter redundancy by limiting the rank of the matrix (the number of non-zero singular values), thereby reducing model complexity. MoE includes multiple low-rank expert networks, each of which adopts an autoencoder structure and limits the feature representation capability through low-rank matrix constraints. That is, each expert network maps the input features to a low-rank space (e.g., reducing the dimension from 640 to 32) through convolutional layers, and then maps them back to the original feature dimension, thereby reducing redundant information.

[0077] Step S40: Quantize the decomposed features to generate quantized features, and entropy encode the quantized features using a super-prior network to obtain compressed features.

[0078] It should be noted that quantization refers to converting continuous feature values ​​into discrete symbols to reduce data precision; super-prior networks refer to models that model the probability distribution of features through additional information (super-prior); and entropy coding refers to a coding method that uses probability distribution characteristics to compress data length.

[0079] After decomposition, the features first undergo uniform quantization, converting continuous values ​​into discrete symbols to generate quantized features. Then, a hyperprior network is used to probabilistically model the quantized features. The hyperprior network extracts the spatial context information of the quantized features through an encoder to generate hyperprior features, which are then predicted by a decoder to predict the probability distribution of the quantized features. Finally, entropy coding (such as arithmetic coding) converts the quantized features into a compressed bitstream, obtaining compressed features and reducing the amount of data required for storage and transmission.

[0080] It should be understood that the feature decomposition and compression method of this application can be widely applied to bandwidth-sensitive real-world scenarios such as smart cities, industrial IoT, and autonomous driving. In smart cities, edge cameras need to perform multiple tasks simultaneously, such as people counting and abnormal behavior detection. This application can compress the amount of transmitted data by dynamically decomposing semantic segmentation and target detection features, ensuring the smoothness of real-time monitoring. In industrial IoT scenarios, robotic arm vision guidance needs to take into account both part positioning and defect detection tasks. This application reduces the bandwidth of cloud feature distribution through task-aware quantization encoding, while ensuring that the edge reconstruction accuracy meets assembly requirements. In autonomous driving scenarios, vehicle-road cooperative systems need to simultaneously process road sign recognition and obstacle prediction tasks. This application optimizes the code stream allocation through a priori network, achieving lossless collaborative transmission of multi-task features under preset bandwidth limitations, significantly improving the timeliness of edge-cloud collaborative decision-making.

[0081] This embodiment generates expert activation weights corresponding to downstream prediction tasks through a hybrid expert network. It decomposes the initial features based on multiple expert networks, explicitly removing task-irrelevant information while retaining key task features and reducing redundant data. The decomposed feature inputs are quantized and encoded to achieve adaptive bitstream and reduce bandwidth consumption. The expert weight combination is dynamically adjusted according to task requirements to adapt to different task scenarios. Furthermore, convolutional layers are first used to process the spatial dimension of the initial features, effectively extracting local features and obtaining intermediate features. Layer normalization stabilizes the distribution of intermediate features, accelerating model convergence and generating normalized features. These normalized features are then subjected to a nonlinear transformation using an activation function, giving the model nonlinear expressive capabilities and yielding expert activation weights that accurately adapt to downstream prediction tasks. An image dataset with multi-task annotations is acquired, and a large visual model is constructed using models trained on historical tasks, improving the accuracy of initial feature extraction.

[0082] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. On this basis, refer to Figure 2 Step S30 may include steps S301 to S303:

[0083] Step S301: The initial features in the first feature space are mapped to the second feature space through the first convolutional layer of the autoencoder to obtain low-dimensional features, wherein the dimension of the second feature space is much smaller than the dimension of the first feature space.

[0084] It should be understood that after the initial features are input into the autoencoder, they first pass through the first convolutional layer for spatial dimension compression. That is, through local filtering operations, the high-dimensional initial features (such as 640 dimensions) are mapped to a low-dimensional feature space (such as 32 dimensions) so as to remove redundant information in the input by learning local feature patterns and generating a more compact feature representation.

[0085] Step S302: The low-dimensional features are mapped back to the first feature space through the second convolutional layer of the autoencoder to generate expert feature components.

[0086] When low-dimensional features undergo spatial dimensionality expansion in the second convolutional layer, a learnable convolutional kernel is first used to remap the features from the compressed low-dimensional space to the same dimension as the initial features. This process employs a reverse mapping mechanism, learning the reconstruction relationship from low-dimensional to high-dimensional through the decoding structure of the autoencoder. This ensures that the expanded features not only retain key task-related information (such as boundary features required for semantic segmentation or geometric structures required for surface normal prediction), but also enhance the feature representations that are highly relevant to the current downstream prediction task through parameterized transformations.

[0087] Each feature component generated by the expert network possesses a dual nature: on the one hand, it preserves a concise representation of the original features through compression in the first convolutional layer, removing redundant information; on the other hand, the dimensionality-increasing operation in the second convolutional layer maintains semantic consistency through reconstruction constraints of the autoencoder (such as mean squared error loss), ensuring that the features do not deviate from the task objective while restoring dimensionality. This design provides interpretable feature units for subsequent weighted combinations, meaning that each expert's output represents knowledge of a specific subtask while ensuring compatibility with features from other experts through dimensional alignment.

[0088] The core purpose of dimensionality upscaling is threefold: First, to remap the compressed semantic information back to the original feature space, making the output features directly compatible with subsequent task decoders (such as single-convolutional layer prediction heads); second, to enhance task-related features through parameterized transformations (such as learning the weights of convolutional kernels), for example, by enhancing spatial continuity or geometric details; and finally, to add the initial features to the upscaled features using a residual connection mechanism (if skip connections exist), supplementing information that may be lost during compression while maintaining the original feature structure, thereby avoiding feature degradation.

[0089] Step S303: Based on the expert activation weights, the feature components are weighted and combined to obtain the decomposed features.

[0090] Based on the expert activation weights generated by the routing module, the feature components output by each expert network are linearly weighted and combined, enabling MoE to dynamically fuse multi-expert features and generate decomposed features that only contain task-related information, while removing irrelevant content and improving the task adaptability of the features.

[0091] In one implementation, step S303 in this embodiment may include: when there are multiple downstream prediction tasks, averaging the expert activation weights and using the average value as the final expert activation weights; multiplying the feature components and the corresponding expert activation weights element by element to determine the weighted feature components; and summing all the weighted feature components to obtain the decomposed features.

[0092] In its implementation, when multiple downstream prediction tasks exist, MoE needs to dynamically coordinate the allocation of expert resources across different tasks. First, the routing module generates a set of expert activation weights independently for each task. Then, by averaging, the multiple sets of weights are merged into a unified weight vector, ensuring that the contributions of each expert are balanced in a multi-task scenario. This averaging process avoids a single task from excessively dominating expert selection and improves computational efficiency through weight sharing.

[0093] After obtaining the fused expert activation weights, the system performs element-wise multiplication of the feature components generated by each expert network with their corresponding weights. This process can adjust the strength of the feature components through weight scaling, for example, assigning higher weights to expert features highly relevant to the current task while suppressing the outputs of irrelevant experts. The element-wise multiplication mechanism ensures the structural consistency of the feature space, allowing the weighted features to maintain their original spatial arrangement and channel relationships. Finally, all weighted feature components are summed, i.e., the dispersed expert knowledge is integrated into a unified feature representation through linear combination. For example, the outputs of spatial structure experts and geometric feature experts are superimposed to generate decomposed features that simultaneously contain semantic and geometric information.

[0094] This embodiment uses the first convolutional layer of the autoencoder to force a narrow channel, creating an information bottleneck and filtering out redundant information irrelevant to downstream prediction tasks. In the low-dimensional space, feature computation is reduced, and different semantic components are more easily separated. The second convolutional layer of the autoencoder remaps the compressed semantic information back to the original feature space, maintaining compatibility with subsequent modules, preserving the original feature structure, and avoiding information loss. Furthermore, when facing multiple downstream prediction tasks, averaging the expert activation weights comprehensively considers the characteristics of each task, resulting in a more universal final weight. This weight is then multiplied element-wise with the feature components to determine the weighted components, and finally summed to obtain the decomposed features for fusing multi-task information, enhancing the comprehensiveness and effectiveness of feature representation.

[0095] Based on the first embodiment of this application, in the third embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 After step S40, steps S411 to S414 may also be included:

[0096] Step S411: Input the compressed features into the corresponding task decoder to obtain the prediction result of the downstream prediction task.

[0097] It should be understood that after the compressed features undergo the encoding process, they need to be converted into specific prediction outputs by a task decoder. The task decoder consists of lightweight convolutional layers or fully connected layers, and its structure is closely related to the type of downstream prediction task. For example, a semantic segmentation task decoder restores the spatial resolution of the features through upsampling operations, generating pixel-level classification results; a surface normal prediction task decoder outputs a three-dimensional direction vector through a regression layer.

[0098] Step S412: Based on the prediction results, determine the loss value of the downstream prediction task using a loss function.

[0099] Select the appropriate loss function based on the task type (e.g., cross-entropy loss for classification tasks, mean squared error loss for regression tasks) and calculate the difference between the predicted result and the true label.

[0100] Step S413: Determine the bitstream loss value of the compressed features.

[0101] It should be noted that the bitstream loss value measures the degree of information loss during feature compression and can be calculated by comparing the feature distribution or reconstruction error before and after compression. For example, in the entropy coding stage, a super-prior network generates a compressed bitstream, and the bitstream loss can be quantized by the quantization noise or the negative log-likelihood function of the probability distribution. This value is directly related to compression efficiency and feature fidelity, ensuring that sufficient information is retained to support task prediction while reducing storage or transmission costs.

[0102] In one implementation, step S413 in this embodiment may include: quantizing the compressed features to obtain quantized features and determining the probability distribution of the quantized features; representing the difference between the probability distribution and the Gaussian distribution using a negative log-likelihood function, and using the difference as the bitstream loss value of the compressed features.

[0103] In practical implementation, the compressed features first undergo secondary quantization, mapping them from a continuous value space to a discrete symbol space to generate quantized features. Then, a priori network or statistical model (such as histogram estimation) is used to analyze the probability distribution of the quantized features, obtaining the probability value of each discrete symbol. This distribution reflects the degree of information redundancy of the features in the compressed domain. Next, the difference between this probability distribution and the theoretical Gaussian distribution is quantified using the negative log-likelihood function. The Gaussian distribution is often considered an ideal reference due to its entropy-maximizing characteristic. The negative log-likelihood function measures the degree to which the actual distribution deviates from the ideal distribution by calculating the sum of the log probability differences between the observed distribution and the Gaussian distribution for each symbol. Finally, this difference value is defined as the bitstream loss value. A larger value indicates higher information loss or redundancy in the compressed features, thus providing a key indicator for optimizing compression efficiency and feature fidelity.

[0104] Step S414: Perform joint optimization on the prediction result based on the loss value and the bitstream loss value to obtain the optimized prediction result.

[0105] The joint optimization phase balances task prediction performance and compression efficiency through a multi-task learning framework. Specifically, the system weights and sums task loss values ​​(e.g., classification error) and bitstream loss values ​​(e.g., compression noise) to form the total loss function. The backpropagation algorithm simultaneously adjusts the parameters of the large visual model, the MoE (Modal of the Environment), and the super-prior network, enabling the model to improve prediction accuracy while optimizing the bitrate-distortion performance of compressed features. For example, during training, if excessive bitstream loss leads to severe loss of feature information, the optimizer reduces quantization intensity to retain more details; if task loss dominates the total loss, priority is given to improving the task adaptability of features. Finally, iterative optimization yields an optimized result that balances compression efficiency and prediction accuracy.

[0106] This embodiment inputs the compressed features into the task decoder to quickly obtain preliminary results for the downstream prediction task. The prediction loss value is calculated using a loss function, and the bitstream loss value is determined simultaneously. Combining these two methods optimizes the results in terms of both prediction accuracy and feature compression efficiency, effectively improving model performance and yielding better prediction results. Furthermore, by quantizing the compressed features and determining the probability distribution, and using the negative log-likelihood to measure the difference from the Gaussian distribution as the bitstream loss value, the compression loss can be accurately evaluated, improving the quality of feature compression.

[0107] For ease of understanding, please refer to Figure 4-5 ( Figure 4 This is a schematic diagram of the training process for the feature decomposition and compression method of this application; Figure 5 The following diagram illustrates the reasoning process of the feature decomposition and compression method of this application, but it is not intended to limit the feature decomposition and compression method of this application.

[0108] A feature decomposition and compression method based on a hybrid low-rank expert strategy is proposed. Given image data and several downstream prediction tasks, the image is first used to extract features through an existing large-scale visual model. These features are then input into a MoE (Modal of Image) for feature decomposition. Finally, the decomposed features are compressed for subsequent transmission and storage. In this process, a feature decomposition and compression network is trained. Four downstream prediction tasks are selected: semantic segmentation, human body parsing, saliency detection, and surface normal prediction (the selection of downstream prediction tasks can be based on requirements; the example in this embodiment is for illustrative purposes only).

[0109] Includes the following steps:

[0110] 1) Collect image data with the labels of the aforementioned downstream prediction tasks to construct a dataset;

[0111] 2) Construct and load the pre-trained large visual model. Specifically, use the ViT (Vision Transformer)-Base trained in the downstream prediction task mentioned above for construction.

[0112] 3) Input image data into a large visual model for processing to obtain a general visual feature (initial feature), which is used to complete the downstream prediction task mentioned above;

[0113] 4) Randomly sample a specific combination of task requirements from the above 4 tasks, such as {semantic segmentation, surface normal prediction}, and input the features obtained in step 3) into a feature decomposition network based on hybrid low-rank experts. Remove information irrelevant to the specific task and retain information relevant to the specific task to obtain the decomposed features.

[0114] 5) Input the decomposed features obtained in step 4) into the feature compression network to obtain compressed features, and input them into the decoder of the specific task to output the prediction results of the downstream prediction task.

[0115] 6) Model inference: Input image data and specific task requirements, the model outputs compressed features for subsequent transmission and storage.

[0116] To make the above features and advantages of this application more apparent and understandable, specific embodiments are described below, along with... Figure 4 and Figure 5A detailed explanation is provided below. It should be noted that the hyperparameter settings given in the following embodiments are merely preferred implementations and are not intended to limit the scope. Those skilled in the art can select the appropriate number and set different layers according to actual needs, which should be understood. Specific details are as follows:

[0117] Step 1: Collect and construct a training dataset consisting of images with K downstream prediction task labels;

[0118] Step 2: Build and load the pre-trained large visual model;

[0119] Step 3: Build a feature decomposition and compression network.

[0120] Specifically, the feature decomposition network consists of two modules: a routing module and a feature extraction module. The routing module comprises two convolutional layers followed by a normalization layer and an activation function layer, outputting the weights of different experts (i.e., expert activation weights). In the implementation, a routing module is set up for each downstream prediction task to model the expert combination required for that task. The feature extraction module consists of 15 low-rank expert networks. Each expert network E_i consists of an autoencoder, which maps the features to C dimensions through a convolutional layer and then maps them back to the original feature dimension D = 640 (C < 0.05). <D)。

[0121] The feature compression network adopts a compression network structure based on super-prior, taking the features to be compressed (i.e., the decomposed features) as input and outputting the quantized and compressed features as output.

[0122] Step 4: Randomly sample n downstream prediction tasks (n<=K) and image data, first input them into the large visual model to obtain initial features, then input them again into the feature decomposition network, obtain the expert activation weights for each of the n downstream prediction tasks through the corresponding routing modules, and then calculate the average to obtain the final expert activation weight w, and then compare it with the feature components F obtained from different expert processing. i Linear combination yields the decomposed features:

[0123] F out =∑ i w i *F i (1)

[0124] Among them, F i and w i Let F be the output feature and activation weight corresponding to the i-th expert. out These are the features after decomposition for the n downstream prediction tasks.

[0125] Step 5: Input the decomposed features obtained in Step 4 into a feature compression network to obtain compressed features, and then input these compressed features into a task-specific decoder to output the prediction results for the downstream prediction task. The decoder can be selected based on existing work; here, a simple single convolutional layer is used to predict the final result.

[0126]

[0127] in, For the final predicted output of task t, conv t is the convolutional layer decoder for task t.

[0128] Step 6: Calculate the loss of each of the n downstream prediction tasks, and at the same time calculate the bitstream loss of feature compression, and perform joint optimization.

[0129] Specifically, for semantic segmentation, human body parsing, and saliency prediction tasks, pixel-level cross-entropy loss is used for optimization; for surface normal prediction tasks, L1 distance loss is used for optimization. Here, this is uniformly expressed in the following form: L t The loss function used to optimize task t:

[0130]

[0131] Feature compression uses per-pixel bitstream loss to measure the bitstream size required to encode feature F. In actual computation, compression is used to obtain the original quantized feature F of vector F. q The negative log-likelihood function between the probability distribution p and the Gaussian distribution q is expressed as:

[0132] L compress =-plogq (4)

[0133] The total optimization loss can be expressed as:

[0134] L = L task +λL compress (5)

[0135] Where λ is the loss weight, which is set to 10.0 in this example.

[0136] Step 7: Inference stage, input image data, the model outputs compressed features, which can be efficiently used by a specific task decoder to predict the final result.

[0137] Compared to baseline methods that directly compress without feature decomposition, this application improves the bitstream of features while enhancing the performance of the required downstream prediction tasks. On the multi-task PASCAL-Context dataset, the average intersection-over-union (mIoU) is used for semantic segmentation and human body parsing tasks; the maximum F-value (maxF) is used for saliency detection; and the average error (mErr) is used for surface normal prediction. For compression performance, the loss per bit per pixel (BPP) is used as the metric. Specific results are shown in Tables 1 and 2. Table 1 assumes that all downstream prediction tasks need to be completed simultaneously, while Table 2 assumes that only one downstream prediction task needs to be completed. Test results are shown in the following table:

[0138] Table 1. Comparison results between this application and the baseline method (user requests all tasks simultaneously)

[0139]

[0140] Table 2 shows the comparison results between this application and the baseline method (each user requests a task individually).

[0141]

[0142]

[0143] As can be seen, this application can achieve improvements in both downstream prediction tasks and bit-per-pixel metrics compared to baseline methods that directly perform feature compression.

[0144] In summary, the innovation of the feature decomposition and compression method in this application lies primarily in the following: by setting up multiple linear low-rank expert networks and performing task-related semantic activation through a task-specific routing module network, new decomposed features are obtained through mixing. These features remove task-irrelevant knowledge, thereby achieving better downstream prediction task performance and compression ratio. Compared to previous feature compression techniques, this application supports task-adaptive bitstream transmission. As the number of requested tasks increases, the required amount of feature information increases, and the bitstream increases accordingly, thus achieving adaptive and efficient compression.

[0145] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the feature disassembly and compression method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0146] This application also provides a feature disassembly and compression device, please refer to... Figure 6 The feature disassembly and compression device includes:

[0147] The general feature acquisition module 10 is used to acquire the initial features of the image data to be predicted, wherein the image data to be predicted is associated with at least one downstream prediction task.

[0148] The expert weight determination module 20 is used to determine the expert activation weights corresponding to the downstream prediction task based on the hybrid expert network.

[0149] The feature decomposition module 30 is used to process the initial features through multiple expert networks in the hybrid expert network to generate decomposed features, wherein the multiple expert networks are networks constrained by a low-rank matrix;

[0150] The feature compression module 40 is used to quantize the decomposed features to generate quantized features, and to entropy encode the quantized features through a super-prior network to obtain compressed features.

[0151] The feature disassembly and compression apparatus provided in this application, employing the feature disassembly and compression methods described in the above embodiments, can solve the technical problem that existing technologies struggle to efficiently disassemble and compress features to reduce bandwidth usage and adapt to different task requirements. Compared with existing technologies, the beneficial effects of the feature disassembly and compression apparatus provided in this application are the same as those of the feature disassembly and compression methods described in the above embodiments, and other technical features in the feature disassembly and compression apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0152] This application provides a feature disassembly and compression device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the feature disassembly and compression method in the first embodiment described above.

[0153] The following is for reference. Figure 7 The diagram illustrates a structural schematic suitable for implementing the feature disassembly and compression device of the embodiments of this application. The feature disassembly and compression device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Desction), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The disassembly and compression device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0154] like Figure 7 As shown, the feature disassembly and compression device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in the read-only memory 1002 or a program loaded from the storage device 1003 into the random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the feature disassembly and compression device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, touch screens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the feature disassembly and compression device to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows feature disassembly and compression devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0155] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0156] The feature disassembly and compression device provided in this application, employing the feature disassembly and compression method described in the above embodiments, can solve the technical problem that existing technologies struggle to efficiently disassemble and compress features to reduce bandwidth usage and adapt to different task requirements. Compared with the prior art, the beneficial effects of the feature disassembly and compression device provided in this application are the same as those of the feature disassembly and compression method described in the above embodiments, and other technical features in this feature disassembly and compression device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0157] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0158] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0159] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the feature disassembly and compression method described in the above embodiments.

[0160] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0161] The aforementioned computer-readable storage medium may be included in the feature disassembly and compression device; or it may exist independently and not assembled into the feature disassembly and compression device.

[0162] The aforementioned computer-readable storage medium carries one or more programs that, when executed by the feature decomposition and compression device, cause the feature decomposition and compression device to: acquire initial features of image data to be predicted, wherein the image data to be predicted is associated with at least one downstream prediction task; determine the expert activation weights corresponding to the downstream prediction task based on a hybrid expert network; process the initial features through multiple expert networks in the hybrid expert network to generate decomposed features, wherein the multiple expert networks are networks constrained by a low-rank matrix; quantize the decomposed features to generate quantized features; and entropy encode the quantized features through a super-prior network to obtain compressed features.

[0163] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, entirely on a remote computer or server, or on an ARM (Advanced RISC Machines) development board. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0164] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0165] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0166] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described feature decomposition and compression method. This solves the technical problem in the prior art of efficiently performing feature decomposition and compression to reduce bandwidth usage and adapt to different task requirements. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the feature decomposition and compression method provided in the above embodiments, and will not be repeated here.

[0167] The above description is only a part of the embodiments of this application and does not limit the scope of protection of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included in the scope of protection of this application.

Claims

1. A method for feature decomposition and compression, characterized in that, The method includes the following steps: Obtain initial features of the image data to be predicted, wherein the image data to be predicted is associated with at least one downstream prediction task; Determine the expert activation weights for downstream prediction tasks based on a hybrid expert network; The initial features are processed by multiple expert networks in the hybrid expert network to generate decomposed features, wherein the multiple expert networks are networks constrained by a low-rank matrix; The decomposed features are quantized to generate quantized features, and then entropy-encoded using a super-prior network to obtain compressed features.

2. The feature decomposition and compression method as described in claim 1, characterized in that, The hybrid expert network includes a routing module, which includes at least one convolutional layer. The step of determining the expert activation weights corresponding to the downstream prediction task based on the hybrid expert network includes: The initial features are processed spatially using convolutional layers to obtain intermediate features. The intermediate features are subjected to layer normalization to generate normalized features; The normalized features are nonlinearly transformed by an activation function to obtain the expert activation weights for the downstream prediction task.

3. The feature decomposition and compression method as described in claim 1, characterized in that, The hybrid expert network further includes a feature extraction module, which comprises multiple expert networks, each consisting of an autoencoder. The step of processing the initial features through the multiple expert networks in the hybrid expert network to generate decomposed features includes: The first convolutional layer of the autoencoder maps the initial features in the first feature space to the second feature space to obtain low-dimensional features, where the dimension of the second feature space is much smaller than the dimension of the first feature space. The low-dimensional features are mapped back to the first feature space through the second convolutional layer of the autoencoder to generate expert feature components. The feature components are weighted and combined based on the expert activation weights to obtain the decomposed features.

4. The feature decomposition and compression method as described in claim 3, characterized in that, The step of weighting and combining the feature components based on the expert activation weights to obtain the decomposed features includes: When there are multiple downstream prediction tasks, the average value of the expert activation weights is calculated, and the average value is used as the final expert activation weights. The weighted feature components are determined by multiplying the feature components and their corresponding expert activation weights element by element. Summing is performed on all weighted feature components to obtain the decomposed features.

5. The feature decomposition and compression method according to any one of claims 1 to 4, characterized in that, After the steps of quantizing the decomposed features to generate quantized features, and entropy encoding the quantized features using a super-prior network to obtain compressed features, the method further includes: The compressed features are input into the corresponding task decoder to obtain the prediction results of the downstream prediction task; Based on the prediction results, the loss value of the downstream prediction task is determined by a loss function; Determine the bitstream loss value of the compressed features; The prediction result is jointly optimized based on the loss value and the bitstream loss value to obtain the optimized prediction result.

6. The feature decomposition and compression method as described in claim 5, characterized in that, The step of determining the bitstream loss value of the compressed features includes: The compressed features are quantized to obtain quantized features, and the probability distribution of the quantized features is determined. The difference between the probability distribution and the Gaussian distribution is represented by the negative log-likelihood function, and this difference is used as the bitstream loss value of the compressed features.

7. The feature decomposition and compression method according to any one of claims 1 to 4, characterized in that, Before the step of obtaining the initial features of the image data to be predicted, the method further includes: Obtain a labeled image dataset containing multiple downstream prediction tasks; Based on the labeled image dataset, a large visual model is constructed using the visual base model trained in historical downstream prediction tasks. Accordingly, the step of obtaining the initial features of the image data to be predicted includes: The image data to be predicted is input into the large visual model to obtain initial features.

8. A feature disassembly and compression device, characterized in that, The feature disassembly and compression device includes: A general feature acquisition module is used to acquire initial features of the image data to be predicted, wherein the image data to be predicted is associated with at least one downstream prediction task; The expert weight determination module is used to determine the expert activation weights corresponding to downstream prediction tasks based on a hybrid expert network. The feature decomposition module is used to process the initial features through multiple expert networks in the hybrid expert network to generate decomposed features, wherein the multiple expert networks are networks constrained by a low-rank matrix; The feature compression module is used to quantize the decomposed features to generate quantized features, and then entropy-encode the quantized features through a super-prior network to obtain compressed features.

9. A feature disassembly and compression device, characterized in that, The feature disassembly and compression device includes: a memory, a processor, and a feature disassembly and compression program stored in the memory and executable on the processor. When the feature disassembly and compression program is executed by the processor, it implements the feature disassembly and compression method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores a feature disassembly and compression program, which, when executed by a processor, implements the feature disassembly and compression method as described in any one of claims 1 to 7.

Images