Feature coding and decoding method and system based on causality and adaptive update

Abstract

The application discloses a feature coding and decoding method and system based on causality and adaptive updating, which are corresponding schemes, in the schemes: prefix causal constraints are applied at the coding side and the reconstruction side at the same time, so that the coding features and the reconstruction features of each token position are not affected by the features of future positions, the causal relationship required by large model training is maintained, and the inconsistency between training and inference information caused by feature compression is reduced; and according to the feature reconstruction error and the actual code rate, the change of the intermediate feature distribution in the training process is monitored, and only the historical data generated before the update trigger time is used for parameter updating and verification, so that the feature coding and decoding performance can be dynamically recovered without introducing future training data; in addition, the prefix causal feature transformation, the adaptive updating and the complete quantization entropy coding link are combined, the communication overhead of the edge cloud cooperative large model training is reduced while the model training effect is maintained.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of feature encoding and distributed training, and in particular to a feature encoding and decoding system based on causality and adaptive updating. Background Technology

[0002] As the scale of model parameters and training data continues to grow, a single computing node often struggles to independently handle the entire training task. Edge-cloud collaborative training divides the model between edge and cloud computing nodes, deploying the front-end network at the edge and the back-end network in the cloud. This allows training data to undergo partial network computation at the edge before intermediate features generated at the model split points are transmitted to the cloud for further computation. This approach protects data privacy at the edge while leveraging the powerful computing capabilities of the cloud, but the model split points introduce new communication interfaces and overhead.

[0003] Intermediate features at model segmentation points typically have long sequence lengths and high feature dimensions, and are repeatedly transmitted during consecutive training steps. When bandwidth between edge clouds is limited and the training scale is large, intermediate feature transmission becomes a major bottleneck affecting training efficiency. To reduce communication overhead, existing solutions usually employ low-precision quantization and sparsification methods to compress intermediate features. However, these methods have limited feature compression strength, and no scheme combining encoding has been proposed yet. Directly using the feature encoder-decoder scheme from inference time for training presents the following two problems.

[0004] Autoregressive large models require that the prediction of the t-th token position can only depend on information from the 1st to the t-th token positions. To ensure that feature compression transmission does not disrupt this information dependency, the feature encoder / decoder should not utilize intermediate features after the t-th position when generating the encoded and reconstructed features for the t-th token position. Even if the front-end network of the model employs causal attention, if the feature encoder / decoder uses symmetric convolution, bidirectional attention, or other transformations that access future positions along the token sequence dimension, the encoded or reconstructed features at the current position may still be mixed with information after the t-th position. This information leakage disrupts the original prefix causality of the large model and causes inconsistencies between the training process and the autoregressive inference process.

[0005] On the other hand, large model training continuously updates the parameters of the front-end network, and the distribution of intermediate features at the model's segmentation points also changes as the training progresses. A feature codec trained only once using intermediate features from before formal training and then used consistently may gradually deviate from the current feature distribution, leading to increased feature reconstruction errors, decreased entropy model matching accuracy, and increased actual transmission bitrate.

[0006] Existing technologies have not yet proposed a feature encoding / decoding scheme for edge-cloud collaborative large-scale model training that can simultaneously meet the following requirements: strictly maintaining the causal relationship between the current position and only the prefixes of the current and historical positions in the token sequence dimension; and dynamically updating based on changes in the distribution of intermediate features during training. Therefore, it is necessary to propose a feature encoding / decoding technique that truly suits large-scale model training scenarios, taking into account causal preservation, dynamic adaptation, and low communication overhead.

[0007] In view of this, the present invention is hereby proposed. Summary of the Invention

[0008] The purpose of this invention is to provide a feature encoding and decoding method and system based on causality and adaptive update. It unifies the application of prefix causal constraints and the adaptive update mechanism based on historical features to the intermediate feature encoding and decoding process of edge-cloud collaborative large model training, so that feature compression transmission does not change the original information dependency relationship of the autoregressive large model, and the feature encoder and decoder can be updated in a controlled manner according to the feature distribution changes during the training process.

[0009] The objective of this invention is achieved through the following technical solution: A feature encoding / decoding method based on causality and adaptive updating includes: The large model to be trained is divided into a front network and a back network, and deployed on different computing nodes. The position between the front network and the back network is determined as the model split position. The intermediate feature sequence output by the front network at the model split position is collected, and the feature codec containing prefix causal transformation is trained to obtain the initial feature codec. The encoding side parameters of the initial feature codec are deployed on the computing node where the front network is located, and the reconstruction side parameters are deployed on the computing node where the back network is located. During the training of the large model, the current intermediate feature sequence output by the front-end network at the model segmentation position and the feature prefix up to the t-th token position are obtained. Using the feature prefix at the t-th token position as a constraint, the feature sequence is subjected to prefix causal coding transformation based on the encoding side parameters. Then, quantization and entropy coding are performed to obtain the transmission bitstream, which is then transmitted to the computing node where the back-end network is located. The computing node where the back-end network is located performs entropy decoding and dequantization on the transmission bitstream. Based on the reconstruction side parameters, the current intermediate feature sequence is reconstructed through prefix causal reconstruction transformation and then input into the back-end network. The feature reconstruction error is calculated using the acquired current intermediate feature sequence and the reconstructed current intermediate feature sequence. The actual transmission bit rate is calculated using the amount of data in the transmitted bit stream and compared with the corresponding reference values. The initial feature codec is then adaptively updated based on the comparison results.

[0010] A feature encoding / decoding system based on causality and adaptive updating, for implementing the aforementioned method, includes: The initial feature codec training unit is used to divide the large model to be trained into a front network and a back network, and deploy them on different computing nodes. The position between the front network and the back network is determined as the model split position. The intermediate feature sequence output by the front network at the model split position is collected and the feature codec containing prefix causal transformation is trained to obtain the initial feature codec. The encoding side parameters of the initial feature codec are deployed on the computing node where the front network is located, and the reconstruction side parameters are deployed on the computing node where the back network is located. The intermediate feature encoding and reconstruction unit is used during the large model training process to obtain the current intermediate feature sequence output by the front-end network at the model splitting position and the feature prefix up to the t-th token position. Using the initial feature encoder and decoder with the feature prefix at the t-th token position as a constraint, the unit performs prefix causal encoding transformation on the current intermediate feature sequence based on the encoding side parameters, and then performs quantization and entropy encoding to obtain the transmission bitstream, which is then transmitted to the computing node where the back-end network is located. The computing node where the back-end network is located performs entropy decoding and dequantization on the transmission bitstream, and reconstructs the current intermediate feature sequence based on the reconstruction side parameters through prefix causal reconstruction transformation, and then inputs it into the back-end network. The feature codec adaptive update unit is used to calculate the feature reconstruction error using the acquired current intermediate feature sequence and the reconstructed current intermediate feature sequence, calculate the actual transmission bit rate using the amount of data in the transmitted bit stream, and compare it with the corresponding reference values ​​respectively, and adaptively update the initial feature codec based on the comparison results.

[0011] As can be seen from the technical solution provided by the present invention above: 1) By applying prefix causal constraints simultaneously on the encoding and reconstruction sides, the encoded features and reconstructed features of each token position are not affected by future position features, thus maintaining the causal relationship required for large model training and reducing the inconsistency between training and inference information due to feature compression; 2) By monitoring the changes in intermediate feature distribution during training based on feature reconstruction error and actual bit rate, and using only historical data generated before the update trigger time for parameter updates and verification, feature encoding and decoding performance can be dynamically restored without introducing future training data; 3) By combining prefix causal feature transformation, adaptive update, and complete quantization entropy encoding link, the communication overhead of edge-cloud collaborative large model training is reduced while maintaining the model training effect, forming a comprehensive technical capability with industry-leading level in causal preservation, dynamic adaptation, and communication compression collaboration. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 The flowchart illustrates a feature encoding / decoding method based on causality and adaptive updating, as provided in an embodiment of the present invention.

[0014] Figure 2 This is a schematic diagram of the overall framework and method flow for edge-cloud collaborative large model training feature compression provided in the embodiments of the present invention.

[0015] Figure 3 This is a schematic diagram of the feature encoder-decoder network structure and prefix causality mechanism provided in an embodiment of the present invention.

[0016] Figure 4 This is a schematic diagram of the adaptive update process of the feature codec provided in an embodiment of the present invention.

[0017] Figure 5 This is a schematic diagram of a feature encoding and decoding system based on causality and adaptive updating, provided in an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0019] First, the following explanations are provided for the terms that may be used in this article: The term "and / or" means that either or both can be achieved simultaneously. For example, X and / or Y means that it includes both "X" or "Y" as well as the three cases of "X and Y".

[0020] The terms “comprising,” “including,” “containing,” “having,” or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, “comprising a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.)” should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0021] The term "composed of" excludes any technical features not expressly listed. When used in a claim, it closes the claim to exclude all technical features other than those expressly listed, except for associated conventional impurities. If the term appears only in a clause of a claim, it limits the claim to the elements expressly listed in that clause; elements recited in other clauses are not excluded from the overall claim.

[0022] The following provides a detailed description of a feature encoding and decoding method and system based on causality and adaptive updating provided by this invention. Contents not described in detail in the embodiments of this invention are prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of this invention, they are performed according to conventional conditions in the art or conditions recommended by the manufacturer. Instruments used in the embodiments of this invention, unless otherwise specified by the manufacturer, are all conventional products that can be purchased commercially.

[0023] Example 1 like Figure 1 The diagram shown is a flowchart of a feature encoding / decoding method based on causality and adaptive updating provided by an embodiment of the present invention, which mainly includes the following steps: Step 1: Obtain initial intermediate features and train the initial feature encoder / decoder.

[0024] In this step: the large model to be trained is divided into a front network and a back network, and deployed on different computing nodes. The position between the front network and the back network is determined as the model splitting position. The intermediate feature sequence output by the front network at the model splitting position is collected, and the feature codec containing prefix causal transformation is trained to obtain the initial feature codec. The encoding side parameters of the initial feature codec are deployed on the computing node where the front network is located, and the reconstruction side parameters are deployed on the computing node where the back network is located.

[0025] In this embodiment of the invention, before training the large model, training data is input into the front-end network, and intermediate feature sequences output by the front-end network at the model splitting positions are collected to form an initial intermediate feature set; wherein, the intermediate feature sequence corresponding to the i-th training data is... Represented as: ; Where i represents the training data index, t represents the token position index, d represents the intermediate feature dimension of each token position, and T represents the length of the intermediate feature sequence. This represents the intermediate feature of the i-th training data at the t-th token position; the initial intermediate feature set contains intermediate feature sequences of multiple training samples, used to learn the compressed representation of the intermediate features and their probability distribution; The initial intermediate feature set is used to train a feature codec containing prefix causal transformation. When training the initial feature codec, the goal is to reduce feature reconstruction error and the coding rate. When the feature reconstruction error and the coding rate meet the set conditions, the training is stopped, and the initial feature codec C0 is obtained.

[0026] In this embodiment of the invention, the feature codec containing prefix causal transformation includes an encoding part and a decoding part; wherein, the encoding part sequentially performs feature normalization, prefix causal encoding transformation, quantization, and entropy encoding; the decoding part sequentially performs entropy decoding, inverse quantization, prefix causal reconstruction transformation, and feature inverse normalization; the feature normalization adopts predetermined statistical parameters or position-by-position processing method, and the quantization adopts element-by-element or group-by-group discretization method.

[0027] Step 2: Use the initial feature encoder / decoder to encode and reconstruct the intermediate features at the model segmentation locations.

[0028] In this step: During the training of the large model, the current intermediate feature sequence output by the front-end network at the model splitting position and the feature prefix up to the t-th token position are obtained. Using the feature prefix at the t-th token position as a constraint, the feature sequence is transformed by prefix causal coding based on the encoding side parameters. Then, quantization and entropy coding are performed to obtain the transmission bitstream, which is then transmitted to the computing node where the back-end network is located. The computing node where the back-end network is located performs entropy decoding and dequantization on the transmission bitstream. Based on the reconstruction side parameters, the current intermediate feature sequence is reconstructed by prefix causal reconstruction transformation and then input into the back-end network.

[0029] In this embodiment of the invention, the step of using the initial feature codec with the feature prefix at the t-th token position as a constraint and performing prefix causal coding transformation on the current intermediate feature sequence based on the encoding side parameters includes: For the k-th training step, the current intermediate feature sequence Its characteristic prefix up to the t-th token position is represented as: ; ; in, Let be the normalized feature of the t-th token position. The feature prefix up to the t-th token position is the feature prefix formed by the current intermediate feature sequence from the 1st token position to the t-th token position. This represents the normalized feature of the 1st token position, the 2nd token position, ..., the Tth token position; The prefix causal coding transformation is performed using the following formula: ; ; Where A represents the prefix causal coding transformation, This represents the encoding side parameters used in the k-th training step. This represents the prefix causal coding transformation performed at the t-th token position, where the input range of the prefix causal coding transformation is restricted to 0. Obtain the coding feature representation of the prefix causal coding transform. ; The encoded feature sequence consists of the encoded feature representations of all T token positions (calculated by the second formula).

[0030] In this embodiment of the invention, the quantization and entropy encoding are implemented in the following manner: ; ; Where Q represents quantization and E represents entropy encoding. This represents the quantization encoding feature of the t-th token position; For transmitting bitstream.

[0031] In this embodiment of the invention, the reconstruction of the current intermediate feature sequence based on the reconstruction side parameters through prefix causal reconstruction transformation is represented as follows: ; ; Where S represents the prefix causal reconstruction transformation, This represents the reconstruction side parameters used in the k-th training step. This represents the prefix causal reconstruction transformation performed at the t-th token position, whose input uses the encoded features from the 1st to the t-th token positions obtained through inverse quantization. , The feature of the t-th token position reconstructed by the prefix causal reconstruction transformation performed at the t-th token position; This is the current intermediate feature sequence for reconstruction.

[0032] Step 3: Monitor and utilize historical intermediate features during the training of the large model to adaptively update the feature encoder and decoder.

[0033] In this step: the feature reconstruction error is calculated using the acquired current intermediate feature sequence and the reconstructed current intermediate feature sequence; the actual transmission bit rate is calculated using the data volume of the transmitted bit stream and compared with the corresponding reference values; and the initial feature codec is adaptively updated based on the comparison results.

[0034] In this embodiment of the invention, the step of calculating the feature reconstruction error using the acquired current intermediate feature sequence and the reconstructed current intermediate feature sequence, and calculating the actual transmission bitrate using the data volume of the transmitted bitstream, is expressed as follows: ; ; in, This represents the feature reconstruction error at the k-th training step, where the subscript F in the formula indicates that the Frobenius norm is used. This represents the actual transmission bitrate of the k-th training step. Indicates the transmitted bitstream Number of bytes, Indicates the connection with the transmitted bitstream The corresponding number of intermediate feature scalars, multiplied by 8, is used to convert the number of bytes into the number of bits.

[0035] In this embodiment of the invention, the step of comparing with corresponding reference values ​​and adaptively updating the initial feature codec based on the comparison results includes: The feature reconstruction error reference value is determined using the initial feature codec or the feature codec updated in the last update. Compared with the actual transmission bit rate reference value Specifically, if this is the first update, the initial feature encoder / decoder is used to determine the feature reconstruction error reference value. Compared with the actual transmission bit rate reference value Otherwise, use the feature codec updated in the last update to determine the feature reconstruction error reference value. Compared with the actual transmission bit rate reference value Compare using the following formula: ; ; in, and These represent the feature reconstruction errors of the k-th training step, respectively. and the actual transmission bit rate of the k-th training step The percentage change relative to the reference value; Set feature reconstruction error thresholds respectively and actual transmission bit rate threshold When at least one metric reaches its corresponding threshold, the current feature codec is deemed to meet the update condition, as shown below: ; in, This represents the update condition decision result of the k-th training step, denoted by [symbol]. The symbol indicates if and only if. Indicates logical OR;

[0036] When the update conditions are met, the initial feature codec update process is initiated.

[0037] In this embodiment of the invention, the step of initiating the initial feature codec update process when the update conditions are met includes: When the update condition is met, the historical training feature set for parameter update and the historical verification feature set for performance verification are obtained from the intermediate feature sequence generated and saved before the update trigger time, satisfying the following: ; in, This represents the historical training feature set used when the update is triggered in the k-th training step. The corresponding historical verification feature set is represented by j≤k, which means that the data in both feature sets were generated and saved before the update trigger time, and the source training step is no later than the kth training step. This represents the intermediate feature sequence of the j-th training step; The parameters used for prefix causal coding transform and prefix causal reconstruction transform in the current feature encoder-decoder. as initial value If k=1, then For the relevant parameters of the initial feature encoder and decoder, if k>1, then These are the parameters from the last update; the parameters used for prefix causal coding transformation and prefix causal reconstruction transformation in the feature encoder-decoder are updated using the historical training feature set. The updated parameters to be verified are represented as follows: ; in, This represents the parameter to be verified obtained after the update is triggered in the k-th training step. The superscript u: update means "updated". The training loss for the feature encoder / decoder; The historical verification feature set is used to verify the feature reconstruction performance and encoding performance of the parameters to be verified; when the verification passes, Used for subsequent training; otherwise, continue using the parameters. , is represented as: ; ; in, This indicates the verification result. This indicates that the verification has passed. This indicates that the verification failed. This represents the parameters used in the (k+1)th training step.

[0038] The method provided in this embodiment of the invention can avoid introducing information about future locations into the feature encoder-decoder process and maintain the causal relationship in the training process of large models; at the same time, it can adapt to changes in intermediate features during model training, and reduce the communication overhead of edge-cloud collaborative large model training while maintaining the model training effect.

[0039] To more clearly demonstrate the technical solution and its effects provided by the present invention, the method provided by the embodiments of the present invention will be described in detail below with reference to specific examples.

[0040] This invention provides a feature encoding / decoding method based on causality and adaptive updating, which is a feature encoding / decoding scheme applied to edge-cloud collaborative training scenarios, such as... Figure 2 As shown, the overall framework includes at least edge computing nodes, cloud computing nodes, and the data transmission link between them. Edge computing nodes perform front-end network computation and intermediate feature encoding for the large model, while cloud computing nodes receive the transmitted bitstream and complete intermediate feature reconstruction and back-end network computation. The large model still performs loss calculation and parameter updates according to the established training algorithm. The feature encoder / decoder is used to compress intermediate features at the model's segmentation points without altering the large model's training task itself. The entire method mainly consists of three steps:

[0041] Step 1: Obtain initial intermediate features and train the initial feature encoder / decoder.

[0042] Step 1.1: Obtain initial intermediate features.

[0043] Step 1.1.1: In this embodiment, the large model to be trained is divided into a front-end network and a back-end network. The front-end network is deployed on edge computing nodes, and the back-end network is deployed on cloud computing nodes. The position between the front-end network and the back-end network is determined as the model splitting position. The model splitting position can be set after the network layer that can output the middle representation of the sequence. The edge and cloud nodes respectively store the model parameters corresponding to the network computation they undertake.

[0044] Step 1.1.2: Before formally training the large model, input the training data into the front-end network and collect the intermediate feature sequences output by the front-end network at the model splitting positions to form the initial intermediate feature set. The intermediate feature sequence corresponding to the i-th training sample is represented as: ; Where i represents the training sample index, t represents the token position index, d represents the intermediate feature dimension of each token position, T represents the length of the intermediate feature sequence, and R is the symbol of the real number set; This represents the intermediate feature of the i-th training sample at the t-th token position. The initial intermediate feature set contains intermediate feature sequences from multiple training samples, used to learn the compressed representation of the intermediate features and their probability distribution.

[0045] Step 1.2: Train the initial feature encoder / decoder.

[0046] Step 1.2.1: Train a feature codec including prefix causal transformation using the initial intermediate feature set obtained in Step 1.1. The feature codec sequentially performs feature normalization, prefix causal coding transformation, quantization, and entropy coding, and on the receiving side, it performs entropy decoding, inverse quantization, prefix causal reconstruction transformation, and inverse normalization. Feature normalization uses predetermined statistical parameters or a position-by-position processing method, and quantization uses an element-by-element or group-by-group discretization method to avoid using features from future token positions for processing at the current position.

[0047] When training the initial feature codec, the joint objectives are to reduce feature reconstruction error and reduce coding rate. Training converges and stops when the feature reconstruction error and coding rate meet set conditions, yielding the initial feature codec C0. The coding-side parameters of the initial feature codec are deployed on the computation nodes that generate intermediate features, while the reconstruction-side parameters are deployed on the computation nodes that receive the transmitted bitstream.

[0048] This part requires updating the parameters of each component within the feature codec to obtain the initial feature codec for training. In the subsequent adaptive update process of the feature codec, the parameters of the prefix causal coding transform and the prefix causal reconstruction transform are mainly updated.

[0049] Step 2: Use a feature encoder-decoder that includes prefix causal transformation to encode and reconstruct the intermediate features at the model segmentation locations.

[0050] Step 2.1: Obtain the current intermediate features.

[0051] Step 2.1.1: During the formal training of the large model, the current training data is input into the front-end network, and network calculations are performed before the model splitting position to obtain the intermediate features of the current training step. The front-end network can be continuously updated under a given training algorithm, so the distribution of intermediate features generated in different training steps can change.

[0052] Step 2.1.2: Obtain the current intermediate feature sequence output by the front-end network at the model splitting position. For the k-th training step, the current intermediate feature sequence and its feature prefix up to the t-th token position are represented as follows: ; ; Where k represents the sequence number of the large model training step. This represents the feature prefix formed by the current intermediate feature sequence from the 1st token position to the tth token position. The causality described in this invention refers to prefix causality in the token sequence dimension, that is, the encoding and decoding result of the tth token position cannot utilize intermediate features after the tth position, rather than the statistical causal relationship between training samples.

[0053] Step 2.2: Perform prefix causal encoding and decoding on the current intermediate features.

[0054] Step 2.2.1: Normalize the current intermediate feature sequence and encode it using the prefix causal coding transform in the current feature encoder-decoder to obtain the encoded feature sequence. The encoded feature at the t-th token position is represented as: ; ; Where A represents the encoding-side prefix causal transformation, This represents the encoding side parameters used in the k-th training step. This represents the prefix causal encoding change performed at the t-th token position. The input range of this prefix causal encoding change is restricted to... ,therefore It is determined solely by intermediate features from the current position and its preceding positions. The encoding side can employ causal convolutions along the token sequence direction, mask mapping, or other trainable transformations that satisfy the same input range constraint.

[0055] Step 2.2.2: Quantize the encoded feature sequence to map continuous encoded features into discrete quantized encoded features; then perform entropy encoding on the quantized encoded features to generate the transmission bitstream. And send it to the cloud computing node. The quantization and entropy encoding process is represented as: ; ; Where Q represents quantization and E represents entropy encoding. This represents the quantization encoding feature at the t-th token position. Entropy coding can employ a lossless coding method that matches the probability distribution of the quantization encoding feature; when it is necessary to recover the encoding feature in the order of token positions, the probability information used to decode the current position is determined only based on the already obtained quantization encoding feature.

[0056] The cloud computing node receives the transmitted bitstream Entropy decoding is performed to recover the quantized encoded feature sequence, and dequantization is completed. Then, the prefix causal reconstruction transform in the current feature encoder-decoder is used to generate the reconstructed intermediate feature sequence. The reconstruction process is represented as follows: ; ; Where S represents the reconstruction side prefix causal transformation, In the above formula, the reconstructed side parameters used in the k-th training step are represented. and These represent the reconstructed intermediate features and the reconstructed intermediate feature sequence, respectively. The reconstructed feature at the t-th token position uses only the dequantized encoded features. to The encoded features after the t-th position are not used. After reconstruction, the reconstruction result is subjected to inverse normalization processing corresponding to the encoding-side normalization.

[0057] For any two intermediate feature sequences with the same feature prefix, the feature codec should produce the same encoding / decoding output prefix, expressed as: ; in, This represents the feature encoding / decoding mapping consisting of encoding, quantization, entropy encoding, entropy decoding, dequantization, and reconstruction processes. The above formula shows that as long as the features of two input sequences up to the t-th token position are the same, their encoding / decoding outputs up to the t-th token position will also be the same; changes in features at future positions will not affect the results at the current or historical positions.

[0058] Step 2.2.3: Input the reconstructed intermediate feature sequence obtained in Step 2.2.2 into the cloud-based backend network for network computation after the model segmentation position in the current training step, and obtain the large model training loss accordingly. The large model can update the front and backend networks according to the predetermined backpropagation and parameter optimization process; within the current training step, the encoding and decoding parameters involved in feature transmission remain consistent.

[0059] Step 3: Monitor and dynamically update the feature encoder and decoder using historical feature data during the training of the large model.

[0060] Step 3.1: Monitor feature encoding and decoding performance.

[0061] Step 3.1.1: Based on the original intermediate feature sequence of the current training step. and reconstructing intermediate feature sequences Calculate the feature reconstruction error of the current feature codec; based on the actual transmitted bitstream generated in the current training step. The actual transmission rate is calculated based on the amount of data, and is expressed as follows: ; ; in, This represents the average feature reconstruction error at the k-th training step. The subscript F in the formula indicates that the Frobenius norm is used. This indicates the corresponding actual transmission bit rate. Indicates the transmitted bitstream Number of bytes, This indicates the number of intermediate feature scalars corresponding to the bitstream, with a multiplier of 8 used to convert the number of bytes to the number of bits. The actual transmission bitrate is calculated directly based on the length of the bitstream generated and transmitted by the feature codec, without further breaking down the internal components of the bitstream.

[0062] Step 3.1.2: Compare the feature reconstruction error and actual transmission bit rate obtained in Step 3.1.1 with the current reference value. and The comparison is then performed. The reference value can be taken from the initial feature codec, or from the most recently validated and enabled feature codec, resulting in: ; ; in, and These represent the changes in the current feature reconstruction error and the actual transmission bit rate relative to the reference value, respectively. Using relative changes can reduce the impact of different model segmentation positions, feature dimensions, or quantization configurations on the threshold setting.

[0063] Step 3.1.3: Set the feature reconstruction error thresholds respectively. and actual transmission bit rate threshold When at least one encoding / decoding performance metric reaches its corresponding threshold, the current feature codec is deemed to meet the update condition, as expressed as: ; in, This represents the update condition determination result of the k-th training step. The logical OR in the formula indicates that an update is triggered if at least one of the reconstruction error condition and the transmission bit rate condition is met. If... If the value is not equal to 1, the current feature encoder / decoder will continue to be used, and monitoring will continue in subsequent training steps. Monitoring can be performed in each training step, or the metrics of multiple training steps can be statistically analyzed at preset intervals.

[0064] Step 3.2: Dynamically update the feature encoder and decoder using historical intermediate features.

[0065] Step 3.2.1: When step 3.1.3 determines that the update condition is met, obtain the historical training feature set for parameter update and the historical verification feature set for performance verification from the intermediate features generated and saved before the update trigger time, satisfying: ; in, This represents the historical training feature set used when the update is triggered in the k-th training step. Let j ≤ k represent the corresponding historical validation feature set. This indicates that the data in both feature sets were generated and saved before the update trigger time, and their source training steps are no later than the k-th training step. The historical training feature set and the historical validation feature set are separated from each other to avoid directly using parameter update data to evaluate candidate parameters.

[0066] Step 3.2.2: Using the current feature codec parameters as initial values, update the parameters in the feature codec used for prefix causal coding transformation and prefix causal reconstruction transformation using the historical training feature set. The updated parameters to be verified are represented as follows: ; in, This represents the updated parameters to be verified obtained after the update is triggered in the k-th training step. The above formula represents the training loss of the feature encoder-decoder. It indicates that candidate parameters are obtained by optimizing the encoder-decoder loss on the historical training feature set, but does not mean that the parameters to be verified immediately replace the current parameters. During parameter updates, the parameters related to feature normalization, quantization, and entropy encoding remain unchanged, and the main training process continues to use the current feature encoder-decoder to complete feature compression and reconstruction.

[0067] Step 3.2.3: Utilize the historical verification feature set to verify the feature reconstruction performance and coding performance of the parameters to be updated. Verification conditions may include reconstruction error not exceeding a set upper limit, actual transmission bit rate not exceeding a set upper limit, or a combined indicator of both meeting the set requirements. Based on the verification results, select the parameters to be used subsequently, expressed as: ; in, This represents the feature encoder / decoder parameters used in the (k+1)th training step. This indicates the verification result of the parameter to be updated. This indicates that the verification has passed, and subsequent training will use [the appropriate method / method]. ; This indicates that the verification failed and the parameters before the update will continue to be used. Once verification is successful, the matching encoding and reconstruction parameters will be enabled synchronously between the end of the current training step and the start of the next training step to avoid inconsistent encoding and decoding parameter versions within the same training step.

[0068] Steps 2 and 3 are repeated during the training of the large model until training is complete. After each successful validation and parameter update, the reconstruction error and actual transmission bitrate obtained from the historical validation feature set can be used to update the parameters. and This is so that subsequent monitoring can reflect performance changes relative to the most recently effective codec.

[0069] like Figure 3 As shown, the prefix causal transformation employs a one-dimensional dilated causal convolution along the token sequence direction. The 3*1 convolution kernel has a length of 3 along the token direction and a width of 1 along the hidden layer direction, and uses padding only on the sequence history side to make the first... The output of each token position depends only on the current position and historical positions. Represents the expansion factor, the first The dilation factor of a convolutional layer is denoted as Its convolution sampling locations include , and In embodiments where the dilation rates are set sequentially to 1, 2, and 4, multiple convolutional layers are stacked to form a 15-token receptive field covering the current position and the previous 14 token positions. The input-output dependency of the transformation exhibits a lower triangular or finite-bandwidth lower triangular structure (i.e., Figure 3 The transformation dependency matrix in the middle blocks the feature transmission path from the future token position to the current position.

[0070] like Figure 4 As shown, the dynamic monitoring and adaptive update process can be executed in parallel with the main training flow of the large model. The main flow continuously uses the current feature encoder-decoder C during training step k. k The update process utilizes historical features generated and saved before the update trigger time to obtain the update parameters to be verified, and performs verification on the historical verification features. Only after successful verification and reaching the boundary of the adjacent training step, do the edge and cloud synchronously use the new parameter version. This ensures that the update process does not interrupt the current training step, and also ensures that the bitstream generated on the encoding side can be correctly recovered by the matching reconstruction side.

[0071] Through the above description of the embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.), including several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0072] Example 2 This invention also provides a feature encoding / decoding system based on causality and adaptive updating, which is mainly used to implement the methods provided in the foregoing embodiments, such as... Figure 5 As shown, the system mainly includes: The initial feature codec training unit is used to divide the large model to be trained into a front network and a back network, and deploy them on different computing nodes. The position between the front network and the back network is determined as the model split position. The intermediate feature sequence output by the front network at the model split position is collected and the feature codec containing prefix causal transformation is trained to obtain the initial feature codec. The encoding side parameters of the initial feature codec are deployed on the computing node where the front network is located, and the reconstruction side parameters are deployed on the computing node where the back network is located. The intermediate feature encoding and reconstruction unit is used during the large model training process to obtain the current intermediate feature sequence output by the front-end network at the model splitting position and the feature prefix up to the t-th token position. Using the initial feature encoder and decoder with the feature prefix at the t-th token position as a constraint, the unit performs prefix causal encoding transformation on the current intermediate feature sequence based on the encoding side parameters, and then performs quantization and entropy encoding to obtain the transmission bitstream, which is then transmitted to the computing node where the back-end network is located. The computing node where the back-end network is located performs entropy decoding and dequantization on the transmission bitstream, and reconstructs the current intermediate feature sequence based on the reconstruction side parameters through prefix causal reconstruction transformation, and then inputs it into the back-end network. The feature codec adaptive update unit is used to calculate the feature reconstruction error using the acquired current intermediate feature sequence and the reconstructed current intermediate feature sequence, calculate the actual transmission bit rate using the amount of data in the transmitted bit stream, and compare it with the corresponding reference values ​​respectively, and adaptively update the initial feature codec based on the comparison results.

[0073] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0074] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art.

Claims

1. A feature encoding / decoding method based on causality and adaptive updating, characterized in that, include: The large model to be trained is divided into a front network and a back network, and deployed on different computing nodes. The position between the front network and the back network is determined as the model split position. The intermediate feature sequence output by the front network at the model split position is collected, and the feature codec containing prefix causal transformation is trained to obtain the initial feature codec. The encoding side parameters of the initial feature codec are deployed on the computing node where the front network is located, and the reconstruction side parameters are deployed on the computing node where the back network is located. During the training of the large model, the current intermediate feature sequence output by the front-end network at the model segmentation position and the feature prefix up to the t-th token position are obtained. Using the feature prefix at the t-th token position as a constraint, the feature sequence is subjected to prefix causal coding transformation based on the encoding side parameters. Then, quantization and entropy coding are performed to obtain the transmission bitstream, which is then transmitted to the computing node where the back-end network is located. The computing node where the back-end network is located performs entropy decoding and dequantization on the transmission bitstream. Based on the reconstruction side parameters, the current intermediate feature sequence is reconstructed through prefix causal reconstruction transformation and then input into the back-end network. The feature reconstruction error is calculated using the acquired current intermediate feature sequence and the reconstructed current intermediate feature sequence. The actual transmission bit rate is calculated using the amount of data in the transmitted bit stream and compared with the corresponding reference values. The initial feature codec is then adaptively updated based on the comparison results.

2. The feature encoding and decoding method based on causality and adaptive updating according to claim 1, characterized in that, The process involves acquiring intermediate feature sequences output by the front-end network at model segmentation locations and training a feature codec containing prefix causal transformation to obtain an initial feature codec, including: Before training the large model, training data is input into the front-end network, and intermediate feature sequences output by the front-end network at the model splitting positions are collected to form an initial intermediate feature set; among them, the intermediate feature sequence corresponding to the i-th training data is... Represented as: ； Where i represents the training data index, t represents the token position index, d represents the intermediate feature dimension of each token position, T represents the length of the intermediate feature sequence, and R is the symbol for the real number set. This represents the intermediate feature of the i-th training data at the t-th token position; the initial intermediate feature set contains intermediate feature sequences of multiple training samples, used to learn the compressed representation of the intermediate features and their probability distribution; The initial intermediate feature set is used to train a feature codec containing prefix causal transformation. When training the initial feature codec, the goal is to reduce feature reconstruction error and the coding rate. When the feature reconstruction error and the coding rate meet the set conditions, the training is stopped, and the initial feature codec C0 is obtained.

3. The feature encoding and decoding method based on causality and adaptive updating according to claim 1, characterized in that, The feature codec containing prefix causal transformation includes: an encoding part and a decoding part; The encoding part sequentially performs feature normalization, prefix causal coding transformation, quantization, and entropy coding; the decoding part sequentially performs entropy decoding, dequantization, prefix causal reconstruction transformation, and feature inverse normalization. The feature normalization adopts predetermined statistical parameters or position-by-position processing, and the quantization adopts element-by-element or group-by-group discretization.

4. The feature encoding and decoding method based on causality and adaptive updating according to claim 1, characterized in that, The step of using the initial feature encoder-decoder with the feature prefix at the t-th token position as a constraint, and performing prefix causal coding transformation on the current intermediate feature sequence based on the encoding side parameters, includes: For the k-th training step, the current intermediate feature sequence Its characteristic prefix up to the t-th token position is represented as: ； ； in, Let be the normalized feature of the t-th token position. The feature prefix up to the t-th token position is the feature prefix formed by the current intermediate feature sequence from the 1st token position to the t-th token position. This represents the normalized feature of the 1st token position, the 2nd token position, ..., the Tth token position; The prefix causal coding transformation is performed using the following formula: ； ； Where A represents the prefix causal coding transformation, This represents the encoding side parameters used in the k-th training step. This represents the prefix causal coding transformation performed at the t-th token position, where the input range of the prefix causal coding transformation is restricted to 0. Obtain the coding feature representation of the prefix causal coding transform. ; The encoded feature sequence consists of the encoded feature representations of all T token positions.

5. The feature encoding and decoding method based on causality and adaptive updating according to claim 4, characterized in that, The quantization and entropy encoding are implemented in the following way: ； ； Where Q represents quantization and E represents entropy encoding. This represents the quantization encoding feature of the t-th token position; For transmitting bitstream.

6. The feature encoding and decoding method based on causality and adaptive updating according to claim 5, characterized in that, The current intermediate feature sequence reconstructed based on the reconstruction side parameters through prefix causal reconstruction transformation is represented as follows: ； ； Where S represents the prefix causal reconstruction transformation, This represents the reconstruction side parameters used in the k-th training step. This represents the prefix causal reconstruction transformation performed at the t-th token position, whose input uses the encoded features from the 1st to the t-th token positions obtained through inverse quantization. , The feature of the t-th token position reconstructed by the prefix causal reconstruction transformation performed at the t-th token position; This is the current intermediate feature sequence for reconstruction.

7. The feature encoding and decoding method based on causality and adaptive updating according to claim 1, characterized in that, The feature reconstruction error is calculated by comparing the acquired current intermediate feature sequence with the reconstructed current intermediate feature sequence, and the actual transmission bit rate is calculated using the data volume of the transmitted bitstream, expressed as follows: ； ； in, This represents the feature reconstruction error at the k-th training step, where the subscript F in the formula indicates that the Frobenius norm is used. This represents the actual transmission bitrate of the k-th training step. Indicates the transmitted bitstream Number of bytes, Indicates the connection with the transmitted bitstream The corresponding number of intermediate feature scalars, multiplied by 8, is used to convert the number of bytes into the number of bits.

8. The feature encoding and decoding method based on causality and adaptive updating according to claim 1, characterized in that, The step of comparing the initial feature encoder / decoder with the corresponding reference values ​​and adaptively updating the initial feature encoder / decoder based on the comparison results includes: The feature reconstruction error reference value is determined using the initial feature codec or the feature codec updated in the last update. Compared with the actual transmission bit rate reference value The following formula is used for comparison: ； ； in, and These represent the feature reconstruction errors of the k-th training step, respectively. and the actual transmission bit rate of the k-th training step The percentage change relative to the reference value; Set feature reconstruction error thresholds respectively and actual transmission bit rate threshold When at least one metric reaches its corresponding threshold, the current feature codec is deemed to meet the update condition, as shown below: ； in, This represents the update condition decision result of the k-th training step, denoted by [symbol]. The symbol indicates if and only if. Indicates logical OR; When the update conditions are met, the initial feature codec update process is initiated.

9. A feature encoding / decoding method based on causality and adaptive updating according to claim 8, characterized in that, The step of initiating the initial feature codec update process when the update conditions are met includes: When the update condition is met, the historical training feature set for parameter update and the historical verification feature set for performance verification are obtained from the intermediate feature sequence generated and saved before the update trigger time, satisfying the following: ； in, This represents the historical training feature set used when the update is triggered in the k-th training step. The corresponding historical verification feature set is represented by j≤k, which means that the data in both feature sets were generated and saved before the update trigger time, and the source training step is no later than the kth training step. This represents the intermediate feature sequence of the j-th training step; The parameters used for prefix causal coding transform and prefix causal reconstruction transform in the current feature encoder-decoder. as initial value If k=1, then For the relevant parameters of the initial feature encoder and decoder, if k>1, then These are the parameters from the last update; the parameters used for prefix causal coding transformation and prefix causal reconstruction transformation in the feature encoder-decoder are updated using the historical training feature set. The updated parameters to be verified are represented as follows: ； in, This represents the parameter to be verified obtained after the update is triggered in the k-th training step. The training loss for the feature encoder / decoder; The historical verification feature set is used to verify the feature reconstruction performance and encoding performance of the parameters to be verified; when the verification passes, Used for subsequent training; otherwise, continue using the parameters. , is represented as: ； ； in, This indicates the verification result. This indicates that the verification has passed. This indicates that the verification failed. This represents the parameters used in the (k+1)th training step.

10. A feature encoding / decoding system based on causality and adaptive updating, used to implement the method according to any one of claims 1 to 9, characterized in that, include: The initial feature codec training unit is used to divide the large model to be trained into a front network and a back network, and deploy them on different computing nodes. The position between the front network and the back network is determined as the model split position. The intermediate feature sequence output by the front network at the model split position is collected and the feature codec containing prefix causal transformation is trained to obtain the initial feature codec. The encoding side parameters of the initial feature codec are deployed on the computing node where the front network is located, and the reconstruction side parameters are deployed on the computing node where the back network is located. The intermediate feature encoding and reconstruction unit is used during the large model training process to obtain the current intermediate feature sequence output by the front-end network at the model splitting position and the feature prefix up to the t-th token position. Using the initial feature encoder and decoder with the feature prefix at the t-th token position as a constraint, the unit performs prefix causal encoding transformation on the current intermediate feature sequence based on the encoding side parameters, and then performs quantization and entropy encoding to obtain the transmission bitstream, which is then transmitted to the computing node where the back-end network is located. The computing node where the back-end network is located performs entropy decoding and dequantization on the transmission bitstream, and reconstructs the current intermediate feature sequence based on the reconstruction side parameters through prefix causal reconstruction transformation, and then inputs it into the back-end network. The feature codec adaptive update unit is used to calculate the feature reconstruction error using the acquired current intermediate feature sequence and the reconstructed current intermediate feature sequence, calculate the actual transmission bit rate using the amount of data in the transmitted bit stream, and compare it with the corresponding reference values ​​respectively, and adaptively update the initial feature codec based on the comparison results.

Images