Context compression method for deep learning network and deep learning network

Abstract

A deep learning network for context compression includes a context learning network for learning a context representation from an input text sequence; a compression network for performing compression processing on the context representation; the generation network is used for generating a target text sequence based on the compressed context representation; wherein the compression network comprises a first compression network which is used for carrying out local compression processing on the context representation to obtain a first compression context representation which retains local semantics of the context representation; and the second compression network is used for carrying out global compression processing on the context representation to obtain a second compression context representation which retains the global semantics of the context representation.

Description

Technical Field

[0001] The embodiments of this specification belong to the field of artificial intelligence, and in particular, relate to a context compression method for deep learning networks and a deep learning network. Background Art

[0002] LLM (Large Language Model) is a deep neural network model based on the Transformer architecture. It demonstrates powerful capabilities in processing text tasks because it can capture long-range dependencies in text sequences using a special self-attention mechanism and retain the order information of text sequences through positional encoding.

[0003] However, with the growing demand for reasoning over long text sequences, when using LLM to process some long text sequences, due to the complexity of the LLM's self-attention mechanism itself, there are usually problems of low computational efficiency and difficulty in learning effective contextual information from overly long text sequences. Summary of the Invention

[0004] This specification proposes a context compression method for a deep learning network, wherein the deep learning network uses a hybrid compression mechanism that supports both local and global compression to compress context representations learned from an input text sequence; the deep learning network includes a context learning network; a compression network; and a generation network; wherein the compression network includes a first compression network for locally compressing the context representation; and a second compression network for globally compressing the context representation; the method includes:

[0005] Get text sequence;

[0006] Inputting the text sequence into the context learning network to learn context representation from the text sequence by the context learning;

[0007] Obtaining the context representation output by the context learning network, and inputting the context representation into the first compression network and the second compression network, respectively, so that the first compression network performs local compression processing on the context representation to obtain a first compressed context representation that retains the local semantics of the context representation, and the second compression network performs global compression processing on the context representation to obtain a second compressed context representation that retains the global semantics of the context representation;

[0008] The first compression context representation output by the first compression network and the second compression context representation output by the second compression network are obtained, and the first compression context representation and the second compression context representation are further input into the generation network so that the generation network generates a target text sequence based on the first compression context representation and the second compression context representation.

[0009] Optionally, the text sequence includes a plurality of text words; the context representation includes a plurality of feature representations corresponding one-to-one to the plurality of text words;

[0010] The local compression processing includes: calculating the importance index of each feature representation included in the context representation, and filtering out some feature representations included in the context representation whose values ​​of the importance index are lower than a threshold based on the importance index;

[0011] The global compression process includes: further extracting global context information from the feature representation included in the context representation, and encoding the global context information into a low-dimensional feature representation.

[0012] Optionally, the first compression network includes a classification network; the classification network is used to calculate the retention probability of each feature representation contained in the context representation, and assign classification labels to each feature representation contained in the context representation based on the retention probability; the classification label includes a first classification label for indicating that the feature representation is retained and a second classification label for indicating that the feature representation is not retained; and, the feature representation contained in the context representation and assigned the second classification label is filtered out.

[0013] Optionally, the text sequence includes text sequence content and text instructions corresponding to the text sequence content;

[0014] Calculating the retention probability of each feature representation included in the context representation includes:

[0015] Calculating the relevance between each feature representation included in the context representation and the text instruction;

[0016] The retention probability of each feature representation included in the context representation is further calculated based on the correlation; wherein the retention probability is positively correlated with the correlation.

[0017] Optionally, the second compression network includes a first attention network, a second attention network and a weight network;

[0018] The first attention network is used to divide the feature representations included in the context representation into multiple groups, extract local context information of each feature representation included in each group in the multiple groups based on the attention mechanism, and encode the local context information into a low-dimensional local feature representation;

[0019] The second attention network is used to extract global context information of each feature representation included in the context representation in the context representation based on the attention mechanism, and encode the global context information into a low-dimensional global feature representation;

[0020] The weight network is used to learn compression weights corresponding to the first attention network and the second attention network respectively based on the context representation, and perform weighted calculation on the local feature representation and the global feature representation based on the learned compression weights to obtain the second compressed context representation.

[0021] Optionally, the text sequence includes text sequence content and text instructions corresponding to the text sequence content; the attention mechanism includes a hybrid attention mechanism that integrates a cross-attention mechanism between each feature representation included in the context representation and the text instructions;

[0022] Extracting local context information of each feature representation contained in each of the plurality of groups in the group to which it belongs based on the attention mechanism, including:

[0023] Fusing the feature representations contained in each of the plurality of groups into a combined feature representation corresponding to each group, and calculating a cross-attention weight between the combined feature representation corresponding to each group and the text instruction;

[0024] Using the cross attention weight as a calculation parameter, further calculating the self-attention weight of each feature representation contained in each group in the group to which it belongs, and calculating the local context information of each feature representation contained in each group in the group to which it belongs based on the self-attention weight;

[0025] Extracting global context information of each feature representation contained in the context representation based on the attention mechanism, including:

[0026] Calculating cross-attention weights between each feature representation included in the context representation and the text instruction;

[0027] The cross-attention weight is used as a calculation parameter to further calculate the self-attention weight of each feature representation contained in the context representation in the context representation, and the global context information of each feature representation contained in the context representation in the context representation is calculated based on the self-attention weight.

[0028] Optionally, the cross attention weight is used as a calculation parameter to further calculate the self-attention weight of each feature representation contained in each group in the group to which it belongs, including:

[0029] Using the cross attention weight as the query vector corresponding to each feature representation contained in each group, further calculating the self-attention weight of each feature representation contained in each group in the group to which it belongs;

[0030] Using the cross attention weight as a calculation parameter, further calculating the self-attention weight of each feature representation included in the context representation in the context representation, including:

[0031] The cross-attention weight is used as the query vector corresponding to each feature representation contained in the context representation, and the self-attention weight of each feature representation contained in the context representation is further calculated.

[0032] Optionally, the training process of the compression network includes: pre-training; and fine-tuning training performed in a target scenario;

[0033] The training of the compression network includes:

[0034] First stage of training: first remove the first compression network and pre-train the second compression network independently;

[0035] The second stage of training includes: after pre-training the second compressed network is completed, adding the first compressed network and freezing the parameters of the second compressed network to pre-train the first compressed network;

[0036] In the third stage of training, after pre-training of the first compression network and the second compression network is completed, fine-tuning training is performed on the first compression network and the second compression network together in the target application scenario.

[0037] Optionally, the pre-training task corresponding to the first compression network includes a completion task; the completion task is a task of completing missing information in the input original context representation; accordingly, the optimization goal of pre-training the first compression network includes: minimizing the error between the information completed by the first compression network and the information actually missing in the input original context representation; the pre-training loss corresponding to the completion task includes: the error between the information completed by the first compression network and the information actually missing in the input original context representation;

[0038] The pre-training task corresponding to the second compression network includes a paraphrase task; wherein the paraphrase task is a task of generating a compressed context representation that is semantically similar to the original context representation based on the input; accordingly, the optimization goal of pre-training the second compression network includes minimizing the similarity error between the original context representation of the input and the compressed context representation generated by the second compression network based on the original context representation; the pre-training loss corresponding to the paraphrase task includes minimizing the similarity error between the original context representation of the input and the compressed context representation generated by the second compression network based on the original context representation;

[0039] The fine-tuning task corresponding to the first compression network and the second compression network includes a sequence generation task corresponding to the target application scenario; accordingly, the optimization objective of fine-tuning the first compression network and the second compression network includes minimizing the distribution difference between a first probability distribution of a sequence generated by executing the sequence generation task with the original context representation as input data and a second probability distribution of a sequence generated by executing the sequence generation task with the compressed context representation as input data. Accordingly, the fine-tuning training loss corresponding to the sequence generation task may specifically include the distribution difference between the first probability distribution and the second probability distribution.

[0040] Optionally, the optimization objectives of fine-tuning the first compression network and the second compression network include: minimizing a first distribution difference between the first probability distribution and the second probability distribution; and, using the original context representation as input data of another pre-trained teacher model, a second distribution difference between a third probability distribution of a sequence generated by the teacher model performing the sequence generation task and the second probability distribution; the fine-tuning training loss corresponding to the sequence generation task includes: the first distribution difference and the second distribution difference.

[0041] This specification also proposes a deep learning network for context compression, including:

[0042] Contextual learning network, used to learn contextual representations from input text sequences;

[0043] A compression network, wherein the compression network includes a first compression network and a second compression network; the first compression network is used to perform local compression processing on the input context representation to obtain a first compressed context representation that retains the local semantics of the context representation; the second compression network is used to perform global compression processing on the context representation to obtain a second compressed context representation that retains the global semantics of the context representation;

[0044] A generative network is used to generate a target text sequence based on the first compressed context representation and the second compressed context representation of the input.

[0045] Optionally, the text sequence includes a plurality of text words; the context representation includes a plurality of feature representations corresponding one-to-one to the plurality of text words;

[0046] The local compression processing includes: calculating the importance index of each feature representation included in the context representation, and filtering out some feature representations included in the context representation whose values ​​of the importance index are lower than a threshold based on the importance index;

[0047] The global compression process includes: further extracting global context information from the feature representation included in the context representation, and encoding the global context information into a low-dimensional feature representation.

[0048] Optionally, the first compression network includes a classification network; the classification network is used to calculate the retention probability of each feature representation contained in the context representation, and assign classification labels to each feature representation contained in the context representation based on the retention probability; the classification label includes a first classification label for indicating that the feature representation is retained and a second classification label for indicating that the feature representation is not retained; and, the feature representation contained in the context representation and assigned the second classification label is filtered out.

[0049] Optionally, the second compression network includes a first attention network, a second attention network and a weight network;

[0050] The first attention network is used to divide the feature representations included in the context representation into multiple groups, extract local context information of each feature representation included in each group in the multiple groups based on the attention mechanism, and encode the local context information into a low-dimensional local feature representation;

[0051] The second attention network is used to extract global context information of each feature representation included in the context representation in the context representation based on the attention mechanism, and encode the global context information into a low-dimensional global feature representation;

[0052] The weight network is used to learn compression weights corresponding to the first attention network and the second attention network respectively based on the context representation, and perform weighted calculation on the local feature representation and the global feature representation based on the learned compression weights to obtain the second compressed context representation.

[0053] Optionally, the second compression network includes a hybrid expert network; wherein the hybrid expert network includes the first attention network as an expert network, the second attention network as an expert network, and the weight network as a gating network.

[0054] Optionally, the first attention network includes an MLP network that introduces an attention mechanism; the second attention network includes a Q-former network.

[0055] Optionally, the deep learning network is an LLM using a Transformer architecture; the context learning network is an encoder of the LLM; the generative network is a decoder of the LLM; or,

[0056] The deep learning network is a cross-model network composed of two different LLMs; the context learning network is a first LLM used to learn context representation from an input text sequence; and the generation network is a second LLM used to generate a target text sequence based on the compressed context representation.

[0057] In the above embodiment, by simultaneously introducing a network structure for performing local compression processing and global compression processing on the context representation learned from the text sequence into the deep learning network, the compressed context representation can retain both local semantics and global semantics, so that on the basis of effective compression of the context representation, the local and global semantics in the context representation can be integrated to the greatest extent, which not only improves the computational efficiency of the deep learning network, but also ensures the semantic integrity of the context representation learned by the deep learning network from a longer text sequence to the greatest extent, thereby avoiding the problem of semantic loss in the learned context representation. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0059] Figure 1 This is a flowchart of a context compression method for deep learning networks shown in an embodiment of this specification;

[0060] Figure 2 is a schematic diagram of a deep learning network shown in an embodiment of this specification;

[0061] Figure 3 is a schematic diagram of another deep learning network shown in an embodiment of this specification;

[0062] Figure 4 is a schematic diagram of another deep learning network shown in an embodiment of this specification;

[0063] Figure 5 is a schematic diagram of another deep learning network shown in an embodiment of this specification;

[0064] Figure 6 is a schematic diagram of another deep learning network shown in an embodiment of this specification;

[0065] Figure 7 is a schematic structural diagram of an electronic device shown in an embodiment of this specification;

[0066] Figure 8 This is a block diagram of a context compression device for deep learning networks shown in one embodiment of this specification. DETAILED DESCRIPTION

[0067] To help those skilled in the art better understand the technical solutions in this specification, the following will provide a clear and complete description of the technical solutions in the embodiments of this specification, in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of them. All other embodiments derived by those skilled in the art based on the embodiments in this specification without creative effort shall fall within the scope of protection of this specification.

[0068] When using LLM to process some long text sequences, due to the complexity of LLM's Self-Attention mechanism itself, there are usually problems such as low computational efficiency and difficulty in learning effective contextual information from overly long text sequences.

[0069] For example, the time complexity of the Self-Attention mechanism of LLM is usually O(n 2 ) level of complexity. When processing text sequences containing tens of thousands of tokens (i.e., word units), on the one hand, this leads to low computational efficiency and high computational cost. On the other hand, if the input text sequence is noisy or too long, the Self-Attention mechanism cannot effectively focus on relevant information from the long text sequence, resulting in semantic loss and the inability to learn the complete semantics contained in the long text sequence.

[0070] To solve the above problems, context compression technology has emerged and has become an effective strategy to improve the long context understanding ability of LLM while reducing computational costs.

[0071] Context compression methods of targets are mainly divided into two categories: local compression (also called hard compression) and global compression (also called soft compression).

[0072] Among them, local compression usually relies on LLM to measure the importance of each token contained in a long text sequence (such as logits or perplexity, which can measure the importance of each token) to filter out the feature representations corresponding to those unimportant tokens contained in the learned context representation, and then eliminate the redundant content contained in the learned context representation as much as possible to achieve local compression of the context representation.

[0073] Although local compression allows the compressed context representation to retain local semantics, it may sacrifice fluency and context coherence, and the filtered content is likely to contain key semantics, resulting in local semantic loss.

[0074] Global compression can usually extract global context information from the feature representation contained in the context representation from a global perspective, and then encode the extracted global context information into a low-dimensional feature representation to achieve global compression of the context representation.

[0075] Global compression can achieve higher compression rates and scalability compared to local compression, but global compression often disrupts the sequential structure of the original text sequence, does not reduce the interpretability of the feature representation after compression, and may also ignore local key semantics.

[0076] It can be seen that the current compression method for context representation cannot guarantee the semantic integrity of the compressed context representation, and there may be problems with semantic loss.

[0077] Based on this, this specification proposes a deep learning network that adopts a hybrid compression mechanism that supports both local compression and global compression.

[0078] The deep learning network may include a context learning network for learning context representation from an input text sequence; a compression network for compressing the context representation; and a generation network for generating a target text sequence based on the compressed context representation; wherein the compression network includes a first compression network for locally compressing the context representation; and a second compression network for globally compressing the context representation;

[0079] When using the deep learning network to process a long text sequence, first, the text sequence can be input into a context learning network so that the context learning network learns a context representation from the text sequence and obtains the context representation output by the context learning network. The context representation is then input into a first compression network and a second compression network included in a compression network, respectively, so that the first compression network performs local compression processing on the context representation to obtain a first compressed context representation that retains the local semantics of the context representation, and the second compression network performs global compression processing on the context representation to obtain a second compressed context representation that retains the global semantics of the context representation.

[0080] Secondly, the first compressed context representation output by the first compression network and the second compressed context representation output by the second compression network can be obtained, and the first compressed context representation and the second compressed context representation can be further input into the generation network so that the generation network can generate a target text sequence based on the first compressed context representation and the second compressed context representation.

[0081] In this deep learning network, by simultaneously introducing a network structure for performing local and global compression processing on the context representation learned from the text sequence, the compressed context representation can retain both local and global semantics. Therefore, on the basis of effective compression of the context representation, the local and global semantics in the context representation can be integrated to the greatest extent. This not only improves the computational efficiency of the deep learning network, but also ensures the semantic integrity of the context representation learned by the deep learning network from a longer text sequence to the greatest extent, avoiding the problem of semantic loss in the learned context representation.

[0082] See Figure 1 , Figure 1This specification provides a flowchart of a context compression method for a deep learning network; wherein the deep learning network uses a hybrid compression mechanism that supports both local and global compression to compress context representations learned from an input text sequence; the deep learning network includes a context learning network; a compression network; and a generation network; wherein the compression network includes a first compression network for locally compressing the context representation; and a second compression network for globally compressing the context representation; the method includes the following execution process:

[0083] Step 102, obtaining a text sequence;

[0084] The above-mentioned text sequence may specifically include a long text sequence; for example, it may be a text sequence containing a relatively large number of tokens (ie, text words); for example, it may be a text sequence containing tens of thousands or even more tokens.

[0085] The specific type of the above text sequence is not particularly limited in this specification.

[0086] For example, in actual applications, the above text sequence may specifically include long literary works, legal documents, instruction manuals, code documents, etc., which will not be listed one by one in this specification.

[0087] It should be noted that when using the above-mentioned deep learning network to process long text sequences, the roles played by the obtained text sequence to be processed are usually different in the settlement phase of deep learning network training and the inference calculation phase based on the completed deep learning phase.

[0088] For example, during the training phase of the aforementioned deep learning network, the text sequences to be processed typically serve as training samples. For example, assuming that the training of the deep learning network includes pretraining and fine-tuning phases, the text sequences to be processed during the pretraining phase can serve as pretraining samples. During the fine-tuning phase, the text sequences to be processed can serve as fine-tuning training samples.

[0089] After the training of the above-mentioned deep learning network is completed, the deep learning network is used to perform inference calculations. At this time, the text sequence to be processed obtained usually serves as an inference sample.

[0090] That is to say, the text sequence obtained when executing step 102 can be a text sequence used as a training sample in the stage of training the above-mentioned deep learning network; and can be a text sequence used as an inference sample in the stage of using the deep learning network for inference calculation.

[0091] Step 104: input the text sequence into the context learning network so that the context learning network learns context representation from the text sequence;

[0092] Step 106: Obtain the context representation output by the context learning network, and input the context representation into the first compression network and the second compression network, respectively, so that the first compression network performs local compression processing on the context representation to obtain a first compressed context representation that retains the local semantics of the context representation, and the second compression network performs global compression processing on the context representation to obtain a second compressed context representation that retains the global semantics of the context representation;

[0093] See Figure 2 , Figure 2 A schematic diagram of a deep learning network shown in this specification;

[0094] like Figure 2 As shown, in this deep learning network, a hybrid compression mechanism that supports both local and global compression processing can be used to compress the context representation learned from the input text sequence, which can specifically include a context learning network, a compression network, and a generation network.

[0095] The above-mentioned context learning network can be specifically used to learn context representation from an input text sequence.

[0096] It should be noted that contextual representations generally refer to vectors encoded from contextual information learned from the input text sequence. For example, if the deep learning network described above is an LLM using the Transformer architecture, the full name of the contextual representations is context-aware representations.

[0097] In some embodiments, the above context representation may generally include a plurality of feature representations corresponding one-to-one to each token in the text sequence.

[0098] It should be explained that the above-mentioned feature representation specifically refers to the feature vector obtained by encoding the features (also called hidden states) learned from each token contained in the text sequence.

[0099] For example, taking the aforementioned deep learning network as an LLM using the Transformer architecture, the LLM's Self-Attention mechanism typically calculates a query vector, key vector, and value vector for each token. Based on the query vector and key vector calculated for each token, an attention weight is calculated for that token. This attention weight is then weighted with the value vector calculated for that token to obtain the feature representation corresponding to that token. The value vector typically corresponds to the features learned from each token, and the feature representation corresponding to that token typically refers to the feature vector obtained by weighting the value vector corresponding to that token and the attention weight corresponding to that token.

[0100] The compression network is used to further compress the context representation output by the context learning network. In practical applications, a hybrid compression mechanism that supports both local and global compression can be introduced into the compression network, which can specifically include a first compression network and a second compression network.

[0101] The first compression network is used to perform local compression processing on the context representation to obtain a first compressed context representation that retains the local semantics of the context representation. The second compression network is used to perform global compression processing on the context representation to obtain a second compressed context representation that retains the global semantics of the context representation.

[0102] It should be noted that the first compressed context representation refers to the compressed context representation obtained by the context learning network through local compression of the original context representation, while preserving the local semantics of the context representation. Correspondingly, the second compressed context representation refers to the compressed context representation obtained by the compression network through global compression of the original context representation, while preserving the global semantics of the context representation.

[0103] In some embodiments, the local compression processing may include calculating an importance index for each feature representation included in the context representation, and filtering, based on the importance index, the feature representations included in the context representation whose values ​​for the importance index are below a threshold. The global compression processing may include further extracting global context information from the feature representations included in the context representation, and encoding the global context information into a low-dimensional feature representation.

[0104] Specifically, the aforementioned importance indicators can be metrics that the deep learning network itself has for measuring the importance of each token contained in a long text sequence. For example, if the deep learning network is an LLM using the Transformer architecture, the aforementioned importance indicators can be metrics such as logits or perplexity that the LLM itself has, which can measure the importance of each token. Of course, in practical applications, a mapping function based on several learnable parameters can also be introduced into the classification layer, and the feature representation can be directly mapped to the importance indicator through this mapping function.

[0105] The above-mentioned generative network is used to generate a target text sequence based on the context representation compressed by the compression network.

[0106] For example, in practical applications, the generation network may use the first compression context representation output by the first compression network and the second compression context representation output by the second compression network as input data to generate a target text sequence.

[0107] It needs to be emphasized that Figure 2 The specific network types of the context learning network, compression network, and generation network in the deep learning network shown are no longer specifically limited in this specification. In practical applications, they can be flexibly designed based on specific needs.

[0108] See Figure 3 , Figure 3 This is a schematic diagram of another deep learning network shown in this specification.

[0109] like Figure 3 As shown, in some embodiments, the first compression network included in the above compression network may specifically be a classification network.

[0110] For example, in practical applications, in order to simplify the network, a multi-layer classification network can be avoided and a single-layer classification network such as a classification head can be used instead.

[0111] The above-mentioned classification network can be specifically used to calculate the retention probability of each feature representation contained in the above-mentioned context representation, and assign classification labels to each feature representation contained in the context representation based on the calculated retention probability, and then filter out some feature representations with low retention probability based on the classification label to complete local compression of the context representation.

[0112] In some embodiments, the importance index used by the classification network when performing local compression processing on the above-mentioned context representation can specifically adopt the index of the deep learning network itself that measures the importance of each token contained in the long text sequence; for example, taking the above-mentioned deep learning network as an LLM using the Transformer architecture as an example, the above-mentioned importance index can adopt the logits or perplexity of the LLM itself that can measure the importance of each token.

[0113] In this case, the classification network can directly use the indicator as the retention probability, or further process the indicator to convert it into a numerical value in the range of [0,1] as the retention probability, and then assign classification labels to each feature representation contained in the context representation based on the retention probability, and then filter out some feature representations with lower retention probabilities based on the classification labels to complete local compression of the context representation.

[0114] In some embodiments, the classification network can also specifically introduce some learnable parameters for mapping feature representations into importance indicators, and construct a mapping function based on these learnable parameters, and then the feature representation can be directly mapped into retention probability through the mapping function.

[0115] The specific type of the mapping function and the types of learnable parameters contained in the mapping function are no longer specifically limited in this specification. In practical applications, they can be flexibly designed based on specific needs.

[0116] For example, in one example, assume that the above context is expressed as follows:

[0117] H={h1,h2,…h i}

[0118] Among them, hi represents the feature representation corresponding to the i-th token in the text sequence.

[0119] In this case, the classification network can be specifically a linear projection layer, in which a mapping function of the following form can be introduced:

[0120] P=σ(WH+b)

[0121] In this mapping function, P represents the retention probability; σ represents the Sigmoid function, which is used to map the output value to the range of [0, 1]; W and b represent the weight matrix and bias vector used by the linear projection layer, respectively, which are learnable parameters that map the feature representation to the retention probability.

[0122] Finally, the linear projection layer can perform mapping calculations on each feature representation contained in the context representation through the mapping function to obtain the retention probability vector corresponding to the context representation:

[0123] P={p1,p2,…pi}

[0124] Where pi represents the retention probability corresponding to the i-th token in the text sequence.

[0125] In some embodiments, the above text sequence may include not only text sequence content but also text instructions corresponding to the text sequence content. The text instructions are generally used to guide the learning process for the text sequence content.

[0126] In this scenario, the retention probability of each feature representation contained in the context representation can be specifically determined based on the relevance of each feature representation with the above-mentioned text instruction. In the process of calculating the retention probability of each feature representation contained in the context representation, the classification network can first calculate the relevance of each feature representation contained in the context representation with the above-mentioned text instruction; then, based on the calculated relevance, the retention probability of each feature representation contained in the context representation can be further calculated; wherein, the retention probability of each feature representation can generally be positively correlated with the relevance of each feature representation with the above-mentioned text instruction. The specific calculation method for calculating the relevance of each feature representation contained in the context representation with the above-mentioned text instruction will not be introduced in detail in this specification, and those skilled in the art can refer to the records in the relevant technology.

[0127] In this way, the retention probability of each feature representation can be positively correlated with the relevance of each feature representation to the above-mentioned text instructions, so that the classification network can give priority to retaining the feature representations that are more relevant to the above-mentioned text instructions, that is, more important to the text instructions, during the local compression process of the context representation.

[0128] In some embodiments, the classification network may be a binary classification network, and the classification labels used by the binary classification network may include a first classification label for indicating a retained feature representation and a second classification label for indicating a non-retained feature representation; for example, Figure 3 As shown, the classification label 1 can be used to indicate that the feature representation is retained, and the classification label 0 can be used to indicate that the feature representation is not retained.

[0129] In some embodiments, when the classification network assigns classification labels to each feature representation contained in the context representation based on the calculated retention probability, it can first determine the proportion of feature representations that need to be retained contained in the context representation based on a preset compression rate; for example, assuming that the preset compression rate is K%, the proportion of feature representations that need to be retained contained in the context representation is K%.

[0130] Then, based on the above proportion, the number of feature representations that need to be retained contained in the context representation can be further determined, and the feature representations with the highest retention probability can be screened from the context representation according to the data; for example, assuming that the proportion is K%, it means that the top K% feature representations with the highest retention probability contained in the context representation need to be retained, and the classification network can screen out the top K% feature representations with the highest retention probability based on the proportion.

[0131] Finally, the first type of assigned labels may be assigned to the filtered feature representations, and the second type of assigned labels may be assigned to the feature representations other than the filtered feature representations.

[0132] For example, assuming that the top K% feature representations with the highest retention probability contained in the context representation need to be retained, the classification network can assign the first category classification label to the top K% feature representations with the highest retention probability that are screened out, and assign the second category classification label to other feature representations other than the top K% feature representations with the highest retention probability that are screened out.

[0133] In this case, when the classification network performs local compression processing on the context representation, the feature representation assigned the second classification label contained in the context representation can be filtered out based on the classification label, and only the feature representation assigned the first classification representation is retained.

[0134] In some embodiments, the second compression network may be an attention network that can utilize an attention mechanism to further extract global context information from feature representations included in the context representation.

[0135] Among them, in order to enable the second compression network to comprehensively extract global context information from the feature representation, the above-mentioned second compression network can be designed as a network composed of multiple attention networks, so that the multiple attention networks can be used to extract context from different perspectives, and then the context extracted by the multiple attention networks from different perspectives are integrated as the output of the second compression network.

[0136] Please continue to see Figure 3 In some embodiments, the second compression network may specifically include a first attention network, a second attention network, and a weight network.

[0137] Among them, the above-mentioned first attention network is responsible for extracting contextual information from a local perspective. Specifically, it can be used to divide the feature representations contained in the context representation into multiple groups, and extract the local context information of each feature representation contained in each group in the group to which it belongs based on the attention mechanism, and then encode the extracted local context information into a low-dimensional local feature representation.

[0138] The second attention network is responsible for extracting contextual information from a global perspective. Specifically, it can be used to extract the global contextual information of each feature representation contained in the context representation based on the attention mechanism, and encode the global contextual information into a low-dimensional global feature representation.

[0139] By introducing a first attention network that extracts context information from a local perspective and a second attention network that extracts context information from a global perspective into the second compression network, the context information further extracted from the local and global perspectives can be effectively balanced, thereby improving the effectiveness of the second compression network in performing global compression processing on the context representation and avoiding the loss of semantics after global compression processing of the context representation; for example, by integrating the context information extracted by the first attention network and the second attention network, the context information finally extracted includes both local context and global context.

[0140] The above-mentioned weight network is used to further learn the compression weights corresponding to the first attention network and the second attention network respectively based on the context representation learned by the context learning network from the text sequence, and perform weighted calculation on the local feature representation output by the first attention network and the global feature representation output by the second attention network based on the learned compression weights to obtain the above-mentioned second compressed context representation.

[0141] Please continue to see Figure 3 In some embodiments, when learning the compression weights corresponding to the first attention network and the second attention network respectively, the above-mentioned weight network can specifically use the context representation learned by the context learning network from the text sequence as input, and dynamically learn the compression weights corresponding to the first attention network and the second attention network respectively from the context representation.

[0142] In other words, the weight network can dynamically learn compression weights corresponding to the first and second attention networks based on the actual semantics of the text sequence. When the contextual representation input to the weight network changes, the compression weights corresponding to the first and second attention networks learned by the weight network based on this contextual representation will also change dynamically.

[0143] For example, in practical applications, a weight function based on several learnable parameters can be introduced into the weight network, and the context representation learned by the context learning network from the text sequence can be used as the input of the weight function to calculate the corresponding compression weights for the first attention network and the second attention network respectively.

[0144] In some embodiments, the weight network may specifically use the following weight function to calculate the corresponding compression weights for the first attention network and the second attention network respectively:

[0145]

[0146] in, represents the compression weight, Softmax refers to the normalized exponential function, V refers to the above context representation, W g refers to the learnable weight matrix.

[0147] In some embodiments, in order to prevent the weight network from being excessively biased towards one of the first attention network and the second attention network during the calculation of the compression weight, learnable noise can be introduced into the weight function. In this case, the weight network can specifically calculate the corresponding compression weights for the first attention network and the second attention network using the following weight function:

[0148]

[0149] In the above formula, is the noise term, given by and Softplus((V·W noise )) consists of two parts. refers to the standard normal distribution noise, Softplus refers to the nonlinear activation function, W noise is the learnable noise weight matrix.

[0150] It should be noted that in practical applications, W g and W noise Both can be a shared weight matrix, but the first attention network and the second attention network can be respectively in W g and W noise The corresponding different rows or columns in , so that different compression weights can be calculated for the first attention network and the second attention network based on the above weight function.

[0151] For example, in practical applications, a parameter i can be introduced into the above weight function to distinguish the weight matrix corresponding to the first attention network and the second attention network from the noise weight matrix. In this case, when calculating the compression weight of the first attention network based on the above weight function, the weight function can be expressed as follows:

[0152]

[0153] The above formula means that W g The first row or column in will be used as the weight matrix of the first attention network to be input into the above weight function to participate in the calculation, W noise The first row or column in will be used as the noise weight matrix of the first attention network and input into the above weight function to participate in the calculation.

[0154] Similarly, when calculating the compression weight of the first attention network based on the above weight function, the weight function can be expressed as follows:

[0155]

[0156] The above formula means that W g The second row or column in will be used as the weight matrix of the second attention network to be input into the above weight function to participate in the calculation, W noise The first row or column in will be used as the noise weight matrix of the second attention network and input into the above weight function to participate in the calculation.

[0157] After the above-mentioned weight network calculates the compression weights corresponding to the first attention network and the second attention network respectively based on the above-mentioned weight function, the local feature representation output by the first attention network and the global feature representation output by the second attention network can be weightedly calculated based on the learned compression weights to obtain the above-mentioned second compressed context representation.

[0158] For example, suppose you use Denotes the compression weight corresponding to the first attention network above, and represents the compression weight corresponding to the first attention network. At this time, the second compressed context representation finally output by the second compression network can be calculated as follows:

[0159]

[0160] Among them, f1(V) represents the local feature representation learned by the first attention network, and f2(V) represents the global feature representation learned by the second attention network.

[0161] Please continue to see Figure 3In some embodiments, in addition to the text sequence content, the above-mentioned text sequence may also include text instructions corresponding to the text sequence content. In this case, in order to enhance the deep learning network's perception of the above-mentioned text instructions, the attention mechanism adopted by the above-mentioned first attention network and the above-mentioned second attention network can specifically be a hybrid attention mechanism that integrates the cross-attention mechanism between the various feature representations contained in the context representation and the text instructions.

[0162] That is to say, the cross-attention mechanism between the various feature representations contained in the above-mentioned context representation and the text instruction can be integrated into the self-attention mechanism adopted by the above-mentioned first attention network and the above-mentioned second attention network.

[0163] By adopting this hybrid attention mechanism, deep learning networks can focus on the contextual dependencies within the input text sequence when processing text sequences, and capture the correlation between the input text sequence and text instructions, thereby improving the deep learning network's understanding and reasoning ability of task intent.

[0164] In some embodiments, when extracting local context information of each feature representation contained in each of the plurality of groups based on the attention mechanism, the first attention network may specifically fuse the feature representations contained in each of the plurality of groups into a combined feature representation corresponding to each group.

[0165] For example, in one example, average pooling can be used to fuse the feature representations contained in each group into one feature representation as a combined feature representation that can represent each group.

[0166] Furthermore, based on the above-mentioned hybrid attention mechanism, the cross-attention weights between the combined feature representations corresponding to each group and the above-mentioned text instructions can be first calculated, and then the calculated cross-attention weights can be used as calculation parameters to further calculate the self-attention weights (which can be called local self-attention weights) of each feature representation contained in each group in the group to which it belongs. Then, based on the self-attention weights, the local context information of each feature representation contained in each group in the group to which it belongs can be calculated.

[0167] In some embodiments, when the first attention network uses the above-mentioned cross-attention weight as a calculation parameter to further calculate the self-attention weight of each feature representation contained in each group in the group to which it belongs, the cross-attention weight can be used as a query vector corresponding to each feature representation contained in each group to further calculate the self-attention weight of each feature representation contained in each group in the group to which it belongs.

[0168] For example, in one example, the first attention network can use the following formula to calculate the local context information of each feature contained in each group in the group to which it belongs:

[0169]

[0170] In the above formula, f1(V) represents the local context information calculated by the first attention network; Attn represents the standard self-attention mechanism; CrossAttn represents the cross-attention mechanism; Assuming that a total of i groups are divided, V i represents the i-th group, represents the combined feature representation corresponding to the i-th group; C represents the above text instruction; Represents the cross attention weight between the combined feature representation corresponding to the i-th group and the above text instructions. This cross attention weight will be used as the self-attention weight of the query vector calculation standard.

[0171] In some embodiments, when the second attention network extracts the global context information of each feature representation contained in the context representation based on the attention mechanism, it can first calculate the cross-attention weights between each feature representation contained in the context representation and the text instruction based on the hybrid attention mechanism, and then use the cross-attention weight as a calculation parameter to further calculate the self-attention weights of each feature representation contained in the context representation in the context representation (which can be called global self-attention weights), and then calculate the global context information of each feature representation contained in the context representation in the context representation based on the self-attention weights.

[0172] In some embodiments, when the second attention network uses the cross-attention weight as a calculation parameter to further calculate the self-attention weights of each feature representation contained in the context representation in the context representation, it can also specifically use the cross-attention weight as a query vector corresponding to each feature representation contained in the context representation to further calculate the self-attention weights of each feature representation contained in the context representation in the context representation.

[0173] It should be noted that, in practical applications, if the relative position of each token in the text sequence is not considered in the above-mentioned second attention network, the relative position of each token in the text sequence can be introduced when calculating the self-attention weights of each feature representation contained in the context representation in the context representation to enhance the calculation.

[0174] For example, in one example, the second attention network can specifically use the following formula to calculate the global context information of each feature representation contained in the context representation:

[0175]

[0176] In the above formula, f2(V) represents the global context information calculated by the second attention network; Attn represents the standard self-attention mechanism; CrossAttn represents the cross-attention mechanism; V represents the feature representation corresponding to each token contained in the above context representation; C represents the above text instruction; CrossAttn(V, C) represents the cross-attention weight between the feature representation corresponding to each token contained in the context representation and the above text instruction, and the cross-attention weight will be used as the standard self-attention weight for Query vector calculation; Pos(V) represents the relative position of the token corresponding to each feature representation contained in the context representation in the text sequence, and the relative position can be specifically added to the key vector as an enhancement item.

[0177] See Figure 4 , Figure 4 A schematic diagram of another deep learning network shown in this specification

[0178] like Figure 4 As shown, in some embodiments, the compression network may specifically be a hybrid expert network using a MoE (Mixture of experts) architecture.

[0179] In this scenario, the hybrid expert network may specifically include a first attention network as an expert network, a second attention network as an expert network, and a weight network as a gating network.

[0180] In other words, when the compression network adopts the MoE architecture, the first and second attention networks can function as the expert networks within the hybrid expert network, while the weight network can function as the gating network within the hybrid expert network. The gating network learns a set of weights for each expert network to represent its contribution to the output of the hybrid expert network. The learned weights are then used to perform a weighted calculation on the outputs of each expert network to produce the output of the hybrid expert network.

[0181] It should be noted that the specific network types of the first attention network and the second attention network are no longer specifically limited in this specification. In practical applications, they can be flexibly designed based on specific needs.

[0182] For example, in practical applications, in order to simplify the network, the first attention network, the second attention network, and the weight network can avoid using networks with complex structures.

[0183] In some embodiments, the first attention network may include an MLP (Multilayer Perceptron) network that incorporates an attention mechanism; the second attention network may include a Q-former network. The MLP network is a classic feedforward artificial neural network whose core feature is the nonlinear mapping of input data through multiple layers of connected neurons, ultimately outputting a prediction result. The Q-former network is a lightweight transformer-based network.

[0184] MLP networks typically have efficient local feature extraction capabilities, but they often lack a global perspective. Unlike MLP networks, Q-former networks can capture global semantics through their inherent attention mechanism, but this comes with high computational complexity. Therefore, by using an MLP network with an attention mechanism as the first attention network and a Q-former network as the second attention network, the characteristics of these two networks can be complemented and balanced, significantly improving the second compression network's ability to extract global semantics.

[0185] The above-mentioned weight network can specifically include a general FFN (Feedforward Neural Network) network. The FFN can normalize its output into a probability distribution through a softmax function, thereby dynamically assigning weights to the outputs of the above-mentioned first attention network and the above-mentioned second attention network.

[0186] Step 108: Obtain the first compression context representation output by the first compression network and the second compression context representation output by the second compression network, and further input the first compression context representation and the second compression context representation into the generation network so that the generation network generates a target text sequence based on the first compression context representation and the second compression context representation.

[0187] After using the compression network to compress the context representation learned by the context learning network from the input text sequence, the first compressed context representation output by the first compression network and the second compressed context representation output by the second compression network can be obtained. Then, the obtained first compressed context representation and the second compressed context representation can be further input into the generation network, so that the three networks can generate the target text sequence based on the first compressed context representation and the second compressed context representation.

[0188] Among them, the specific process of generating the target text sequence by the generation network based on the first compressed context representation and the second compressed context representation will not be described in detail in this specification. Those skilled in the art can refer to the records in the relevant technology.

[0189] It should be noted that the above compression network can be used as a standalone context compression framework, adapting to different deep learning networks. In practical applications, it can be embedded within a deep learning network as a standalone context compression framework, or it can be connected to different deep learning networks across networks.

[0190] See Figure 5 , Figure 5 This is a schematic diagram of another deep learning network shown in this specification;

[0191] like Figure 5 As shown, in some embodiments, the above-mentioned deep learning network can specifically be an LLM using a Transformer architecture.

[0192] In this case, the context learning network may specifically be the encoder of the LLM; correspondingly, the generation network may specifically be the decoder of the LLM.

[0193] That is to say, if the above compression network is understood as an independent context compression framework, then the context compression framework can be implanted into the LLM using the Transformer architecture to compress the context representation learned by the LLM encoder from the text sequence.

[0194] See Figure 6 , Figure 6 This is a schematic diagram of another deep learning network shown in this specification;

[0195] like Figure 6 As shown, in some embodiments, the above-mentioned deep learning network can also be a cross-model network composed of two different LLMs.

[0196] In this case, the context learning network may specifically be a first LLM for learning context representation from an input text sequence; correspondingly, the generation network may specifically be a second LLM for generating a target text sequence based on the compressed context representation.

[0197] That is to say, if the above compression network is understood as an independent context compression framework, then the context compression network can be connected with two different LLMs to achieve complementarity in model functions between the two LLMs.

[0198] For example, suppose that the first LLM is an LLM with a relatively strong ability to learn contextual representations from the input text sequence, but the LLM has a relatively weak ability to generate a target text sequence based on the learned contextual representations; and the second LLM is an LLM with a relatively strong ability to generate a target text sequence based on the learned contextual representations, but its own ability to learn contextual representations from the input text sequence is relatively weak.

[0199] In this scenario, the first LLM and the second LLM are connected through the above-mentioned context compression network. On the one hand, the first LLM can be used to learn context representation from the input text sequence. On the other hand, after compressing the context representation learned by the first LLM from the input text sequence, the context compression network can also utilize the second LLM's ability to generate a target text sequence based on the context representation, and generate a target text sequence based on the compressed context representation, thereby achieving complementarity between the first LLM and the second LLM in the two aspects of learning context representation and generating target text sequences.

[0200] In some embodiments, to improve the scalability of the deep learning network, the compression network can be trained independently from the context learning network and the generative network included in the deep learning network during training. That is, while the compression network can be used as an independent context compression to adapt to different deep learning networks, it can be trained independently during training, without the need to train the compression network as a whole with the context learning network and the generative network included in the deep learning network.

[0201] When the compression network is trained independently, since the first compression network and the second compression network included in the compression network perform context compression from a local and a global perspective respectively, if the first compression network and the second compression network included in the compression network are trained as a whole, it may usually make it difficult to achieve optimal convergence in the training process of the compression network.

[0202] For example, in practical applications, if the first compression network and the second compression network contained in the compression network are trained as a whole, the model may prioritize learning the global semantics of the context representation (the global semantics that need to be retained are easier to learn than the local semantics), resulting in difficulty in achieving optimal convergence in the training process of the compression network.

[0203] In some embodiments, the training process for the compressed network typically includes pre-training and fine-tuning training in the target scenario. To ensure optimal convergence during the training process for the compressed network, a three-stage training strategy can be designed, where training for the first compressed network is restricted to the first stage, and training for the second compressed network is restricted to a second stage after the first stage.

[0204] Based on this three-stage training strategy, the training process of the compression network may specifically include:

[0205] First stage of training: first remove the first compression network and pre-train the second compression network independently;

[0206] In the second stage of training, after pre-training the second compressed network is completed, the first compressed network can be added to the compressed network, and the parameters contained in the pre-trained second compressed network are frozen, and the pre-training of the first compressed network is continued;

[0207] In the third stage of training, after the pre-training of the first compression network and the second compression network is completed, the first compression network and the second compression network can be further fine-tuned together in the target application scenario.

[0208] It should be noted that during the training of the first and second compressed networks, the tasks performed by the second compressed networks can be flexibly designed in practical applications. For example, tasks related to the functions and capabilities of the networks can be designed for the first and second compressed networks.

[0209] In some embodiments, the pre-training task corresponding to the first compression network may specifically include a completion task.

[0210] Among them, the completion task can specifically refer to the task of completing the missing information in the original context representation of the input; for example, in practical applications, a mask can be used to mask part of the information in the original context representation of the input, and the first compression network can be used to complete the masked information.

[0211] In this case, the optimization objective of pre-training the first compression network may include minimizing the error between the information completed by the first compression network and the information actually missing from the input original context representation. Accordingly, the pre-training loss corresponding to the completion task may specifically include minimizing the error between the information completed by the first compression network and the information actually missing from the input original context representation.

[0212] In some embodiments, the pre-training task corresponding to the second compression network may specifically include a paraphrase task.

[0213] The interpretation task may specifically refer to a task of generating a compressed context representation having semantic similarity to the original context representation based on the input original context representation.

[0214] In this case, the optimization goal of pre-training the second compression network may include: minimizing the similarity error between the original context representation of the input and the compressed context representation generated by the second compression network based on the original context representation; wherein the similarity can generally be a similarity value between 0 and 1, and the similarity error can specifically refer to the difference between the similarity between the original context representation of the input and the compressed context representation generated by the second compression network based on the original context representation and 1 (that is, 100%).

[0215] Correspondingly, the pre-training loss corresponding to the interpretation task may specifically include: a similarity error between the original context representation of the input and the compressed context representation generated by the second compression network based on the original context representation.

[0216] In some embodiments, the fine-tuning tasks corresponding to the first compression network and the second compression network may specifically include sequence generation tasks corresponding to the target application scenario; it should be noted that the application scenario for fine-tuning the compression network usually depends on specific needs in actual applications and will not be specifically restricted in this specification.

[0217] In this case, the optimization goal of fine-tuning the first compression network and the second compression network can specifically include: minimizing the distribution difference between a first probability distribution (that is, conditional distribution) of a sequence generated by performing the sequence generation task under the condition that the original context representation is used as input data, and a second probability distribution of a sequence generated by performing the sequence generation task under the condition that the compressed context representation is used as input data.

[0218] For example, in practical applications, the difference between the first probability distribution and the second probability distribution can be represented by calculating a divergence index (such as KL divergence) between the first probability distribution and the second probability distribution.

[0219] It should be noted that since the function of the compression network is to compress the contextual representation of the input, the final output of the compression network performing the sequence generation task may have the risk of semantic loss, resulting in the omission of some important contextual information.

[0220] In some embodiments, when fine-tuning the first compression network and the second compression network together, a sequence generated by another pre-trained deep learning model that performs a sequence generation task as a teacher model can be introduced as a constraint item in the optimization objective of the fine-tuning training.

[0221] In this case, the optimization objectives for fine-tuning the first and second compression networks may specifically include minimizing a first distribution difference between the first and second probability distributions, and minimizing a second distribution difference between a third probability distribution of a sequence generated by another pre-trained teacher model performing the sequence generation task using the original context representation as input data. Accordingly, the fine-tuning training loss corresponding to the sequence generation task may specifically include the first and second distribution differences.

[0222] In this way, the sequence generated by the compression network based on the compressed context representation to perform the sequence generation task can be forced to align with the sequence generated by the teacher model based on the original context representation to perform the sequence generation task, thereby improving the fine-tuning training effect of the compression network.

[0223] It should be noted that, in practical applications, the specific type of the above-mentioned teacher model is no longer specifically limited in this specification; for example, in one example, the above-mentioned teacher model may specifically include RAG (Retrieval-augmented Generation, retrieval enhancement generation model).

[0224] In the above technical solution, by simultaneously introducing a network structure for performing local compression and global compression processing on the context representation learned from the text sequence into the deep learning network, the compressed context representation can retain both local and global semantics. Therefore, on the basis of effective compression of the context representation, the local and global semantics in the context representation can be integrated to the greatest extent. This not only improves the computational efficiency of the deep learning network, but also ensures the semantic integrity of the context representation learned by the deep learning network from a longer text sequence to the greatest extent, thereby avoiding the problem of semantic loss in the learned context representation.

[0225] Corresponding to the embodiments of the aforementioned method, this specification also provides embodiments of an apparatus, an electronic device, and a storage medium.

[0226] Figure 7 This is a schematic structural diagram of an electronic device provided by an exemplary embodiment. Figure 7At the hardware level, the device includes a processor 702, an internal bus 704, a network interface 706, a memory 708, and a non-volatile memory 710, and may also include other required hardware. One or more embodiments of this specification can be implemented based on software, such as the processor 702 reading the corresponding computer program from the non-volatile memory 710 into the memory 708 and then running it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0227] like Figure 8 As shown, Figure 8 This is a block diagram of a context compression device for deep learning networks according to an exemplary embodiment of the present specification. The device can be operated in Figure 7 The electronic device shown in the figure implements the technical solution of this specification. The deep learning network uses a hybrid compression mechanism that supports both local compression and global compression to compress the context representation learned from the input text sequence; the deep learning network includes a context learning network; a compression network; and a generation network; the compression network includes a first compression network for locally compressing the context representation; and a second compression network for globally compressing the context representation; the device 80 includes:

[0228] Acquisition module 801, acquiring a text sequence;

[0229] A first input module 802 inputs the text sequence into the context learning network so that the context learning network learns context representation from the text sequence;

[0230] A second input module 803 acquires the context representation output by the context learning network, and inputs the context representation into the first compression network and the second compression network, respectively, so that the first compression network performs local compression processing on the context representation to obtain a first compressed context representation that retains the local semantics of the context representation, and the second compression network performs global compression processing on the context representation to obtain a second compressed context representation that retains the global semantics of the context representation;

[0231] The third input module 804 obtains the first compression context representation output by the first compression network and the second compression context representation output by the second compression network, and further inputs the first compression context representation and the second compression context representation into the generation network, so that the generation network generates a target text sequence based on the first compression context representation and the second compression context representation.

[0232] Accordingly, this specification also provides an electronic device, which includes a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the steps in all the method flows described above.

[0233] Accordingly, this specification also provides a computer-readable storage medium on which executable computer program instructions are stored; wherein, when the instructions are executed by a processor, the steps in all the method flows described above are implemented.

[0234] Accordingly, this specification also provides a computer program product having executable computer program instructions stored thereon; wherein, when the computer program instructions are executed by a processor, the steps in all the method flows described above are implemented.

[0235] In the 1990s, technological improvements could be clearly distinguished as either hardware improvements (for example, improvements to circuit structures like diodes, transistors, and switches) or software improvements (improvements to process flows). However, with the advancement of technology, many process flow improvements today can now be considered direct improvements to hardware circuit structures. Designers almost always create the corresponding hardware circuit structure by programming the improved process flow into the hardware circuit. Therefore, it cannot be said that a process flow improvement cannot be implemented using hardware modules. For example, a programmable logic device (PLD), such as a field programmable gate array (FPGA), is an integrated circuit whose logical function is determined by user programming. Designers can "integrate" a digital system on a PLD through their own programming, without having to hire a chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly done using "logic compiler" software. This is similar to the software compiler used when developing programs. Before compilation, the original code must also be written in a specific programming language, called a hardware description language (HDL). There is not just one HDL, but many, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art will also understand that by simply programming the method flow in one of these hardware description languages ​​and then programming it into an integrated circuit, a hardware circuit that implements the logic method flow can be easily obtained.

[0236] The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that in addition to implementing the controller in a purely computer-readable program code format, the controller can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules that implement the method and structures within the hardware component.

[0237] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a server system. Of course, this application does not exclude that with the future development of computer technology, the computer that implements the functions of the above embodiments may be, for example, a personal computer, a laptop computer, an in-vehicle human-computer interaction device, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0238] Although one or more embodiments of this specification provide method operation steps as described in the embodiments or flow charts, more or fewer operation steps may be included based on conventional or non-creative means. The order of steps listed in the embodiments is only one way of executing the order of many steps and does not represent the only execution order. When the device or terminal product in practice is executed, it can be executed in sequence or in parallel according to the method shown in the embodiments or the drawings (for example, a parallel processor or a multi-threaded processing environment, or even a distributed data processing environment). The term "comprise", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, product or equipment including a series of elements includes not only those elements, but also includes other elements that are not clearly listed, or also includes elements inherent to such process, method, product or equipment. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or equipment including the elements. For example, if the words first, second, etc. are used to represent the name, they do not represent any particular order.

[0239] For the convenience of description, the above devices are described in terms of functions divided into various modules. Of course, when implementing one or more of the present specifications, the functions of each module can be implemented in the same or multiple software and / or hardware, or the module that implements the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0240] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0241] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0242] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0243] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0244] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0245] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage, graphene storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0246] Those skilled in the art will appreciate that one or more embodiments of this specification may be provided as a method, system, or computer program product. Thus, one or more embodiments of this specification may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0247] One or more embodiments of this specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. One or more embodiments of this specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0248] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between the various embodiments can be referenced across them. Each embodiment focuses on the differences from the other embodiments. In particular, since the system embodiments are generally similar to the method embodiments, their description is relatively simple. For relevant parts, reference can be made to the description of the method embodiments. Throughout this specification, reference to the terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of this specification. In this specification, the schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, those skilled in the art may combine and integrate the different embodiments or examples, and features of different embodiments or examples, described in this specification, without conflict.

[0249] The foregoing is merely an example of one or more embodiments of this specification and is not intended to limit the one or more embodiments of this specification. It will be apparent to those skilled in the art that various modifications and variations may be made to one or more embodiments of this specification. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this specification shall be included within the scope of the claims.

Claims

1. A context compression method for a deep learning network, wherein the deep learning network uses a hybrid compression mechanism that supports both local and global compression to compress context representations learned from an input text sequence; the deep learning network includes a context learning network; Compressed network; and generating a network; wherein the compression network includes a first compression network for performing local compression processing on the context representation; and a second compression network for performing global compression processing on the context representation; the method includes: Get text sequence; Inputting the text sequence into the context learning network to learn context representation from the text sequence by the context learning; Obtaining the context representation output by the context learning network, and inputting the context representation into the first compression network and the second compression network, respectively, so that the first compression network performs local compression processing on the context representation to obtain a first compressed context representation that retains the local semantics of the context representation, and the second compression network performs global compression processing on the context representation to obtain a second compressed context representation that retains the global semantics of the context representation; The first compression context representation output by the first compression network and the second compression context representation output by the second compression network are obtained, and the first compression context representation and the second compression context representation are further input into the generation network so that the generation network generates a target text sequence based on the first compression context representation and the second compression context representation.

2. The method according to claim 1, wherein the text sequence comprises a plurality of text words; and the context representation comprises a plurality of feature representations corresponding one-to-one to the plurality of text words. in, The local compression processing includes: calculating an importance index of each feature representation included in the context representation, and filtering, based on the importance index, some feature representations included in the context representation whose values ​​of the importance index are lower than a threshold; The global compression process includes: further extracting global context information from the feature representation included in the context representation, and encoding the global context information into a low-dimensional feature representation.

3. In the method as claimed in claim 2, the first compression network includes a classification network; the classification network is used to calculate the retention probability of each feature representation contained in the context representation, and assign classification labels to each feature representation contained in the context representation based on the retention probability; the classification label includes a first classification label for indicating that the feature representation is retained and a second classification label for indicating that the feature representation is not retained; and, the feature representation contained in the context representation and assigned the second classification label is filtered out.

4. The method according to claim 3, wherein the text sequence comprises text sequence content and text instructions corresponding to the text sequence content; Calculating the retention probability of each feature representation included in the context representation includes: Calculating the relevance between each feature representation included in the context representation and the text instruction; The retention probability of each feature representation included in the context representation is further calculated based on the correlation; wherein the retention probability is positively correlated with the correlation.

5. The method of claim 3, wherein the second compression network comprises a first attention network, a second attention network, and a weight network; in, The first attention network is used to divide the feature representations included in the context representation into multiple groups, extract local context information of each feature representation included in each group in the multiple groups based on the attention mechanism, and encode the local context information into a low-dimensional local feature representation; The second attention network is used to extract global context information of each feature representation included in the context representation in the context representation based on the attention mechanism, and encode the global context information into a low-dimensional global feature representation; The weight network is used to learn compression weights corresponding to the first attention network and the second attention network respectively based on the context representation, and perform weighted calculation on the local feature representation and the global feature representation based on the learned compression weights to obtain the second compressed context representation.

6. The method of claim 5, wherein the text sequence comprises text sequence content and text instructions corresponding to the text sequence content; and the attention mechanism comprises a hybrid attention mechanism that integrates a cross-attention mechanism between each feature representation included in the context representation and the text instructions; Extracting local context information of each feature representation contained in each of the plurality of groups in the group to which it belongs based on the attention mechanism, including: Fusing the feature representations contained in each of the plurality of groups into a combined feature representation corresponding to each group, and calculating a cross-attention weight between the combined feature representation corresponding to each group and the text instruction; Using the cross attention weight as a calculation parameter, further calculating the self-attention weight of each feature representation contained in each group in the group to which it belongs, and calculating the local context information of each feature representation contained in each group in the group to which it belongs based on the self-attention weight; Extracting global context information of each feature representation contained in the context representation based on the attention mechanism, including: Calculating a cross-attention weight between each feature representation included in the context representation and the text instruction; The cross-attention weight is used as a calculation parameter to further calculate the self-attention weight of each feature representation contained in the context representation in the context representation, and the global context information of each feature representation contained in the context representation in the context representation is calculated based on the self-attention weight.

7. The method of claim 6, further calculating the self-attention weight of each feature representation contained in each group in the group to which it belongs, using the cross-attention weight as a calculation parameter, comprising: Using the cross attention weight as the query vector corresponding to each feature representation contained in each group, further calculating the self-attention weight of each feature representation contained in each group in the group to which it belongs; Using the cross attention weight as a calculation parameter, further calculating the self-attention weight of each feature representation included in the context representation in the context representation, including: The cross-attention weight is used as the query vector corresponding to each feature representation contained in the context representation, and the self-attention weight of each feature representation contained in the context representation is further calculated.

8. The method of claim 6, wherein the training process of the compression network comprises: Pre-training; and, fine-tuning training in the target scenario; The training of the compression network includes: First stage of training: first remove the first compression network and pre-train the second compression network independently; The second stage of training includes: after pre-training the second compressed network is completed, adding the first compressed network and freezing the parameters of the second compressed network to pre-train the first compressed network; In the third stage of training, after pre-training of the first compression network and the second compression network is completed, fine-tuning training is performed on the first compression network and the second compression network together in the target application scenario.

9. The method of claim 8, wherein the pre-training task corresponding to the first compression network includes a completion task; the completion task is a task of completing missing information in the original context representation of the input; accordingly, the optimization objectives for pre-training the first compression network include: Minimizing the error between the information completed by the first compression network and the information actually missing in the input original context representation; The pre-training loss corresponding to the completion task includes: the error between the information completed by the first compression network and the information actually missing in the input original context representation; The pre-training task corresponding to the second compression network includes a paraphrase task; wherein the paraphrase task is a task of generating a compressed context representation that is semantically similar to the original context representation based on the input; accordingly, the optimization goal of pre-training the second compression network includes minimizing the similarity error between the original context representation of the input and the compressed context representation generated by the second compression network based on the original context representation; the pre-training loss corresponding to the paraphrase task includes minimizing the similarity error between the original context representation of the input and the compressed context representation generated by the second compression network based on the original context representation; The fine-tuning tasks corresponding to the first compression network and the second compression network include a sequence generation task corresponding to the target application scenario; accordingly, the optimization objectives of fine-tuning training of the first compression network and the second compression network include: minimizing the distribution difference between the first probability distribution of the sequence generated by executing the sequence generation task under the condition that the original context representation is used as input data, and the second probability distribution of the sequence generated by executing the sequence generation task under the condition that the compressed context representation is used as input data; accordingly, the fine-tuning training loss corresponding to the sequence generation task can specifically include: the distribution difference between the above-mentioned first probability distribution and the above-mentioned second probability distribution.

10. The method of claim 9, wherein the optimization objectives for fine-tuning the first compression network and the second compression network include: minimizing a first distribution difference between the first probability distribution and the second probability distribution; and, using the original context representation as input data of another pre-trained teacher model, and calculating a second distribution difference between a third probability distribution of a sequence generated by the teacher model when performing the sequence generation task and the second probability distribution; The fine-tuning training loss corresponding to the sequence generation task includes: the first distribution difference and the second distribution difference.

11. A deep learning network for context compression, comprising: Contextual learning network, used to learn contextual representations from input text sequences; A compression network, wherein the compression network includes a first compression network and a second compression network; the first compression network is used to perform local compression processing on the input context representation to obtain a first compressed context representation that retains the local semantics of the context representation; the second compression network is used to perform global compression processing on the context representation to obtain a second compressed context representation that retains the global semantics of the context representation; A generative network is used to generate a target text sequence based on the first compressed context representation and the second compressed context representation of the input.

12. The deep learning network of claim 11, wherein the text sequence comprises a plurality of text tokens; and the context representation comprises a plurality of feature representations corresponding one-to-one to the plurality of text tokens. in, The local compression processing includes: calculating an importance index of each feature representation included in the context representation, and filtering, based on the importance index, some feature representations included in the context representation whose values ​​of the importance index are lower than a threshold; The global compression process includes: further extracting global context information from the feature representation included in the context representation, and encoding the global context information into a low-dimensional feature representation.

13. The deep learning network as described in claim 12, wherein the first compression network includes a classification network; the classification network is used to calculate the retention probability of each feature representation contained in the context representation, and assign classification labels to each feature representation contained in the context representation based on the retention probability; the classification labels include a first classification label for indicating that the feature representation is retained and a second classification label for indicating that the feature representation is not retained; and, the feature representation contained in the context representation and assigned the second classification label is filtered out.

14. The deep learning network of claim 13, wherein the second compression network comprises a first attention network, a second attention network, and a weight network; in, The first attention network is used to divide the feature representations included in the context representation into multiple groups, extract local context information of each feature representation included in each group in the multiple groups based on the attention mechanism, and encode the local context information into a low-dimensional local feature representation; The second attention network is used to extract global context information of each feature representation included in the context representation in the context representation based on the attention mechanism, and encode the global context information into a low-dimensional global feature representation; The weight network is used to learn compression weights corresponding to the first attention network and the second attention network respectively based on the context representation, and perform weighted calculation on the local feature representation and the global feature representation based on the learned compression weights to obtain the second compressed context representation.

15. The deep learning network of claim 14, wherein the second compression network comprises a mixture of experts network; The hybrid expert network includes the first attention network as an expert network, the second attention network as an expert network, and the weight network as a gating network.

16. The deep learning network of claim 15, wherein the first attention network comprises an MLP network incorporating an attention mechanism; and the second attention network comprises a Q-former network.

17. The deep learning network of claim 13, wherein the deep learning network is an LLM using a Transformer architecture; the context learning network is an encoder of the LLM; the generative network is a decoder of the LLM; or The deep learning network is a cross-model network composed of two different LLMs; the context learning network is a first LLM used to learn context representation from an input text sequence; and the generation network is a second LLM used to generate a target text sequence based on the compressed context representation.

18. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 are implemented.

19. A computer program product comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 10.

Images