Data processing method and device, storage medium and computing device cluster

Abstract

The invention provides a data processing method and device, a storage medium and a computing device cluster, which are used for reducing the first word delay of a large model. After a target sample set is obtained, feature extraction is performed on samples in the target sample set to obtain a target vector set, then the target vector set is input into a converter block to obtain a reference vector output by an Nth layer of the converter block, and the reference vector is input into a first prediction module to obtain a reference score set; the reference score set comprises an importance score of a sample corresponding to each vector in the reference vector set, inputting the reference score set and the reference vector set into a first cutting module to obtain an updated reference vector set, and analyzing the updated reference vector set to obtain a target prediction result. The samples in the target sample set are cut at any layer of the converter block, so that the calculation amount of analyzing and processing the target sample set by the converter block can be effectively reduced, and the first word time delay of the target prediction result is effectively reduced.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more particularly to a data processing method, apparatus, storage medium, and computing device cluster. Background Technology

[0002] As the application scenarios of large language models (LLM) continue to expand, the application of LLM has gradually evolved from text-based dialogue to document analysis and reasoning.

[0003] Because document size is significantly larger than that of text-based dialogues, it results in a longer first-word latency. To improve the inference efficiency of large models, various solutions have been developed to enhance their analysis efficiency. For example, in document analysis scenarios, attention scores are used to determine the importance of tokens to the model's analysis results. Key-value caches (KV caches) that retain important tokens are used to compress the KV cache and reduce memory usage.

[0004] Since this method uses the attention score of a token as the basis for judging the importance of the token to the model analysis results, each token still needs to participate in the attention calculation, making it difficult to significantly reduce the first-word latency of large models. Summary of the Invention

[0005] This application provides a data processing method, apparatus, storage medium, and computing device cluster for reducing the number of tokens used when performing inference on large models, thereby reducing the time required for large model analysis and lowering the first-word latency of large models.

[0006] Firstly, this application provides a data processing method, including:

[0007] Obtain the target sample set;

[0008] Feature extraction is performed on samples in the target sample set to obtain the target vector set;

[0009] Input the target vector set into the transformer block to obtain the reference vector set of the Nth layer output of the transformer block, where N is a positive integer;

[0010] The reference vector set is input into the first prediction module to obtain the reference score set. The first prediction module is used to score the input vectors to obtain the importance score of the input vectors. The reference score set includes the importance score of the sample corresponding to each vector in the reference vector set obtained by analyzing the reference vector set.

[0011] The reference score set and reference vector set are input into the first pruning module to obtain the updated reference vector set. The first pruning module is used to remove the vector corresponding to the sample with the lowest importance score in the vector set corresponding to the input importance score set based on the input importance score set.

[0012] The updated reference vector set is input into the N+1th layer of the transformer block to obtain the preset feature vector set;

[0013] The target prediction result is obtained by processing the preset feature vector set.

[0014] In this embodiment, after obtaining the target sample set, feature extraction is performed on the samples in the target sample set to obtain a target vector set. The target vector set is then input into the transformer block to obtain the reference vector output of the Nth layer of the transformer block. The first prediction module uses the reference vector set to score the importance of the vectors in the reference vector set to the analysis results, obtaining a reference score set. The reference score set includes the importance score of the sample corresponding to each vector in the reference vector set. The first pruning module prunes (deletes) the vectors in the reference vector set based on the reference score set to obtain an updated reference vector set. The updated reference vector set is input into the N+1th layer of the transformer block. After processing by the remaining transformer block layers, a preset feature vector is obtained. Processing the preset feature vector yields the target prediction result. Pruning the samples in the target sample set at any layer of the transformer block can effectively reduce the computational load of the transformer block in analyzing and processing the target sample set, and effectively reduce the first-word latency of the target prediction result.

[0015] In one possible implementation of the first aspect, inputting the updated reference vector set into the (N+1)th layer of the transformer block to obtain a preset feature vector set includes:

[0016] The updated reference vector set is input into the (N+1)th layer of the converter block to obtain the process vector set output by the (N+a)th layer of the converter block, where a is a positive integer;

[0017] The process vector set is input into the second prediction module to obtain the process score set. The second prediction module is used to score the input vectors to obtain the importance score of the input vectors. The process score set includes the importance score of the sample corresponding to each vector in the process vector set obtained by analyzing the process vector set.

[0018] The process score set and process score are input into the second pruning module. The second pruning module is used to remove the vector corresponding to the sample with the lowest importance score in the process vector set to obtain the updated process vector set.

[0019] The updated process vector set is input into the N+a+1 layer of the transformer block to obtain the preset feature vector set.

[0020] In this embodiment, by setting multiple sets of prediction modules and pruning modules between different transformer sub-blocks of the transformer block, the number of samples in the target sample set is reduced, avoiding the impact of pruning too many samples at once on the analysis results, increasing the fault tolerance of the scheme to the prediction module, and improving the inference accuracy of the scheme.

[0021] In one possible implementation of the first aspect, before extracting features from samples in the target sample set to obtain the target vector set, the method further includes:

[0022] Obtain the training sample set;

[0023] Feature extraction is performed on the training sample set to obtain the training vector set;

[0024] Input the training vector set into the transformer block to obtain a first saliency score set and a first vector set. The first saliency score set includes the saliency score of each sample in the training sample set at the Nth layer of the transformer block, and the first vector set includes the output vector of each sample in the training sample set at the Nth layer of the transformer block.

[0025] The first vector set is input into the first prediction module to be trained to obtain the first importance score set, which includes the importance score of each sample in the training sample set obtained by analyzing the first vector set.

[0026] Based on the first saliency score set and the first importance score set, supervised training is performed on the first prediction module to be trained to obtain the first prediction module.

[0027] In this embodiment, a first prediction module to be trained is set in the large model. It analyzes the first vector set output by the Nth layer of the LLM model's transformer block to obtain a first importance score set. Supervised training of the first prediction module to be trained is then completed based on the saliency score corresponding to each vector in the Nth layer of the transformer block and the first importance score set. By training the first prediction module, the accuracy of the importance scores in the first importance score set obtained by the first prediction module from analyzing the first vector set is effectively improved.

[0028] In one possible implementation of the first aspect, after inputting a first vector set into a first prediction module to be trained and obtaining a first importance score set, the method further includes:

[0029] Input the first importance score set and the first vector set into the first clipping module to obtain the updated first vector set;

[0030] The updated first vector set is input into the N+1 layer of the transformer block to obtain the second saliency score set and the second vector set. The second saliency score set includes the saliency score of the first sample set in the N+a layer of the transformer block, and the second vector set includes the output vector of each sample in the first sample set in the N+a layer of the transformer block. The first sample set includes the sample corresponding to each vector in the second vector set.

[0031] The second vector set is input into the second prediction module to be trained to obtain the second importance score set, which includes the importance score of each sample in the first sample set obtained by analyzing the second vector set.

[0032] Based on the second saliency score set and the second importance score set, supervised training is performed on the second prediction module to be updated to obtain the second prediction module.

[0033] In this embodiment, a pruning component including a first prediction module and a first pruning module is provided between the Nth and N+1th layers of the transformer block of the LLM model, and a pruning component including a second prediction module and a second pruning module is provided between the N+ath and N+a+1th layers of the transformer block. During the analysis of the LLM model, at least two sample prunings are performed. Through progressive multi-stage pruning, the number of samples that need to be inferred during the inference process can be further reduced, thereby reducing the inference computation and memory usage.

[0034] In one possible implementation of the first aspect, the first prediction module to be trained is subjected to supervised training based on a first saliency score set and a first importance score set to obtain the first prediction module, including:

[0035] Based on the first significance score set, the samples in the training sample set are sorted in descending order of significance score to obtain the first sequence;

[0036] Based on the first importance score set, the samples in the training sample set are sorted in descending order of importance score to obtain the second sequence;

[0037] By increasing the similarity between the first sequence and the second sequence, the first importance score set is updated to obtain the first prediction module.

[0038] In this embodiment, the samples in the training sample set are sorted in descending order of saliency score based on the first saliency score set to obtain a first sequence. The samples in the training sample set are sorted in descending order of importance score based on the first importance score set to obtain a second sequence. The first importance score set is updated by increasing the similarity between the first sequence and the second sequence to complete the update of the first prediction module. The first prediction module to be trained is trained using the similarity between the first sequence and the second sequence, which effectively avoids the problem of inaccurate analysis caused by importance score bias and improves the training reliability of the first prediction module to be trained.

[0039] In one possible implementation of the first aspect, inputting the training vector set into a transformer block to obtain a first saliency score set and a first vector set includes:

[0040] Input the training vectors into the transformer block to obtain the first vector set and the first gradient value. The first gradient is the gradient value of the Nth layer of the transformer block in backpropagation.

[0041] The first saliency score set is obtained by multiplying each vector in the first vector set with the first gradient value.

[0042] In this embodiment of the application, when calculating the saliency score based on the integral gradient, the method of calculating the saliency score by multiplying the integral gradient and the output vector of the Nth layer of the transformer block uses the intermediate data of the model to calculate the saliency score without introducing new calculations, which effectively improves training efficiency.

[0043] In one possible implementation of the first aspect, inputting the training vector set into a transformer block to obtain a first saliency score set and a first vector set includes:

[0044] The training vectors are input into the transformer block to obtain a first vector set and a first attention score, which includes the attention score of each sample in each layer of the transformer block.

[0045] The attention scores of the first sample in each layer of the transformer block are accumulated to obtain the first saliency score. The first saliency score is included in the first saliency score set, and the first sample is any sample in the training sample set.

[0046] In this embodiment, the saliency score is calculated based on the attention score. The attention score during the forward reasoning process is accumulated according to the sample, and the accumulated attention score is used as the saliency score of the sample. This effectively evaluates the influence of the sample on the output through the saliency score, while also effectively improving the interpretability of the saliency score.

[0047] Secondly, this application provides a data processing method, including:

[0048] Obtain the training sample set;

[0049] Feature extraction is performed on the training sample set to obtain the training vector set;

[0050] Input the training vector set into the transformer block to obtain the first saliency score set and the first vector set. The first saliency score set includes the saliency score of each sample in the training sample set at the Nth layer of the transformer block, and the first vector set includes the output vector of each sample in the training sample set at the Nth layer of the transformer block, where N is a positive integer.

[0051] The first vector set is input into the first prediction module to be trained to obtain the first importance score set, which includes the importance score of each sample in the training sample set obtained by analyzing the first vector set.

[0052] Based on the first saliency score set and the first importance score set, supervised training is performed on the first prediction module to be trained to obtain the first prediction module.

[0053] In one possible implementation of the second aspect, after inputting the first vector set into the first prediction module to be trained and obtaining the first importance score set, the method further includes:

[0054] Input the first importance score set and the first vector set into the first clipping module to obtain the updated first vector set;

[0055] The updated first vector set is input into the N+1 layer of the transformer block to obtain the second saliency score and the second vector set. The second saliency score set includes the saliency score of the first sample in the N+a layer of the transformer block, and the second vector set includes the output vector of the first sample set in the N+a layer of the transformer block. The first sample set is the sample contained in the second vector set, and a is a positive integer.

[0056] The second vector set is input into the second prediction module to be trained to obtain the second importance score set, which includes the importance score of each sample in the first sample set obtained by analyzing the second vector set.

[0057] Based on the second saliency score set and the second importance score set, supervised training is performed on the second prediction module to be updated to obtain the second prediction module.

[0058] In one possible implementation of the second aspect, the first prediction module to be trained is subjected to supervised training based on a first saliency score set and a first importance score set to obtain the first prediction module, including:

[0059] Based on the first significance score set, the samples in the training sample set are sorted in descending order of significance score to obtain the first sequence;

[0060] Based on the first importance score set, the samples in the training sample set are sorted in descending order of importance score to obtain the second sequence;

[0061] By increasing the similarity between the first sequence and the second sequence, the first importance score set is updated to obtain the first prediction module.

[0062] In one possible implementation of the second aspect, inputting the training vector set into the transformer block to obtain the first saliency score set and the first vector set includes:

[0063] Input the training vectors into the transformer block to obtain the first vector set and the first gradient value. The first gradient is the gradient value of the Nth layer of the transformer block in backpropagation.

[0064] The first saliency score set is obtained by multiplying each vector in the first vector set with the first gradient value.

[0065] In one possible implementation of the second aspect, inputting the training vector set into the transformer block to obtain the first saliency score set and the first vector set includes:

[0066] Input the training vectors into the transformer block to obtain the first vector set and the first attention score. The first attention score includes the attention score of the first sample in each layer of the transformer block. The first sample is any sample in the training sample set.

[0067] The attention scores of the first sample at each layer of the transformer block are accumulated in the first attention score to obtain the first saliency score, which is contained in the first saliency score set.

[0068] Thirdly, embodiments of this application provide a data processing apparatus, including:

[0069] The acquisition unit is used to acquire the target sample set;

[0070] The processing unit is used to extract features from samples in the target sample set to obtain the target vector set;

[0071] The processing unit is also used to input the target vector set into the transformer block to obtain the reference vector set of the Nth layer output of the transformer block, where N is a positive integer;

[0072] The processing unit is also used to input the reference vector set into the first prediction module to obtain a reference score set. The first prediction module is used to score the input vector to obtain the importance score of the input vector. The reference score set includes the importance score of the sample corresponding to each vector in the reference vector set obtained by analyzing the reference vector set.

[0073] The processing unit is also used to input the reference score set and the reference vector set into the first pruning module to obtain the updated reference vector set. The first pruning module is used to remove the vector corresponding to the sample with the lowest importance score in the vector set corresponding to the input importance score set based on the input importance score set.

[0074] The processing unit is also used to input the updated reference vector set into the N+1th layer of the transformer block to obtain a preset feature vector set;

[0075] The processing unit is also used to process a preset feature vector set to obtain target prediction results.

[0076] Fourthly, this application provides a data processing apparatus, comprising:

[0077] The acquisition unit is used to acquire the training sample set;

[0078] The processing unit is used to extract features from the training sample set to obtain the training vector set;

[0079] The processing unit is also configured to input the training vector set into the transformer block to obtain a first saliency score set and a first vector set. The first saliency score set includes the saliency score of each sample in the training sample set at the Nth layer of the transformer block, and the first vector set includes the output vector of each sample in the training sample set at the Nth layer of the transformer block, where N is a positive integer.

[0080] The processing unit is also configured to input the first vector set into the first prediction module to be trained to obtain a first importance score set, which includes the importance score of each sample in the training sample set obtained by analyzing the first vector set.

[0081] The processing unit is also used to perform supervised training on the first prediction module to be trained based on the first saliency score set and the first importance score set, so as to obtain the first prediction module.

[0082] Fifthly, this application provides a data processing apparatus, including a processor and a memory, wherein the processor stores instructions, and when the instructions stored in the memory are executed on the processor, the methods shown in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect are implemented.

[0083] Sixthly, this application provides a computer-readable storage medium storing instructions that, when executed on a processor, implement the methods shown in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0084] In a seventh aspect, this application provides a computing device cluster, including at least one computing device, each computing device including a processor and memory;

[0085] A processor of at least one computing device is configured to execute instructions stored in the memory of at least one computing device to cause a cluster of computing devices to perform the methods shown in the first aspect, any possible implementation of the first aspect, the second aspect, or any possible implementation of the second aspect.

[0086] The beneficial effects shown in any of the second to seventh aspects are similar to those of the first aspect or any possible implementation of the first aspect, and will not be repeated here. Attached Figure Description

[0087] Figure 1 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0088] Figure 2 Another schematic flowchart illustrating the data processing method provided in this application embodiment;

[0089] Figure 3 Another schematic flowchart illustrating the data processing method provided in this application embodiment;

[0090] Figure 4 A schematic diagram of an architecture of an LLM model including SDTP components provided for an embodiment of this application;

[0091] Figure 5 Another architectural diagram of an LLM model including SDTP components provided for embodiments of this application;

[0092] Figure 6a A schematic diagram of experimental data from a control experiment provided in an embodiment of this application;

[0093] Figure 6b Another schematic diagram of experimental data for the control experiment provided in the embodiments of this application;

[0094] Figure 6c Another schematic diagram of experimental data for the control experiment provided in the embodiments of this application;

[0095] Figure 6d Another schematic diagram of experimental data for the control experiment provided in the embodiments of this application;

[0096] Figure 6e Another schematic diagram of experimental data for the control experiment provided in the embodiments of this application;

[0097] Figure 7 A schematic diagram of the structure of a data processing apparatus provided in an embodiment of this application;

[0098] Figure 8 Another schematic diagram of the data processing apparatus provided in the embodiments of this application;

[0099] Figure 9 This is another schematic diagram of the data processing apparatus provided in the embodiments of this application. Detailed Implementation

[0100] This application provides a data processing method, apparatus, storage medium, and computing device cluster for reducing the number of tokens used in analyzing large models, reducing the number of events required for model execution, and lowering the first-word latency of large models.

[0101] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0102] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of units is not necessarily limited to those units, but may include other units not explicitly listed or inherent to those processes, methods, products, or apparatuses. Additionally, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can be expressed as: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0103] As AI applications continue to expand, the application scenarios of LLM (Limited Language Management) have gradually evolved from simple text dialogues or text question-answering to document analysis and reasoning. Furthermore, to improve the output quality of LLM, prompt engineering is being more widely applied in LLM inference. Compared to simple text dialogues or text question-answering, the length of the input text in prompt engineering and document-based analysis and reasoning in LLM increases exponentially, and long sequence inference in LLM is accompanied by an increase in first-word latency.

[0104] To reduce the first-word latency of LLM inference, algorithms have been developed that compress the original input context to reduce the first-word latency of LLM in long sequence scenarios.

[0105] In algorithms that compress the original input context, a small model is used to explicitly learn the prompt compression module. However, the operation of the prompt compression module still takes a lot of time, resulting in limited reduction of the first-word latency in LLM long sequence inference by this compression method.

[0106] In the key-value cache compression scheme based on token importance, the accumulated attention of tokens is used as a reference for token importance, and the key-value cache of tokens is then retained based on the importance of the tokens. However, this scheme still requires attention calculation for all tokens in the input sequence. The process of performing attention calculation for all tokens in the input sequence consumes a lot of time and computing power, and has limited effect on reducing the first word latency of LLM long sequence inference.

[0107] To address the above issues, this application proposes inserting a prediction module and a pruning module between layers N and N+1 of the transformer block, building upon existing large-scale models. The prediction module analyzes the importance score of the samples (tokens) corresponding to the input vectors. The pruning module removes the least important samples from layer N of the transformer block based on the prediction module's analysis results, obtaining samples for layer N+1. The vectors corresponding to these samples are then input into layer N+1 of the transformer block to obtain the target prediction result. This ensures that the analysis of layer N+1 and subsequent layers is performed on the reduced vectors, thus effectively reducing the first-word latency of LLM long sequence inference.

[0108] The solutions provided in this application can be divided into applications of the pruning component in the training phase and applications of the pruning component in the inference phase, depending on the application scenario. The following sections will describe these two scenarios in detail:

[0109] First, we will introduce the application of the clipping component in the inference phase. Please refer to [link to relevant documentation]. Figure 1 , Figure 1 This is a schematic flowchart of a data processing method provided in an embodiment of this application.

[0110] S110. Obtain the target sample set;

[0111] The target sample set can be documents in document question answering scenarios, documents to be translated in language translation scenarios, code requirements in code generation scenarios, sample documents in few-shot learning, etc. The target sample set includes multiple samples, which are the basic units of text fragments, i.e., tokens. A sample can be a Chinese character, an English sequence, or a sequence of multiple characters, etc., without any restrictions.

[0112] S120. Extract features from the samples in the target sample set to obtain the target vector set;

[0113] There is a one-to-one correspondence between the vectors in the target vector set and the samples in the target sample set. For example, sample A (tokenA) corresponds to vector A in the target vector set.

[0114] Specifically, samples in the target sample set can be identified as numerical vectors based on word embedding techniques, such as Word2Vec or GloVe, without limitation.

[0115] S130. Input the target vector set into the transformer block to obtain the reference vector set of the Nth layer output of the transformer block;

[0116] Where N is a positive integer, N is a pre-defined number of layers, and when the converter block contains M layers of converter sub-blocks, M is a positive integer greater than 2 and N is a positive integer less than M.

[0117] To facilitate understanding, the transformer block will be introduced. The transformer block is an important component of the transformer model architecture. The transformer model architecture includes a tokenizer, which divides the input content into multiple tokens; an embedding layer, which converts the tokens from natural language to machine language; a transformer block, which predicts the next word, consisting of an attention block and a feedforward block. The attention block adds context to the text, and the feedforward block guesses the next word; and a softmax layer, which converts scores into probabilities to sample the next word.

[0118] A converter block can contain multiple sub-converter blocks, which are called layers of the converter block. A converter block can have multiple layers, and each layer of the converter block contains an attention sub-block and a feedforward sub-block.

[0119] Specifically, after inputting the target vector set into the transformer block, the reference vector set of the Nth layer output of the transformer block is obtained. Each layer of the transformer block will output a set of vectors. N can be any layer before the first or last layer of the transformer block, and N is a positive integer.

[0120] It should be noted that the Nth layer of the converter block here is a pre-defined layer number, which can be set according to the specific application scenario in actual applications, and there is no restriction here.

[0121] S140. Input the reference vector set into the first prediction module to obtain the reference score set;

[0122] The first prediction module is used to score the input vector and obtain the importance score of the input vector. The reference score set includes the importance score of the sample corresponding to each vector in the reference vector set obtained by the first prediction module.

[0123] S150. Input the reference score set and reference vector set into the first clipping module to obtain the updated reference vector set;

[0124] Since the samples in the reference score set and the reference vector set are in one-to-one correspondence, the first pruning module is used to remove the vector corresponding to the sample with the lowest importance score in the reference vector set based on the reference score set.

[0125] Specifically, while the first prediction module may have biases in its analysis of the importance scores of samples in the target sample set, the ranking of sample importance scores is consistent under the same analysis model. Therefore, the ranking of sample importance scores in the target sample set has a certain degree of reliability.

[0126] This application proposes that the first pruning module can update the reference vector set based on the importance score ranking of the samples. The first pruning module can obtain the updated reference vector set by pruning the vectors corresponding to the 10% of samples with the lowest importance scores.

[0127] In this embodiment, the importance scores of the samples are ranked using a reference score set, and the ranking results are used to prune the reference vector set, removing the vectors with the lowest importance. This effectively avoids the inaccuracy of pruning caused by the bias of importance scores, which is introduced by pruning based on importance score values, and effectively reduces the impact of pruning on the analysis results.

[0128] S160. Input the updated reference vector set into the N+1th layer of the transformer block to obtain the preset feature vector;

[0129] Specifically, after the updated reference vector set is input into the N+1 layer of the transformer block, the updated reference vector set is processed by multiple transformer sub-blocks from the N+1 layer to the M layer of the transformer block to obtain the preset feature vector.

[0130] S170. Process the preset feature vector to obtain the target prediction result.

[0131] Specifically, the preset feature vectors are processed in a single layer to obtain the target prediction result.

[0132] In this embodiment, after obtaining the target sample set, feature extraction is performed on the samples in the target sample set to obtain a target vector set. The target vector set is then input into the transformer block to obtain the reference vector output of the Nth layer of the transformer block. The first prediction module uses the reference vector set to score the importance of the vectors in the reference vector set to the analysis results, obtaining a reference score set. The reference score set includes the importance score of the sample corresponding to each vector in the reference vector set. The first pruning module prunes (deletes) the vectors in the reference vector set based on the reference score set to obtain an updated reference vector set. The updated reference vector set is input into the N+1th layer of the transformer block. After processing by the remaining transformer block layers, a preset feature vector is obtained. Processing the preset feature vector yields the target prediction result. Pruning the samples in the target sample set at any layer of the transformer block can effectively reduce the computational load of the transformer block in analyzing and processing the target sample set, and effectively reduce the first-word latency of the target prediction result.

[0133] In practical applications, the pruning of the target sample set can be done in one step or multiple steps. The following section further describes the solution provided in this application for the scenario of multiple pruning. Based on the aforementioned steps S110 to S170, step S160 may include steps S161 to S164. Please refer to... Figure 2 The following sections will describe these steps in detail:

[0134] S161. Input the updated reference vector set into the N+1th layer of the converter block to obtain the process vector set output by the N+ath layer of the converter block.

[0135] Where a is a positive integer, and a is a pre-set value. In the case where the transformer block contains M transformer sub-blocks, since the first prediction module and the first trimming module are set between two transformer sub-blocks, and the second prediction module and the second trimming module are also set between two transformer sub-blocks, and there is at least one transformer sub-block between the first prediction module and the second prediction module, it is obvious that in this case, M is an integer greater than 3.

[0136] Furthermore, the description of setting prediction modules and clipping modules in the converter block in this embodiment is only an example. In actual applications, the prediction modules and clipping modules set in the converter block can be one, two, or even more. When the converter block includes X groups of prediction modules and clipping modules, M is an integer greater than or equal to X+1. In actual applications, the settings should be combined with the specific application scenario, and no restrictions are imposed here.

[0137] Based on this, when multiple prediction modules and trimming modules are evenly distributed in the transformer block, 'a' can be the number of transformer sub-blocks between two sets of prediction modules and trimming modules. The number of transformer sub-blocks between multiple sets of prediction modules and trimming modules can also be different. In practical applications, it can be set according to the specific application scenario, and there is no restriction here.

[0138] S162. Input the process vector set into the second prediction module to obtain the process score set;

[0139] The second prediction module is used to score the input vector to obtain the importance score of the input vector. The process score set includes the importance score of the sample corresponding to each vector in the process vector set obtained by the second prediction module from analyzing the process vector set.

[0140] S163. Input the process scoring set and process vector set into the second trimming module to obtain the updated process vector set;

[0141] Since the samples in the process score set and the process vector set are in one-to-one correspondence, the second pruning module can sort the vectors in the process vector set by the importance score of each sample in the process score set, so as to remove the vector corresponding to the sample with the lowest importance score in the process vector set.

[0142] Specifically, since the second prediction model may have biases in its analysis of the importance scores of samples in the target sample set, the importance scores of samples are consistent under the same analysis mode. Therefore, the ranking of importance scores is more reliable than the ranking of importance scores.

[0143] Based on the above reasoning, this application proposes that the second pruning module can prune the process vector set based on the importance score ranking of the samples. The second pruning module can obtain the updated process vector set by pruning the vectors corresponding to the 10% of samples with the lowest importance scores.

[0144] It is understood that the explanation of pruning the vectors corresponding to the samples with the lowest importance scores here is only an example. In practical applications, the vectors corresponding to the samples with the lowest importance scores, or other proportions of vectors, can be pruned. There are no restrictions here.

[0145] S164. Input the updated process vector into the N+a+1th layer of the transformer block to obtain the preset feature vector set.

[0146] Specifically, after the updated process vector set is input into the N+a+1th layer of the converter block, the updated process vector set is processed by multiple converter sub-blocks from the N+a+1th layer to the Mth layer of the converter block to obtain the preset feature vector.

[0147] In this embodiment, by setting multiple sets of prediction modules and pruning modules between different transformer sub-blocks of the transformer block, the number of samples in the target sample set is reduced, avoiding the impact of pruning too many samples at once on the analysis results, and increasing the fault tolerance of the scheme for the accuracy of the prediction module.

[0148] Then, the operation of the solution provided in this application during the training phase will be explained. Please refer to [link / reference]. Figure 3 , Figure 3 This is another schematic flowchart illustrating the data processing method provided in the embodiments of this application.

[0149] S201. Obtain the training sample set;

[0150] The training sample set can be a document containing multiple fields, or multiple documents containing multiple fields. Each document contains multiple tokens, which can be a Chinese character, an English sequence, or a sequence of multiple characters, etc. Each token is a sample.

[0151] S202. Extract features from the training sample set to obtain the training vector set;

[0152] Specifically, the samples in the training sample set can be converted from text into vectors that computers can understand based on word embedding technology. For example, word embedding technology can be Word2Vec technology or GloVe technology, without limitation here.

[0153] S203. Input the training vector set into the transformer block to obtain the first saliency score set and the first vector set;

[0154] The first saliency score set includes the saliency score of each sample in the training sample set at the Nth layer of the transformer block, and the first vector set includes the output vector of each sample in the training sample set at the Nth layer of the transformer block.

[0155] Specifically, the significance score corresponding to a sample is used to evaluate the magnitude of the sample's influence on the LLM output. Depending on different needs, different methods can be used to calculate the significance score. For example, a significance score based on integral gradient, a significance score based on attention, or a significance score based on layer-by-layer correlation propagation can be used. The significance score can also be obtained based on the calculation method of feature attribution interpretability. There are no restrictions here.

[0156] For example, when the saliency score is calculated based on attention scores, after the training vector is input into the transformer block, the first vector set and the first attention score set of the Nth layer output of the transformer block are collected. The first attention score set includes the attention score of each sample in the training sample set at each layer of the M layers of the transformer block. Then, the attention score of the first sample in the first attention score set at each layer of the transformer block is scored to obtain the first saliency score corresponding to the first sample. The first saliency score is included in the first saliency score set, and the first sample is any sample in the training sample set. After multiple calculations, all saliency scores in the first saliency score set can be obtained.

[0157] In this embodiment, the saliency score is calculated based on the attention score. The attention score during the forward reasoning process is accumulated according to the sample, and the accumulated attention score is used as the saliency score of the sample. This effectively evaluates the influence of the sample on the output through the saliency score, while also effectively improving the interpretability of the saliency score.

[0158] For example, when the saliency score is calculated based on the integral gradient, after the training vector is input into the transformer block, the first vector set of the output of the Nth layer of the transformer block and the first gradient value of the Nth layer of the transformer block during backpropagation are collected. Multiplying the first gradient value by any vector in the first vector set yields the saliency score corresponding to that vector. The multiple saliency scores obtained by multiplying the first gradient value by each vector in the first vector set constitute the first saliency score set.

[0159] In this embodiment of the application, when calculating the saliency score based on the integral gradient, the method of calculating the saliency score by multiplying the integral gradient and the output vector of the Nth layer of the transformer block uses the intermediate data of the model to calculate the saliency score without introducing new calculations, which effectively improves training efficiency.

[0160] S204. Input the first vector set into the first prediction module to be trained to obtain the first importance score set;

[0161] The first prediction module to be trained is used to evaluate the input vector and obtain the importance score of the input vector. In this application scenario, since the input vector of the first prediction module is the first vector set, the obtained first importance score set includes the importance score of the output vector of each sample in the Nth layer of the transformer block.

[0162] The importance scores in the first importance score set are used to describe the influence of each sample in the training sample set on the LLM output.

[0163] S205. Based on the first saliency score set and the first importance score set, supervised training is performed on the first prediction module to be trained to obtain the first prediction module;

[0164] The first prediction module to be trained is a two-layer multilayer perceptron (MLP) structure.

[0165] Specifically, while model analysis may contain errors, the shifts in sample importance scores are consistent when analyzed using the same model, making the ranking of sample importance scores more reliable. Similarly, the ranking of sample significance scores is more reliable than the numerical values ​​of the sample significance scores.

[0166] Based on the above viewpoints, this application proposes that, when supervising the training of the first prediction module to be trained based on the first saliency score set and the first importance score set:

[0167] First, based on the first saliency score set, the samples in the training sample set are sorted in descending order of saliency score to obtain the first sequence.

[0168] Then, based on the first importance score set, the samples in the training sample set are sorted in descending order of importance score to obtain the second sequence.

[0169] Understandably, in practical applications, the samples in the training sample set can be sorted first based on the first significance score set, or the samples in the training sample set can be sorted first based on the first importance score set. The explanation here is just an example. In practical applications, it can be adjusted according to the specific situation without restriction.

[0170] The first sequence is then used as a supervision signal to supervise the training of the first prediction module to be trained. The parameters of the first loss function are updated by increasing the similarity between the first and second sequences, thereby completing the training of the first prediction module to be trained and obtaining the first prediction module. The first loss function is the loss function of the first prediction module to be trained.

[0171] In this embodiment, the samples in the training sample set are sorted in descending order of saliency score based on the first saliency score set to obtain a first sequence. The samples in the training sample set are sorted in descending order of importance score based on the first importance score set to obtain a second sequence. The first importance score set is updated by increasing the similarity between the first sequence and the second sequence to complete the update of the first prediction module. The first prediction module to be trained is trained using the similarity between the first sequence and the second sequence, which effectively avoids the problem of inaccurate analysis caused by importance score bias and improves the training reliability of the first prediction module to be trained.

[0172] It is understood that the description of the structure of the first prediction module to be trained here is only an example. In actual applications, it can be set and adjusted according to the specific application scenario, and there are no restrictions here.

[0173] S206. Input the first importance score set and the first vector set into the first clipping module to obtain the updated first vector set;

[0174] The first pruning module is used to remove the vector corresponding to the sample with the lowest importance score from the vector set corresponding to the input importance score set based on the input importance score set.

[0175] Specifically, the scores in the first importance score set are the scores of each sample in the training sample set, and the vectors in the first vector set are the output vectors of each sample in the training sample set at the Nth layer of the transformer block. Therefore, each score in the first importance score set can correspond to a vector in the first vector set. The first pruning module is used to prune the vectors in the first vector set based on the scores in the first importance score set to obtain the updated first vector set.

[0176] For example, the first pruning module can prune the first vector set based on the importance score of the samples. The first pruning module can obtain the updated first vector set by pruning the vectors corresponding to the 10% of samples with the lowest importance scores.

[0177] It should be noted that the updated first vector set obtained by pruning the first vector set is achieved by masking the vectors corresponding to the samples with the lowest importance scores in the first vector set. In practice, other methods can also be used, and no restrictions are placed here.

[0178] S207. Input the updated first vector set into the N+1 layer of the transformer block to obtain the second saliency score set and the second vector set.

[0179] The second saliency score set includes the saliency scores of the first sample set at the (N+a)th layer of the transformer block, the second vector set includes the output vector of each sample in the first sample set at the (N+a)th layer of the transformer block, and the first sample set includes the sample corresponding to each vector in the second vector set.

[0180] Specifically, when the updated first vector set is input into the N+1 layer of the transformer block to obtain the second saliency score set and the second vector set, different methods can be used to calculate the saliency score. For example, saliency scores based on integral gradients, attention-based saliency scores, or saliency scores based on layer-by-layer correlation propagation can be used. The saliency score can be obtained based on the saliency score calculation method of feature attribution interpretability. There are no restrictions here. The specific operation method is similar to the method of obtaining the first saliency score and the first vector set in the aforementioned step S203, and will not be repeated here.

[0181] S208. Input the second vector set into the second prediction module to be trained to obtain the second importance score set;

[0182] The second prediction module to be trained is used to evaluate the input vector and obtain the importance score of the input vector. Since the input vector of the second prediction module to be trained is the second vector set, and the first sample set includes the sample corresponding to each vector in the second vector set, the second importance score set includes the importance score of each sample in the first sample set obtained by analyzing the second vector set.

[0183] The importance scores in the second importance score set are used to describe the influence of each sample in the first sample set on the LLM output.

[0184] S209. Based on the second saliency score set and the second importance score set, supervised training is performed on the second prediction module to be updated to obtain the second prediction module;

[0185] The second prediction module to be trained is a two-layer MLP structure.

[0186] Specifically, ranking based on sample importance scores has higher reliability than ranking based on sample importance scores, and ranking based on sample significance scores has higher reliability than ranking based on sample significance scores. This application proposes that when supervising the training of the second prediction module based on the second significance score set and the second importance score set:

[0187] First, based on the second significance score set, the samples in the training sample set are sorted in descending order of significance score to obtain the second sequence.

[0188] Then, based on the second importance score set, the samples in the training sample set are sorted in descending order of importance score to obtain the second sequence.

[0189] It is understandable that in practical applications, the samples in the training sample set can be sorted first based on the second significance score set, or the samples in the training sample set can be sorted first based on the second importance score set. The explanation here is only an example. In practical applications, it can be adjusted according to the specific situation without restriction.

[0190] The second sequence is then used as a supervision signal to supervise the training of the second prediction module. The parameters of the second loss function are updated by increasing the similarity between the two sequences, thus completing the training of the second prediction module and obtaining the second prediction module. The second loss function is the loss function of the second prediction module to be trained.

[0191] It is understood that the description of the structure of the second prediction module to be trained here is only an example. In actual applications, it can be set and adjusted according to the specific application scenario, and there are no restrictions here.

[0192] 210. Input the second importance score set and the second vector set into the second pruning module to obtain the updated second vector set, so that the N+a+1th layer of the transformer block is analyzed based on the updated second vector set.

[0193] Where a is a positive integer, the second pruning module is used to remove the vector corresponding to the sample with the lowest importance score from the vector set corresponding to the input importance score set based on the input importance score set.

[0194] Specifically, the scores in the second importance score set are the scores of each sample in the first sample set, and the vectors in the second vector set are the output vectors of each sample in the first sample set at the (N+a)th layer of the transformer block. Therefore, each score in the second importance score set can correspond to a vector in the second vector set. The second pruning module is used to prune the vectors in the second vector set based on the scores in the second importance score set to obtain the updated second vector set.

[0195] For example, the interval between the second trimming module and the first trimming module can be 4 layers of transformer sub-blocks, that is, 'a' can be 4. In practical applications, a component consisting of a prediction module and a trimming module can be set every 4 layers of transformer sub-blocks, which is not limited here.

[0196] The second pruning module can prune the second vector set based on the importance scores of the samples. It can prune the vectors corresponding to the 10% of samples with the lowest importance scores to obtain the updated second vector set. In other words, the training sample set is pruned by 10% in the first pruning module to obtain the first sample set, and then the first sample set is pruned by 10% in the second pruning module to obtain the updated first sample set.

[0197] In the scenario of S-stage cropping, after S-stage cropping, the number of remaining samples is: (0.9S) * the number of samples in the training sample set.

[0198] It is understood that the description of the value of 'a' and the cropping ratio here is only an example. In actual applications, the settings should be combined with the specific application scenario. No restrictions are imposed here.

[0199] Furthermore, the updated second vector set obtained by pruning the second vector set is achieved by masking the vectors corresponding to the samples with the lowest importance scores in the second vector set. In practice, other methods can also be used, and no restrictions are placed here.

[0200] The updated second vector set will then be input into the N+a+1th layer of the transformer block for further analysis; no restrictions are imposed here.

[0201] In this embodiment, a pruning component is set between the Nth and N+1th layers of the transformer block of the LLM model, and a pruning component is also set between the N+ath and N+a+1th layers of the transformer block. During the analysis of the LLM model, at least two sample prunings are performed. Through progressive multi-stage pruning, the number of samples that need to be inferred during the inference process can be further reduced, thereby reducing the inference computation and memory usage.

[0202] Furthermore, in this embodiment, steps S201 to S206 can also be implemented independently. A first prediction module to be trained is set up in the large model, and the first vector set output from the Nth layer of the LLM model's transformer block is analyzed to obtain a first importance score set. Supervised training of the first prediction module to be trained is then completed based on the saliency score corresponding to each vector in the Nth layer of the transformer block and the first importance score set. By training the first prediction module, the accuracy of the importance scores in the first importance score set obtained by the first prediction module from analyzing the first vector set is effectively improved.

[0203] It should be noted that steps S201 to S210 can be implemented before step S120 or can be implemented separately; there is no restriction here.

[0204] To facilitate understanding of the changes to the LLM architecture proposed in this application, the following will be combined with... Figure 4 The data processing method provided in this application involves a first prediction module and a first cropping module to be trained.

[0205] Depend on Figure 4 It can be seen that LLM contains an embedding layer, a transformer block, and a normalization layer (softmax layer). The transformer block contains multiple transformer sub-blocks, and a pruning component consisting of a first prediction module and a first pruning module to be trained is set between the Nth layer and the N+1th layer of the transformer block.

[0206] After the training sample set is input into the LLM, it first passes through an embedding layer, a transformer block, and a normalization layer to obtain a first saliency score set. Then, it passes through the embedding layer to obtain the initial vector corresponding to each sample. The initial vectors corresponding to each sample in the training sample set form an initial vector set, which is then input into the first layer of the transformer block. The initial vectors of each sample are processed through layers one through N of the transformer block to obtain a first vector set (the vector output from the Nth layer of the transformer block). This first vector set is then input into the first prediction module to be trained to obtain a first importance score set. The first importance score set includes the importance score corresponding to each sample in the training sample set. The first pruning module prunes the first vector set according to the input first importance score set to obtain an updated first vector set. Based on the first saliency score and the first importance score set, the first prediction module to be trained is trained to obtain the first prediction module.

[0207] The dashed lines describe the process of analyzing and obtaining the first saliency score set before pruning, while the solid lines describe the process of pruning to obtain the updated first vector set.

[0208] The solution provided in this application does not require modification of existing parameters in the LLM; it only requires training the parameters set in the newly added prediction module, which significantly reduces the amount of computation during training and improves training efficiency.

[0209] Furthermore, in combination Figure 5 The data processing method provided in this application also includes a second prediction module and a second pruning module to be trained.

[0210] In the data processing method involving a first prediction module to be trained, a second prediction module to be trained, a first pruning module, and a second pruning module, a pruning component consisting of a second prediction module to be trained and a second pruning module is also set between the N+a layer and the N+a+1 layer of the transformer block. After the updated first vector set is input into the N+1 layer of the transformer block, the second vector set output by the N+a layer of the transformer block is collected.

[0211] In this scenario, after the training sample set is input into the LLM, a second saliency score set can be obtained through the embedding layer, transformer block, and normalization layer. This second vector set is then input into the second prediction module to be trained to obtain a second importance score set. The second importance score set includes the importance score corresponding to each sample in the first sample set, which is the set of samples corresponding to each vector in the updated first vector set.

[0212] Then, supervised training is performed on the second prediction module to be trained based on the second importance score set and the second significance score set to obtain the second prediction module.

[0213] The dashed lines describe the process of obtaining the first and second significance score sets before pruning. The solid lines represent the processes of pruning the first vector set based on the first significance score set to obtain the updated first vector set, and pruning the second vector set based on the second significance score set to obtain the updated second vector set.

[0214] The solution provided in this application uses the resampling technique gumbel-softmax to enable the training of multiple prediction modules at once, reducing the number of training iterations and improving training efficiency.

[0215] To evaluate the generalization ability of the solution provided in this application across different tasks, multiple sets of comparative experiments were conducted based on the solution in different application scenarios. The results of the comparative experiments will be presented below in conjunction with different application scenarios:

[0216] For ease of explanation, the scheme provided in this application will be referred to as saliency-driven dynamic token pruning (SDTP).

[0217] First, the solution provided in this application is applied to the scenario of pruning intermediate layer tokens in LLMLingua2. It is compared with the LLMLingua2 pruning solution using prompt pruning from the perspectives of analysis accuracy and speed. In the same dataset, the analysis results obtained by prompt pruning and the solution provided in this application are as follows: Figure 6a As shown.

[0218] For evaluation in long sequence scenarios, the LongBench open test set for long context understanding is used as the input sample set for testing. LongBench covers single-document question answering (singledoc question & answer, singledocQA), multi-document question answering (multidoc question & answer, multidoc QA), summarization, few-shot learning, and code completion.

[0219] Using training with all tokens as a control group (original token), compared to schemes that prune the prompt, the proposed solution (SDTP) achieves higher accuracy (AVG) and speedup, such as... Figure 6a The experimental results show that, with a compression ratio of 1.61x, the accuracy of the SDTP scheme is 39.82, while the accuracy of the scheme with the prompt trimmed is 33.94. Compared to the control group, the SDTP scheme has a 1.71x speedup in full inference (prefill) and a 1.31x speedup in end-to-end (end2end) inference. The prompt trimmed scheme, compared to the control group, has a 1.16x speedup in full inference and a 1.31x speedup in end-to-end inference.

[0220] As can be seen from the foregoing comparison, the solution provided in this application has both better acceleration effect and higher analysis accuracy.

[0221] Then, the latency and floating-point operations per second (FLOPs) of the scheme provided in this application under different sequence lengths were analyzed. The impact of using the SDTP scheme on latency and FLOPs when different sequence lengths are input as prompts to the model are analyzed, and the results are as follows. Figure 6b As shown.

[0222] Depend onFigure 6b It can be seen that the length of the input prompt includes several different cases such as 4k, 8k, 16k, 32k, 64k and 128k. The input sequence of each length is input into the Mistral-7B model (large language model) containing the SDTP component, or into the Mistral-7B model (large language model) without the SDTP component.

[0223] Depend on Figure 6b It can be seen that when the prompt length is 128k, introducing SDTP in LLM can reduce FLOPs by 47%, reduce latency by 1.75 times, and reduce peak memory usage by 34.26%.

[0224] SDTP was then applied to Llama2-7B, Mitral-7B, and BLOOM-7B to verify whether the LLM model incorporating SDTP retains the strong generalization ability of the LLM model. In this validation, the generalization strength of the LLM model incorporating SDTP was verified on multiple 5-shot downstream tasks. The experimental results are as follows: Figure 6c As shown.

[0225] To comprehensively verify the impact of the SDTP component on the generalization performance of LLM models, accuracy was validated using both standard LLM models and LLM models incorporating the SDTP component in various scenarios: Choice of Plausible Alternatives (COPA), Physical Interaction Question Answering (PIQA), WinoGrande (an adversarial Winograd schema challenge at scale), Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms (MathQA), Boolean Questions (BoolQ), the Word-in-Context Dataset (WiC), the Winograd Schema Challenge Dataset (WsC), and the Commitment Bank (CB). The LLM model incorporating the SDTP component cumulatively pruned 65% of the tokens. Compared to the standard LLM model that retains all tokens, the LLM model including the SDTP component still exhibits comparable accuracy. This result, demonstrated by its performance on different datasets, fully illustrates that the introduction of the SDTP component not only significantly reduces the analysis time of the LLM model but also preserves the strong generalization ability inherent in the LLM model itself.

[0226] Finally, based on the Llama-7B model, the Llama-7B model was used as a control group. The SDTP component was added to the Llama-7B model, and a simplified version of the SDTP component was added to the Llama-7B model. In the simplified version of the SDTP component, during the model training process, the token's significance score value was used to supervise the training of the token's importance score value.

[0227] Based on a 5-shot evaluation task, ablation experiments were conducted in the COPA, PIQA, WinoGrande, and MathQA scenarios. The aforementioned three Llama-7B models constitute the three experimental groups for the ablation experiments. The experimental results are as follows: Figure 6d As shown.

[0228] Depend onFigure 6d It can be seen that, compared to the Llama-7B model (w / o Rank & MSE), the Llama-7B model with the simplified SDTP component (w / o Rank) exhibits better accuracy, indicating that the Llama-7B model with the simplified SDTP component can effectively retain important tokens and reduce the impact of redundant tokens on the analysis results. Furthermore, the model trained based on importance score ranking and significance score ranking shows even better accuracy, indicating that the ranking-based optimization strategy can further improve the accuracy of the SDTP component in pruning redundant tokens.

[0229] Furthermore, another set of comparative experiments was conducted based on the Mistral-7B model (without Rank & MSE), with the addition of the SDTP component (SDTP) to the Mistral-7B model, and with the addition of a simplified version of the SDTP component (without Rank) to the Mistral-7B model. These comparative experiments were performed on the LongBench dataset, and the results are as follows: Figure 6e As shown.

[0230] Depend on Figure 6e It can be seen that, compared to the Mistral-7B model, the model with the simplified SDTP component added to the Mistral-7B model exhibits better accuracy, indicating that adding the simplified SDTP component to the Mistral-7B model can effectively retain important tokens and reduce the impact of redundant tokens on the analysis results. Furthermore, the model trained based on importance score ranking and significance score ranking shows even better accuracy, indicating that the ranking-based optimization strategy can further improve the accuracy of the SDTP component in pruning redundant tokens.

[0231] The data processing method provided in this application has been described above. The data processing apparatus provided in this application will now be described in conjunction with the accompanying drawings:

[0232] Please see Figure 7 , Figure 7 A schematic diagram of a data processing apparatus provided in this application.

[0233] The data processing apparatus 10 includes:

[0234] Acquisition unit 110 is used to acquire the target sample set;

[0235] The processing unit 120 is used to extract features from samples in the target sample set to obtain a target vector set;

[0236] The processing unit 120 is also used to input the target vector set into the converter block to obtain the reference vector set of the Nth layer output of the converter block, where N is a positive integer;

[0237] Processing unit 120 is also used to input the reference vector set into the first prediction module to obtain a reference score set. The first prediction module is used to score the input vector to obtain the importance score of the input vector. The reference score set includes the importance score of the sample corresponding to each vector in the obtained reference vector set by analyzing the reference vector set.

[0238] The processing unit 120 is also used to input the reference score set and the reference vector set into the first pruning module to obtain the updated reference vector set. The first pruning module is used to remove the vector corresponding to the sample with the lowest importance score in the vector set corresponding to the input importance score set based on the input importance score set.

[0239] The processing unit 120 is also used to input the updated reference vector set into the N+1th layer of the transformer block to obtain a preset feature vector set;

[0240] The processing unit 120 is also used to process a preset feature vector set to obtain the target prediction result.

[0241] Optionally, the processing unit 120 is specifically used for:

[0242] The updated reference vector set is input into the (N+1)th layer of the converter block to obtain the process vector set output by the (N+a)th layer of the converter block, where a is a positive integer;

[0243] The process vector set is input into the second prediction module to obtain the process score set. The second prediction module is used to score the input vectors to obtain the importance score of the input vectors. The process score set includes the importance score of the sample corresponding to each vector in the process vector set obtained by analyzing the process vector set.

[0244] The process score set and process score are input into the second pruning module. The second pruning module is used to remove the vector corresponding to the sample with the lowest importance score in the process vector set to obtain the updated process vector set.

[0245] The updated process vector set is input into the N+a+1 layer of the transformer block to obtain the preset feature vector set.

[0246] Optionally, the acquisition unit can also be used to acquire the training sample set;

[0247] Processing unit 120 is also used for:

[0248] Feature extraction is performed on the training sample set to obtain the training vector set;

[0249] Input the training vector set into the transformer block to obtain a first saliency score set and a first vector set. The first saliency score set includes the saliency score of each sample in the training sample set at the Nth layer of the transformer block, and the first vector set includes the output vector of each sample in the training sample set at the Nth layer of the transformer block.

[0250] The first vector set is input into the first prediction module to be trained to obtain the first importance score set, which includes the importance score of each sample in the training sample set obtained by analyzing the first vector set.

[0251] Based on the first saliency score set and the first importance score set, supervised training is performed on the first prediction module to be trained to obtain the first prediction module.

[0252] Optionally, the processing unit 120 is also used for:

[0253] Input the first importance score set and the first vector set into the first clipping module to obtain the updated first vector set;

[0254] The updated first vector set is input into the N+1 layer of the transformer block to obtain the second saliency score set and the second vector set. The second saliency score set includes the saliency score of the first sample set in the N+a layer of the transformer block, and the second vector set includes the output vector of each sample in the first sample set in the N+a layer of the transformer block. The first sample set includes the sample corresponding to each vector in the second vector set.

[0255] The second vector set is input into the second prediction module to be trained to obtain the second importance score set, which includes the importance score of each sample in the first sample set obtained by analyzing the second vector set.

[0256] Based on the second saliency score set and the second importance score set, supervised training is performed on the second prediction module to be updated to obtain the second prediction module.

[0257] Optionally, the processing unit 120 is specifically used for:

[0258] Based on the first significance score set, the samples in the training sample set are sorted in descending order of significance score to obtain the first sequence;

[0259] Based on the first importance score set, the samples in the training sample set are sorted in descending order of importance score to obtain the second sequence;

[0260] By increasing the similarity between the first sequence and the second sequence, the first importance score set is updated to obtain the first prediction module.

[0261] Optionally, the processing unit 120 is specifically used for:

[0262] Input the training vectors into the transformer block to obtain the first vector set and the first gradient value. The first gradient is the gradient value of the Nth layer of the transformer block in backpropagation.

[0263] The first saliency score set is obtained by multiplying each vector in the first vector set with the first gradient value.

[0264] Optionally, the processing unit 120 is specifically used for:

[0265] The training vectors are input into the transformer block to obtain a first vector set and a first attention score, which includes the attention score of each sample in each layer of the transformer block.

[0266] The attention scores of the first sample in each layer of the transformer block are accumulated to obtain the first saliency score. The first saliency score is included in the first saliency score set, and the first sample is any sample in the training sample set.

[0267] Please see Figure 8 , Figure 8 Another schematic diagram of the data processing apparatus provided in this application.

[0268] The data processing apparatus 20 includes:

[0269] Acquisition unit 210 is used to acquire the training sample set;

[0270] Processing unit 220 is used to extract features from the training sample set to obtain a training vector set;

[0271] The processing unit 220 is further configured to input the training vector set into the transformer block to obtain a first saliency score set and a first vector set. The first saliency score set includes the saliency score of each sample in the training sample set at the Nth layer of the transformer block, and the first vector set includes the output vector of each sample in the training sample set at the Nth layer of the transformer block, where N is a positive integer.

[0272] The processing unit 220 is further configured to input the first vector set into the first prediction module to be trained to obtain a first importance score set, the first importance score set including the importance score of each sample in the training sample set obtained by analyzing the first vector set;

[0273] The processing unit 220 is also used to perform supervised training on the first prediction module to be trained based on the first saliency score set and the first importance score set, so as to obtain the first prediction module.

[0274] Optionally, the processing unit 220 is also used for:

[0275] Input the first importance score set and the first vector set into the first clipping module to obtain the updated first vector set;

[0276] The updated first vector set is input into the N+1 layer of the transformer block to obtain the second saliency score and the second vector set. The second saliency score set includes the saliency score of the first sample in the N+a layer of the transformer block, and the second vector set includes the output vector of the first sample set in the N+a layer of the transformer block. The first sample set is the sample contained in the second vector set, and a is a positive integer.

[0277] The second vector set is input into the second prediction module to be trained to obtain the second importance score set, which includes the importance score of each sample in the first sample set obtained by analyzing the second vector set.

[0278] Based on the second saliency score set and the second importance score set, supervised training is performed on the second prediction module to be updated to obtain the second prediction module.

[0279] Optionally, the processing unit 220 is specifically used for:

[0280] Based on the first significance score set, the samples in the training sample set are sorted in descending order of significance score to obtain the first sequence;

[0281] Based on the first importance score set, the samples in the training sample set are sorted in descending order of importance score to obtain the second sequence;

[0282] By increasing the similarity between the first sequence and the second sequence, the first importance score set is updated to obtain the first prediction module.

[0283] Optionally, the processing unit 220 is specifically used for:

[0284] Input the training vectors into the transformer block to obtain the first vector set and the first gradient value. The first gradient is the gradient value of the Nth layer of the transformer block in backpropagation.

[0285] The first saliency score set is obtained by multiplying each vector in the first vector set with the first gradient value.

[0286] Optionally, the processing unit 220 is specifically used for:

[0287] Input the training vectors into the transformer block to obtain the first vector set and the first attention score. The first attention score includes the attention score of the first sample in each layer of the transformer block. The first sample is any sample in the training sample set.

[0288] The attention scores of the first sample at each layer of the transformer block are accumulated in the first attention score to obtain the first saliency score, which is contained in the first saliency score set.

[0289] Please see Figure 9 , Figure 9 Another schematic diagram of the data processing apparatus provided in the embodiments of this application.

[0290] like Figure 9 As shown, the computing device 300 includes a bus 303, a memory 304, a processor 305, and a communication interface 306. The processor 305, the memory 304, and the communication interface 306 communicate with each other via the bus 303. The computing device 300 can be a server or a terminal device. It should be understood that the present invention does not limit the number of processors and memories in the computing device 300.

[0291] Bus 303 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 9 The bus 303 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 303 may include a path for transmitting information between various components of the computing device 300 (e.g., memory 304, processor 305, communication interface 306).

[0292] Processor 305 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0293] Memory 304 may include volatile memory, such as random access memory (RAM). Processor 305 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0294] The memory 304 stores executable program code to implement the data processing method. That is, the memory 304 stores instructions from the cloud service platform for executing the data processing provided in this embodiment.

[0295] The communication interface 306 uses transceiver units such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 300 and other devices or communication networks.

[0296] The computing device 300 is used to execute the operations performed by the first server in the foregoing embodiments to implement the screen sharing method provided in the embodiments of this application.

[0297] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0298] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0299] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0300] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0301] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method of data processing, characterized by, The method comprises the following steps: obtaining a target sample set; performing feature extraction on the samples in the target sample set to obtain a target vector set; inputting the target vector set into a transformer block to obtain a reference vector set output by the Nth layer of the transformer block, wherein N is a positive integer; inputting the reference vector set into a first prediction module to obtain a reference score set, the first prediction module being configured to score the input vector to obtain an importance score of the input vector, and the reference score set comprising the importance score of each sample corresponding to each vector in the reference vector set obtained by analyzing the reference vector set; inputting the reference score set and the reference vector set into a first pruning module to obtain an updated reference vector set, the first pruning module being configured to remove the vector corresponding to the sample with the lowest importance score from the vector set corresponding to the input importance score set based on the input importance score set; inputting the updated reference vector set into the N+1th layer of the transformer block to obtain a preset feature vector set; processing the preset feature vector set to obtain a target prediction result.

2. The method of claim 1, wherein, The method of inputting the updated reference vector set into the N+1th layer of the transformer block to obtain a preset feature vector set comprises the following steps: inputting the updated reference vector set into the N+1th layer of the transformer block to obtain a process vector set output by the N+a layer of the transformer block, wherein a is a positive integer; inputting the process vector set into a second prediction module to obtain a process score set, the second prediction module being configured to score the input vector to obtain an importance score of the input vector, and the process score set comprising the importance score of each sample corresponding to each vector in the process vector set obtained by analyzing the process vector set; inputting the process score set and the process score into a second pruning module, the second pruning module being configured to remove the vector corresponding to the sample with the lowest importance score from the process vector set to obtain an updated process vector set; inputting the updated process vector set into the N+a+1th layer of the transformer block to obtain the preset feature vector set.

3. The method according to claim 1 or 2, characterized in that, Before the step of performing feature extraction on the samples in the target sample set to obtain a target vector set, the method further comprises the following steps: obtaining a training sample set; performing feature extraction on the training sample set to obtain a training vector set; inputting the training vector set into the transformer block to obtain a first saliency score set and a first vector set, the first saliency score set comprising the saliency score of each sample in the training sample set at the Nth layer of the transformer block, and the first vector set comprising the output vector of each sample in the training sample set at the Nth layer of the transformer block; inputting the first vector set into a first prediction module to be trained to obtain a first importance score set, the first importance score set comprising the importance score of each sample in the training sample set obtained by analyzing the first vector set; based on the first saliency score set and the first importance score set, supervising and training the first prediction module to be trained to obtain the first prediction module.

4. The method of claim 3, wherein, The method further comprises: inputting the first importance score set and the first vector set into the first pruning module to obtain an updated first vector set; inputting the updated first vector set into an N+1th layer of the transformer block to obtain a second saliency score set and a second vector set, the second saliency score set comprising saliency scores of first samples in the first sample set at an N+a layer of the transformer block, the second vector set comprising output vectors of each sample in the first sample set at the N+a layer of the transformer block, the first sample set comprising samples corresponding to each vector in the second vector set; inputting the second vector set into a second prediction module to be trained to obtain a second importance score set, the second importance score set comprising importance scores of each sample in the first sample set obtained by analyzing the second vector set; based on the second saliency score set and the second importance score set, performing supervised training on the second prediction module to be updated to obtain the second prediction module.

5. The method according to claim 3 or 4, characterized in that, The method further comprises: based on the first saliency score set, sorting the samples in the training sample set in descending order of saliency scores to obtain a first sequence; based on the first importance score set, sorting the samples in the training sample set in descending order of importance scores to obtain a second sequence; by improving the similarity between the first sequence and the second sequence, updating the first importance score set to obtain the first prediction module.

6. The method according to any one of claims 3 to 5, characterized in that, The method further comprises: inputting the training vector set into the transformer block to obtain a first saliency score set and a first vector set, the first saliency score set comprising saliency scores of each vector in the first vector set, the saliency score of each vector being a product of the vector and a gradient value of an Nth layer of the transformer block in back propagation; based on the product of each vector in the first vector set and the first gradient value, obtaining the first saliency score set.

7. The method according to any one of claims 3 to 5, characterized in that, The method further comprises: inputting the training vector set into the transformer block to obtain a first saliency score set and a first vector set, the first saliency score set comprising saliency scores of each vector in the first vector set, the saliency score of each vector being a product of the vector and a gradient value of an Nth layer of the transformer block in back propagation; based on the product of each vector in the first vector set and the first gradient value, obtaining the first saliency score set.

8. A method of data processing, characterized by, The method further comprises: inputting the training vector set into the transformer block to obtain a first saliency score set and a first vector set, the first saliency score set comprising saliency scores of each vector in the first vector set, the saliency score of each vector being a product of the vector and a gradient value of an Nth layer of the transformer block in back propagation; based on the product of each vector in the first vector set and the first gradient value, obtaining the first saliency score set. The method further comprises: obtaining a training sample set; performing feature extraction on the training sample set to obtain a training vector set; inputting the training vector set into the transformer block to obtain a first saliency score set and a first vector set, the first saliency score set comprising a saliency score of each sample in the training sample set at an Nth layer of the transformer block, the first vector set comprising an output vector of each sample in the training sample set at the Nth layer of the transformer block, N being a positive integer; inputting the first vector set into a first prediction module to be trained to obtain a first importance score set, the first importance score set comprising an importance score of each sample in the training sample set obtained by analyzing the first vector set; supervising training of the first prediction module to be trained based on the first saliency score set and the first importance score set to obtain a first prediction module.

9. The method of claim 8, wherein, After the first vector set is inputted into the first prediction module to be trained to obtain the first importance score set, the method further comprises: inputting the first importance score set and the first vector set into the first pruning module to obtain an updated first vector set; inputting the updated first vector set into an (N+1)th layer of the transformer block to obtain a second saliency score set and a second vector set, the second saliency score set comprising a saliency score of the first sample at an (N+a)th layer of the transformer block, the second vector set comprising an output vector of the first sample set at the (N+a)th layer of the transformer block, the first sample set being a sample contained in the second vector set, a being a positive integer; inputting the second vector set into a second prediction module to be trained to obtain a second importance score set, the second importance score set comprising an importance score of each sample in the first sample set obtained by analyzing the second vector set; supervising training of the second prediction module to be updated based on the second saliency score set and the second importance score set to obtain a second prediction module.

10. The method according to claim 8 or 9, characterized in that, The supervising training of the first prediction module to be trained based on the first saliency score set and the first importance score set to obtain the first prediction module comprises: sorting samples in the training sample set in descending order of saliency scores based on the first saliency score set to obtain a first sequence; sorting samples in the training sample set in descending order of importance scores based on the first importance score set to obtain a second sequence; updating the first importance score set by improving similarity between the first sequence and the second sequence to obtain the first prediction module.

11. The method according to any one of claims 8 to 10, characterized in that, The inputting the training vector set into the transformer block to obtain a first saliency score set and a first vector set comprises: inputting the training vector into the transformer block to obtain the first vector set and a first gradient value, the first gradient being a gradient value of the Nth layer of the transformer block in back propagation; obtaining the first saliency score set based on a product of each vector in the first vector set and the first gradient value.

12. The method according to any one of claims 8 to 10, characterized in that, The inputting the training vector set into the transformer block to obtain a first saliency score set and a first vector set comprises: inputting the training vector set into the transformer block to obtain the first vector set and a first attention score, the first attention score comprising an attention score of a first sample in each layer of the transformer block, the first sample being any sample in the training sample set; accumulating the attention scores of the first sample in each layer of the transformer block in the first attention score to obtain the first saliency score, the first saliency score being included in the first saliency score set.

13. An apparatus for data processing, characterized by The method comprises the following steps: an acquisition unit is configured to acquire a target sample set; a processing unit is configured to perform feature extraction on samples in the target sample set to obtain a target vector set; the processing unit is further configured to input the target vector set into a transformer block to obtain a reference vector set output by an Nth layer of the transformer block, N being a positive integer; the processing unit is further configured to input the reference vector set into a first prediction module to obtain a reference score set, the first prediction module being configured to score an input vector to obtain an importance score of the input vector, the reference score set comprising an importance score of a sample corresponding to each vector in the reference vector set obtained by analyzing the reference vector set; the processing unit is further configured to input the reference score set and the reference vector set into a first pruning module to obtain an updated reference vector set, the first pruning module being configured to remove, based on an importance score set input, a vector corresponding to a sample with the lowest importance score in the vector set corresponding to the importance score set input; the processing unit is further configured to input the updated reference vector set into an (N+1)th layer of the transformer block to obtain a preset feature vector set; the processing unit is further configured to process the preset feature vector set to obtain a target prediction result.

14. An apparatus for data processing, characterized by The method comprises the following steps: an acquisition unit is configured to acquire a training sample set; a processing unit is configured to perform feature extraction on the training sample set to obtain a training vector set; the processing unit is further configured to input the training vector set into a transformer block to obtain a first saliency score set and a first vector set, the first saliency score set comprising a saliency score of each sample in the training sample set at an Nth layer of the transformer block, the first vector set comprising an output vector of each sample in the training sample set at the Nth layer of the transformer block, N being a positive integer; the processing unit is further configured to input the first vector set into a first prediction module to be trained to obtain a first importance score set, the first importance score set comprising an importance score of each sample in the training sample set obtained by analyzing the first vector set; the processing unit is further configured to supervise training of the first prediction module to be trained based on the first saliency score set and the first importance score set to obtain the first prediction module.

15. An apparatus for data processing, characterized by The device comprises a processor and a memory, the memory being configured to store instructions, and the processor being configured to execute the instructions, when the processor executes the instructions, performing the method of any one of claims 1 to 12.

16. A storage medium, characterized by The program, when executed on a computer, causes the computer to perform the method of any one of claims 1 to 12.

17. A cluster of computing devices, characterized in that, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method of any one of claims 1 to 12.

Images