Prompt compression method and device, computer equipment and storage medium

Abstract

The invention belongs to the technical field of artificial intelligence, and relates to a prompt compression method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining a distillation language model, and a vocabulary and an embedded matrix corresponding to the distillation language model, the attention mask of the distillation language model being a pressing mask; respectively adding key point marks in the vocabulary and the embedding matrix to obtain a mark vocabulary and a mark embedding matrix; obtaining a first task sequence, and inserting k continuous key point marks between a task text and an input text of the first task sequence to obtain a first prediction sequence; inputting the first prediction sequence into a distillation language model for prediction operation to obtain output data which is composed of a key point mark group and an input text; vocabularies corresponding to the output data are obtained in the vocabulary, and a target prediction text is obtained. The method effectively reduces the calculation and storage cost, improves the available space in the input context window, and improves the output quality.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a prompt compression method, device, computer equipment and storage medium. Background Art

[0002] Hints are the main way to exploit the multi-task capability of language models (LMs), but hints take up valuable space in the input context window, and repeatedly encoding the same hint is computationally inefficient. Fine-tuning and distillation methods can train specialized LMs without hints, but require retraining the model for each task.

[0003] Existing hint compression techniques, such as prefix tuning or reference encoders, require learning a prefix or reference vector via gradient descent for each task, or distillation over synthetic data.

[0004] However, traditional hint compression methods all increase the training cost and complexity. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a prompt compression method, apparatus, computer device and storage medium to solve the problem that traditional prompt compression methods increase training cost and complexity.

[0006] In order to solve the above technical problems, the embodiment of the present application provides a prompt compression method, which adopts the following technical solution:

[0007] Obtaining a distilled language model and a vocabulary and an embedding matrix corresponding to the distilled language model, wherein the attention mask of the distilled language model is a presting mask;

[0008] Add key point tags to the vocabulary and the embedding matrix respectively to obtain a tag vocabulary and a tag embedding matrix;

[0009] Obtain a first task sequence, insert k consecutive key point markers between the task text and the input text of the first task sequence, and obtain a first prediction sequence;

[0010] Inputting the first prediction sequence into the distilled language model for prediction operation to obtain output data, wherein the output data consists of a key point tag group and the input text;

[0011] The vocabulary corresponding to the output data is obtained in the vocabulary table to obtain the target predicted text.

[0012] Furthermore, the step of inputting the first prediction sequence into the distilled language model for prediction operation to obtain output data specifically includes the following steps:

[0013] Inputting the task text of the first prediction sequence into the distillation language model to obtain a first output matrix;

[0014] Extracting first k output vectors in the first output matrix in order as embedding vectors of the key point marker group;

[0015] Performing a concatenation operation on the embedding vector of the key point marker group and the input text to obtain a concatenation matrix;

[0016] Inputting the concatenated matrix into the distilled language model to obtain a second output matrix;

[0017] The k+1th and subsequent output vectors are extracted from the second output matrix in order as the output data.

[0018] Furthermore, after the step of inputting the first prediction sequence into the distilled language model for prediction operation to obtain output data, the following steps are also included:

[0019] Obtaining a Transformer activation value corresponding to the embedding vector of the key point tag group;

[0020] The task text of the first prediction sequence is used as a key and the Transformer activation value is used as a value to store in a hash table;

[0021] When a second task sequence with the same task is obtained next time, the second task text corresponding to the second task sequence is retrieved from the hash table;

[0022] Concatenate the second Transformer activation value corresponding to the second task text with the second input text in the second task sequence to obtain a second key matrix and a second value matrix;

[0023] Calculate the second key matrix and the second value matrix based on the self-attention mechanism to obtain second output data;

[0024] The vocabulary corresponding to the second output data is obtained from the vocabulary table to obtain a second predicted text.

[0025] Furthermore, the calculation formula of the self-attention mechanism is expressed as:

[0026]

[0027] Where M is a Q×K-dimensional mask matrix, Q and K represent query and key respectively; d represents the dimension of each head.

[0028] Furthermore, before the step of obtaining the distilled language model and the vocabulary and embedding matrix corresponding to the distilled language model, the following steps are also included:

[0029] Call the trained original language model;

[0030] The original language model is distilled according to the context distillation objective function to obtain the distilled language model.

[0031] Furthermore, the context distillation objective function L G (pG,T) is expressed as:

[0032] L G (p G , T)=E t～T，x [D KL (p LM (y|t,x)||p G (y|G(t),x))]

[0033] Among them, p LM (y|t,x) represents the trained original language model, p G (y|G(t),x) represents the distilled language model after context distillation, D KL represents the indicator used to measure the difference between two probability distributions, E t～T,x represents the expected value operator, which means averaging x and t.

[0034] In order to solve the above technical problems, the embodiment of the present application further provides a prompt compression device, which adopts the following technical solution:

[0035] A distillation model acquisition module, used to acquire a distillation language model and a vocabulary and an embedding matrix corresponding to the distillation language model, wherein the attention mask of the distillation language model is a presting mask;

[0036] A key point tag adding module, used to add key point tags to the vocabulary and the embedding matrix respectively, to obtain a tag vocabulary and a tag embedding matrix;

[0037] A first prediction sequence acquisition module is used to acquire a first task sequence, insert k consecutive key point marks between the task text and the input text of the first task sequence, and obtain a first prediction sequence;

[0038] A prediction module, configured to input the first prediction sequence into the distilled language model for prediction operation to obtain output data, wherein the output data consists of a key point tag group and the input text;

[0039] The target predicted text acquisition module is used to acquire the vocabulary corresponding to the output data in the vocabulary table to obtain the target predicted text.

[0040] Furthermore, the prediction module includes:

[0041] A task text input submodule, used for inputting the task text of the first prediction sequence into the distillation language model to obtain a first output matrix;

[0042] An embedding vector extraction submodule, used to extract first k output vectors in the first output matrix in order as embedding vectors of the key point marking group;

[0043] A splicing submodule, used for performing a splicing operation on the embedding vector of the key point mark group and the input text to obtain a splicing matrix;

[0044] A concatenated matrix input submodule, used for inputting the concatenated matrix into the distilled language model to obtain a second output matrix;

[0045] The output data acquisition submodule is used to extract the k+1th and subsequent output vectors in the second output matrix in sequence as the output data.

[0046] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0047] The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the prompt compression method as described above when executing the computer-readable instructions.

[0048] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0049] The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the prompt compression method described above are implemented.

[0050] The present application provides a prompt compression method, comprising: obtaining a distillation language model and a vocabulary and an embedding matrix corresponding to the distillation language model, wherein the attention mask of the distillation language model is a presting mask; adding key point tags to the vocabulary and the embedding matrix respectively to obtain a tag vocabulary and a tag embedding matrix; obtaining a first task sequence, inserting k consecutive key point tags between the task text and the input text of the first task sequence, and obtaining a first prediction sequence; inputting the first prediction sequence into the distillation language model for prediction operation to obtain output data, wherein the output data consists of a key point tag group and the input text; obtaining a vocabulary corresponding to the output data in the vocabulary to obtain a target prediction text. Compared with the prior art, the present application compresses prompts into smaller key point tags, and can reuse cached activation values, thereby reducing computational and storage costs. In addition, prompts take up less space, thereby increasing the available space in the input context window, and key point tags can retain the most important and relevant information in the prompts, thereby improving output quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0052] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0053] Figure 2 This is a flowchart of the implementation of the prompt compression method provided in Example 1 of the present application;

[0054] Figure 3 is a schematic diagram of the structure of the prompt compression device provided in Example 2 of the present application;

[0055] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0057] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0058] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0059] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0060] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0061] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to the laptop computer 1011, tablet computer 1012 or mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0062] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0063] It should be noted that the prompt compression method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the prompt compression device is generally set in the server / terminal device.

[0064] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0065] Embodiment 1

[0066] Continue to refer Figure 2 , shows a flow chart of an embodiment of a prompt compression method according to the present application. The prompt compression method comprises: step S201, step S202, step S203, step S204 and step S205.

[0067] In step S201, a distilled language model and a vocabulary and an embedding matrix corresponding to the distilled language model are obtained, wherein the attention mask of the distilled language model is a presting mask.

[0068] In the embodiment of the present application, the mask matrix is ​​a technique for controlling which positions can be noticed, which can be used to implement different attention modes, such as causal attention or global attention. The attention mechanism is a technique for calculating the correlation between each position in the input sequence and other positions, which can be used to extract important information in the input sequence and generate an output sequence.

[0069] In an embodiment of the present application, in order to implement the presting technology, the present application needs to modify the causal mask and the global mask so that the part after the key point marker in the input sequence cannot notice the part before the key point marker (but can notice the key point marker itself). The present application calls this modified mask a presting mask. Based on the causal mask or the global mask, the presting mask disconnects the part before the key point marker from the part after the key point marker to form an "L" shaped area. In this way, LM can only obtain the information in the prompt through the key point marker, thereby achieving the purpose of prompt compression.

[0070] In step S202, key point tags are added to the vocabulary and the embedding matrix respectively to obtain a tag vocabulary and a tag embedding matrix.

[0071] In an embodiment of the present application, a special key point marker is added to the vocabulary and embedding matrix of the LM. This key point marker can be represented by a special symbol, such as [CLS], and does not repeat with other vocabularies in the LM. The embedding matrix of the LM is a matrix of dimension V×d, where V is the size of the vocabulary and d is the dimension of the embedding vector. After adding the key point marker, the size of the vocabulary becomes V+1, and the embedding matrix becomes (V+1)×d, where the embedding vector corresponding to the key point marker can be randomly initialized or inherited from a pre-trained LM.

[0072] In step S203, a first task sequence is obtained, and k consecutive key point tags are inserted between the task text of the first task sequence and the input text to obtain a first prediction sequence.

[0073] In an embodiment of the present application, given a (task, input) pair (t, x), k consecutive key points are inserted between t and x: (t, g1, ..., gk, x), such as "Translated into French: The cat". In this way, the length of the input sequence is |t| + k + |x|, where |t| and |x| represent the number of words in the task and input, respectively. If the task or input is empty, the corresponding part is omitted. For example, if the task is empty, the input sequence is (g1, ..., gk, x).

[0074] In step S204, the first prediction sequence is input into the distillation language model for prediction operation to obtain output data, wherein the output data consists of a key point tag group and an input text.

[0075] In the embodiment of the present application, this sequence is input into the LM, but the part of the input sequence after the key mark cannot notice the part before the key mark (but can notice the key mark itself). This forces the LM to compress the information in the prompt into the key mark, because the input x (and output y) cannot notice the prompt t. Specifically, the present application implements this restriction by modifying the attention mask of the Transformer.

[0076] In step S205, the vocabulary corresponding to the output data is obtained in the vocabulary table to obtain the target predicted text.

[0077] In an embodiment of the present application, given a new task t and input x, LM is used to predict the corresponding key point tag G(t), and G(t) and x are used as input to generate output y. Specifically, the present application first inputs the task t into the LM to obtain an output matrix O of |t|×d dimensions. Then, the present application selects the first k output vectors from O as the embedding vectors corresponding to the key point tag G(t). Next, the present application concatenates G(t) and x to form an input matrix I of (k+|x|)×d dimensions, and inputs it into the LM to obtain an output matrix O' of (k+|x|)×d dimensions. Finally, the present application selects the k+1th and subsequent output vectors from O' as the embedding vectors corresponding to the output y, and converts them into corresponding words in the vocabulary.

[0078] In the embodiment of the present application, for example, in the above example, assuming that LM predicts that the key point tag G(t) is "traduire en frant ais", the sequence s becomes "traduireen frant ais The cat is on the mat". This sequence is input into LM, and it is assumed that the output y obtained is "Le chat est sur le tapis", which is the French translation of "The cat is on the mat".

[0079] This is an example of using presting technology for prompt compression and multi-task reasoning. In this way, the application can effectively improve the computational efficiency and multi-task capability of LM.

[0080] In practical applications, assume that this application has a pre-trained language model LM, which can perform a variety of different tasks, such as translation, summarization, question answering, etc. This application hopes to use presting technology to compress prompts to improve the computational efficiency and multi-task capability of LM, specifically:

[0081] First, the application needs to add a special key point marker to the vocabulary and embedding matrix of the LM. Then, for each task t and input x, the application inserts k consecutive key point markers between t and x to form a sequence s. For example, if task t is "translate into French" and input x is "The cat is on the mat", the sequence s is "translate into French The cat is on the mat". Next, the application inputs the sequence s into the LM and uses a presting mask to restrict the part of the input sequence after the key point marker from not noticing the part before the key point marker (but can notice the key point marker itself). This forces the LM to compress the information in the cue into the key point marker, because the input x (and output y) cannot notice the cue t.

[0082] In an embodiment of the present application, a prompt compression method is provided, including: obtaining a distillation language model and a vocabulary and an embedding matrix corresponding to the distillation language model, wherein the attention mask of the distillation language model is a presting mask; adding key point tags to the vocabulary and the embedding matrix respectively, obtaining a tag vocabulary and a tag embedding matrix; obtaining a first task sequence, inserting k consecutive key point tags between the task text and the input text of the first task sequence, obtaining a first prediction sequence; inputting the first prediction sequence into the distillation language model for prediction operation, obtaining output data, wherein the output data consists of a key point tag group and an input text; obtaining vocabulary corresponding to the output data in the vocabulary, obtaining a target prediction text. Compared with the prior art, the present application compresses prompts into smaller key point tags, and can reuse cached activation values, thereby reducing computational and storage costs. In addition, prompts take up less space, thereby increasing the available space in the input context window, and key point tags can retain the most important and relevant information in the prompts, thereby improving output quality.

[0083] In some optional implementations of the embodiments of the present application, the step of inputting the first prediction sequence into the distillation language model for prediction operation to obtain output data specifically includes the following steps:

[0084] Inputting the task text of the first prediction sequence into the distillation language model to obtain a first output matrix;

[0085] Extract the first k output vectors in the first output matrix in order as the embedding vectors of the key point tag group;

[0086] Perform concatenation operation on the embedding vector of the key point tag group and the input text to obtain a concatenation matrix;

[0087] Input the concatenated matrix into the distillation language model to obtain a second output matrix;

[0088] In the second output matrix, the k+1th and subsequent output vectors are extracted in sequence as output data.

[0089] In an embodiment of the present application, the present application first inputs the task t into the LM to obtain an output matrix O of |t|×d dimensions. Then, the present application selects the first k output vectors from O as the embedding vectors corresponding to the key point marker G(t). Next, the present application concatenates G(t) and x to form an input matrix I of (k+|x|)×d dimensions, and inputs it into the LM to obtain an output matrix O' of (k+|x|)×d dimensions. Finally, the present application selects the k+1th and subsequent output vectors from O' as the embedding vectors corresponding to the output y.

[0090] In some optional implementations of the embodiments of the present application, after the step of inputting the first prediction sequence into the distillation language model for prediction operation to obtain output data, the following steps are also included:

[0091] Get the Transformer activation value corresponding to the embedding vector of the key point token group;

[0092] The task text of the first prediction sequence is used as the key and the Transformer activation value is used as the value in the hash table;

[0093] When a second task sequence with the same task is obtained next time, the second task text corresponding to the second task sequence is retrieved from the hash table;

[0094] Concatenate the second Transformer activation value corresponding to the second task text with the second input text in the second task sequence to obtain a second key matrix and a second value matrix;

[0095] Calculate a second key matrix and a second value matrix based on the self-attention mechanism to obtain second output data;

[0096] The vocabulary corresponding to the second output data is obtained in the vocabulary table to obtain the second predicted text.

[0097] In an embodiment of the present application, in order to improve computational efficiency, the Transformer activation values ​​(e.g., key and value matrices) corresponding to G(t) are cached and reused the next time the same task t is encountered. Specifically, the present application first calculates the key and value matrices K and V corresponding to G(t), and stores them in a hash table H, with task t as the key and (K, V) as the value. Then, the next time the same task t is encountered, the present application directly retrieves (K, V) from H, and concatenates them with the key and value matrices corresponding to the input x to form a (k+|x|)×d-dimensional key matrix K' and a (k+|x|)×d-dimensional value matrix V'. Next, the present application uses K' and V' to perform self-attention calculations to obtain the output matrix O'. In this way, the present application avoids repeatedly calculating the key and value matrices corresponding to G(t), thereby saving computing time and resources.

[0098] In some optional implementations of the embodiments of the present application, the calculation formula of the above self-attention mechanism is expressed as:

[0099]

[0100] Where M is a Q×K-dimensional mask matrix, Q and K represent query and key respectively; d represents the dimension of each head.

[0101] In the embodiment of the present application, the mask matrix is ​​a technique for controlling which positions can be noticed, which can be used to implement different attention modes, such as causal attention or global attention. The attention mechanism is a technique for calculating the correlation between each position in the input sequence and other positions, which can be used to extract important information in the input sequence and generate an output sequence.

[0102] In an embodiment of the present application, it is assumed that the present application has an input sequence s=(s1, s2, s3, s4), where each si represents a word or a symbol. The present application wants to use a language model (LM) to generate an output sequence y=(y1, y2, y3, y4), where each yi represents a word or a symbol. In order to do this, the present application needs to use LM to calculate the correlation between each position and other positions in the input sequence s, that is, the attention weights. The attention weight can be represented by a matrix A, where Aij represents the attention weight of the i-th position to the j-th position. For example, A12 represents the attention weight of the 1st position to the 2nd position, A23 represents the attention weight of the 2nd position to the 3rd position, and so on. The attention weight matrix A can be calculated using the following formula.

[0103] In the embodiment of the present application, the Transformer consists of L layers, each layer has M multi-head self-attention mechanisms. Each self-attention mechanism receives a Q×d-dimensional query matrix Q, a K×d-dimensional key matrix K, and a V×d-dimensional value matrix V, and outputs a Q×d-dimensional output matrix O. Where Q, K, V, and O represent query, key, value, and output, respectively, and d is the dimension of each head. The calculation formula of the self-attention mechanism is as follows:

[0104]

[0105] Among them, M is a Q×K-dimensional mask matrix, which is used to control which positions can be noticed. The elements in M ​​take values ​​of 0 or negative infinity, 0 means that it can be noticed, and negative infinity means that it cannot be noticed. When there is no mask, all elements in M ​​are 0. When there is a mask, some elements in M ​​are negative infinity, so that the corresponding position outputs a weight close to 0 after softmax. For decoder models (such as the GPT series), a causal mask is usually used, that is, only the current position and its previous position are allowed to be noticed. For encoder-decoder models (such as the T5 series), the encoder usually has no mask, the decoder uses a causal mask, and a global mask is used when cross-attention is performed between the decoder and the encoder, that is, all positions are allowed to be noticed.

[0106] In some optional implementations of the embodiments of the present application, before the step of obtaining the distilled language model and the vocabulary and embedding matrix corresponding to the distilled language model, the following steps are also included:

[0107] Call the trained original language model;

[0108] The original language model is distilled according to the context distillation objective function to obtain a distilled language model.

[0109] In some optional implementations of the embodiments of the present application, the context distillation objective function L G (p G ,T) is expressed as:

[0110] L G (p G ,T)=E t～T，x [D KL (p LM (y|t,x)||p G (y|G(t),x)

[0111] Among them, p LM (y|t,x) represents the trained original language model, p G(y|G(t),x) represents the distilled language model after context distillation, D KL represents the indicator used to measure the difference between two probability distributions, E t～T,x represents the expected value operator, which means averaging x and t.

[0112] In an embodiment of the present application, this formula represents the objective function of the context distillation of the present application, which is a method for training a new language model (LM) so that it can approximate the behavior of the original LM without the need for a prompt ("context") t.

[0113] In an embodiment of the present application, the objective function of the context distillation of the present application is a method for training a new language model (LM) so that it can approximate the behavior of the original LM without the need for a prompt ("context") t.

[0114] In an embodiment of the present application, given a fixed prompt t, the present application wants to train a new language model pG so that it can generate an output y that is as similar or identical to the original language model pLM as possible without the need for t. To do this, the present application needs to minimize the loss function LG(pG,T), that is, minimize the KL divergence between pLM(y|t,x) and pG(y|G(t),x). In this way, context distillation can be achieved, that is, the dependence on the prompt t is removed.

[0115] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0116] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0117] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0118] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0119] Embodiment 2

[0120] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a prompt compression device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0121] like Figure 3 As shown, the prompt compression device 200 of the embodiment of the present application includes:

[0122] A distillation model acquisition module 210 is used to acquire a distillation language model and a vocabulary and an embedding matrix corresponding to the distillation language model, wherein the attention mask of the distillation language model is a presting mask;

[0123] A key point tag adding module 220, used to add key point tags to the vocabulary and the embedding matrix respectively to obtain a tag vocabulary and a tag embedding matrix;

[0124] A first prediction sequence acquisition module 230 is used to acquire a first task sequence, insert k consecutive key point marks between the task text of the first task sequence and the input text, and obtain a first prediction sequence;

[0125] A prediction module 240 is used to input the first prediction sequence into the distillation language model for prediction operation to obtain output data, wherein the output data consists of a key point tag group and an input text;

[0126] The target predicted text acquisition module 250 is used to acquire words corresponding to the output data in the vocabulary table to obtain the target predicted text.

[0127] In an embodiment of the present application, a prompt compression device 200 is provided, including: a distillation model acquisition module 210, used to obtain a distillation language model and a vocabulary and an embedding matrix corresponding to the distillation language model, wherein the attention mask of the distillation language model is a presting mask; a key point mark adding module 220, used to add key point marks to the vocabulary and the embedding matrix respectively, to obtain a mark vocabulary and a mark embedding matrix; a first prediction sequence acquisition module 230, used to obtain a first task sequence, insert k consecutive key point marks between the task text and the input text of the first task sequence, to obtain a first prediction sequence; a prediction module 240, used to input the first prediction sequence into the distillation language model for prediction operation, to obtain output data, wherein the output data consists of a key point mark group and an input text; a target prediction text acquisition module 250, used to obtain vocabulary corresponding to the output data in the vocabulary, to obtain a target prediction text. Compared with the prior art, the present application compresses prompts into smaller key point tags and can reuse cached activation values, thereby reducing computational and storage costs. In addition, prompts take up less space, thereby increasing the available space in the input context window. Key point tags can retain the most important and relevant information in the prompt, thereby improving output quality.

[0128] In some optional implementations of the embodiments of the present application, the prediction module includes:

[0129] A task text input submodule, used for inputting the task text of the first prediction sequence into the distillation language model to obtain a first output matrix;

[0130] An embedding vector extraction submodule, used to extract the first k output vectors in the first output matrix in order as the embedding vectors of the key point marker group;

[0131] A concatenation submodule is used to concatenate the embedding vectors of the key point tag group and the input text to obtain a concatenation matrix;

[0132] A concatenated matrix input submodule, used for inputting the concatenated matrix into the distillation language model to obtain a second output matrix;

[0133] The output data acquisition submodule is used to extract the k+1th and subsequent output vectors in the second output matrix in order as output data.

[0134] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of a computer device according to an embodiment of the present application.

[0135] The computer device 300 includes a memory 310, a processor 320, and a network interface 330 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 300 having components 310-330, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0136] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0137] The memory 310 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 310 can be an internal storage unit of the computer device 300, such as a hard disk or memory of the computer device 300. In other embodiments, the memory 310 can also be an external storage device of the computer device 300, such as a plug-in hard disk equipped on the computer device 300, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card, etc. Of course, the memory 310 can also include both the internal storage unit of the computer device 300 and its external storage device. In the embodiment of the present application, the memory 310 is generally used to store the operating system and various application software installed on the computer device 300, such as computer-readable instructions for prompting compression methods, etc. In addition, the memory 310 can also be used to temporarily store various data that have been output or are to be output.

[0138] The processor 320 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 320 is generally used to control the overall operation of the computer device 300. In the embodiment of the present application, the processor 320 is used to run the computer-readable instructions stored in the memory 310 or process data, such as running the computer-readable instructions of the prompt compression method.

[0139] The network interface 330 may include a wireless network interface or a wired network interface. The network interface 330 is generally used to establish a communication connection between the computer device 300 and other electronic devices.

[0140] The computer device provided by the present application compresses prompts into smaller key point tags and can reuse cached activation values, thereby reducing computing and storage costs. In addition, the prompts take up less space, thereby increasing the available space in the input context window. The key point tags can retain the most important and relevant information in the prompts, thereby improving output quality.

[0141] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the prompt compression method as described above.

[0142] The computer-readable storage medium provided by the present application compresses prompts into smaller key point tags and can reuse cached activation values, thereby reducing computing and storage costs. In addition, the prompts take up less space, thereby increasing the available space in the input context window. The key point tags can retain the most important and relevant information in the prompts, thereby improving the output quality.

[0143] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0144] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A prompt compression method, characterized in that: The steps include: Obtaining a distilled language model and a vocabulary and an embedding matrix corresponding to the distilled language model, wherein the attention mask of the distilled language model is a presting mask; Add key point tags to the vocabulary and the embedding matrix respectively to obtain a tag vocabulary and a tag embedding matrix; Obtain a first task sequence, insert k consecutive key point markers between the task text and the input text of the first task sequence, and obtain a first prediction sequence; Inputting the first prediction sequence into the distilled language model for prediction operation to obtain output data, wherein the output data consists of a key point tag group and the input text; The vocabulary corresponding to the output data is obtained in the vocabulary table to obtain the target predicted text.

2. The prompt compression method according to claim 1, characterized in that: The step of inputting the first prediction sequence into the distilled language model for prediction operation to obtain output data specifically includes the following steps: Inputting the task text of the first prediction sequence into the distillation language model to obtain a first output matrix; Extracting first k output vectors in the first output matrix in order as embedding vectors of the key point marker group; Performing a concatenation operation on the embedding vector of the key point marker group and the input text to obtain a concatenation matrix; Inputting the concatenated matrix into the distilled language model to obtain a second output matrix; The k+1th and subsequent output vectors are extracted from the second output matrix in order as the output data.

3. The prompt compression method according to claim 2, characterized in that: After the step of inputting the first prediction sequence into the distilled language model for prediction operation to obtain output data, the following steps are also included: Obtaining a Transformer activation value corresponding to the embedding vector of the key point tag group; The task text of the first prediction sequence is used as a key and the Transformer activation value is used as a value to store in a hash table; When a second task sequence with the same task is obtained next time, the second task text corresponding to the second task sequence is retrieved from the hash table; Concatenate the second Transformer activation value corresponding to the second task text with the second input text in the second task sequence to obtain a second key matrix and a second value matrix; Calculate the second key matrix and the second value matrix based on the self-attention mechanism to obtain second output data; The vocabulary corresponding to the second output data is obtained from the vocabulary table to obtain a second predicted text.

4. The prompt compression method according to claim 3, characterized in that: The calculation formula of the self-attention mechanism is expressed as: Where M is a Q×K-dimensional mask matrix, Q and K represent query and key respectively; d represents the dimension of each head.

5. The prompt compression method according to claim 1, characterized in that: Before the step of obtaining the distilled language model and the vocabulary and embedding matrix corresponding to the distilled language model, the following steps are also included: Call the trained original language model; The original language model is distilled according to the context distillation objective function to obtain the distilled language model.

6. The prompt compression method according to claim 5, characterized in that: The context distillation objective function L G (p G ,T) is expressed as: L G (p G ,T)=E t～T,x [D KL (p LM (y|t,x)‖p G (y|G(t),x))] Among them, p LM (y|t,x) represents the trained original language model, p G (y|G(t),x) represents the distilled language model after context distillation, D KL represents the indicator used to measure the difference between two probability distributions, E t～T,x represents the expected value operator, which means averaging x and t.

7. A prompt compression device, characterized in that: include: A distillation model acquisition module, used to acquire a distillation language model and a vocabulary and an embedding matrix corresponding to the distillation language model, wherein the attention mask of the distillation language model is a presting mask; A key point tag adding module, used to add key point tags to the vocabulary and the embedding matrix respectively, to obtain a tag vocabulary and a tag embedding matrix; A first prediction sequence acquisition module is used to acquire a first task sequence, insert k consecutive key point marks between the task text and the input text of the first task sequence, and obtain a first prediction sequence; A prediction module, configured to input the first prediction sequence into the distilled language model for prediction operation to obtain output data, wherein the output data consists of a key point tag group and the input text; The target predicted text acquisition module is used to acquire the vocabulary corresponding to the output data in the vocabulary table to obtain the target predicted text.

8. The prompt compression device according to claim 7, characterized in that: The prediction module comprises: A task text input submodule, used for inputting the task text of the first prediction sequence into the distillation language model to obtain a first output matrix; An embedding vector extraction submodule, used to extract first k output vectors in the first output matrix in order as embedding vectors of the key point marking group; A splicing submodule, used for performing a splicing operation on the embedding vector of the key point mark group and the input text to obtain a splicing matrix; A concatenated matrix input submodule, used for inputting the concatenated matrix into the distilled language model to obtain a second output matrix; The output data acquisition submodule is used to extract the k+1th and subsequent output vectors in the second output matrix in sequence as the output data.

9. A computer device comprising a memory and a processor, characterized in that: The memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the prompt compression method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the prompt compression method according to any one of claims 1 to 6 are implemented.

Images