A text representation method, device and electronic device based on information compression

Abstract

The present invention discloses a text representation method, device and electronic device based on information compression, and relates to the field of artificial intelligence technology. The method comprises: segmenting and segmenting the current text respectively; converting the numbers of each segmentation and the segmentation corresponding to the segmentation into vector representations and summing them to obtain the text representation of the current text; numbering the representations of several historical paragraphs closest to the current text of the previous text respectively, converting the numbers of each historical paragraph representation into vector representations and summing them with the corresponding historical paragraph representations to obtain the input representation of the historical paragraph; splicing the input representation of the historical paragraph and the text representation of the current text to obtain the hidden state representation; after the language model is calculated, the final encoding representation of each word of the current text is output. This method saves video memory usage, greatly increases the length of the model processing window, and has great advantages in computing consumption; at the same time, it enables the self-attention mechanism to focus on the target word more accurately.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a text representation method, device and electronic device based on information compression. Background Art

[0002] In text generation tasks such as human-computer dialogue and news summaries, deep neural network models based on the Transformer structure are now widely used. Among them, the self-attention module is the core module for completing semantic associations, reference resolution, and logical reasoning between words. Its main operating principle is to calculate the relevance weight of the current word and all the words on its left, perform weighted summation of all the words on the left (including itself), and calculate the information representation of the current word.

[0003] However, the operation mode of the above self-attention module has many challenges and problems when processing long texts with complex logic and many paragraphs:

[0004] First, as the generated text continues to grow, all the word information KV cache (Key and Value cache information) of the left text must be saved (to be used in subsequent calculations), which will inevitably result in extremely high video memory usage requirements, thereby limiting the maximum inference length and maximum field of view; second, during the inference process, the calculation of the self-attention mechanism is O(N^2) complexity related to the text length N, so the longer the text, the exponential growth of computing resource consumption; third, when the text is long and has many paragraphs, the information attention for a single word still only stays at the word level, which will inevitably cause problems such as inaccurate information association and omission of important information.

[0005] The most popular attention design based on rotational position encoding has not effectively solved the above problems. When processing very long texts, it will attenuate the attention to distant information to a certain extent, which will inevitably aggravate the third problem mentioned above. Summary of the invention

[0006] In order to solve the problems existing in the prior art, the present invention provides the following technical solutions.

[0007] A first aspect of the present invention provides a text representation method based on information compression, comprising:

[0008] The current text is segmented and divided into segments; each segment is numbered, and the numbers of each segment and the segment corresponding to the segment are converted into vector representations and summed to obtain the text representation of the current text;

[0009] The historical paragraph representations of the previous text that are closest to the current text are numbered respectively, and the numbers of the historical paragraph representations are converted into vector representations and summed with the corresponding historical paragraph representations to obtain the input representation of the historical paragraph;

[0010] The input representation of the historical paragraph and the text representation of the current text are concatenated to obtain the hidden state representation; the hidden state representation is input into the language model, and after calculation, the final encoding representation of each word of the current text is output;

[0011] The final encoded representation of the last word in each paragraph of the current text is used as the historical paragraph representation of the corresponding paragraph and saved for the next text calculation.

[0012] Preferably, converting the serial number of each historical paragraph representation into a vector representation and summing the serial number with the corresponding historical paragraph representation comprises:

[0013] The historical paragraph representation is subjected to a linear transformation to obtain the linearly transformed historical paragraph representation;

[0014] The numbers of the historical paragraph representations before the linear transformation are converted into vector representations and summed with the corresponding historical paragraph representations after the linear transformation to obtain the input representation of the historical paragraph.

[0015] Preferably, numbering each segment separately includes: using one number for one segment.

[0016] Preferably, the language model adopts a Transformer model.

[0017] Preferably, the step of inputting the hidden state representation into the language model and, after calculation, outputting the final encoded representation of each word of the current text comprises:

[0018] At the attention layer of the current Block layer of the language model, the hidden state representation is used to calculate the word-level query matrix, key matrix, and value matrix, as well as the paragraph-level query matrix and key matrix;

[0019] The paragraph mask difference matrix is ​​obtained by subtracting the paragraph-level mask matrix from the word-level lower triangular mask matrix; wherein the word-level lower triangular mask matrix is ​​a matrix with all 1s on the lower left, 0s on the upper right, and 1s on the diagonal; and the paragraph-level mask matrix is ​​a matrix with all 1s in the paragraph and the rest 0s.

[0020] The output hidden state of the final attention layer is calculated using the following formula:

[0021] ；

[0022] in, is the output hidden state of the final attention layer; , , They are word-level query matrix, key matrix, and value matrix respectively; , They are paragraph-level query matrix and key matrix respectively; for The transposed matrix of for The transposed matrix of is the paragraph mask difference matrix; d is the dimension of the attention head; M is the autoregressive attention matrix in the language model;

[0023] The output hidden state of the final attention layer is input to the feedforward layer of the current Block layer of the language model, and the output result of the feedforward layer is used as the output of the current Block layer;

[0024] The data of multiple Block layers are passed and calculated in sequence, and finally the final encoded representation of each word in the current text is output.

[0025] Preferably, calculating the paragraph-level query matrix and key matrix using the hidden state representation includes:

[0026] ;

[0027] ;

[0028] in, and All are transformation matrices; linear is a linear transformation; is a function that copies the hidden state representation to each word in the paragraph; It is the set of the last vector of each paragraph interval in the hidden state representation, which is called the paragraph hidden state representation; It means the hidden state of the next paragraph is represented by the subtraction of the hidden state of the previous paragraph.

[0029] A second aspect of the present invention provides a text representation device based on information compression, comprising:

[0030] The text representation module of the current text is used to segment the current text into words and segments; number each segment respectively, convert the number of each word and the segment corresponding to the word into a vector representation and sum them up to obtain the text representation of the current text;

[0031] The historical paragraph input representation module is used to number the historical paragraph representations of the previous text that are closest to the current text, convert the number of each historical paragraph representation into a vector representation, and sum it with the corresponding historical paragraph representation to obtain the input representation of the historical paragraph;

[0032] A calculation module is used to concatenate the input representation of the historical paragraph and the text representation of the current text to obtain a hidden state representation; the hidden state representation is input into the language model, and after calculation, the final encoded representation of each word of the current text is output;

[0033] The historical paragraph representation cache module is used to use the final encoded representation of the last word in each paragraph of the current text as the historical paragraph representation of the corresponding paragraph and save it for the next text calculation.

[0034] The present invention also provides a memory storing a plurality of instructions, wherein the instructions are used to implement the text representation method based on information compression as described in the first aspect.

[0035] The present invention also provides an electronic device, comprising a processor and a memory connected to the processor, wherein the memory stores a plurality of instructions, and the instructions can be loaded and executed by the processor so that the processor can execute the text representation method based on information compression as described in the first aspect.

[0036] The beneficial effects of the present invention are:

[0037] (1) The technical solution provided by the present invention only needs to save the overall representation of each historical paragraph for historical information, without saving the KV cache (key and value cache information) of the word, which can greatly save the usage of video memory and greatly increase the theoretical length of the window processed by the model.

[0038] (2) During the reasoning process, under the same text field width, the present invention has a great advantage in computing consumption.

[0039] (3) During the text reasoning and generation process, in addition to judging the relevance based on the meaning of each word, the algorithm also pays attention to the expression of the paragraph and calculates the relevance of the target word in a hierarchical manner, so that the self-attention mechanism can focus on the target word more accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A schematic diagram of the flow of the text representation method based on information compression according to the present invention;

[0041] Figure 2 It is a schematic diagram of the overall structure of the text representation method according to a specific embodiment of the present invention;

[0042] Figure 3 It is a schematic diagram of a method for calculating a paragraph mask difference matrix according to a specific embodiment of the present invention;

[0043] Figure 4 This is a functional structural diagram of the text representation device based on information compression described in the present invention. DETAILED DESCRIPTION

[0044] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0045] The method provided by the present invention can be implemented in the following terminal environment, and the terminal may include one or more of the following components: a processor, a memory, and a display screen. The memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the method described in the following embodiment.

[0046] The processor may include one or more processing cores. The processor uses various interfaces and lines to connect various parts of the entire terminal, and executes various functions of the terminal and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory.

[0047] The memory may include random access memory (RAM) or read-only memory (ROM). The memory may be used to store instructions, programs, codes, code sets or instructions.

[0048] The display screen is used to display the user interface of each application.

[0049] In addition, those skilled in the art can understand that the structure of the above terminal does not constitute a limitation on the terminal, and the terminal may include more or fewer components, or combine certain components, or arrange the components differently. For example, the terminal also includes components such as a radio frequency circuit, an input unit, a sensor, an audio circuit, and a power supply, which will not be described in detail here.

[0050] The core design idea of ​​the present invention is to introduce paragraph-level representation to enhance the single-level representation scheme that relies only on word representation. (1) By controlling the attention mask matrix in the attention mechanism, an independent representation of each paragraph is calculated; (2) The KV cache information of words at very long distances is no longer saved, but is replaced by paragraph representation; (3) The attention mechanism with paragraph weight is adopted to make the attention calculation fully consider the semantic relevance of the paragraph.

[0051] Embodiment 1

[0052] like Figure 1 As shown, an embodiment of the present invention provides a text representation method based on information compression, including:

[0053] S101, segmenting the current text into words and segments; numbering each segment, converting the numbers of each word and the segment corresponding to the word into vector representations and summing them up to obtain a text representation of the current text;

[0054] S102, numbering several historical paragraph representations of the previous text that are closest to the current text, converting the numbers of the historical paragraph representations into vector representations and summing them with the corresponding historical paragraph representations to obtain an input representation of the historical paragraph;

[0055] S103, concatenating the input representation of the historical paragraph and the text representation of the current text to obtain a hidden state representation; inputting the hidden state representation into the language model, and outputting the final encoded representation of each word of the current text after calculation;

[0056] S104, taking the final encoded representation of the last word in each paragraph of the current text as the historical paragraph representation of the corresponding paragraph and saving it for the next text calculation.

[0057] The converting the serial number of each historical paragraph representation into a vector representation and summing the serial number with the corresponding historical paragraph representation comprises:

[0058] The historical paragraph representation is subjected to a linear transformation to obtain the linearly transformed historical paragraph representation;

[0059] The numbers of the historical paragraph representations before the linear transformation are converted into vector representations and summed with the corresponding historical paragraph representations after the linear transformation to obtain the input representation of the historical paragraph.

[0060] The numbering of each segment includes: using one number for each segment.

[0061] In a specific embodiment, Figure 2 As shown, the current input text is: "The weather is nice, we should go to the suburbs to enjoy the beauty of nature", the obtained word segmentation includes: ["weather", "nice", ",","we", "should", "go", "suburbs", "enjoy", "for a while", "nature", "of", "beautiful"]; the obtained segments include: ["the weather is nice," "we should go", "enjoy the suburbs", "the beauty of nature"]; the segments are numbered 1, 2, 3, 4 from left to right. Figure 2 middle, ~ They represent the vector representation of the corresponding word segmentation respectively; S-3, S-2, S-1 represent the historical paragraph representations of the previous text closest to the current text after a linear transformation, and their corresponding numbers are 0; R0 ~ R11 represent the final encoding representation of each word after the language model calculation; ~ Respectively represent the final encoding representation based on the word and the probability distribution of the next word prediction; , R and R Respectively represent several historical paragraphs of the previous text that are closest to the current text.

[0062] In a preferred embodiment of the present invention, the language model adopts a Transformer model.

[0063] The step of inputting the hidden state representation into the language model and, after calculation, outputting the final encoded representation of each word of the current text comprises:

[0064] At the attention layer of the current Block layer of the language model, the hidden state representation is used to calculate the word-level query matrix, key matrix, and value matrix, as well as the paragraph-level query matrix and key matrix;

[0065] The paragraph mask difference matrix is ​​obtained by subtracting the paragraph mask matrix from the word-level lower triangular mask matrix (see Figure 3 ); wherein the word-level lower triangular mask matrix is ​​a matrix with all 1s in the lower left, 0s in the upper right, and 1s in the diagonal; the paragraph-level mask matrix is ​​a matrix with all 1s in the paragraph and the rest 0s;

[0066] The output hidden state of the final attention layer is calculated using the following formula:

[0067] ；

[0068] in, is the output hidden state of the final attention layer; , , They are word-level query matrix, key matrix, and value matrix respectively; , They are paragraph-level query matrix and key matrix respectively; for The transposed matrix of for The transposed matrix of is the paragraph mask difference matrix; d is the dimension of the attention head; M is the autoregressive attention matrix in the language model;

[0069] The output hidden state of the final attention layer is input to the feedforward layer of the current Block layer of the language model, and the output result of the feedforward layer is used as the output of the current Block layer;

[0070] The data of multiple Block layers are passed and calculated in sequence, and finally the final encoded representation of each word in the current text is output.

[0071] Specifically, the paragraph-level query matrix and key matrix are calculated using the hidden state representation, including:

[0072] ;

[0073] ;

[0074] in, and All are transformation matrices; linear is a linear transformation; is a function that copies the hidden state representation to each word in the paragraph; It is the set of the last vector of each paragraph interval in the hidden state representation, which is called the paragraph hidden state representation; It means the hidden state of the next paragraph is represented by the subtraction of the hidden state of the previous paragraph.

[0075] Embodiment 2

[0076] like Figure 4 As shown, another aspect of the present invention also includes a functional module architecture that is completely consistent with the aforementioned method flow, that is, the embodiment of the present invention also provides a text representation device based on information compression, including:

[0077] The text representation module 401 of the current text is used to segment the current text into words and segments; number each segment respectively, convert the number of each word and the segment corresponding to the word into a vector representation and sum them up to obtain the text representation of the current text;

[0078] The historical paragraph input representation module 402 is used to number the historical paragraph representations of the previous text that are closest to the current text, convert the number of each historical paragraph representation into a vector representation, and sum it with the corresponding historical paragraph representation to obtain the historical paragraph input representation;

[0079] The calculation module 403 is used to concatenate the input representation of the historical paragraph and the text representation of the current text to obtain a hidden state representation; input the hidden state representation into the language model, and after calculation, output the final encoded representation of each word of the current text;

[0080] The historical paragraph representation cache module 404 is used to use the final encoded representation of the last word in each paragraph of the current text as the historical paragraph representation of the corresponding paragraph and save it for the next text calculation.

[0081] Furthermore, in the input representation module of the historical paragraph, the converting of the numbers of each historical paragraph representation into a vector representation and summing them with the corresponding historical paragraph representation includes: subjecting the historical paragraph representation to a linear transformation to obtain a linearly transformed historical paragraph representation; converting the numbers of each historical paragraph representation before the linear transformation into a vector representation and summing them with the corresponding linearly transformed historical paragraph representation to obtain the input representation of the historical paragraph.

[0082] Furthermore, in the text representation module of the current text, numbering each segment includes: using one number for one segment.

[0083] Furthermore, in the calculation module, the language model adopts the Transformer model. The hidden state representation is input into the language model, and after calculation, the final encoding representation of each word of the current text is output, including: in the attention layer of the current Block layer of the language model, the word-level query matrix, key matrix and value matrix, as well as the paragraph-level query matrix and key matrix are calculated using the hidden state representation; the paragraph-level mask difference matrix is ​​obtained by subtracting the paragraph-level mask matrix from the word-level lower triangular mask matrix; wherein the word-level lower triangular mask matrix is ​​a matrix with all 1s in the lower left, 0s in the upper right, and 1s in the diagonal; the paragraph-level mask matrix is ​​a matrix with all 1s in the forward direction within the paragraph and the rest being 0s; the output hidden state of the final attention layer is calculated using the following formula:

[0084] ；

[0085] in, is the output hidden state of the final attention layer; , , They are word-level query matrix, key matrix, and value matrix respectively; , They are paragraph-level query matrix and key matrix respectively; for The transposed matrix of for The transposed matrix of is the paragraph mask difference matrix; d is the dimension of the attention head; M is the autoregressive attention matrix in the language model;

[0086] The output hidden state of the final attention layer is input into the feedforward layer of the current Block layer of the language model, and the output result of the feedforward layer is used as the output of the current Block layer; the data of multiple Block layers are passed and calculated in sequence, and finally the final encoded representation of each word in the current text is output.

[0087] Furthermore, the paragraph-level query matrix and key matrix are calculated using the hidden state representation, including:

[0088] ;

[0089] ;

[0090] in, and All are transformation matrices; linear is a linear transformation; is a function that copies the hidden state representation to each word in the paragraph; It is the set of the last vector of each paragraph interval in the hidden state representation, which is called the paragraph hidden state representation; It means the hidden state of the next paragraph is represented by the subtraction of the hidden state of the previous paragraph.

[0091] The device can be implemented by the text representation method based on information compression provided in the above-mentioned embodiment 1. The specific implementation method can be found in the description of embodiment 1 and will not be repeated here.

[0092] The present invention also provides a memory storing a plurality of instructions, wherein the instructions are used to implement the text representation method based on information compression as described in the first embodiment.

[0093] The present invention also provides an electronic device, comprising a processor and a memory connected to the processor, wherein the memory stores a plurality of instructions, and the instructions can be loaded and executed by the processor so that the processor can execute the text representation method based on information compression as described in Example 1.

[0094] Although preferred embodiments of the present invention have been described, additional changes and modifications may be made to these embodiments by those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A text representation method based on information compression, characterized in that: include: The current text is segmented and divided into segments; each segment is numbered, and the numbers of each segment and the segment corresponding to the segment are converted into vector representations and summed to obtain the text representation of the current text; The historical paragraph representations of the previous text that are closest to the current text are numbered respectively, and the historical paragraph representations are subjected to a linear transformation to obtain the historical paragraph representation after the linear transformation; the numbers of the historical paragraph representations before the linear transformation are converted into vector representations and summed with the corresponding historical paragraph representations after the linear transformation to obtain the input representation of the historical paragraph; The input representation of the historical paragraph and the text representation of the current text are concatenated to obtain the hidden state representation; the hidden state representation is input into the language model, and after calculation, the final encoding representation of each word of the current text is output; The final encoding representation of the last word in each paragraph of the current text is used as the historical paragraph representation of the corresponding paragraph and saved for the next text calculation; The step of inputting the hidden state representation into the language model and, after calculation, outputting the final encoded representation of each word of the current text comprises: At the attention layer of the current Block layer of the language model, the hidden state representation is used to calculate the word-level query matrix, key matrix, and value matrix, as well as the paragraph-level query matrix and key matrix; The paragraph mask difference matrix is ​​obtained by subtracting the paragraph-level mask matrix from the word-level lower triangular mask matrix; wherein the word-level lower triangular mask matrix is ​​a matrix with all 1s on the lower left, 0s on the upper right, and 1s on the diagonal; and the paragraph-level mask matrix is ​​a matrix with all 1s in the paragraph and the rest 0s. The output hidden state of the final attention layer is calculated using the following formula: ； in, is the output hidden state of the final attention layer; , , They are word-level query matrix, key matrix, and value matrix respectively; , They are paragraph-level query matrix and key matrix respectively; for The transposed matrix of for The transposed matrix of is the paragraph mask difference matrix; d is the dimension of the attention head; M is the autoregressive attention matrix in the language model; The output hidden state of the final attention layer is input to the feedforward layer of the current Block layer of the language model, and the output result of the feedforward layer is used as the output of the current Block layer; The data of multiple Block layers are passed and calculated in sequence, and finally the final encoded representation of each word in the current text is output.

2. The text representation method based on information compression as claimed in claim 1, characterized in that: The numbering of each segment includes: using one number for each segment.

3. The text representation method based on information compression as claimed in claim 1, characterized in that: The language model adopts the Transformer model.

4. The text representation method based on information compression as claimed in claim 1, characterized in that: The paragraph-level query matrix and key matrix calculated using the hidden state representation include: ； ； in, and All are transformation matrices; linear is a linear transformation; is a function that copies the hidden state representation to each word in the paragraph; It is the set of the last vector of each paragraph interval in the hidden state representation, which is called the paragraph hidden state representation; It means the hidden state of the next paragraph is represented by the subtraction of the hidden state of the previous paragraph.

5. A text representation device based on information compression, characterized in that: include: The text representation module of the current text is used to segment the current text into words and segments; number each segment respectively, convert the number of each word and the segment corresponding to the word into a vector representation and sum them up to obtain the text representation of the current text; The input representation module of the historical paragraph is used to number the historical paragraph representations of the previous text that are closest to the current text, and the historical paragraph representations are subjected to a linear transformation to obtain the historical paragraph representation after the linear transformation; the numbers of the historical paragraph representations before the linear transformation are converted into vector representations and summed with the corresponding historical paragraph representations after the linear transformation to obtain the input representation of the historical paragraph; A calculation module is used to concatenate the input representation of the historical paragraph and the text representation of the current text to obtain a hidden state representation; the hidden state representation is input into the language model, and after calculation, the final encoded representation of each word of the current text is output; A historical paragraph representation cache module is used to use the final encoding representation of the last word in each paragraph of the current text as the historical paragraph representation of the corresponding paragraph and save it for the next text calculation; The step of inputting the hidden state representation into the language model and, after calculation, outputting the final encoded representation of each word of the current text comprises: At the attention layer of the current Block layer of the language model, the hidden state representation is used to calculate the word-level query matrix, key matrix, and value matrix, as well as the paragraph-level query matrix and key matrix; The paragraph mask difference matrix is ​​obtained by subtracting the paragraph-level mask matrix from the word-level lower triangular mask matrix; wherein the word-level lower triangular mask matrix is ​​a matrix with all 1s on the lower left, 0s on the upper right, and 1s on the diagonal; and the paragraph-level mask matrix is ​​a matrix with all 1s in the paragraph and the rest 0s. The output hidden state of the final attention layer is calculated using the following formula: ； in, is the output hidden state of the final attention layer; , , They are word-level query matrix, key matrix, and value matrix respectively; , They are paragraph-level query matrix and key matrix respectively; for The transposed matrix of for The transposed matrix of is the paragraph mask difference matrix; d is the dimension of the attention head; M is the autoregressive attention matrix in the language model; The output hidden state of the final attention layer is input to the feedforward layer of the current Block layer of the language model, and the output result of the feedforward layer is used as the output of the current Block layer; The data of multiple Block layers are passed and calculated in sequence, and finally the final encoded representation of each word in the current text is output.

6. A memory, characterized in that: A plurality of instructions are stored, and the instructions are used to implement the text representation method based on information compression as described in any one of claims 1-4.

7. An electronic device, characterized in that: It comprises a processor and a memory connected to the processor, wherein the memory stores a plurality of instructions, and the instructions can be loaded and executed by the processor so that the processor can execute the text representation method based on information compression as described in any one of claims 1-4.

Images