Text coding method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, can be applied to business system platforms of financial science and technology, medical health and the like, and discloses a text coding method and device, computer equipment and a storage medium, and the method comprises the following steps: obtaining an input text sequence, and generating a key vector matrix corresponding to the input text sequence, the key vector matrix comprises a query matrix, a key matrix and a value matrix; obtaining the attention degree of each element of the input text sequence, and pruning the query matrix based on the attention degree and the query matrix; and based on the query matrix after pruning processing, the key matrix and the value matrix, calculating an attention score of the input text sequence so as to encode the input text sequence. According to the method and the device, the technical problem that the operand of attention calculation and memory occupation cannot be reduced while accurate text coding cannot be ensured in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence technology, and in particular to a text encoding method, device, computer equipment and storage medium. Background Art

[0002] With the development of deep learning models, attention mechanisms, in particular, have become a key component of these models. For example, in the healthcare field, attention mechanisms can be applied to the parsing of input text by intelligent medical consultation robots. By encoding the patient's input text, the attention mechanism can then identify the patient's question, replacing some of the doctor's work, reducing their stress, and providing convenience for users.

[0003] In the field of financial technology, the attention mechanism can be applied to intelligent customer service. The attention mechanism encodes the text input by the patient, and then analyzes the patient's problem, which is convenient for users while reducing the workload of manual customer service. In addition, the attention mechanism can also be used in real-time financial news sentiment analysis and risk warning, and the attention mechanism can capture the complex semantics and emotional information in financial texts.

[0004] While attention mechanisms offer significant advantages for capturing textual information, their computational complexity and storage requirements increase dramatically with the length of the input sequence when processing long or high-frequency real-time data, placing significant pressure on hardware devices like GPUs. Therefore, ensuring accurate text encoding while reducing the computational complexity and memory usage of attention calculations has become a pressing issue.

[0005] Existing optimization methods, such as sparse attention, low-rank approximation, and hierarchical attention, typically reduce computational complexity by structurally adjusting the model architecture. However, these methods often require compromises in accuracy or face significant engineering challenges during implementation, making them difficult to directly apply to financial scenarios that require both real-time performance and high accuracy. Summary of the Invention

[0006] The purpose of the present invention is to overcome the above-mentioned technical deficiencies and provide a text encoding method, device, computer equipment and storage medium to solve the technical problem that the existing technology cannot ensure the accuracy of text encoding while reducing the computational complexity and memory usage of attention calculation.

[0007] In order to achieve the above technical objectives, the present invention adopts the following technical solutions:

[0008] In a first aspect, the present invention provides a text encoding method, comprising the following steps:

[0009] Obtain an input text sequence and generate a key vector matrix corresponding to the input text sequence, wherein the key vector matrix includes a query matrix, a key matrix, and a value matrix;

[0010] Obtaining the attention of each element of the input text sequence, and performing pruning processing on the query matrix based on the attention and the query matrix;

[0011] Based on the query matrix after pruning, the key matrix and the value matrix, an attention score of the input text sequence is calculated to achieve encoding of the input text sequence.

[0012] In some embodiments, the step of obtaining an input text sequence and generating a key vector matrix corresponding to the input text sequence, wherein the key vector matrix includes a query matrix, a key matrix, and a value matrix, includes:

[0013] Get the input text sequence;

[0014] Obtain the pre-trained weight matrix of the query matrix, the weight matrix of the key matrix, and the weight matrix of the value matrix;

[0015] Based on the input text, the weight matrix of the query matrix, the weight matrix of the key matrix and the weight matrix of the value matrix, a key vector matrix corresponding to the input text sequence is performed through matrix operation.

[0016] In some embodiments, obtaining the attention of each element of the input text sequence and pruning the query matrix based on the attention and the query matrix includes:

[0017] Obtaining the attention of each element of the input text sequence through a pre-trained attention model;

[0018] Calculating the importance of each element based on the attention and the query matrix;

[0019] The query matrix is pruned based on a preset importance threshold and the calculated importance of each element.

[0020] In some embodiments, obtaining the attention level of each element of the input text sequence includes:

[0021] Get the text content of each element and the context information of the element;

[0022] Extract features from the acquired text content and context, and input the extracted features into the attention evaluation model;

[0023] The attention level of each element of the input text sequence is determined by an attention level evaluation model.

[0024] In some embodiments, the pruning of the query matrix based on a preset importance threshold and the calculated importance of each element includes:

[0025] Comparing the preset importance threshold with the calculated importance of each element to obtain a comparison result;

[0026] Based on the comparison result, the query matrix is pruned.

[0027] In some embodiments, pruning the query matrix based on the comparison result includes:

[0028] Based on the comparison result, filter out elements whose importance is less than the preset importance threshold; remove the query vectors corresponding to the elements whose importance is less than the preset importance threshold in the query matrix;

[0029] The pruned query matrix is reconstructed to achieve pruning of the query matrix.

[0030] In some embodiments, the calculating the attention score of the input text sequence based on the pruned query matrix, the key matrix, and the value matrix to achieve encoding of the input text sequence includes:

[0031] Calculating an attention weight matrix for the input text sequence based on the pruned query matrix and the key matrix;

[0032] Calculating an attention score for the input text sequence based on the attention weight matrix and the value matrix;

[0033] The calculated attention score is quantized and encoded to generate a final text encoding, thereby completing the encoding of the input text sequence.

[0034] In a second aspect, the present invention further provides a text encoding device, comprising:

[0035] A matrix calculation module, configured to obtain an input text sequence and generate a key vector matrix corresponding to the input text sequence, wherein the key vector matrix includes a query matrix, a key matrix, and a value matrix;

[0036] a pruning module, configured to obtain the attention of each element of the input text sequence, and perform pruning processing on the query matrix based on the attention and the query matrix;

[0037] An encoding module is used to calculate the attention score of the input text sequence based on the query matrix after pruning, the key matrix and the value matrix to achieve encoding of the input text sequence.

[0038] In a third aspect, the present invention further provides a computer device comprising a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the text encoding method as described above when executing the computer-readable instructions.

[0039] In a fourth aspect, the present invention further provides a computer-readable storage medium having computer-readable instructions stored thereon, which implement the steps of the text encoding method described above when executed by a processor.

[0040] Compared to existing technologies, the text encoding method, apparatus, computer device, and storage medium provided by the present invention first obtain an input text sequence and generate a key vector matrix corresponding to the input text sequence, wherein the key vector matrix includes a query matrix, a key matrix, and a value matrix. The attention level of each element in the input text sequence is then obtained and, based on the attention level and the query matrix, the query matrix is pruned. Finally, an attention score for the input text sequence is calculated based on the pruned query matrix, the key matrix, and the value matrix to achieve encoding of the input text sequence. This invention improves upon the traditional self-attention mechanism by dynamically determining whether to retain the attention connection at each position based on the context and complexity of the input sequence. This adaptive pruning mechanism avoids the potential loss of accuracy associated with fixed pruning rate methods. The model dynamically determines whether to retain the attention connection at each position based on the complexity and context of the input data, thereby significantly reducing computing resources and storage space usage while ensuring accurate text encoding recognition. This optimization method is suitable for large-scale, real-time financial data inference on edge devices or in resource-constrained environments, providing financial institutions with more efficient, low-latency real-time monitoring and decision support. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the solutions in the present invention, a brief introduction is given below to the drawings required for use in describing the embodiments of the present invention. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0042] Figure 1 is an exemplary system architecture diagram in which the present invention may be applied;

[0043] Figure 2 is a flowchart of an embodiment of a text encoding method according to the present invention;

[0044] Figure 3 yes Figure 2A flowchart of a specific embodiment of step S100 is shown;

[0045] Figure 4 yes Figure 2 A flowchart of a specific embodiment of step S200 is shown;

[0046] Figure 5 yes Figure 4 A flowchart of a specific embodiment of step S230 is shown;

[0047] Figure 6 yes Figure 2 A flowchart of a specific embodiment of step S300 is shown;

[0048] Figure 7 is a structural diagram of an embodiment of a text encoding device according to the present invention;

[0049] Figure 8 is a schematic structural diagram of an embodiment of a computer device according to the present invention;

[0050] Figure 9 FIG. 1 is a schematic structural diagram of another embodiment of a computer device according to the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0052] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present invention. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute a separate or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0053] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0054] The text encoding method based on artificial intelligence provided by the embodiment of the present invention can be applied in Figure 1In an application environment, a client communicates with a server via a network. The server can first obtain an input text sequence through the client and generate a key vector matrix corresponding to the input text sequence, where the key vector matrix includes a query matrix, a key matrix, and a value matrix. The server then obtains the attention level of each element of the input text sequence and prunes the query matrix based on the attention level and the query matrix. Finally, the server calculates an attention score for the input text sequence based on the pruned query matrix, the key matrix, and the value matrix to encode the input text sequence. This invention improves on the traditional self-attention mechanism by dynamically determining whether to retain the attention connection at each position based on the context and complexity of the input sequence. This adaptive pruning mechanism avoids the potential loss of accuracy associated with fixed pruning rate methods. The model dynamically determines whether to retain the attention connection at each position based on the complexity and context of the input data, thereby significantly reducing computing resources and storage space usage while ensuring accurate text encoding recognition. This optimization method is suitable for large-scale, real-time financial data inference on edge devices or in resource-constrained environments, providing financial institutions with more efficient, low-latency real-time monitoring and decision support. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablet computers, and portable wearable devices. The server can be implemented as an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0055] See also Figure 2 , Figure 2 A flowchart of an embodiment of a text encoding method according to the present invention is shown. The text encoding method is applicable to machine customer service in financial scenarios, capturing complex semantic and emotional information in financial texts, or intelligent consultation robots in medical scenarios, and includes steps S100 to S300.

[0056] S100: Acquire an input text sequence, and generate a key vector matrix corresponding to the input text sequence, wherein the key vector matrix includes a query matrix, a key matrix, and a value matrix.

[0057] In this embodiment, the query matrix plays a role in the self-attention mechanism by performing similarity calculations with the key matrix to determine the contribution of each input position to the final output. The query matrix calculates a similarity score by performing a dot product with the key matrix, which is then normalized using a softmax function to obtain attention weights. These weights are used to perform a weighted summation of the value vector to generate a context-sensitive representation. The key matrix helps calculate the similarity between the query matrix and each input position, thereby determining the contribution of each input position to the final output. The key matrix calculates a similarity score by performing a dot product with the query matrix, which is then normalized using a softmax function to obtain attention weights. The value matrix plays a role in the self-attention mechanism by providing information about the actual content of the input sequence. After calculating the attention weights, the value matrix generates the final attention output through a weighted summation. The value matrix contains information about the actual content of the input sequence. Through the attention mechanism, this information can be transferred and shared between different positions. The value matrix and value vector are combined with the attention weights through a weighted summation to generate a context-sensitive representation. This is particularly important for understanding and generating natural language text, as natural language processing tasks often require consideration of contextual information.

[0058] Example description: In the medical field, users consult with intelligent medical consultation robots and input text information into the intelligent medical consultation robot, such as the user's medical condition information. After receiving the text information input by the user, the system calculates its corresponding query matrix (Query), key matrix (Key), and value matrix (Value).

[0059] In the financial field, taking financial news text information as an example, the system receives real-time captured financial news text information and calculates its corresponding query matrix (Query), key matrix (Key) and value matrix (Value).

[0060] S200: Obtain the attention of each element of the input text sequence, and perform pruning processing on the query matrix based on the attention and the query matrix.

[0061] In this embodiment, to reduce computational burden, the query matrix is pruned by removing less important tokens. This reduces the amount of attention required for unimportant content, thereby reducing overall computational effort and accelerating the overall model's inference speed. The attention level reflects the importance of each element, and the importance score is calculated based on the attention level, thereby achieving pruning of the query matrix.

[0062] S300. Based on the query matrix after pruning, the key matrix and the value matrix, calculate the attention score of the input text sequence to achieve encoding of the input text sequence.

[0063] In this embodiment, after obtaining the query matrix after pruning, the attention score of the input text sequence can be calculated based on the query matrix, the key matrix and the value matrix. The attention score is the final encoding result and can be used for subsequent decoding operations.

[0064] The present invention additionally introduces an adaptive attention pruning algorithm into the attention calculation process of the original transformer architecture. The algorithm dynamically adjusts the attention distribution of each layer according to the contextual features of the corresponding text (such as the financial news text here) to achieve the purpose of saving computing resources.

[0065] It can be understood that the encoding method described in the embodiment of the present invention can be applied to semantic recognition scenarios in medical health, financial technology, etc. For example, in the financial field, it can be applied to the above-mentioned financial news text recognition scenario. In the medical health field, it can be applied to the intelligent medical consultation robot scenario. The user can input the condition through voice or text, and the intelligent medical consultation robot will encode the condition entered by the user. For example, the user enters the condition description sentence "I have had stomach pain for several consecutive days and have a poor appetite." When the intelligent medical consultation robot recognizes the condition entered by the user, it will first perform text encoding and text decoding. Among them, the text encoding method can adopt the text encoding method described in the embodiment of the present invention, calculate the key vector matrix of the text entered by the user through the attention mechanism, and then prune the query matrix in the key vector matrix. Finally, the attention score is calculated through the pruned query matrix, value matrix and key matrix, thereby completing the encoding operation.

[0066] In an embodiment of the present invention, an input text sequence is first acquired and a key vector matrix corresponding to the input text sequence is generated, wherein the key vector matrix includes a query matrix, a key matrix, and a value matrix. The attention level of each element of the input text sequence is then obtained and, based on the attention level and the query matrix, the query matrix is pruned. Finally, an attention score for the input text sequence is calculated based on the pruned query matrix, the key matrix, and the value matrix to achieve encoding of the input text sequence. This invention improves upon the traditional self-attention mechanism by dynamically determining whether to retain the attention connection at each position based on the context and complexity of the input sequence. This adaptive pruning mechanism avoids the potential loss of accuracy associated with fixed pruning rate methods. The model dynamically determines whether to retain the attention connection at each position based on the complexity and context of the input data, thereby significantly reducing computing resources and storage space usage while ensuring accurate text encoding recognition. This optimization method is suitable for large-scale, real-time financial data inference on edge devices or in resource-constrained environments, providing financial institutions with more efficient, low-latency real-time monitoring and decision support.

[0067] In some embodiments, see Figure 3 , the step S100 specifically includes:

[0068] S110, obtaining an input text sequence;

[0069] S120, obtaining a weight matrix of a query matrix, a weight matrix of a key matrix, and a weight matrix of a value matrix obtained by pre-training;

[0070] S130 , performing matrix operations on a key vector matrix corresponding to the input text sequence based on the input text sequence, the weight matrix of the query matrix, the weight matrix of the key matrix, and the weight matrix of the value matrix.

[0071] In this embodiment, the query matrix, key matrix, and value matrix are calculated using a pre-trained weight matrix. The weight matrix of the query matrix, the weight matrix of the key matrix, and the weight matrix of the value matrix are all obtained through training. The weight matrix of the query matrix is multiplied by the input data to generate the query matrix. The main function of the weight matrix of the query matrix is to map the input data to a query space through a linear transformation, so that it can be used for subsequent similarity calculations with the key matrix. Specifically, it helps the model dynamically focus on different parts of the input sequence and capture the relationship between elements that are far apart in the sequence. The weight matrix of the key matrix is a learnable parameter matrix that is multiplied by the input data to generate the key matrix. This key matrix represents the identity or characteristics of each element in the sequence. They are used to perform similarity calculations with the query matrix, thereby determining the contribution of each input position to the final output. The main function of the weight matrix of the key matrix is to help calculate the similarity between the query vector and each input position, thereby determining the contribution of each input position to the final output. The similarity score is calculated by performing a dot product with the query matrix and then normalized using the softmax function to obtain attention weights. These weights are used to perform a weighted summation of the value vector to generate the final attention output. The weight matrix of the value matrix. When multiplied by the input data, the value matrix is generated. The value matrix contains the actual information or features of each element in the sequence. This information is aggregated according to the attention weights to influence the final output. The main function of the weight matrix of the value matrix is to carry the actual information or features of the input sequence.

[0072] For example, given an input sequence X = {x1, x2, ..., x n ) (for example, real-time captured financial news text), the model first calculates its corresponding query matrix (Query), key matrix (Key), and value matrix (Value). The specific calculation formula is as follows:

[0073] Q=XW Q ,K=XW K ,V=XWV ,

[0074] Where W Q 、W K 、W V It is the weight matrix of the query matrix, key matrix and value matrix obtained through training. Q represents the query matrix, K represents the key matrix and V represents the value matrix.

[0075] In some embodiments, see Figure 4 , the step S200 specifically includes:

[0076] S210, obtaining the attention of each element of the input text sequence through a pre-trained attention model;

[0077] S220, calculating the importance of each element based on the attention degree and the query matrix;

[0078] S230 , performing pruning processing on the query matrix based on a preset importance threshold and the calculated importance of each element.

[0079] In this embodiment, the importance of each token is dynamically evaluated based on the contextual information of the input text sequence (e.g., financial text), and then the query vector to be retained is calculated based on the importance distribution. The calculation formula is as follows:

[0080]

[0081] Among them, f(i) represents token i The attention level in the current financial news context is dynamically adjusted based on the text content and contextual information. Specifically, different tokens and their contexts determine the attention intensity required for the token. Symbol represents an element-wise multiplication operation, and Q represents the query matrix. In this way, the entire query matrix can be pruned—removing unimportant tokens and retaining and gathering important tokens as Q for subsequent attention score calculation.

[0082] In some embodiments, step S210 specifically includes:

[0083] Obtaining the text content of each element and the context information of the element; extracting features from the obtained text content and context information, and inputting the extracted features into the attention evaluation model;

[0084] The attention level of each element of the input text sequence is determined by an attention level evaluation model.

[0085] In this embodiment, LLMs typically employ a self-attention or cross-attention mechanism to calculate the relevance between tokens in the input sequence. This mechanism enables the model to identify which tokens are more important to the current task and dynamically assign more attention weight to these tokens. For example, when processing natural language text, the model may pay more attention to nouns, verbs, and other words with key semantic information, while paying relatively less attention to stop words. The importance of each token depends not only on its own characteristics but also on the context in which it is used. LLMs consider the tokens before and after it to determine its specific meaning and importance in the current context. For example, in the sentence "I like watching movies," the importance of the token "movie" is increased by the actions of "liking" and "watching." In addition to the content of a token, its position in the input sequence is also a key factor influencing its attention. LLMs typically assign a positional code to each token to indicate its relative position in the sequence. Tokens closer to the current processing position may receive higher attention because they are more likely to contain information relevant to the current task.

[0086] In some embodiments, see Figure 5 , the step S230 specifically includes:

[0087] S231, comparing the preset importance threshold with the calculated importance of each element to obtain a comparison result;

[0088] S232: Based on the comparison result, perform pruning processing on the query matrix.

[0089] In this embodiment, in order to implement the pruning processing of the query matrix, an importance threshold is preset. The importance threshold is compared with the calculated importance of each element to obtain a comparison result. The comparison result can provide a theoretical basis for the pruning processing of the query matrix, and then determine whether the attention connection of each position is retained, thereby ensuring the accuracy of text encoding while greatly reducing the computing resources and storage space occupied.

[0090] In some embodiments, step S232 specifically includes:

[0091] Based on the comparison result, filter out elements whose importance is less than the preset importance threshold;

[0092] Removing query vectors corresponding to elements in the query matrix whose importance is less than the preset importance threshold;

[0093] The pruned query matrix is reconstructed to achieve pruning of the query matrix.

[0094] In this embodiment, the comparison results are screened according to the importance threshold. If the comparison result is less than the preset importance threshold, it means that the importance of the token is low and will not affect the final encoding result. Therefore, the attention calculation of the token can be deleted. If the comparison result is not less than the preset importance threshold, it means that the importance of the token is high and will affect the final encoding result. Therefore, the attention of the token needs to be calculated.

[0095] In some embodiments, see Figure 6 , the step S300 specifically includes:

[0096] S310, calculating the attention weight matrix of the input text sequence based on the query matrix after pruning and the key matrix;

[0097] S320, calculating an attention score of the input text sequence based on the attention weight matrix and the value matrix;

[0098] S330: quantize and encode the calculated attention score to generate a final text encoding, thereby completing the encoding of the input text sequence.

[0099] In this embodiment, the attention weight matrix is usually obtained by calculating the similarity between the query vector (Query) and the key vector (Key). Specifically, it measures the degree of association between the Query and each Key through a dot product operation to generate a score matrix. Each element in this score matrix represents the attention score between the Query and the corresponding Key, reflecting the closeness of the association between them. The main function of the attention weight matrix is to provide input for the subsequent Softmax function, which converts the score into a probability distribution to determine the attention weight that each element should receive. These weights are then used to perform a weighted summation on the value vector (Value) to generate the final attention output. In this way, the model can dynamically adjust the degree of attention to different elements based on the contextual information of the input sequence, thereby improving the understanding and processing capabilities of the sequence.

[0100] Specifically, the calculation method of attention score is as follows:

[0101] Similarity calculation: The similarity between the query vector (Query, abbreviated as Q) and the key vector (Key, abbreviated as K) is calculated using the dot product (also known as the inner product). Specifically, Q and K are matrix multiplied to produce a Q × K attention weight matrix. Each element in this matrix represents the degree of association between the corresponding query and key, with larger values indicating a closer connection.

[0102] Scaling: To prevent problems such as gradient disappearance or explosion, the similarity score is usually scaled. A common practice is to divide the similarity score by in It is the dimension of the Key.

[0103] Masking: When working on certain tasks, such as sequence labeling in natural language processing, it may be necessary to mask out certain elements that are not of interest. In this case, a masking operation is used to add a mask matrix to the scaled attention scores. The mask matrix is typically a matrix of the same dimensions as the scores matrix, containing values of 0 or -∞, which specifies which elements should be masked out.

[0104] Normalization: Use the Softmax function to normalize the scaled attention scores and convert them into a probability distribution. In this way, the sum of the attention scores of each row is 1, which enables the model to learn the attention weights more stably.

[0105] Through the above steps, the model can dynamically adjust the allocation of attention according to the importance of different parts of the input data, thereby improving its ability to understand and process the data.

[0106] Optionally, the calculation formula of the attention matrix is:

[0107]

[0108] The Softmax function maps each element in the input vector to the (0, 1) interval and makes the sum of these elements equal to 1.

[0109] The calculation formula of the attention score is:

[0110] Attention Scores=Attention Weights×V.

[0111] The technical solution provided by the present invention first obtains an input text sequence and generates a key vector matrix corresponding to the input text sequence, wherein the key vector matrix includes a query matrix, a key matrix, and a value matrix. The attention level of each element in the input text sequence is then obtained and, based on the attention level and the query matrix, the query matrix is pruned. Finally, an attention score for the input text sequence is calculated based on the pruned query matrix, the key matrix, and the value matrix to achieve encoding of the input text sequence. This invention improves on the traditional self-attention mechanism by dynamically determining whether to retain the attention connection at each position based on the context and complexity of the input sequence. This adaptive pruning mechanism avoids the potential loss of accuracy associated with fixed pruning rate methods. The model dynamically determines whether to retain the attention connection at each position based on the complexity and context of the input data, thereby significantly reducing computing resources and storage space usage while ensuring accurate text encoding recognition. This optimization method is suitable for large-scale, real-time financial data inference on edge devices or in resource-constrained environments, providing financial institutions with more efficient, low-latency real-time monitoring and decision support.

[0112] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0113] Another embodiment of the present invention provides a text encoding device, which corresponds to the text encoding method in the above embodiment. Figure 7 The text encoding device includes a matrix calculation module 11, a pruning module 12 and an encoding module 13. The functional modules are described in detail as follows:

[0114] The matrix calculation module 11 is used to obtain an input text sequence and generate a key vector matrix corresponding to the input text sequence, wherein the key vector matrix includes a query matrix, a key matrix and a value matrix.

[0115] The pruning module 12 is configured to obtain the attention of each element of the input text sequence, and perform pruning processing on the query matrix based on the attention and the query matrix.

[0116] The encoding module 13 is used to calculate the attention score of the input text sequence based on the query matrix after pruning, the key matrix and the value matrix, so as to achieve encoding of the input text sequence.

[0117] In some embodiments, the matrix calculation module 11 specifically includes an input text sequence acquisition unit, a weight matrix acquisition unit, and a key vector matrix acquisition unit, wherein:

[0118] The input text sequence acquisition unit is used to acquire the input text sequence;

[0119] The weight matrix acquisition unit is used to obtain the weight matrix of the query matrix, the weight matrix of the key matrix and the weight matrix of the value matrix obtained in advance;

[0120] The key vector matrix acquisition unit is used to obtain the key vector matrix corresponding to the input text sequence through matrix operation based on the input text sequence, the weight matrix of the query matrix, the weight matrix of the key matrix and the weight matrix of the value matrix.

[0121] In some embodiments, the pruning module 12 includes an attention acquisition unit, an importance calculation unit, and a pruning unit, wherein:

[0122] The attention degree acquisition unit is used to acquire the attention degree of each element of the input text sequence through a pre-trained attention degree model;

[0123] The importance calculation unit is used to calculate the importance of each element based on the attention degree and the query matrix;

[0124] The pruning unit is used to perform pruning processing on the query matrix based on a preset importance threshold and the calculated importance of each element.

[0125] In some embodiments, the attention acquisition unit is specifically configured to:

[0126] Get the text content of each element and the context information of the element;

[0127] Extract features from the acquired text content and context, and input the extracted features into the attention evaluation model;

[0128] The attention level of each element of the input text sequence is determined by an attention level evaluation model.

[0129] In some embodiments, the pruning unit specifically includes a comparison subunit and a processing subunit, wherein:

[0130] The comparison subunit is used to compare the preset importance threshold with the calculated importance of each element to obtain a comparison result;

[0131] The processing subunit is configured to perform pruning processing on the query matrix based on the comparison result.

[0132] In some embodiments, the processing subunit is specifically configured to:

[0133] Based on the comparison result, filter out elements whose importance is less than the preset importance threshold;

[0134] Removing query vectors corresponding to elements in the query matrix whose importance is less than the preset importance threshold;

[0135] The pruned query matrix is reconstructed to achieve pruning of the query matrix.

[0136] In some embodiments, the encoding module 13 includes an attention score matrix calculation unit, an attention score calculation unit and an encoding unit, wherein:

[0137] The attention score matrix calculation unit is used to calculate the attention weight matrix of the input text sequence based on the query matrix after pruning and the key matrix;

[0138] An attention score calculation unit is used to calculate the attention score of the input text sequence based on the attention weight matrix and the value matrix to achieve encoding of the input text sequence;

[0139] The encoding unit is used to quantize and encode the calculated attention score to generate the final text encoding, thereby completing the encoding of the input text sequence.

[0140] In an embodiment of the present invention, an input text sequence is first acquired and a key vector matrix corresponding to the input text sequence is generated, wherein the key vector matrix includes a query matrix, a key matrix, and a value matrix. The attention level of each element of the input text sequence is then obtained and, based on the attention level and the query matrix, the query matrix is pruned. Finally, an attention score for the input text sequence is calculated based on the pruned query matrix, the key matrix, and the value matrix to achieve encoding of the input text sequence. This invention improves upon the traditional self-attention mechanism by dynamically determining whether to retain the attention connection at each position based on the context and complexity of the input sequence. This adaptive pruning mechanism avoids the potential loss of accuracy associated with fixed pruning rate methods. The model dynamically determines whether to retain the attention connection at each position based on the complexity and context of the input data, thereby significantly reducing computing resources and storage space usage while ensuring accurate text encoding recognition. This optimization method is suitable for large-scale, real-time financial data inference on edge devices or in resource-constrained environments, providing financial institutions with more efficient, low-latency real-time monitoring and decision support.

[0141] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware via computer-readable instructions. The computer-readable instructions can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes in the above-described method embodiments. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0142] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0143] For the specific definition of the text encoding device, please refer to the definition of the text encoding method above and will not be repeated here. The various modules in the above-mentioned text encoding device can be implemented in whole or in part through software, hardware, or a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of the above modules.

[0144] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device includes a processor, memory, network interface and database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, it implements the functions or steps on the server side of an artificial intelligence-based text encoding method.

[0145] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 9 As shown. The computer device includes a processor, memory, network interface, display screen and input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a client side of a text encoding method.

[0146] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0147] Obtain input data, and based on the type of the input data, use a corresponding feature extraction model to extract features from the input data to generate a patient health profile;

[0148] Obtain an input text sequence and generate a key vector matrix corresponding to the input text sequence, wherein the key vector matrix includes a query matrix, a key matrix, and a value matrix;

[0149] Obtaining the attention of each element of the input text sequence, and performing pruning processing on the query matrix based on the attention and the query matrix;

[0150] Based on the query matrix after pruning, the key matrix and the value matrix, an attention score of the input text sequence is calculated to achieve encoding of the input text sequence.

[0151] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0152] Obtain an input text sequence and generate a key vector matrix corresponding to the input text sequence, wherein the key vector matrix includes a query matrix, a key matrix, and a value matrix;

[0153] Obtaining the attention of each element of the input text sequence, and performing pruning processing on the query matrix based on the attention and the query matrix;

[0154] Based on the query matrix after pruning, the key matrix and the value matrix, an attention score of the input text sequence is calculated to achieve encoding of the input text sequence.

[0155] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can be found in the relevant descriptions of the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0156] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchl ink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0157] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0158] In summary, the text encoding method, apparatus, computer device, and storage medium provided by the present invention first obtain an input text sequence and generate a key vector matrix corresponding to the input text sequence, wherein the key vector matrix includes a query matrix, a key matrix, and a value matrix. Next, the attention level of each element of the input text sequence is obtained and, based on the attention level and the query matrix, the query matrix is pruned. Finally, based on the pruned query matrix, the key matrix, and the value matrix, an attention score for the input text sequence is calculated to achieve encoding of the input text sequence. This invention improves upon the traditional self-attention mechanism by dynamically determining whether to retain the attention connection at each position based on the context and complexity of the input sequence. This adaptive pruning mechanism avoids the potential loss of accuracy associated with fixed pruning rate methods. The model dynamically determines whether to retain the attention connection at each position based on the complexity and context of the input data, thereby significantly reducing computing resources and storage space usage while ensuring accurate text encoding recognition. This optimization method is suitable for large-scale, real-time financial data inference on edge devices or in resource-constrained environments, providing financial institutions with more efficient, low-latency real-time monitoring and decision support.

[0159] It should be noted that if software tools or components other than those of our company appear in the embodiments of this application, they are only used for illustration and do not represent actual use.

[0160] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A text encoding method, characterized in that: The steps include: Obtain an input text sequence and generate a key vector matrix corresponding to the input text sequence, wherein the key vector matrix includes a query matrix, a key matrix, and a value matrix; Obtaining the attention of each element of the input text sequence, and performing pruning processing on the query matrix based on the attention and the query matrix; Based on the query matrix after pruning, the key matrix and the value matrix, an attention score of the input text sequence is calculated to achieve encoding of the input text sequence.

2. The text encoding method according to claim 1, characterized in that The step of obtaining an input text sequence and generating a key vector matrix corresponding to the input text sequence includes: Get the input text sequence; Obtain the pre-trained weight matrix of the query matrix, the weight matrix of the key matrix, and the weight matrix of the value matrix; Based on the input text sequence, the weight matrix of the query matrix, the weight matrix of the key matrix and the weight matrix of the value matrix, a key vector matrix corresponding to the input text sequence is performed through matrix operation.

3. The text encoding method according to claim 1, wherein: The obtaining of the attention of each element of the input text sequence and performing pruning on the query matrix based on the attention and the query matrix includes: Obtaining the attention of each element of the input text sequence through a pre-trained attention model; Calculating the importance of each element based on the attention and the query matrix; The query matrix is pruned based on a preset importance threshold and the calculated importance of each element.

4. The text encoding method according to claim 1 or 3, characterized in that: The obtaining of the attention level of each element of the input text sequence includes: Get the text content of each element and the context information of the element; Extract features from the acquired text content and context, and input the extracted features into the attention evaluation model; The attention level of each element of the input text sequence is determined by an attention level evaluation model.

5. The text encoding method according to claim 3, wherein: The pruning process of the query matrix based on the preset importance threshold and the calculated importance of each element includes: Comparing the preset importance threshold with the calculated importance of each element to obtain a comparison result; Based on the comparison result, the query matrix is pruned.

6. The text encoding method according to claim 5, characterized in that The pruning of the query matrix based on the comparison result includes: Based on the comparison result, filter out elements whose importance is less than the preset importance threshold; Removing query vectors corresponding to elements in the query matrix whose importance is less than the preset importance threshold; The pruned query matrix is reconstructed to achieve pruning of the query matrix.

7. The text encoding method according to claim 1, wherein: The step of calculating the attention score of the input text sequence based on the query matrix after pruning, the key matrix, and the value matrix to encode the input text sequence includes: Calculating an attention weight matrix for the input text sequence based on the pruned query matrix and the key matrix; Calculating an attention score for the input text sequence based on the attention weight matrix and the value matrix; The calculated attention score is quantized and encoded to generate a final text encoding, thereby completing the encoding of the input text sequence.

8. A text encoding device, characterized in that include: A matrix calculation module, configured to obtain an input text sequence and generate a key vector matrix corresponding to the input text sequence, wherein the key vector matrix includes a query matrix, a key matrix, and a value matrix; a pruning module, configured to obtain the attention of each element of the input text sequence, and perform pruning processing on the query matrix based on the attention and the query matrix; An encoding module is used to calculate the attention score of the input text sequence based on the query matrix after pruning, the key matrix and the value matrix to achieve encoding of the input text sequence.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the text encoding method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the text encoding method according to any one of claims 1 to 7.

Images