Text processing method and device based on converter algorithm

Abstract

The invention provides a text processing method and device based on a converter algorithm. The method comprises the following steps: receiving first text data; determining a first text representation vector corresponding to each sentence in the first text data according to the word corresponding to each sentence in the first text data; determining a first query matrix and a first key matrix corresponding to the first text representation vector according to the first text representation vector and a relative position relationship between words corresponding to each sentence in the first text data; selecting correlation column data from the first key matrix, and determining a second key matrix; the correlation degree of the correlation column data and the first query matrix is greater than a first threshold value; inputting the first query matrix and the second key matrix into a self-attention layer in a converter algorithm to obtain a self-attention result; and determining processed second text data according to the self-attention result and the weighted activation function. According to the scheme, the text processing efficiency and accuracy can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a text processing method and device based on a converter algorithm. Background Art

[0002] Transform algorithms are often used in text processing scenarios, including but not limited to text error correction scenarios and text translation scenarios. Transform algorithms generally include multiple encoders and multiple decoders, where each encoder and decoder includes multiple self-attentions. Each self-attention requires input of the query matrix Q, the key matrix K, and the value matrix V.

[0003] The computational complexity of each self-attention mechanism is O(S 2 ), where S is the maximum length of the input sentence in the transform algorithm; as the length of the input sentence increases, the amount of computation required increases quadratically; and because the complete attention matrix needs to be stored, the amount of memory storage is also proportional to the length of the sentence.

[0004] Moreover, after the query matrix Q is multiplied by the key matrix K, the number of parameters of the algorithm will increase, which may easily cause overfitting of the transform algorithm and reduce its generalization ability outside the training samples.

[0005] In addition, the existing transform algorithm only considers the absolute position of each word, that is, the fixed position of each word in the input sentence, which leads to the lack of context awareness in the algorithm and is not conducive to the transform algorithm learning the deeper semantic structure of the input sentence.

[0006] Again, each self-attention layer is connected to a feedforward layer. The activation function used in the feedforward layer of the existing transform algorithm is the ReLU activation function. The ReLU activation function will output 0 in the case of negative input, which will reduce the expressiveness of the model, causing the gradient to disappear during training, and make the activation parameters sparse, which will lead to information loss and reduced training efficiency. Moreover, the gradient of the ReLU activation function is constant in the case of positive input, which will lead to difficulties in weight initialization and training. It is difficult for the algorithm to capture nonlinear changes between parameters, resulting in low text processing accuracy of the algorithm.

[0007] Therefore, how to improve the efficiency and accuracy of text processing remains to be solved. Summary of the invention

[0008] The present application provides a text processing method and device based on a converter algorithm, which can improve the efficiency and accuracy of text processing.

[0009] In the first aspect, an embodiment of the present application provides a text processing method based on a converter algorithm, which can be executed by a text processing device based on a converter algorithm, and the text processing device based on the converter algorithm can be a terminal device or a module for a terminal device, or a server or a module for a server. The present application does not limit the execution subject of the method. The method includes: receiving first text data; the first text data includes multiple sentences; wherein each sentence can be divided into multiple words; according to the words corresponding to each sentence in the first text data, determining the first text representation vector corresponding to each sentence in the first text data; the first text representation vector includes the word vector corresponding to each word; according to the first text representation vector, and the relative position relationship between the words corresponding to each sentence in the first text data, determining the first query matrix and the first key matrix corresponding to the first text representation vector; the first query matrix and the first key matrix are used to determine the degree of correlation between different words in each sentence; selecting the number of correlation columns from the first key matrix According to the invention, a second key matrix is ​​determined; the correlation degree between the correlation column data and the first query matrix is ​​greater than a first threshold; the first query matrix and the second key matrix are input into the self-attention layer in the converter algorithm to obtain a self-attention result; according to the self-attention result and the weighted activation function, the processed second text data is determined; the weighted activation function is determined according to the weighted values ​​of the maximum activation function, the first normal distribution activation function and the second normal distribution activation function; the first normal distribution activation function is used to describe the nonlinear relationship of the parameters in the negative interval of the self-attention result; the second normal distribution activation function is used to describe the nonlinear relationship of the parameters in the positive interval of the self-attention result.

[0010] In the above scheme, on the one hand, the first query matrix and the first key matrix contain a relative position relationship, which makes the algorithm context-aware and enables the transform algorithm to learn the deeper semantic structure of the input sentence; on the other hand, selecting part of the column data from the first key matrix to construct the second key matrix can not only reduce the computational complexity and memory complexity of the algorithm; but also the selected part of the column data has a large correlation with the first query matrix, which can better maintain the self-attention coefficient of the first key matrix and the first query matrix without losing the accuracy of the data; on the other hand, using the weighted activation function in the feedforward layer can better characterize the nonlinear relationship between parameters, which can make the algorithm output have better statistical properties, improve the generalization ability of the model, help the algorithm capture more complex data patterns, and more effectively simulate noise and outliers in the data. In summary, through the above three aspects, the efficiency and accuracy of text processing can be improved.

[0011] In a possible implementation method, for any sentence in the first text data, if the distance between the words corresponding to the sentence is less than or equal to a distance threshold, the distance vector between the words corresponding to the sentence is set to a first distance vector; the first distance vector is used to indicate the relative position relationship between the words corresponding to the sentence, and the first distance vector is learned based on historical text data; if the distance between the words corresponding to the sentence is greater than the distance threshold, the distance vector between the words corresponding to the sentence is set to a second distance vector based on a decay function; the decay function is used to represent a function that gradually decays with increasing distance; based on the first text representation vector, the first distance vector and the second distance vector, the first query matrix and the first key matrix corresponding to the first text representation vector are determined.

[0012] The above scheme introduces relative position information when the distance between words corresponding to a sentence is less than or equal to the distance threshold, which can capture the relative distance and direction between words in the data on the first text, so that the algorithm can better understand the structure and contextual relationship of the data on the first text; when the distance between words corresponding to a sentence is greater than the distance threshold, a decay function is used to characterize the positional relationship between words, so that the input length of the first text data will not be limited. This scheme does not depend on the maximum length of the first text data and can more flexibly process first text data of different lengths; the algorithm adjusts its understanding of the position according to the actual content of the first text data, which is conducive to learning richer language features and patterns; when processing longer input texts, it helps to capture long-distance dependencies.

[0013] In a possible implementation method, each row of data in the first key matrix is ​​sorted to obtain a third key matrix; the valid values ​​in the first query matrix and the third key matrix are calculated to obtain a first result set; the first result set includes the product of the valid values ​​in the first query matrix and the valid values ​​in the third key matrix; according to the number of occurrences of each column of data in the third key matrix in the first result set, and the product of the valid values ​​in the first query matrix and the valid values ​​in the third key matrix, the correlation column data is selected from the first key matrix to determine the second key matrix.

[0014] The above scheme can accurately and effectively determine the correlation column data. Selecting the correlation column data from the first key matrix can reduce the computational complexity of the algorithm, allowing the algorithm to process longer input sentences; due to the reduction in the amount of calculation, it also reduces the memory usage during the algorithm runtime and improves the efficiency of memory use; it maintains the ability to effectively capture long-distance dependencies and the expression ability of the attention matrix, and can effectively capture the key information in the input sentence; compared with the dense connection of the existing technology, this scheme can effectively avoid overfitting, allowing the algorithm to have better scalability on different tasks and data sets.

[0015] In a possible implementation method, K1 row data and K2 column data are selected from the first key matrix to determine a fourth key matrix; wherein the first key matrix and the fourth key matrix have the same dimension; the K1 row data and the K2 column data have the same position in the first key matrix and the fourth key matrix; except for the K1 row data and the K2 column data in the fourth key matrix, the values ​​of other matrix positions are set to invalid values; the K1 is a positive integer less than the total number of rows of the first key matrix; the K2 is a positive integer less than the total number of columns of the first key matrix; and each row of data of the fourth key matrix is ​​sorted to obtain the third key matrix.

[0016] The above scheme reduces the number of elements of the first key matrix, which can reduce the computational complexity of the algorithm and enable the algorithm to process longer input sentences. Due to the reduction in the amount of calculation, the memory usage during the algorithm operation is also reduced, and the memory usage efficiency is improved. The effective capture capability of long-distance dependencies and the expression capability of the attention matrix are maintained, and the key information in the input sentences can be effectively captured. Compared with the dense connection of the existing technology, this scheme can effectively avoid overfitting and make the algorithm more scalable on different tasks and data sets.

[0017] In a possible implementation method, M1 row data and M2 column data are selected from the first query matrix to determine the second query matrix; wherein the first query matrix and the second query matrix have the same dimension; the positions of the M1 row data and the M2 column data in the first query matrix and the second query matrix are the same; except for the M1 row data and the M2 column data in the second query matrix, the values ​​of other matrix positions are set to invalid values; the M1 is a positive integer less than the total number of rows of the first query matrix; the M2 is a positive integer less than the total number of columns of the first query matrix; the second query matrix and the second key matrix are input into the self-attention layer in the converter algorithm to obtain the self-attention result.

[0018] The above scheme reduces the number of elements in the first query matrix, which can reduce the computational complexity of the algorithm and enable the algorithm to process longer input sentences. Due to the reduction in the amount of calculation, the memory usage during the algorithm operation is also reduced, and the memory usage efficiency is improved. The effective capture capability of long-distance dependencies and the expression capability of the attention matrix are maintained, and the key information in the input sentences can be effectively captured. Compared with the dense connection of the existing technology, this scheme can effectively avoid overfitting and make the algorithm more scalable on different tasks and data sets.

[0019] In one possible implementation method, the second query matrix and the fourth key matrix are input into the first self-attention layer in the converter algorithm to obtain a first self-attention result; the second query matrix and the second key matrix are input into the second self-attention layer in the converter algorithm to obtain a second self-attention result; and the self-attention result is obtained based on the first self-attention result and the second self-attention result.

[0020] The above scheme inputs different query matrices and key matrices to different self-attention layers, which enables the self-attention layer to learn richer content and effectively capture the key information in the input sentence; making the algorithm more scalable on different tasks and data sets.

[0021] In one possible implementation method, N values ​​are selected from each row in the first query matrix and added to the corresponding positions of the second query matrix to determine the third query matrix; wherein N is a positive integer; M values ​​are selected from each row in the first key matrix and added to the corresponding positions of the fourth key matrix to determine the fifth key matrix; wherein M is a positive integer; correlation column data is selected from the fifth key matrix to obtain a sixth key matrix; and the third query matrix and the sixth key matrix are input into the second self-attention layer in the converter algorithm.

[0022] The above scheme adds N elements to each row of data in the second query matrix and M elements to each row of data in the fourth key matrix, which can increase the randomness of the third query matrix and the fifth key matrix, reduce the design of artificial rules, and improve the accuracy of text processing.

[0023] In one possible implementation method, elements whose horizontal coordinates are in the first set are selected from the first key matrix as the K1 row data; the numerical values ​​in the first set are related to the maximum length of the input sentence; and elements whose vertical coordinates are in the first set are selected from the first key matrix as the K2 column data.

[0024] The above scheme selects several global nodes to act as intermediaries for information transmission between other nodes. These nodes will establish relationships with all other nodes, increasing the effectiveness of K1 row data and K2 column data.

[0025] In a possible implementation method, an element whose horizontal coordinate value is equal to a first numerical value is selected from the first key matrix as the first row sub-data; the first numerical value is equal to half of the maximum length of the input sentence; an element whose vertical coordinate value is equal to the first numerical value is selected from the first key matrix as the first column sub-data; an element whose horizontal coordinate and vertical coordinate satisfy a first rule is selected from the first key matrix as the second sub-data; the first rule means that the absolute difference between the horizontal coordinate and the vertical coordinate is less than the first difference; an element whose horizontal coordinate and vertical coordinate satisfy a second rule is selected from the first key matrix as the third sub-data; the second rule means that the sum of the horizontal coordinate and the vertical coordinate is equal to the maximum length of the input sentence; and the K1 row data and the K2 column data are determined respectively according to the first row sub-data, the first column sub-data, the second sub-data and the third sub-data.

[0026] In the above scheme, since most data are localized, each node is connected only to its corresponding word segmentation and the other nodes corresponding to the surrounding word segmentations, which can better facilitate long-distance information exchange. At the same time, in symmetrical positions, both nodes are selected to establish connections to characterize long-distance dependencies.

[0027] In a possible implementation method, elements whose horizontal coordinates and vertical coordinates satisfy a third rule are selected from the first key matrix as fourth sub-data; the third rule means that the remainder of the horizontal coordinate or the vertical coordinate and a second value is a specified value, and / or the horizontal coordinate is the same as the vertical coordinate; elements whose horizontal coordinates and vertical coordinates satisfy a fourth rule are selected from the first key matrix as fifth sub-data; the fourth rule means that the horizontal coordinate and / or the vertical coordinate are located in the vertex range of the first key matrix; and the K1 row data and the K2 column data are determined respectively based on the fourth sub-data and the fifth sub-data.

[0028] The above scheme selects elements within a certain range to establish connections, which can characterize local dependencies; at the same time, the top corners establish full connections, which can describe the long-distance relationship across the beginning and end of the sentence.

[0029] In a possible implementation method, an element whose horizontal coordinate value is equal to the third value and whose vertical coordinate is an odd number is selected from the first key matrix as the sixth sub-data; an element whose horizontal coordinate value is equal to the fourth value and whose vertical coordinate is an even number is selected from the first key matrix as the seventh sub-data; the third value and the fourth value are related to the maximum length of the input sentence, and the difference between the third value and the fourth value is a specified value; four elements located at the top corners are selected from the first key matrix as the eighth sub-data; and the K1 row data and the K2 column data are determined respectively according to the sixth sub-data, the seventh sub-data and the eighth sub-data.

[0030] In a possible implementation method, an element whose ordinate value is equal to the third value and whose abscissa is an odd number is selected from the first key matrix as the sixth sub-data; an element whose ordinate value is equal to the fourth value and whose abscissa is an even number is selected from the first key matrix as the seventh sub-data; four elements located at the top corners are selected from the first key matrix as the eighth sub-data; and the K1 row data and the K2 column data are determined respectively according to the sixth sub-data, the seventh sub-data and the eighth sub-data.

[0031] The above scheme, selecting elements with odd vertical coordinates or selecting elements with even vertical coordinates, can expand the receptive field of the global node; selecting four elements located at the top corners can describe the long-distance relationship across the beginning and end of the sentence.

[0032] In a possible implementation method, the calculation method of the first normal distribution activation function is as follows:

[0033]

[0034] Wherein, x is a parameter in the self-attention result; tanh is a tangent function; ρ1, ρ2 and ρ3 are parameters that need to be trained in the first normal distribution activation function;

[0035] The calculation method of the second normal distribution activation function is as follows:

[0036]

[0037] Wherein, x is a parameter in the self-attention result; tanh is a tangent function; δ1 and δ2 are parameters that need to be trained in the first normal distribution activation function; σ is a distribution function of the logistic distribution; e x is an exponential function.

[0038] The above scheme can accurately and effectively characterize the nonlinear relationship between parameters, and provide a smooth and non-monotonic gradient when calculating the activation value, which helps to alleviate the problem of gradient disappearance or explosion, and is conducive to more stable learning of the algorithm; the output is closer to the Gaussian distribution, which can make the output of the algorithm have better statistical properties, improve the generalization ability of the model, help the algorithm capture more complex data patterns, and more effectively simulate noise and outliers in the data.

[0039] In a second aspect, an embodiment of the present application provides a text processing device based on a converter algorithm, comprising: a receiving unit and a determining unit; the receiving unit is used to receive first text data; the first text data includes multiple sentences; wherein each sentence can be divided into multiple words; the determining unit is used to determine a first text representation vector corresponding to each sentence in the first text data according to the words corresponding to each sentence in the first text data; the first text representation vector includes a word vector corresponding to each word; according to the first text representation vector and the relative position relationship between the words corresponding to each sentence in the first text data, determine a first query matrix and a first key matrix corresponding to the first text representation vector; the first query matrix and the first key matrix are used to determine the words in each sentence. The degree of correlation between different words; selecting correlation column data from the first key matrix to determine the second key matrix; the degree of correlation between the correlation column data and the first query matrix is ​​greater than a first threshold; inputting the first query matrix and the second key matrix into the self-attention layer in the converter algorithm to obtain a self-attention result; determining the processed second text data according to the self-attention result and the weighted activation function; the weighted activation function is determined according to the weighted values ​​of the maximum activation function, the first normal distribution activation function and the second normal distribution activation function; the first normal distribution activation function is used to describe the nonlinear relationship of the parameters in the negative interval of the self-attention result; the second normal distribution activation function is used to describe the nonlinear relationship of the parameters in the positive interval of the self-attention result.

[0040] In a possible implementation method, the determination unit is specifically used to, for any sentence in the first text data, if the distance between the words corresponding to the sentence is less than or equal to a distance threshold, set the distance vector between the words corresponding to the sentence to a first distance vector; the first distance vector is used to indicate the relative position relationship between the words corresponding to the sentence, and the first distance vector is learned based on historical text data; if the distance between the words corresponding to the sentence is greater than the distance threshold, set the distance vector between the words corresponding to the sentence to a second distance vector according to a decay function; the decay function is used to represent a function that gradually decays with increasing distance; based on the first text representation vector, the first distance vector and the second distance vector, determine the first query matrix and the first key matrix corresponding to the first text representation vector.

[0041] In a possible implementation method, the determination unit is specifically used to sort each row of data in the first key matrix to obtain a third key matrix; calculate the valid values ​​in the first query matrix and the third key matrix to obtain a first result set; the first result set includes the product of the valid values ​​in the first query matrix and the valid values ​​in the third key matrix; select the correlation column data from the first key matrix according to the number of occurrences of each column of data in the third key matrix in the first result set, and the product of the valid values ​​in the first query matrix and the valid values ​​in the third key matrix, to determine the second key matrix.

[0042] In a possible implementation method, the determination unit is specifically used to select K1 row data and K2 column data from the first key matrix to determine a fourth key matrix; wherein the first key matrix and the fourth key matrix have the same dimension; the K1 row data and the K2 column data have the same position in the first key matrix and the fourth key matrix; except for the K1 row data and the K2 column data in the fourth key matrix, the values ​​of other matrix positions are set to invalid values; the K1 is a positive integer less than the total number of rows of the first key matrix; the K2 is a positive integer less than the total number of columns of the first key matrix; and each row of data of the fourth key matrix is ​​sorted to obtain the third key matrix.

[0043] In a possible implementation method, the determination unit is also used to select M1 row data and M2 column data from the first query matrix to determine a second query matrix; wherein the first query matrix and the second query matrix have the same dimension; the positions of the M1 row data and the M2 column data in the first query matrix and the second query matrix are the same; except for the M1 row data and the M2 column data in the second query matrix, the values ​​of other matrix positions are set to invalid values; the M1 is a positive integer less than the total number of rows of the first query matrix; the M2 is a positive integer less than the total number of columns of the first query matrix; the second query matrix and the second key matrix are input into the self-attention layer in the converter algorithm to obtain the self-attention result.

[0044] In a possible implementation method, the determination unit is specifically used to input the second query matrix and the fourth key matrix into the first self-attention layer in the converter algorithm to obtain a first self-attention result; input the second query matrix and the second key matrix into the second self-attention layer in the converter algorithm to obtain a second self-attention result; and obtain the self-attention result based on the first self-attention result and the second self-attention result.

[0045] In a possible implementation method, the determination unit is specifically used to select N numerical values ​​from each row in the first query matrix, add them to the corresponding positions of the second query matrix, and determine the third query matrix; wherein N is a positive integer; select M numerical values ​​from each row in the first key matrix, add them to the corresponding positions of the fourth key matrix, and determine the fifth key matrix; wherein M is a positive integer; select correlation column data from the fifth key matrix to obtain a sixth key matrix; and input the third query matrix and the sixth key matrix into the second self-attention layer in the converter algorithm.

[0046] In a possible implementation method, the determination unit is specifically used to select elements whose horizontal coordinates are in a first set from the first key matrix as the K1 row data; the numerical values ​​in the first set are related to the maximum length of the input sentence; and select elements whose vertical coordinates are in the first set from the first key matrix as the K2 column data.

[0047] In a possible implementation method, the determination unit is specifically used to select an element whose horizontal coordinate value is equal to a first numerical value from the first key matrix as the first row sub-data; the first numerical value is equal to half of the maximum length of the input sentence; select an element whose vertical coordinate value is equal to the first numerical value from the first key matrix as the first column sub-data; select an element whose horizontal coordinate and vertical coordinate satisfy a first rule from the first key matrix as the second sub-data; the first rule means that the absolute difference between the horizontal coordinate and the vertical coordinate is less than the first difference; select an element whose horizontal coordinate and vertical coordinate satisfy a second rule from the first key matrix as the third sub-data; the second rule means that the sum of the horizontal coordinate and the vertical coordinate is equal to the maximum length of the input sentence; and determine the K1 row data and the K2 column data respectively according to the first row sub-data, the first column sub-data, the second sub-data and the third sub-data.

[0048] In a possible implementation method, the determination unit is specifically used to select elements whose horizontal coordinates and vertical coordinates satisfy a third rule from the first key matrix as fourth sub-data; the third rule refers to that the remainder of the horizontal coordinate or the vertical coordinate and a second value is a specified value, and / or that the horizontal coordinate is the same as the vertical coordinate; select elements whose horizontal coordinates and vertical coordinates satisfy a fourth rule from the first key matrix as fifth sub-data; the fourth rule refers to that the horizontal coordinate and / or the vertical coordinate are located in the vertex range of the first key matrix; and determine the K1 row data and the K2 column data respectively based on the fourth sub-data and the fifth sub-data.

[0049] In a possible implementation method, the determination unit is specifically used to select from the first key matrix an element whose horizontal coordinate value is equal to the third value and whose vertical coordinate is an odd number as the sixth sub-data; select from the first key matrix an element whose horizontal coordinate value is equal to the fourth value and whose vertical coordinate is an even number as the seventh sub-data; the third value and the fourth value are related to the maximum length of the input sentence, and the difference between the third value and the fourth value is a specified value; select from the first key matrix four elements located at the top corners as the eighth sub-data; and determine the K1 row data and the K2 column data respectively according to the sixth sub-data, the seventh sub-data and the eighth sub-data.

[0050] In a possible implementation method, the determination unit is specifically used to select from the first key matrix an element whose ordinate value is equal to the third value and whose abscissa is an odd number as the sixth sub-data; select from the first key matrix an element whose ordinate value is equal to the fourth value and whose abscissa is an even number as the seventh sub-data; select from the first key matrix four elements located at the top corners as the eighth sub-data; and determine the K1 row data and the K2 column data respectively according to the sixth sub-data, the seventh sub-data and the eighth sub-data.

[0051] In a possible implementation method, the calculation method of the first normal distribution activation function is as follows:

[0052]

[0053] Wherein, x is a parameter in the self-attention result; tanh is a tangent function; ρ1, ρ2 and ρ3 are parameters that need to be trained in the first normal distribution activation function;

[0054] The calculation method of the second normal distribution activation function is as follows:

[0055]

[0056] Wherein, x is a parameter in the self-attention result; tanh is a tangent function; δ1 and δ2 are parameters that need to be trained in the first normal distribution activation function; σ is a distribution function of the logistic distribution; e x is an exponential function.

[0057] In a third aspect, an embodiment of the present application further provides a computing device, including:

[0058] A memory for storing program instructions;

[0059] The processor is used to call the program instructions stored in the memory, and execute any method of implementing the above-mentioned first aspect according to the obtained program instructions.

[0060] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, in which computer-readable instructions are stored. When a computer reads and executes the computer-readable instructions, any method of the above-mentioned first aspect is implemented.

[0061] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising a computer program executable by a computer device, wherein when the program is run on the computer device, the computer device executes any method for implementing the above-mentioned first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 An overall structural diagram of a transformer algorithm (Transformer) provided in an embodiment of the present application;

[0063] Figure 2 A flowchart of a traditional text processing method based on a converter algorithm provided in an embodiment of the present application;

[0064] Figure 3 An internal structure diagram of a Transformer algorithm provided in an embodiment of the present application;

[0065] Figure 4 A structural diagram of a self-attention method provided in an embodiment of the present application;

[0066] Figure 5 A flowchart of a text processing method based on a converter algorithm provided in an embodiment of the present application;

[0067] Figure 6 A structural diagram of a first text representation vector provided in an embodiment of the present application;

[0068] Figure 7 A flowchart of a text processing method based on a converter algorithm provided in an embodiment of the present application;

[0069] Figure 8 A schematic diagram of a flow chart of a method for determining a second key matrix provided in an embodiment of the present application;

[0070] Fig. 9 A schematic diagram of a flow chart of a method for determining a second key matrix provided in an embodiment of the present application;

[0071] Fig.10 A flowchart of a text processing method based on a converter algorithm provided in an embodiment of the present application;

[0072] Fig.11 A schematic diagram of the structure of a text processing device based on a converter algorithm provided in an embodiment of the present application;

[0073] Fig.12 A structural schematic diagram of a text processing device based on a converter algorithm provided in an embodiment of the present application. DETAILED DESCRIPTION

[0074] The following is a brief introduction to the converter algorithm.

[0075] The overall structure of the transformer algorithm is as follows Figure 1 As shown, it includes multiple encoders (Encoders) and multiple decoders (Decoders).

[0076] The Transformer workflow is roughly divided into the following steps: Figure 2 As shown:

[0077] Step 201, obtain the text to be processed, and obtain the representation vector X of each word in each sentence in the text to be processed.

[0078] Among them, the representation vector X of each word is obtained by adding the Embedding of the word and the Embedding of the word position.

[0079] Step 202, construct a representation vector matrix based on the representation vectors of all words, and pass the representation vector matrix into the Encoder. After 6 Encoder blocks, the encoding information matrix C of all words in the sentence can be obtained.

[0080] Step 203: pass the encoding information matrix C output by the Encoder to the Decoder, and the Decoder outputs the processed text.

[0081] In one possible implementation, the internal structure of the Transformer algorithm is as follows: Figure 3 As shown in the figure, the left side is the Encoder block and the right side is the Decoder block. The part in the red circle is the Multi-Head Attention mechanism, which is composed of multiple Self-Attentions. It can be seen that the Encoder block contains one Multi-Head Attention, and the Decoder block contains two Multi-Head Attentions (one of which uses a mask). There is also a residual connection layer Add and a normalization layer Norm above the Multi-Head Attention. The Add layer represents the residual connection (Residual Connection) used to prevent network degradation, and Norm represents LayerNormalization, which is used to normalize the activation values ​​of each layer.

[0082] In one possible implementation method, the structure of self-attention is as follows: Figure 4 As shown in the figure, the input of each Self-Attention is the query matrix Q, the key matrix K, and the value matrix V. Self-Attention receives the input (the matrix X composed of the word representation vector x) or the output of the previous Encoder block.

[0083] The query matrix represents the query representation of each word in the input text data of the Transformer algorithm. In the attention calculation, the query vector is used to match the key vector to determine which words in the sequence are important to the current word, that is, its perspective as the receiver of information.

[0084] The key matrix contains the key representation of each word in the input text data of the Transformer algorithm. The key vector is dot-producted with the query vector to calculate the similarity or match between the two, which determines which words can be "discovered" by the query vector.

[0085] The value matrix is ​​the value representation of each token in the input text data of the Transformer algorithm. The value vector contains the specific information of each token in the sequence, which will be weighted and summed according to the attention score calculated by multiplying the query matrix and the key matrix.

[0086] In a possible implementation method, the calculation method of the self-attention coefficient is shown in the following formula (1):

[0087]

[0088] Among them, Q is the query matrix, K is the key matrix, V is the value matrix, K T is the transpose of the key matrix, d k is the dimension of the key matrix K; softmax is the activation function.

[0089] The text processing method based on the converter algorithm provided in the embodiment of the present application can be applied to the field of text translation technology, such as translating Chinese into English, or translating English into Chinese; it can also be applied to the field of text error correction technology, that is, marking and correcting errors in input text data, etc. Specifically, in the field of electronic contract auditing, the electronic contract to be reviewed is input into the text processing method based on the converter algorithm provided in the embodiment of the present application, the erroneous information in the electronic contract to be reviewed is corrected, and the corrected part is marked to remind the auditor of the risk. Of course, it can also be applied to other technical fields, and this application does not limit this.

[0090] Figure 5 A flowchart of a text processing method based on a converter algorithm provided in an embodiment of the present application, the method can be executed by a text processing device based on a converter algorithm, the text processing device based on a converter algorithm can be a terminal device or a module for a terminal device, or a server or a module for a server. The present application does not limit the execution subject of the method.

[0091] The method comprises the following steps:

[0092] Step 501: Receive first text data.

[0093] The first text data includes a plurality of sentences; each sentence can be divided into a plurality of words.

[0094] In a possible implementation method, the first text data is the text data to be translated, and the second text data is the text data after the translation is completed; or the first text data is the electronic contract to be reviewed, and the second text data is the electronic contract after the review is completed. Of course, the first text data and the second text data can also be other types, which is not limited in this application.

[0095] Step 502: Determine a first text representation vector corresponding to each sentence in the first text data according to the words corresponding to each sentence in the first text data.

[0096] The first text representation vector includes a word vector corresponding to each word.

[0097] In one possible implementation method, each sentence in the first text data is segmented to obtain the words corresponding to each sentence; illustratively, the first text data "I have a cat" is segmented to obtain "I", "have", "one", and "cat" respectively; the first text data "the loan balance is 5200 original" is segmented to obtain "loan", "balance", "for", "original" and "5200" respectively.

[0098] In a possible implementation method, the word vectors corresponding to multiple words in each sentence are determined, and the first text representation vector is obtained according to the word vectors corresponding to the multiple words. Specifically, the first text data "I have a cat" is segmented to obtain "I", "I have", "I", and "I cat" respectively; and the word vectors corresponding to "I", "I have", "I", and "I cat" are obtained respectively through the word2vec algorithm. Assuming that the dimension corresponding to each word vector is 512, the dimension of the first text representation vector is 4*512.

[0099] In a possible implementation method, word vectors corresponding to multiple words in each sentence and the absolute position relationship between words corresponding to each sentence are determined to obtain a first text representation vector. Figure 6 As shown, the Transformer representation x is the first text representation vector.

[0100] In a possible implementation method, the absolute position relationship between the words corresponding to each sentence can be obtained by the following formula (2):

[0101]

[0102] Among them, pos represents the position of the word in the sentence, d represents the dimension of PE (the same as the word Embedding), 2i represents the even dimension, and 2i+1 represents the odd dimension (i.e. 2i≤d, 2i+1≤d).

[0103] Step 503: Determine a first query matrix and a first key matrix corresponding to the first text representation vector according to the first text representation vector and the relative position relationship between words corresponding to each sentence in the first text data.

[0104] The first query matrix and the first key matrix are used to determine the correlation between different words in each sentence.

[0105] In a possible implementation method, a first query matrix, a first key matrix and a first value matrix corresponding to the first text representation vector are determined based on the first text representation vector and the relative position relationship between words corresponding to each sentence in the first text data.

[0106] Step 504: Select correlation column data from the first key matrix to determine a second key matrix.

[0107] The degree of correlation between the correlation column data and the first query matrix is ​​greater than a first threshold.

[0108] In one possible implementation method, the first key matrix and the second key matrix have the same dimension; the correlation column data have the same position in the first key matrix and the second key matrix; except for the correlation column data, the values ​​of other matrix positions in the second key matrix are set to invalid values; and the total number of columns of the correlation column data is less than the total number of columns of the first key matrix.

[0109] In a possible implementation method, the invalid value is negative infinity. Of course, the invalid value may also be other values, which are not limited in this application. The invalid value may not be stored or calculated.

[0110] In a possible implementation method, in the query matrix or the key matrix, except for invalid values, the remaining values ​​are called valid values.

[0111] In a possible implementation method, P columns of data are selected from the first key matrix, and the correlation between the P columns of data and the first query matrix is ​​greater than that between other columns of data in the first key matrix. The P columns of data are called correlation column data, where P is a positive integer.

[0112] Step 505: input the first query matrix and the second key matrix into the self-attention layer in the converter algorithm to obtain a self-attention result.

[0113] Step 506: Determine the processed second text data according to the self-attention result and the weighted activation function.

[0114] Among them, the weighted activation function is determined according to the weighted values ​​of the maximum activation function, the first normal distribution activation function and the second normal distribution activation function; the first normal distribution activation function is used to describe the nonlinear relationship of the parameters in the self-attention result located in the negative interval; the second normal distribution activation function is used to describe the nonlinear relationship of the parameters in the self-attention result located in the positive interval.

[0115] Since the Transform algorithm contains multiple Encoder blocks and Decoder blocks, and each Encoder block and Decoder block contains multiple Multi-Head Attentions, and each Multi-Head Attention contains multiple Self-Attentions; in this application, steps 503 to 505 can be applied to each Self-Attention; and the weighted activation function can also be applied to each Feed Forward layer. In other words, the improvements of this application for Self-Attention and Feed Forward can be applied to the decoder (Decoder block).

[0116] Assuming that the present application is applied to the field of text translation technology, the first text data is "I have a cat", and after the above steps 501 to 506, the second text data obtained is "I have a cat".

[0117] Assuming that the present application is applied to the field of text error correction technology, the first text data is "the loan balance is 5200 original", and after the above steps 501 to 506, the second text data obtained is "the loan balance is 5200 yuan"; among them, "5200 original" is modified to "5200 yuan".

[0118] In the above scheme, on the one hand, the first query matrix and the first key matrix contain a relative position relationship, which makes the algorithm context-aware and enables the transform algorithm to learn the deeper semantic structure of the input sentence; on the other hand, selecting part of the column data from the first key matrix to construct the second key matrix can not only reduce the computational complexity and memory complexity of the algorithm; but also the selected part of the column data has a large correlation with the first query matrix, which can better maintain the self-attention coefficient of the first key matrix and the first query matrix without losing the accuracy of the data; on the other hand, using a weighted activation function in the feedforward layer can better characterize the nonlinear relationship between parameters, which can make the algorithm output have better statistical properties, improve the generalization ability of the model, help the algorithm capture more complex data patterns, and more effectively simulate noise and outliers in the data. Through the above three aspects, the efficiency and accuracy of text processing can be improved.

[0119] In one embodiment, the specific implementation process of step 503 is as follows: Figure 7 As shown, the following steps are included:

[0120] Step 701, determining a distance threshold.

[0121] Step 702: Determine the relationship between the distance between the words corresponding to the sentence and the distance threshold.

[0122] In a possible implementation method, for any sentence in the first text data, if the distance between words corresponding to the sentence is less than or equal to a distance threshold, then execute step 703; if the distance between words corresponding to the sentence is greater than the distance threshold, then execute step 704.

[0123] In a possible implementation method, if the distance between the words corresponding to the sentence is less than or equal to the distance threshold, it means that the algorithm learns the relative position relationship between all the word segments whose distance to a certain word segment is less than or equal to the distance threshold. Specifically, if the distance threshold is equal to 10, that is, only the relative position information between a certain word segment and the 10 word segment segments to the left and right is considered to be learned by the algorithm, and the decay function is used to characterize the distances exceeding 10, so that there is no limit on the length of the text. That is, the application does not limit the length of the input sentence. Regardless of the length of the sentence, the method of the application can be used to characterize the distance relationship between different words.

[0124] Step 703: Set the distance vector between the words corresponding to the sentence as the first distance vector.

[0125] The first distance vector is used to indicate the relative position relationship between words corresponding to a sentence, and the first distance vector is learned based on historical text data.

[0126] Step 704: Set the distance vector between the words corresponding to the sentence as a second distance vector according to the attenuation function.

[0127] The attenuation function is used to characterize a function that gradually attenuates as the distance increases.

[0128] Step 705: Determine the first query matrix and the first key matrix corresponding to the first text representation vector according to the first text representation vector, the first distance vector, and the second distance vector.

[0129] In one possible implementation, the distance vector corresponding to the query matrix Q is R Q , R Q Including the first distance vector and the second distance vector corresponding to the query matrix Q, the first query matrix after adding the relative position relationship is Q+R Q Similarly, the first bond matrix after adding the relative position relationship is K+R K , the first value matrix after adding the relative position relationship is V+R V Among them, R Q is a matrix with the same dimensions as Q, R K is a matrix with the same dimension as K, R V is a matrix of the same dimension as V.

[0130] In one possible implementation method, in the above formula (1), when calculating the product QK of the query matrix Q and the key matrix K T When adding the relative position encoding matrix (Q+R Q )(K+R K ) T .

[0131] In a possible implementation method, in the above formula (1), when calculating the product softmax(·)V of the normalized exponential function and the value matrix V, the relative position encoding matrix softmax(·)(V+R V ).

[0132] In a possible implementation method, taking the key matrix K as an example, the calculation method of the first distance vector and the second distance vector is shown in the following formula (3):

[0133]

[0134] Where k is the distance threshold, (i, j) represents the element located in the i-th row and j-th column of the key matrix K; represents the attenuation function; sin represents the sine function; D represents the dimension of the distance vector (the same as the word Embedding); Represents the first distance vector, whose specific value is learned through back propagation.

[0135] For example,

[0136] If k = 1, R K The dimension is 3:

[0137]

[0138] Among them, R Q , R V Calculation method and R K same.

[0139] In one possible implementation, The calculation of is regarded as a function of (ji) In order to adapt to texts of different lengths, when (ji) is too large or too small, the decay function is used to calculate the position information, that is, Replace with When (ji) is within the algorithm learning range, In R K In the example, Convert to as well as The specific value of is learned as the algorithm parameter through back propagation.

[0140] Optionally, each subsequent attention calculation needs to be embedded in the above steps 701 to 705.

[0141] The above scheme introduces relative position information when the distance between words corresponding to a sentence is less than or equal to the distance threshold, which can capture the relative distance and direction between words in the data on the first text, so that the algorithm can better understand the structure and contextual relationship of the data on the first text; when the distance between words corresponding to a sentence is greater than the distance threshold, a decay function is used to characterize the positional relationship between words, so that the input length of the first text data will not be limited. This scheme does not depend on the maximum length of the first text data and can more flexibly process first text data of different lengths; the algorithm adjusts its understanding of the position according to the actual content of the first text data, which is conducive to learning richer language features and patterns; when processing longer input texts, it helps to capture long-distance dependencies.

[0142] In one embodiment, a weighted activation function is used in the feed forward layer. Feed Forward consists of two layers of fully connected neural networks. The first layer of the fully connected network will expand the matrix, and the second layer will reduce the expanded matrix to ensure that the input and output matrix dimensions of Feed Forward are the same. The weighted activation function is used to calculate the activation value in order to simulate nonlinear information.

[0143] In a possible implementation method, the calculation method of the Feed Forward layer activation function is shown in the following formula (4):

[0144] FFN(x)=MGEAA(xW1+b1)W2+b2……(4)

[0145] Among them, MGEAA represents the weighted activation function, x is the parameter in the self-attention result; W1, W2 and b1, b2 are the parameters to be learned.

[0146] In one possible implementation method, the weighted activation function is determined based on the weighted values ​​of the maximum activation function, the first normal distribution activation function and the second normal distribution activation function; the first normal distribution activation function is used to describe the nonlinear relationship of the parameters in the self-attention result that are located in the negative interval; the second normal distribution activation function is used to describe the nonlinear relationship of the parameters in the self-attention result that are located in the positive interval.

[0147] Optionally, the maximum activation function is calculated as shown in the following formula (5):

[0148] PA0(x)=max(0,x)……(5)

[0149] Among them, max represents the maximum value function.

[0150] Optionally, the calculation method of the first normal distribution activation function is shown in the following formula (6):

[0151]

[0152] Among them, x is a parameter in the self-attention result; tanh is a tangent function; ρ1, ρ2 and ρ3 are parameters that need to be trained in the first normal distribution activation function.

[0153] The activation value of the first normal distribution activation function is calculated in the same way as PA0 in the region of x ≥ 0 to reduce the amount of calculation; when x < 0, an approximate function is used to simulate nonlinear information, and adjustment factors ρ1, ρ2, and ρ3 are added to determine the appropriate values ​​for specific data scenarios, where ρ1, ρ2, and ρ3 are algorithm parameters that participate in algorithm training.

[0154] Optionally, the calculation method of the second normal distribution activation function is shown in the following formula (7):

[0155]

[0156] Wherein, x is a parameter in the self-attention result; tanh is a tangent function; δ1 and δ2 are parameters that need to be trained in the first normal distribution activation function; σ is a distribution function of the logistic distribution; e x is an exponential function.

[0157] The activation value of the second normal distribution activation function uses an approximate function to simulate nonlinear information in the region of x ≥ 0, and adds adjustment factors δ1 and δ2 to determine the appropriate value for the specific data scenario; when x < 0, x·tanh(e x -1) to simulate activation information, reduce computational complexity, and provide nonlinear gradients. δ1 and δ2 are algorithm parameters that participate in algorithm training;

[0158] In one possible implementation method, the weighted activation function is shown in the following formula (8), where w i For adaptive weights, participate in the training process.

[0159] MGEA Activation MGEAA(x)=w1PA0(x)+w2PA1(x)+w3PA2(x)……(8)

[0160] The exact expression of the normal distribution cumulative probability function involves an integral form that has no closed form solution, that is, it cannot be accurately expressed by elementary functions. The exact integral expression is computationally expensive, especially when it needs to be evaluated a large number of times or quickly, so approximate calculation methods are usually used to estimate its value.

[0161] The above scheme can accurately and effectively characterize the nonlinear relationship between parameters, and provide a smooth and non-monotonic gradient when calculating the activation value, which helps to alleviate the problem of gradient disappearance or explosion, and is conducive to more stable learning of the algorithm; the output is closer to the Gaussian distribution, which can make the output of the algorithm have better statistical properties, improve the generalization ability of the model, help the algorithm capture more complex data patterns, and more effectively simulate noise and outliers in the data.

[0162] In one embodiment, in the above step 504, the specific implementation process of selecting correlation column data from the first key matrix and determining the second key matrix is ​​as follows: Figure 8 As shown, the following steps are included:

[0163] Step 801: sort each row of data in the first key matrix to obtain a third key matrix.

[0164] The row sorting refers to sorting the values ​​of the first key matrix row by row, and the sorting includes sorting from large to small, or from small to large. For example, assuming that the first key matrix is Then sort each row of the first key matrix to obtain the third key matrix:

[0165] Step 802: Calculate the valid values ​​in the first query matrix and the third key matrix to obtain a first result set.

[0166] The first result set includes the product of the valid values ​​in the first query matrix and the valid values ​​in the third key matrix.

[0167] Step 803, according to the number of occurrences of each column of the third key matrix in the first result set and the product of the valid values ​​in the first query matrix and the valid values ​​in the third key matrix, select the relevant column data from the first key matrix to determine the second key matrix.

[0168] The above scheme can accurately and effectively determine the correlation column data. Selecting the correlation column data from the first key matrix can reduce the computational complexity of the algorithm, allowing the algorithm to process longer input sentences; due to the reduction in the amount of calculation, it also reduces the memory usage during the algorithm runtime and improves the efficiency of memory use; it maintains the ability to effectively capture long-distance dependencies and the expression ability of the attention matrix, and can effectively capture the key information in the input sentence; compared with the dense connection of the existing technology, this scheme can effectively avoid overfitting, allowing the algorithm to have better scalability on different tasks and data sets.

[0169] The specific implementation process of the above steps 801 to 803 is described in detail below with reference to a specific embodiment.

[0170] For example, the first text data, i.e. the text to be corrected, is "the loan balance is 5200 yuan". After word segmentation processing on the text to be corrected, "loan", "balance", "for", and "original" can be obtained, which correspond to q1, q2, q3, and q4 of the first query matrix and k1 of the first key matrix respectively. T , k2 T 、k3 T 、k4 T Taking "loan" as an example, its first query vector is q1, and the first key vector is k1 T , k2 T 、k3 T 、k4 T The product of q1·k1 T ,q1·k2 T ,q1·k3 T ,q1·k4 T , corresponding to four connections, respectively representing the attention scores of the word “loan”, “balance”, “for”, and “original”.

[0171] In the prior art, each q i and each k j T All are connected, and the corresponding attention score is E i,j ; Among them, E i,j =q i ·k j T This application only selects some attention scores of E i,j , therefore, the optimized attention score matrix E is shown in the following formula (9):

[0172]

[0173] In this application, the unselected attention score is E i,j Set to an invalid value, which can be -∞; of course, it can also be other values, which are not limited in this application. Set the invalid value to -∞, and the elements set to -∞ will not be stored, thereby reducing memory complexity.

[0174] The specific implementation process of the above steps 801 to 803 is as follows: Fig. 9 As shown, the following steps are included:

[0175] Step 901: sort the first key matrix according to the row dimension to obtain a third key matrix.

[0176] Where S represents the maximum length of the model input sentence.

[0177] In one possible implementation, the first key matrix (k1 T ,k2 T ,…k S T ) are sorted in descending order, or, for the first key matrix (k1 T ,k2 T ,…k S T ) Each row of data is sorted in ascending order.

[0178] Step 902: Set a second threshold m1.

[0179] In a possible implementation method, since the key values ​​with smaller values ​​have less influence on the attention coefficient, the key values ​​with smaller values ​​do not need to be calculated; for the first key matrix (k1 T ,k2 T ,…k S T) Sort the rows from large to small by dimension, and only select the first m1 larger key values ​​and the values ​​in the first query matrix for calculation, which can reduce the complexity of attention coefficient calculation and improve the efficiency of text processing. Where m1 is a positive integer.

[0180] Step 903: For each row of elements in the first query matrix, calculate the result of each row of elements and m1 key values.

[0181] Among them, t = 1, 2, ... D, D represents the dimension of the word vector, that is, the capacity of the dictionary.

[0182] In one possible implementation, for each q in the first query matrix i , calculate q i The result q with m1 larger key value it ·k jt .

[0183] In one possible implementation method, if each row of the first key matrix is ​​sorted in descending order, then the calculated That is, the m1 single item result is obtained and the m1 single item result is placed in the target single item queue.

[0184] In one possible implementation method, if each row of the first key matrix is ​​sorted in descending order, then the calculated That is, the Top-m1 single item result is obtained and placed in the target single item queue.

[0185] Step 904, calculate the number of k in the target single-item queue. j T The corresponding single item quantity is normalized and calculated for each k j T The corresponding first score.

[0186] In one possible implementation, the target single-item queue is calculated, and each k j T The corresponding single item quantity cnt=(cnt1,cnt2,…,cnt S ), and use the softmax function for normalization to calculate each k j T The corresponding first score

[0187] In a possible implementation method, an element whose value is greater than the result threshold is selected from the target single item queue, placed in the sub-target single item queue, and steps 904 to 907 are executed on the sub-target single item queue.

[0188] Step 905, calculate the number of k in the target single-item queue.j T The corresponding single cumulative value is normalized and calculated for each k j T The corresponding second score.

[0189] In one possible implementation, the target single-item queue is calculated, and each k j T The corresponding single item cumulative value value=(value1,value2,…,value S ), and use the softmax function for normalization to calculate each k j T The corresponding second score

[0190] Step 906, calculate each k j T The corresponding final score.

[0191] Among them, w1 and w2 are algorithm parameters and participate in algorithm training.

[0192] In one possible implementation, we calculate each k j T The corresponding final score score=w1score1+w2score2.

[0193] Step 907: based on the final score, retain the m2 k with the highest scores j T as the selected key.

[0194] Step 908: Determine a second key matrix according to the keys selected by the elements in each row of the first query matrix.

[0195] In one possible implementation method, according to each q i The selected key determines the second key matrix.

[0196] Specifically, because the correlation measure Among the values, some single items with relatively large values ​​are it ·k jt For the overall metric q i ·k j T The contribution will be more obvious. If we can identify k with larger values ​​and more items, j T , these k j T It is more likely to be the key we want to select than the others.

[0197] Assume S = 4, D = 3, and q1 = (q11 ,q 12 ,q 13 ) as an example, m1=2, m2=2.

[0198]

[0199] After sorting:

[0200]

[0201] Since m1 = 2, each value in Q only needs to be equal to K T The calculation can be performed on the first two columns of .

[0202] Assume q 11 The selected item is q 11 ·k 11 ,q 11 ·k 21 ,q 12 The selected item is q 12 ·k 22 ,q 12 ·k 32 ,q 13 The selected item is q 13 ·k 33 ,q 13 ·k 43 , then K T The number of occurrences of each column of data is (k 11 , (k 21 , k 22 ), (k 32 , k 33 ), k 43 ), then the single item quantity cnt=(1,2,2,1), the single item cumulative value value=(q 11 ·k 11 ,q 11 ·k 21 +q 12 ·k 22 ,q 12 ·k 32 +q 13 ·k 33 ,q 13 ·k 43 ).

[0203] Assume the final score is score = (0.1, 0.4, 0.4, 0.1), since m2 = 2, the key selected by the first query q1 is k2 T 、k3 T .

[0204] Similarly, the key selected by the first query q2 is k1 T 、k3T ; The key selected by the first query q3 is k1 T 、k2 T ; The key selected by the first query q3 is k3 T 、k4 T .

[0205] The above scheme, according to For a certain q i , for each k j T (j=1,2,…S), solve for several single terms q with relatively high values it ·k jt ,t=1,2,…d, put them in the target single-item queue. Then select a part with higher values ​​from the target single-item queue and calculate each k j T Among the contribution values, select several k with higher scores j T , as the selected one can be compared with the query q i The key to establish the connection.

[0206] In one embodiment, Multi-Head Attention includes multiple Self-Attentions, and the above steps 801 to 803, or steps 901 to 908 may be performed for each Self-Attention.

[0207] In another embodiment, some Self-Attentions are selected from Multi-Head Attention, and the selected Self-Attentions are subjected to correlation column data selection operations, i.e., the above steps 801 to 803, or steps 901 to 908 are executed. For example, assuming that a Multi-Head Attention includes 8 Self-Attentions, the 1st, 3rd, 5th, and 7th Self-Attentions are selected to perform correlation column data selection operations; or, the 2nd, 4th, 6th, and 8th Self-Attentions are selected to perform correlation column data selection operations. This application does not limit this.

[0208] In one embodiment, before step 801 or step 901, the method further includes:

[0209] Select K1 row data and K2 column data from the first key matrix to determine a fourth key matrix; wherein the first key matrix and the fourth key matrix have the same dimension; the K1 row data and the K2 column data have the same position in the first key matrix and the fourth key matrix; except for the K1 row data and the K2 column data in the fourth key matrix, the values ​​of other matrix positions are set to invalid values; the K1 is a positive integer less than the total number of rows of the first key matrix; the K2 is a positive integer less than the total number of columns of the first key matrix.

[0210] The step 801, sorting each row of data of the first key matrix to obtain a third key matrix, includes:

[0211] Each row of data in the fourth key matrix is ​​sorted to obtain the third key matrix.

[0212] The above scheme selects some elements from the first key matrix to form the fourth key matrix, and then performs a correlation column data selection operation on the fourth key matrix, which can further reduce the number of elements in the first key matrix. When determining the attention coefficient based on the second query matrix and the key matrix, it can reduce the computational complexity and memory complexity, thereby improving the efficiency of text processing.

[0213] In another embodiment, in addition to reducing the number of elements in the first key matrix, the number of elements in the first query matrix can also be reduced. Specifically, M1 row data and M2 column data are selected from the first query matrix to determine the second query matrix; wherein the dimensions of the first query matrix and the second query matrix are the same; the positions of the M1 row data and the M2 column data in the first query matrix and the second query matrix are the same; in the second query matrix, except for the M1 row data and the M2 column data, the values ​​of other matrix positions are set to invalid values; the M1 is a positive integer less than the total number of rows of the first query matrix; the M2 is a positive integer less than the total number of columns of the first query matrix.

[0214] The above scheme selects some elements from the first query matrix to form the second query matrix, which can reduce the number of elements in the second query matrix. When determining the attention coefficient based on the second query matrix and the key matrix, it can reduce the computational complexity and memory complexity, thereby improving the efficiency of text processing.

[0215] In this embodiment, the above step 505 may also be: inputting the second query matrix and the second key matrix into the self-attention layer in the converter algorithm to obtain the self-attention result.

[0216] In one embodiment, the embodiment is applied to Multi-Head Attention, and a portion of Self-Attention is selected from Multi-Head Attention, and a correlation column data selection operation is performed on the selected Self-Attention. The method is as follows: Fig.10 As shown, the following steps are included:

[0217] Step 1001: Select M1 row data and M2 column data from the first query matrix to determine a second query matrix.

[0218] Step 1002: Select K1 row data and K2 column data from the first key matrix to determine a fourth key matrix.

[0219] Step 1003, selecting correlation column data from the fourth key matrix to determine a second key matrix; the correlation degree between the correlation column data and the second query matrix is ​​greater than a first threshold.

[0220] Step 1004: input the second query matrix and the fourth key matrix into the first self-attention layer in the converter algorithm to obtain a first self-attention result.

[0221] In one possible implementation method, the first self-attention layer includes at least one Self-Attention.

[0222] Step 1005: input the second query matrix and the second key matrix into the second self-attention layer in the converter algorithm to obtain a second self-attention result.

[0223] In a possible implementation method, the second self-attention layer includes at least one Self-Attention. The second self-attention layer uses the second key matrix, which is the result of the fourth key matrix performing a correlation column data selection operation. Therefore, the second self-attention layer is the selected Self-Attention and needs to perform a correlation column data selection operation.

[0224] Step 1006, obtaining the self-attention result according to the first self-attention result and the second self-attention result.

[0225] The above scheme inputs different query matrices and key matrices to different self-attention layers, which enables the self-attention layer to learn richer content and effectively capture the key information in the input sentence; making the algorithm more scalable on different tasks and data sets.

[0226] The above steps 1001 to 1006 are described below using a specific example. Assume that Multi-HeadAttention contains 8 Self-Attentions, where the 1st, 3rd, 5th, and 7th Self-Attentions are the first self-attention layer, i.e., they do not perform the correlation column data selection operation, and the 2nd, 4th, 6th, and 8th Self-Attentions are the second self-attention layer, i.e., they perform the correlation column data selection operation. M1, M2, K1, and K2 are all 3.

[0227] Then the above steps 1001 to 1006 include: selecting 3 row data and 3 column data from the first query matrix to determine the second query matrix; selecting 3 row data and 3 column data from the first key matrix to determine the fourth key matrix; selecting correlation column data from the fourth key matrix to determine the second key matrix; using the second query matrix and the fourth key matrix as inputs of the 1st, 3rd, 5th, and 7th Self-Attentions, and inputting them into the 1st, 3rd, 5th, and 7th Self-Attentions to obtain the first self-attention result; using the second query matrix and the second key matrix as inputs of the 2nd, 4th, 6th, and 8th Self-Attentions, and inputting them into the 2nd, 4th, 6th, and 8th Self-Attentions to obtain the second self-attention result.

[0228] In one embodiment, the inputting of the second query matrix and the second key matrix into the second self-attention layer in the converter algorithm includes: selecting N values ​​from each row in the first query matrix, adding them to the corresponding positions of the second query matrix, and determining a third query matrix; wherein N is a positive integer; selecting M values ​​from each row in the first key matrix, adding them to the corresponding positions of the fourth key matrix, and determining a fifth key matrix; wherein M is a positive integer; selecting correlation column data from the fifth key matrix to obtain a sixth key matrix; and inputting the third query matrix and the sixth key matrix into the second self-attention layer in the converter algorithm.

[0229] The above scheme adds N elements to each row of data in the second query matrix and M elements to each row of data in the fourth key matrix, which can increase the randomness of the third query matrix and the fifth key matrix, reduce the design of artificial rules, and improve the accuracy of text processing.

[0230] In one embodiment, K1 row data and K2 column data are selected from the first key matrix according to certain rules, wherein the rules include but are not limited to the following. The following rules are also applicable to the first query matrix, that is, M1 row data and M2 column data are selected from the first query matrix. Here, only the first key matrix is ​​used as an example for explanation, and no further description is given.

[0231] Rule 1: Select from the first key matrix the elements whose horizontal coordinates are in the first set as the K1 row data; the numerical values ​​in the first set are related to the maximum length of the input sentence; select from the first key matrix the elements whose vertical coordinates are in the first set as the K2 column data.

[0232] Exemplarily, the horizontal coordinate i is selected from the first key matrix to take the value in the first set Or the vertical coordinate j takes the value in the set The elements in , and the rest of the elements are set to -∞; where S is the maximum length of the input sentence. Of course, the first set can also be other values, which are not limited in this application. The overall computational complexity and memory complexity of this method are O(S).

[0233] Rule 1 selects several global nodes to act as intermediaries for information dissemination between other nodes. These nodes will establish relationships with all other nodes. Here, we have selected and As a middleman. Taking S=8 as an example, the optimized matrix E is:

[0234]

[0235] Among them, q2, q6, k2 T and k6 T They are middlemen, and they are connected to each other's nodes.

[0236] Rule 2: Select elements whose horizontal coordinates are equal to the first numerical value from the first key matrix as the first row sub-data; the first numerical value is equal to half of the maximum length of the input sentence; select elements whose vertical coordinates are equal to the first numerical value from the first key matrix as the first column sub-data; select elements whose horizontal coordinates and vertical coordinates satisfy the first rule from the first key matrix as the second sub-data; the first rule means that the absolute difference between the horizontal coordinate and the vertical coordinate is less than the first difference; select elements whose horizontal coordinates and vertical coordinates satisfy the second rule from the first key matrix as the third sub-data; the second rule means that the sum of the horizontal coordinate and the vertical coordinate is equal to the maximum length of the input sentence; determine the K1 row data and the K2 column data respectively according to the first row sub-data, the first column sub-data, the second sub-data and the third sub-data.

[0237] For example, three types of elements are selected, one of which is the value of the horizontal coordinate i Or the value of the ordinate j elements; one type is the elements whose horizontal coordinate i and vertical coordinate j satisfy |ij|≤1, the other type is the elements whose horizontal coordinate i and vertical coordinate j satisfy i+j=S+1, and the rest of the elements are set to -∞. The overall computational complexity and memory complexity of this method is O(S).

[0238] Because most data has the characteristics of locality, we can try to let each node only connect with the corresponding word and the node corresponding to the surrounding word. Here we limit the connection to the node corresponding to the left and right word. In order to better exchange information over long distances, we select the nodes of both parties to establish a connection in a symmetrical position. At the same time, we select several middlemen to characterize the long-distance dependency relationship. Taking S=8 as an example, the optimized matrix E is:

[0239]

[0240] For example, q3 and k2 T 、k3 T and k4 T The connection is established, q8 and k1 are in symmetrical positions T The connection is established, middlemen q4, k4 T Both nodes are connected.

[0241] Rule three: select elements whose horizontal coordinates and vertical coordinates satisfy the third rule from the first key matrix as the fourth sub-data; the third rule means that the remainder of the horizontal coordinate or the vertical coordinate and the second value is a specified value, and / or the horizontal coordinate is the same as the vertical coordinate; select elements whose horizontal coordinates and vertical coordinates satisfy the fourth rule from the first key matrix as the fifth sub-data; the fourth rule means that the horizontal coordinate and / or the vertical coordinate are located in the vertex range of the first key matrix; according to the fourth sub-data and the fifth sub-data, determine the K1 row data and the K2 column data respectively.

[0242] For example, two types of elements are selected. One type is that the element that meets the conditions i mod3=1, j mod3=1 and i=j is p, and the elements in the 3×3 matrix with p as the upper left corner vertex are selected; the other type is the elements in the 3×3 matrix with (S,1) as the lower left corner vertex, or the elements in the 3×3 matrix with (1,S) as the upper right corner vertex, and the remaining elements are set to -∞. The overall computational complexity and memory complexity of this method are O(S);

[0243] Divide the text into groups of three, and the q in each group i and k j T Establish a full connection to characterize local dependencies. At the same time, (q1, q2, q3) and (k S-2T ,k S-1 T ,k S T ) establishes a full connection, (q S-2 ,q S-1 ,q s ) and (k1 T ,k2 T ,k3 T ) establishes a full connection to describe the long-distance relationship between the beginning and the end of the sentence. Taking S=8 as an example, the optimized matrix E is:

[0244]

[0245] For example, (q1,q2,q3) and (k1 T ,k2 T ,k3 T ) establishes a complete connection, (q1,q2,q3) and (k 8-2 T ,k 8-1 T ,k8 T )A full connection is established.

[0246] Rule 4: Select from the first key matrix an element whose abscissa value is equal to the third value and whose ordinate is an odd number as the sixth sub-data; select from the first key matrix an element whose abscissa value is equal to the fourth value and whose ordinate is an even number as the seventh sub-data; the third value and the fourth value are related to the maximum length of the input sentence, and the difference between the third value and the fourth value is a specified value; select from the first key matrix four elements located at the top corners as the eighth sub-data; determine the K1 row data and the K2 column data respectively according to the sixth sub-data, the seventh sub-data and the eighth sub-data;

[0247] or,

[0248] Select from the first key matrix an element whose ordinate value is equal to the third value and whose abscissa is an odd number as the sixth sub-data; select from the first key matrix an element whose ordinate value is equal to the fourth value and whose abscissa is an even number as the seventh sub-data; select from the first key matrix four elements located at the top corners as the eighth sub-data; and determine the K1 row data and the K2 column data respectively according to the sixth sub-data, the seventh sub-data and the eighth sub-data.

[0249] For example, three types of elements are selected. One type is that the horizontal coordinate i takes the value or The element whose ordinate value is an odd number, or the ordinate j value is or The first type is the element whose horizontal coordinate value is an odd number; the second type is the element whose horizontal coordinate i is or and the ordinate value is an even number, or the ordinate j value is or The first type is the elements whose horizontal coordinates are even numbers; the second type is the four vertex position elements, namely (1,1), (1,S), (S,1) and (S,S). The remaining elements are set to -∞. The overall computational complexity and memory complexity of this method are O(S).

[0250] If you choose and As a global node, in order to expand the receptive field of the global node without introducing additional computational complexity, the connection of each global node is dispersed to two nodes, such as The corresponding connections are scattered to and superior, k and j are even j T Connected, k and j are odd numbers j T In addition, increase e 11 、e 1S 、e S1 and e ss Connect to describe the head-to-tail relationship. Taking S=8 as an example, the optimized matrix E is:

[0251]

[0252] For example, the connection corresponding to q4 is distributed to q3 and q4, and the connection corresponding to q3 and k2 T 、k4 T 、k6 T 、k8 T Connected, q4 and k1 T 、k3 T 、k5 T , k7T connected, e 11 、e 18 、e 81 、e 88 Corresponding head-to-tail relationship.

[0253] In a possible implementation method, for any Self-Attention in Multi-Head Attention, any of the above rules can be selected to perform a selection operation on the first query matrix and the first key matrix.

[0254] In another possible implementation method, Multi-Head Attention includes 8 Self-Attentions, among which the 1st and 2nd Self-Attentions adopt the above-mentioned Rule 1 for selection operation, and the 2nd Self-Attention randomly adds N elements to each row for retention on the basis of the above-mentioned Rule 1; similarly, the 3rd and 4th Self-Attentions adopt the above-mentioned Rule 2 for selection operation, and the 4th Self-Attention randomly adds N elements to each row for retention on the basis of the above-mentioned Rule 2; the 5th and 6th Self-Attentions adopt the above-mentioned Rule 3 for selection operation, and the 6th Self-Attention randomly adds N elements to each row for retention on the basis of the above-mentioned Rule 3; the 7th and 8th Self-Attentions adopt the above-mentioned Rule 4 for selection operation, and the 8th Self-Attention randomly adds N elements to each row for retention on the basis of the above-mentioned Rule 4.

[0255] In one possible implementation method, 8 groups of attention outputs are obtained, each group of attention is independent of each other, denoted as head1, head2…head8, and the output size of each group is [S, D]. The 8 attention outputs are concatenated. Due to the concatenation operation, the output size is 8 times the original, that is, the size is [S, 8*D]. Then, linear transformation is used for dimensionality reduction, and the output size is [S, D]. The size of the linear transformation matrix is ​​[8*D, D], so the number of parameters of W is 8*D*D.

[0256] Based on the same technical concept, Fig.11 The text processing device 1100 based on the converter algorithm provided in the embodiment of the present application is exemplarily shown. Fig.11As shown, it includes: a receiving unit 1101 and a determining unit 1102; the receiving unit 1101 is used to receive first text data; the first text data includes a plurality of sentences; wherein each sentence can be divided into a plurality of words; the determining unit 1102 is used to determine a first text representation vector corresponding to each sentence in the first text data according to the words corresponding to each sentence in the first text data; the first text representation vector includes a word vector corresponding to each word; according to the first text representation vector and the relative position relationship between the words corresponding to each sentence in the first text data, a first query matrix and a first key matrix corresponding to the first text representation vector are determined; the first query matrix and the first key matrix are used to determine the relationship between different words in each sentence ; selecting correlation column data from the first key matrix to determine the second key matrix; the correlation degree between the correlation column data and the first query matrix is ​​greater than a first threshold; inputting the first query matrix and the second key matrix into the self-attention layer in the converter algorithm to obtain a self-attention result; determining the processed second text data according to the self-attention result and the weighted activation function; the weighted activation function is determined according to the weighted values ​​of the maximum activation function, the first normal distribution activation function and the second normal distribution activation function; the first normal distribution activation function is used to describe the nonlinear relationship of the parameters in the negative interval of the self-attention result; the second normal distribution activation function is used to describe the nonlinear relationship of the parameters in the positive interval of the self-attention result.

[0257] In a possible implementation method, the determination unit 1102 is specifically used to, for any sentence in the first text data, if the distance between the words corresponding to the sentence is less than or equal to a distance threshold, set the distance vector between the words corresponding to the sentence to a first distance vector; the first distance vector is used to indicate the relative position relationship between the words corresponding to the sentence, and the first distance vector is learned based on historical text data; if the distance between the words corresponding to the sentence is greater than the distance threshold, set the distance vector between the words corresponding to the sentence to a second distance vector according to a decay function; the decay function is used to represent a function that gradually decays with increasing distance; based on the first text representation vector, the first distance vector and the second distance vector, determine the first query matrix and the first key matrix corresponding to the first text representation vector.

[0258] In a possible implementation method, the determination unit 1102 is specifically used to sort each row of data in the first key matrix to obtain a third key matrix; calculate the valid values ​​in the first query matrix and the third key matrix to obtain a first result set; the first result set includes the product of the valid values ​​in the first query matrix and the valid values ​​in the third key matrix; according to the number of occurrences of each column of data in the third key matrix in the first result set, and the product of the valid values ​​in the first query matrix and the valid values ​​in the third key matrix, select the correlation column data from the first key matrix to determine the second key matrix.

[0259] In a possible implementation method, the determination unit 1102 is specifically used to select K1 row data and K2 column data from the first key matrix to determine a fourth key matrix; wherein the first key matrix and the fourth key matrix have the same dimension; the K1 row data and the K2 column data have the same position in the first key matrix and the fourth key matrix; except for the K1 row data and the K2 column data in the fourth key matrix, the values ​​of other matrix positions are set to invalid values; the K1 is a positive integer less than the total number of rows of the first key matrix; the K2 is a positive integer less than the total number of columns of the first key matrix; and each row of data of the fourth key matrix is ​​sorted to obtain the third key matrix.

[0260] In a possible implementation method, the determination unit 1102 is also used to select M1 row data and M2 column data from the first query matrix to determine a second query matrix; wherein the first query matrix and the second query matrix have the same dimension; the positions of the M1 row data and the M2 column data in the first query matrix and the second query matrix are the same; except for the M1 row data and the M2 column data in the second query matrix, the values ​​of other matrix positions are set to invalid values; the M1 is a positive integer less than the total number of rows of the first query matrix; the M2 is a positive integer less than the total number of columns of the first query matrix; the second query matrix and the second key matrix are input into the self-attention layer in the converter algorithm to obtain the self-attention result.

[0261] In a possible implementation method, the determination unit 1102 is specifically used to input the second query matrix and the fourth key matrix into the first self-attention layer in the converter algorithm to obtain a first self-attention result; input the second query matrix and the second key matrix into the second self-attention layer in the converter algorithm to obtain a second self-attention result; and obtain the self-attention result based on the first self-attention result and the second self-attention result.

[0262] In a possible implementation method, the determination unit 1102 is specifically used to select N values ​​from each row in the first query matrix, add them to the corresponding positions of the second query matrix, and determine the third query matrix; wherein N is a positive integer; select M values ​​from each row in the first key matrix, add them to the corresponding positions of the fourth key matrix, and determine the fifth key matrix; wherein M is a positive integer; select correlation column data from the fifth key matrix to obtain a sixth key matrix; and input the third query matrix and the sixth key matrix into the second self-attention layer in the converter algorithm.

[0263] In a possible implementation method, the determination unit 1102 is specifically used to select elements whose horizontal coordinate values ​​are in the first set from the first key matrix as the K1 row data; the numerical values ​​in the first set are related to the maximum length of the input sentence; and select elements whose vertical coordinate values ​​are in the first set from the first key matrix as the K2 column data.

[0264] In a possible implementation method, the determination unit 1102 is specifically used to select an element whose horizontal coordinate value is equal to a first numerical value from the first key matrix as the first row sub-data; the first numerical value is equal to half of the maximum length of the input sentence; select an element whose vertical coordinate value is equal to the first numerical value from the first key matrix as the first column sub-data; select an element whose horizontal coordinate and vertical coordinate satisfy a first rule from the first key matrix as the second sub-data; the first rule means that the absolute difference between the horizontal coordinate and the vertical coordinate is less than the first difference; select an element whose horizontal coordinate and vertical coordinate satisfy a second rule from the first key matrix as the third sub-data; the second rule means that the sum of the horizontal coordinate and the vertical coordinate is equal to the maximum length of the input sentence; and determine the K1 row data and the K2 column data respectively according to the first row sub-data, the first column sub-data, the second sub-data and the third sub-data.

[0265] In a possible implementation method, the determination unit 1102 is specifically used to select elements whose horizontal coordinates and vertical coordinates satisfy a third rule from the first key matrix as fourth sub-data; the third rule refers to the remainder of the horizontal coordinate or the vertical coordinate and the second value is a specified value, and / or the horizontal coordinate is the same as the vertical coordinate; select elements whose horizontal coordinates and vertical coordinates satisfy a fourth rule from the first key matrix as fifth sub-data; the fourth rule refers to the horizontal coordinate and / or the vertical coordinate are located in the vertex range of the first key matrix; and determine the K1 row data and the K2 column data respectively based on the fourth sub-data and the fifth sub-data.

[0266] In a possible implementation method, the determination unit 1102 is specifically used to select from the first key matrix an element whose horizontal coordinate value is equal to the third value and whose vertical coordinate is an odd number as the sixth sub-data; select from the first key matrix an element whose horizontal coordinate value is equal to the fourth value and whose vertical coordinate is an even number as the seventh sub-data; the third value and the fourth value are related to the maximum length of the input sentence, and the difference between the third value and the fourth value is a specified value; select from the first key matrix four elements located at the top corners as the eighth sub-data; and determine the K1 row data and the K2 column data respectively according to the sixth sub-data, the seventh sub-data and the eighth sub-data.

[0267] In a possible implementation method, the determination unit 1102 is specifically used to select from the first key matrix an element whose ordinate value is equal to the third value and whose abscissa is an odd number as the sixth sub-data; select from the first key matrix an element whose ordinate value is equal to the fourth value and whose abscissa is an even number as the seventh sub-data; select from the first key matrix four elements located at the top corners as the eighth sub-data; and determine the K1 row data and the K2 column data respectively according to the sixth sub-data, the seventh sub-data and the eighth sub-data.

[0268] In a possible implementation method, the calculation method of the first normal distribution activation function is as follows:

[0269]

[0270] Wherein, x is a parameter in the self-attention result; tanh is a tangent function; ρ1, ρ2 and ρ3 are parameters that need to be trained in the first normal distribution activation function;

[0271] The calculation method of the second normal distribution activation function is as follows:

[0272]

[0273] Wherein, x is a parameter in the self-attention result; tanh is a tangent function; δ1 and δ2 are parameters that need to be trained in the first normal distribution activation function; σ is a distribution function of the logistic distribution; e x is an exponential function.

[0274] Based on the same technical concept, the embodiment of the present application provides a text processing device 1200 based on a converter algorithm. The text processing device 1200 based on a converter algorithm may be, for example, a computing device. Fig.12As shown, a text processing device 1200 based on a converter algorithm includes at least one processor 1201 and a memory 1202 connected to the at least one processor. The specific connection medium between the processor 1201 and the memory 1202 is not limited in the embodiment of the present application. Fig.12 For example, the processor 1201 and the memory 1202 are connected via a bus. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0275] In an embodiment of the present application, the memory 1202 stores instructions that can be executed by at least one processor 1201. The at least one processor 1201 can execute the above-mentioned text processing method based on the converter algorithm by executing the instructions stored in the memory 1202.

[0276] The processor 1201 is a control center of the text processing device 1200 based on the converter algorithm, and can use various interfaces and lines to connect various parts of the computer device, and perform resource settings by running or executing instructions stored in the memory 1202 and calling data stored in the memory 1202. Optionally, the processor 1201 may include one or more determination units, and the processor 1201 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1201. In some embodiments, the processor 1201 and the memory 1202 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on independent chips.

[0277] Processor 1201 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware processor for execution, or can be executed by a combination of hardware and software modules in the processor.

[0278] Memory 1202, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules. Memory 1202 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (Random Access Memory, RAM), static random access memory (Static Random Access Memory, SRAM), programmable read-only memory (Programmable Read Only Memory, PROM), read-only memory (Read Only Memory, ROM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), magnetic memory, disk, optical disk, etc. Memory 1202 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 1202 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0279] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer-executable program. The computer-executable program is used to enable a computer to execute a text processing method based on a converter algorithm listed in any of the above methods.

[0280] An embodiment of the present application provides a computer program product, including a computer program executable by a computer device. When the program is run on the computer device, the computer device executes a text processing method based on a converter algorithm listed in any of the above-mentioned methods.

[0281] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0282] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0283] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0284] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0285] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A text processing method based on a converter algorithm, characterized in that: include: Receive first text data; the first text data includes a plurality of sentences; wherein each sentence can be divided into a plurality of words; Determine, according to the words corresponding to each sentence in the first text data, a first text representation vector corresponding to each sentence in the first text data; the first text representation vector includes a word vector corresponding to each word; Determine a first query matrix and a first key matrix corresponding to the first text representation vector according to the first text representation vector and the relative position relationship between the words corresponding to each sentence in the first text data; the first query matrix and the first key matrix are used to determine the correlation between different words in each sentence; Selecting correlation column data from the first key matrix to determine a second key matrix; the correlation degree between the correlation column data and the first query matrix is ​​greater than a first threshold; Inputting the first query matrix and the second key matrix into a self-attention layer in a converter algorithm to obtain a self-attention result; According to the self-attention result and the weighted activation function, the processed second text data is determined; the weighted activation function is determined according to the weighted values ​​of the maximum activation function, the first normal distribution activation function and the second normal distribution activation function; the first normal distribution activation function is used to describe the nonlinear relationship of the parameters in the negative interval of the self-attention result; the second normal distribution activation function is used to describe the nonlinear relationship of the parameters in the positive interval of the self-attention result.

2. The method according to claim 1, characterized in that The determining, according to the first text representation vector and the relative position relationship between words corresponding to each sentence in the first text data, a first query matrix and a first key matrix corresponding to the first text representation vector comprises: For any sentence in the first text data, if the distance between words corresponding to the sentence is less than or equal to a distance threshold, setting the distance vector between the words corresponding to the sentence as a first distance vector; the first distance vector is used to indicate the relative position relationship between the words corresponding to the sentence, and the first distance vector is learned based on historical text data; If the distance between the words corresponding to the sentence is greater than the distance threshold, setting the distance vector between the words corresponding to the sentence as a second distance vector according to a decay function; the decay function is used to represent a function that gradually decays as the distance increases; The first query matrix and the first key matrix corresponding to the first text representation vector are determined according to the first text representation vector, the first distance vector, and the second distance vector.

3. The method according to claim 1, characterized in that The step of selecting correlation column data from the first key matrix to determine a second key matrix includes: sorting each row of data of the first key matrix to obtain a third key matrix; Calculating the valid values ​​in the first query matrix and the third key matrix to obtain a first result set; the first result set includes the product of the valid values ​​in the first query matrix and the valid values ​​in the third key matrix; According to the number of occurrences of each column of the third key matrix in the first result set and the product of the valid values ​​in the first query matrix and the valid values ​​in the third key matrix, the relevant column data is selected from the first key matrix to determine the second key matrix.

4. The method according to claim 3, characterized in that Before sorting each row of data of the first key matrix to obtain the third key matrix, the method further includes: Select K1 row data and K2 column data from the first key matrix to determine a fourth key matrix; wherein the dimensions of the first key matrix and the fourth key matrix are the same; the positions of the K1 row data and the K2 column data in the first key matrix and the fourth key matrix are the same; except for the K1 row data and the K2 column data in the fourth key matrix, the values ​​of other matrix positions are set to invalid values; the K1 is a positive integer less than the total number of rows of the first key matrix; the K2 is a positive integer less than the total number of columns of the first key matrix; The step of sorting each row of data of the first key matrix to obtain a third key matrix includes: Each row of data in the fourth key matrix is ​​sorted to obtain the third key matrix.

5. The method according to claim 4, characterized in that The method further comprises: Select M1 row data and M2 column data from the first query matrix to determine a second query matrix; wherein the dimensions of the first query matrix and the second query matrix are the same; the positions of the M1 row data and the M2 column data in the first query matrix and the second query matrix are the same; except for the M1 row data and the M2 column data in the second query matrix, the values ​​of other matrix positions are set to invalid values; the M1 is a positive integer less than the total number of rows of the first query matrix; the M2 is a positive integer less than the total number of columns of the first query matrix; Inputting the first query matrix and the second key matrix into the self-attention layer in the converter algorithm to obtain a self-attention result, including: The second query matrix and the second key matrix are input into the self-attention layer in the converter algorithm to obtain the self-attention result.

6. The method according to claim 5, characterized in that The step of inputting the second query matrix and the second key matrix into the self-attention layer in the converter algorithm to obtain the self-attention result comprises: Inputting the second query matrix and the fourth key matrix into a first self-attention layer in the converter algorithm to obtain a first self-attention result; Inputting the second query matrix and the second key matrix into a second self-attention layer in the converter algorithm to obtain a second self-attention result; The self-attention result is obtained according to the first self-attention result and the second self-attention result.

7. The method according to claim 6, characterized in that The step of inputting the second query matrix and the second key matrix into a second self-attention layer in the converter algorithm comprises: Select N values ​​from each row in the first query matrix, add them to the corresponding positions in the second query matrix, and determine a third query matrix; wherein N is a positive integer; Selecting M values ​​from each row in the first key matrix and adding them to corresponding positions of the fourth key matrix to determine a fifth key matrix; wherein M is a positive integer; Select correlation column data from the fifth key matrix to obtain a sixth key matrix; The third query matrix and the sixth key matrix are input into the second self-attention layer in the converter algorithm.

8. The method according to any one of claims 4 to 7, characterized in that Select K1 row data and K2 column data from the first key matrix, including: Selecting elements whose horizontal coordinate values ​​are in a first set from the first key matrix as the K1 rows of data; the values ​​in the first set are related to the maximum length of the input sentence; Select elements whose ordinate values ​​are in the first set from the first key matrix as the K2 column data.

9. The method according to any one of claims 4 to 7, characterized in that Select K1 row data and K2 column data from the first key matrix, including: Selecting an element whose abscissa value is equal to a first value from the first key matrix as the first row of sub-data; the first value is equal to half of the maximum length of the input sentence; Selecting an element whose ordinate value is equal to the first value from the first key matrix as the first column of sub-data; Selecting elements whose horizontal coordinates and vertical coordinates satisfy a first rule from the first key matrix as second sub-data; the first rule means that the absolute difference between the horizontal coordinate and the vertical coordinate is less than the first difference; Selecting an element whose abscissa and ordinate satisfy a second rule from the first key matrix as the third sub-data; the second rule means that the sum of the abscissa and the ordinate is equal to the maximum length of the input sentence; The K1 row data and the K2 column data are determined respectively according to the first row sub-data, the first column sub-data, the second sub-data and the third sub-data.

10. The method according to any one of claims 4 to 7, characterized in that Select K1 row data and K2 column data from the first key matrix, including: Selecting an element whose abscissa and ordinate satisfy a third rule from the first key matrix as the fourth sub-data; the third rule means that the remainder of the abscissa or the ordinate and a second value is a specified value, and / or the abscissa is the same as the ordinate; Selecting elements whose abscissa and ordinate satisfy a fourth rule from the first bond matrix as fifth sub-data; the fourth rule means that the abscissa and / or the ordinate is located within the vertex range of the first bond matrix; The K1 row data and the K2 column data are determined according to the fourth sub-data and the fifth sub-data, respectively.

11. The method according to any one of claims 4 to 7, characterized in that Select K1 row data and K2 column data from the first key matrix, including: Selecting, from the first key matrix, an element whose abscissa value is equal to the third value and whose ordinate is an odd number as the sixth sub-data; Selecting an element whose abscissa value is equal to the fourth value and whose ordinate is an even number from the first key matrix as the seventh sub-data; the third value and the fourth value are related to the maximum length of the input sentence, and the difference between the third value and the fourth value is a specified value; Select four elements located at the top corners from the first key matrix as the eighth sub-data; Determine the K1 row data and the K2 column data respectively according to the sixth sub-data, the seventh sub-data and the eighth sub-data; or, Selecting, from the first key matrix, an element whose ordinate value is equal to the third value and whose abscissa is an odd number as the sixth sub-data; Selecting, from the first key matrix, an element whose ordinate value is equal to the fourth value and whose abscissa is an even number as the seventh sub-data; Select four elements located at the top corners from the first key matrix as the eighth sub-data; The K1 row data and the K2 column data are determined respectively according to the sixth sub-data, the seventh sub-data and the eighth sub-data.

12. The method according to claim 1, characterized in that The calculation method of the first normal distribution activation function is as follows: Wherein, x is a parameter in the self-attention result; tanh is a tangent function; ρ1, ρ2 and ρ3 are parameters that need to be trained in the first normal distribution activation function; The calculation method of the second normal distribution activation function is as follows: Wherein, x is a parameter in the self-attention result; tanh is a tangent function; δ1 and δ2 are parameters that need to be trained in the first normal distribution activation function; σ is a distribution function of the logistic distribution; e x is an exponential function.

13. A text processing device based on a converter algorithm, characterized in that: It includes a receiving unit and a determining unit: The receiving unit is used to receive first text data; the first text data includes a plurality of sentences; wherein each sentence can be divided into a plurality of words; The determination unit is used to determine the first text representation vector corresponding to each sentence in the first text data according to the words corresponding to each sentence in the first text data; the first text representation vector includes the word vector corresponding to each word; according to the first text representation vector and the relative position relationship between the words corresponding to each sentence in the first text data, determine the first query matrix and the first key matrix corresponding to the first text representation vector; the first query matrix and the first key matrix are used to determine the correlation between different words in each sentence; select correlation column data from the first key matrix to determine the second key matrix; the correlation between the correlation column data and the first query matrix is ​​greater than a first threshold; input the first query matrix and the second key matrix into the self-attention layer in the converter algorithm to obtain a self-attention result; determine the processed second text data according to the self-attention result and the weighted activation function; the weighted activation function is determined according to the weighted values ​​of the maximum activation function, the first normal distribution activation function and the second normal distribution activation function; the first normal distribution activation function is used to describe the nonlinear relationship of the parameters in the negative interval of the self-attention result; the second normal distribution activation function is used to describe the nonlinear relationship of the parameters in the positive interval of the self-attention result.

14. A computing device, characterized in that include: A memory for storing program instructions; A processor is used to call the program instructions stored in the memory, and execute the method according to any one of claims 1 to 12 according to the obtained program instructions.

15. A computer-readable storage medium, characterized in that: The method comprises computer-readable instructions, and when a computer reads and executes the computer-readable instructions, the method according to any one of claims 1 to 12 is implemented.

16. A computer program product, characterized in that The invention comprises a computer program executable by a computer device, and when the program is run on the computer device, the computer device is caused to execute the steps of the method according to any one of claims 1 to 12.

Images