Method, device and storage medium for representing natural language association features

Abstract

The present invention provides a method for representing natural language association features based on linear calculation, including the following steps: S1. Obtaining a natural language word sequence to generate a first feature matrix A with n rows and m columns of the natural language, where n is the number of words in the natural language and m is the number of bits of code for each word; S2. Cumulatively summing each column of the first feature matrix A with n rows and m columns to generate a second feature matrix B with n rows and m columns; S3. Performing a linear transformation of the second feature matrix B with n rows and m columns using a neural network to obtain a third feature matrix C with n rows and m columns; S4. Normalizing the third feature matrix C with n rows and m columns to obtain a fourth feature matrix D with n rows and m columns; the fourth feature matrix D represents the natural language association features. The present invention avoids the large number of matrix multiplication calculations of the attention mechanism, greatly reduces the amount of calculation, effectively improves the linear computing power, and reduces the use of video memory.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, a device and a storage medium for representing natural language association features based on linear calculation. Background Art

[0002] Since there are mutual relationships between n words in natural language, for example, in a sentence: [Tomorrow, the weather is, rainy], "weather" has an important correlation with "rainy day", in order to represent such a correlation, the existing technology generally needs to calculate the correlation features. The current commonly used method is based on the attention mechanism.

[0003] The general calculation steps of the attention mechanism are:

[0004] 1. Linear transformation of input vectors: Three linear transformations are performed on each word vector (e.g., a single word or word embedding) in the input natural language word sequence to obtain the query vector, key vector, and value vector. The weight matrices for these three transformations are learned.

[0005] 2. Calculating Attention Scores: The attention scores are calculated using the dot product of the query vector and the key vector. The dot product attention mechanism takes the dot product of the query vector and each key vector as the attention score. These scores are then scaled (divided by the square root of the vector's dimension) and normalized using the Softmax function to obtain the attention weight for each position.

[0006] 3. Weighted Sum: Multiply the normalized attention weight by the corresponding value vector, then sum the weighted values at all positions. This weighted sum is the output of the attention mechanism. The weighted sum is the natural language association feature represented.

[0007] The existing technology of natural language related feature representation based on the attention mechanism has the following technical problems:

[0008] (1) Computational complexity: The computational complexity of the attention mechanism increases with the length of the word sequence, because each position needs to calculate the attention weight with all other positions. This leads to a significant increase in the computational cost of the model when processing long word sequences, limiting its scalability in practical applications.

[0009] (2) Position Information Processing: The attention mechanism itself does not directly consider the position information of words, so it may perform poorly when processing disordered input (such as aggregate data). To solve this problem, the Transformer neural network (a deep learning neural network model based on the self-attention mechanism) introduced position encoding to inject position information, but this method does not always capture the accurate relationship between all positions.

[0010] (3) Sensitivity to noise: The attention mechanism may become sensitive to a certain degree of noise in the input, resulting in unstable output. Some methods (such as dropout or layer normalization) can alleviate this problem, but they cannot completely solve it.

[0011] The computational complexity of representing natural language association features based on the attention mechanism has seriously affected the training and prediction efficiency of large models of the attention mechanism. Although there have been a lot of explorations in the industry, it has not been possible to improve performance within the scope of linear calculation. Summary of the Invention

[0012] The present invention provides a method, a device and a storage medium for representing natural language association features based on linear calculation to solve the technical problems of the prior art methods for representing natural language association features, such as complex calculations, the need for position information processing and sensitivity to noise.

[0013] One aspect of the present invention is to provide a method for representing natural language association features based on linear calculation, the method comprising the following steps:

[0014] S1. Obtain a word sequence of a natural language and generate a first feature matrix A of the natural language with n rows and m columns, where n is the number of words in the natural language and m is the number of bits of code for each word;

[0015] S2. Cumulatively sum each column of the first characteristic matrix A with n rows and m columns to generate a second characteristic matrix B with n rows and m columns;

[0016] S3, performing a linear transformation of the neural network on the second characteristic matrix B with n rows and m columns to obtain a third characteristic matrix C with n rows and m columns;

[0017] S4. Normalize the third feature matrix C with n rows and m columns to obtain a fourth feature matrix D with n rows and m columns; the fourth feature matrix D represents natural language association features.

[0018] In a preferred embodiment, in step S1, a unique and non-repeating ID number is assigned to each word in the natural language to obtain a word sequence in the natural language;

[0019] For each word ID number, an m-bit code is randomly generated to generate the first feature matrix A of the natural language with n rows and m columns.

[0020] In a preferred embodiment, in step S2, for each element value of each column of the first characteristic matrix A, the previous element value is sequentially accumulated to generate a second characteristic matrix B with n rows and m columns.

[0021] In a preferred embodiment, in step S3, a linear transformation matrix X with m rows and m columns is constructed, and the second characteristic matrix B with n rows and m columns is multiplied by the linear transformation matrix X with m rows and m columns to obtain a third characteristic matrix C with n rows and m columns.

[0022] In a preferred embodiment, in step S4 , the value of each element of the fourth characteristic matrix D with n rows and m columns ranges from -1 to 1.

[0023] In a preferred embodiment, in step S4, the third characteristic matrix C with n rows and m columns is normalized using the Min-Max normalization method or the Z-Score normalization method, so that the value of each element of the obtained fourth characteristic matrix D with n rows and m columns ranges from -1 to 1.

[0024] Another object of the present invention is to provide a representation device for natural language association features based on linear calculation, the representation device comprising:

[0025] A first feature matrix A generation module is used to obtain a natural language word sequence and generate a first feature matrix A of the natural language with n rows and m columns, where n is the number of words in the natural language and m is the number of encoding bits for each word;

[0026] A second characteristic matrix B generating module is used to accumulate and sum each column of the first characteristic matrix A with n rows and m columns to generate a second characteristic matrix B with n rows and m columns;

[0027] A third characteristic matrix C generation module is used to perform a linear transformation of the neural network on the second characteristic matrix B with n rows and m columns to obtain a third characteristic matrix C with n rows and m columns;

[0028] The fourth feature matrix D generation module is used to normalize the third feature matrix C with n rows and m columns to obtain a fourth feature matrix D with n rows and m columns; the fourth feature matrix D represents natural language association features.

[0029] In a preferred embodiment, the first feature matrix A generation module assigns a unique and non-repeating ID number to each word in the natural language to obtain a word sequence in the natural language;

[0030] For each word ID number, an m-bit code is randomly generated to generate the first feature matrix A of the natural language with n rows and m columns.

[0031] In a preferred embodiment, the second characteristic matrix B generating module accumulates the previous element value in each column of the first characteristic matrix A to generate a second characteristic matrix B with n rows and m columns;

[0032] The third characteristic matrix C generation module constructs an m-row and m-column linear transformation matrix X, and multiplies the n-row and m-column second characteristic matrix B by the m-row and m-column linear transformation matrix X to obtain the n-row and m-column third characteristic matrix C;

[0033] The fourth characteristic matrix D generation module normalizes the third characteristic matrix C with n rows and m columns, so that each element value of the obtained fourth characteristic matrix D with n rows and m columns ranges from -1 to 1.

[0034] Another object of the present invention is to provide a storage medium for storing computer-executable instructions, wherein the computer-executable instructions are used to execute the method for representing natural language association features based on linear calculation provided by the present invention.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] The present invention provides a method, device, and storage medium for representing natural language association features based on linear calculations, which avoids the large number of matrix multiplication calculations of the attention mechanism, greatly reduces the amount of calculation, effectively improves the linear computing power, and reduces the use of video memory.

[0037] The present invention provides a method, device, and storage medium for representing natural language association features based on linear calculations. The method directly accumulates and sums each column of the first feature matrix A generated by the word vector. The accumulation process transfers all the influencing features of the previous sequence to the backward sequence, maintaining a good relationship between the previous and next sequences. The features can be widely transmitted, and the influence of position information on the results is weakened. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 This is a flow chart of a method for representing natural language association features based on linear calculation in the present invention.

[0040] Figure 2 This is a structural block diagram of a natural language association feature representation device based on linear calculation in the present invention. DETAILED DESCRIPTION

[0041] In order to make the above and other features and advantages of the present invention more clear, the present invention is further described below with reference to the accompanying drawings. It should be understood that the specific embodiments given herein are for the purpose of explaining to those skilled in the art and are only exemplary and not restrictive.

[0042] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like to indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0043] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0044] like Figure 1 As shown, according to an embodiment of the present invention, a method for representing natural language association features based on linear calculation is provided, comprising the following method steps:

[0045] Step S1: Obtain a word sequence of a natural language and generate a first feature matrix A of the natural language with n rows and m columns, where n is the number of words in the natural language and m is the number of encoding bits for each word.

[0046] Specifically, assign a unique and non-repeating ID number to each word in the natural language, and obtain the word sequence of the natural language.

[0047] For the ID number of each word, randomly generate an m-bit code, and generate the first feature matrix A of n rows and m columns of the natural language.

[0048] For example, for a natural language ["tomorrow", "the", "weather", "is", "rainy day"], there are n = 5 words in total. Assign a unique and non-repeating ID number to each word. For example, "tomorrow" is assigned the ID number 1, "the" is assigned the ID number 2, "weather" is assigned the ID number 3, "is" is assigned the ID number 4, and "rainy day" is assigned the ID number 5, and obtain the natural language sequence [1, 2, 3, 4, 5].

[0049] For the ID number of each word, randomly generate an m-bit code. For example, for the ID number of each word, randomly generate an m = 4-bit code. Then, the word "tomorrow" with the ID number 1 generates a 4-bit code word vector (Embedding): [0.1, 0.2, 0.3, 0.4];

[0050] The word "the" with the ID number 2 generates a 4-bit code word vector (Embedding): [0.2, 0.5, 0.1, 0.3];

[0051] The word "weather" with the ID number 3 generates a 4-bit code word vector (Embedding): [0.5, 0.9, 0.2, 0.3];

[0052] The word "is" with the ID number 4 generates a 4-bit code word vector (Embedding): [0.3, 0.2, 0.1, 0.7];

[0053] The word "rainy day" with the ID number 5 generates a 4-bit code word vector (Embedding): [0.1, 0.4, 0.2, 0.9].

[0054] According to the natural language sequence [1, 2, 3, 4, 5] of n words and the word vectors (Embedding) of m = 4-bit codes, generate the first feature matrix A of n rows and m columns of the natural language. For example, for a natural language ["tomorrow", "the", "weather", "is", "rainy day"], generate a 5-row and 4-column first feature matrix A:

[0055] [[0.1, 0.2, 0.3, 0.4]

[0056] [0.2, 0.5, 0.1, 0.3]

[0057] [0.5, 0.9, 0.2, 0.3]

[0058] [0.3,0.2,0.1,0.7]

[0059] [0.1,0.4,0.2,0.9]].

[0060] It should be noted that the number of encoding bits m of the word embedding generated for each word is randomly initialized, and the element values of the word embedding generated for each word are also randomly initialized. The element values of the word embedding generated for each word with random initialization are continuously adjusted during the reverse learning process of the neural network to optimize the first feature matrix A.

[0061] Step S2: cumulatively sum each column of the first characteristic matrix A with n rows and m columns to generate a second characteristic matrix B with n rows and m columns.

[0062] Specifically, for each element value of each column of the first characteristic matrix A, the previous element value is sequentially accumulated to generate a second characteristic matrix B with n rows and m columns.

[0063] Take the first column [0.1, 0.2, 0.5, 0.3, 0.1] of the first feature matrix A generated by a natural language [tomorrow, the weather is, rainy] as an example.

[0064] The first element value 0.1 remains unchanged, and the cumulative sum is itself, that is, 0.1;

[0065] The second element value is 0.2, which is added to the previous element value, that is, 0.2 + 0.1 = 0.3;

[0066] The third element has a value of 0.5, which is added to the previous element value, i.e. 0.5 + 0.3 = 0.8.

[0067] The fourth element value is 0.3, which is the sum of the previous element value, that is, 0.3 + 0.8 = 1.1;

[0068] The fifth element value is 0.1, which is the sum of the previous element value, that is, 0.1+1.1=1.2.

[0069] By analogy, for each element value in each column of the first characteristic matrix A, the previous element value is accumulated in sequence to obtain the second characteristic matrix B with 5 rows and 4 columns:

[0070] [[0.1000,0.2000,0.3000,0.4000]

[0071] [0.3000,0.7000,0.4000,0.7000]

[0072] [0.8000,1.6000,0.6000,1.0000]

[0073] [1.1000,1.8000,0.7000,1.7000]

[0074] [1.2000,2.2000,0.9000,2.6000]].

[0075] Step S3: Perform a neural network linear transformation on the second characteristic matrix B having n rows and m columns to obtain a third characteristic matrix C having n rows and m columns.

[0076] Specifically, a linear transformation matrix X with m rows and m columns is constructed, and the second characteristic matrix B with n rows and m columns is multiplied by the linear transformation matrix X with m rows and m columns to obtain a third characteristic matrix C with n rows and m columns.

[0077] For example, for a natural language [tomorrow, the, weather, is, rainy], according to the second feature matrix B with 5 rows and 4 columns generated in step S2, a linear transformation matrix X with 4 rows and 4 columns is constructed:

[0078] [[0.7260,0.3795,0.6275,0.8122]

[0079] [0.9521,0.5301,0.5481,0.2933]

[0080] [0.1535,0.7780,0.6424,0.1868]

[0081] [0.3251,0.4054,0.7136,0.0900]].

[0082] Each element value of the linear transformation matrix X is randomly initialized.

[0083] The second characteristic matrix B with 5 rows and 4 columns is subjected to a linear transformation of the neural network. Specifically, the second characteristic matrix B is multiplied by the linear transformation matrix X with 4 rows and 4 columns to obtain a third characteristic matrix C with 5 rows and 4 columns, that is, B×X=C.

[0084] In this embodiment, the obtained 5-row 4-column third characteristic matrix C is:

[0085] [[0.4391,0.5395,0.6505,0.2319]

[0086] [1.1732,1.0799,1.3284,0.5867]

[0087] [2.5214,2.0240,2.4780,1.3211]

[0088] [3.1725,2.6054,3.3396,1.7051]

[0089] [3.9492,3.3759,4.3923,2.0220]].

[0090] Step S4: normalize the third characteristic matrix C with n rows and m columns to obtain a fourth characteristic matrix D with n rows and m columns.

[0091] The fourth feature matrix D with n rows and m columns represents natural language association features, and the value of each element of the fourth feature matrix D with n rows and m columns ranges from -1 to 1.

[0092] Furthermore, the Min-Max (deviation normalization) normalization method or the Z-Score (standard score) normalization method is used to normalize the third characteristic matrix C of n rows and m columns, so that the value of each element of the obtained fourth characteristic matrix D of n rows and m columns ranges from -1 to 1.

[0093] For example, for a natural language [tomorrow, the, weather, is, rainy], the 5-row, 4-column third feature matrix C is normalized using the Min-Max normalization method to obtain the 5-row, 4-column fourth feature matrix D.

[0094] The fourth feature matrix D with 5 rows and 4 columns represents the associated features of a natural language [tomorrow, the, weather, is, rainy day].

[0095] like Figure 2 As shown, according to an embodiment of the present invention, a natural language association feature representation device based on linear calculation is provided, comprising:

[0096] The first feature matrix A generating module 100 is used to obtain a word sequence of a natural language and generate a first feature matrix A of the natural language with n rows and m columns, where n is the number of words in the natural language and m is the number of encoding bits for each word.

[0097] Specifically, the first feature matrix A generating module 100 assigns a unique and non-repeating ID number to each word in the natural language to obtain a word sequence in the natural language.

[0098] For each word ID number, an m-bit code is randomly generated to generate the first feature matrix A of the natural language with n rows and m columns.

[0099] The second characteristic matrix B generating module 200 is configured to perform cumulative summation on each column of the first characteristic matrix A having n rows and m columns to generate a second characteristic matrix B having n rows and m columns.

[0100] Specifically, the second characteristic matrix B generating module 200 sequentially accumulates the previous element value for each element value of each column of the first characteristic matrix A to generate a second characteristic matrix B with n rows and m columns.

[0101] The third characteristic matrix C generating module 300 is used to perform a neural network linear transformation on the second characteristic matrix B with n rows and m columns to obtain a third characteristic matrix C with n rows and m columns.

[0102] Specifically, the third characteristic matrix C generating module 300 constructs a linear transformation matrix X with m rows and m columns, and multiplies the second characteristic matrix B with n rows and m columns by the linear transformation matrix X with m rows and m columns to obtain a third characteristic matrix C with n rows and m columns.

[0103] The fourth feature matrix D generating module 400 is used to perform normalization processing on the third feature matrix C with n rows and m columns to obtain a fourth feature matrix D with n rows and m columns; the fourth feature matrix D represents natural language association features.

[0104] The fourth characteristic matrix D generating module 400 performs normalization processing on the third characteristic matrix C of n rows and m columns, so that each element value of the obtained fourth characteristic matrix D of n rows and m columns ranges from -1 to 1.

[0105] Furthermore, the Min-Max (deviation normalization) normalization method or the Z-Score (standard score) normalization method is used to normalize the third characteristic matrix C of n rows and m columns, so that the value of each element of the obtained fourth characteristic matrix D of n rows and m columns ranges from -1 to 1.

[0106] According to an embodiment of the present invention, a storage medium is provided for storing computer-executable instructions, wherein the computer-executable instructions are used to execute a method for representing natural language association features based on linear calculation provided by the present invention.

[0107] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are exemplary and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A method for representing natural language association features based on linear calculation, characterized in that: The representation method comprises the following steps: S1. Obtain a word sequence of a natural language and generate a first feature matrix A of the natural language with n rows and m columns, where n is the number of words in the natural language and m is the number of bits of code for each word; S2. Cumulatively sum each column of the first characteristic matrix A with n rows and m columns to generate a second characteristic matrix B with n rows and m columns; S3, performing a linear transformation of the neural network on the second characteristic matrix B with n rows and m columns to obtain a third characteristic matrix C with n rows and m columns; S4. Normalize the third feature matrix C with n rows and m columns to obtain a fourth feature matrix D with n rows and m columns; the fourth feature matrix D represents natural language association features.

2. The method according to claim 1, wherein: In step S1, a unique and non-repeating ID number is assigned to each word in the natural language to obtain a word sequence of the natural language; For each word ID number, an m-bit code is randomly generated to generate the first feature matrix A of the natural language with n rows and m columns.

3. The representation method according to claim 1, characterized in that: In step S2, for each element value of each column of the first characteristic matrix A, the previous element value is sequentially accumulated to generate a second characteristic matrix B with n rows and m columns.

4. The representation method according to claim 1, characterized in that: In step S3, a linear transformation matrix X with m rows and m columns is constructed, and the second characteristic matrix B with n rows and m columns is multiplied by the linear transformation matrix X with m rows and m columns to obtain a third characteristic matrix C with n rows and m columns.

5. The representation method according to claim 1, characterized in that: In step S4 , the value of each element of the fourth characteristic matrix D with n rows and m columns ranges from -1 to 1.

6. The representation method according to claim 1, characterized in that: In step S4, the third characteristic matrix C with n rows and m columns is normalized using the Min-Max normalization method or the Z-Score normalization method, so that each element value of the obtained fourth characteristic matrix D with n rows and m columns ranges from -1 to 1.

7. A device for representing natural language association features based on linear calculation, characterized in that: The display device comprises: A first feature matrix A generation module is used to obtain a natural language word sequence and generate a first feature matrix A of the natural language with n rows and m columns, where n is the number of words in the natural language and m is the number of encoding bits for each word; A second characteristic matrix B generating module is used to accumulate and sum each column of the first characteristic matrix A with n rows and m columns to generate a second characteristic matrix B with n rows and m columns; A third characteristic matrix C generation module is used to perform a linear transformation of the neural network on the second characteristic matrix B with n rows and m columns to obtain a third characteristic matrix C with n rows and m columns; The fourth feature matrix D generation module is used to normalize the third feature matrix C with n rows and m columns to obtain a fourth feature matrix D with n rows and m columns; the fourth feature matrix D represents natural language association features.

8. The display device according to claim 7, characterized in that The first feature matrix A generation module assigns a unique and non-repeating ID number to each word in the natural language to obtain the word sequence of the natural language; For each word ID number, an m-bit code is randomly generated to generate the first feature matrix A of the natural language with n rows and m columns.

9. The display device according to claim 7, characterized in that The second characteristic matrix B generation module accumulates the previous element value of each element value in each column of the first characteristic matrix A in sequence to generate a second characteristic matrix B with n rows and m columns; The third characteristic matrix C generation module constructs an m-row and m-column linear transformation matrix X, multiplies the n-row and m-column second characteristic matrix B by the m-row and m-column linear transformation matrix X, and obtains the m-row and m-column third characteristic matrix C; The fourth characteristic matrix D generation module normalizes the third characteristic matrix C with n rows and m columns, so that each element value of the obtained fourth characteristic matrix D with n rows and m columns ranges from -1 to 1.

10. A storage medium, characterized in that: The storage medium is used to store computer-executable instructions, and the computer-executable instructions are used to execute the representation method according to any one of claims 1 to 6.

Images