Semantic understanding and generating method in natural language processing based on self-attention mechanism

Abstract

The invention discloses a semantic comprehension and generation method in natural language processing based on a self-attention mechanism, relates to the technical field of natural language processing, and can effectively capture a long-distance dependency relationship in an input text so as to improve the accuracy of semantic comprehension. A smooth text conforming to input text semantics can be generated; and the calculation cost can be obviously reduced. The key points of the scheme are as follows: segmenting an input text to obtain word or sub-word units; using a pre-trained word embedding model to map the word or sub-word unit into a first vector of a preset dimension; adding the first vector and the position coding vector to obtain a second vector; constructing a multi-layer self-attention mechanism, and performing linear transformation on the second vector of each position to obtain a vector sequence; encoding the data and extracting a semantic vector; and generating an output text related to input text semantics in combination with the multi-layer self-attention mechanism and the semantic vector. The method is used for natural language processing.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and particularly to a semantic understanding and generation method in natural language processing based on the self-attention mechanism. Background Art

[0002] With the rapid development of artificial intelligence technology, natural language processing (NLP) has made remarkable progress in fields such as machine translation, text generation, and question answering systems. Traditional natural language processing methods mainly rely on recurrent neural networks (RNNs) and their variants (such as LSTMs, GRUs), which have problems of gradient disappearance and low computational efficiency when dealing with long texts. In recent years, the self-attention mechanism has been widely used in natural language processing tasks as a new technical means. It can effectively capture long-range dependencies in text and significantly improve the training speed and performance of the model.

[0003] Existing models based on the self-attention mechanism still have some limitations in semantic understanding and generation tasks. For example, when dealing with complex semantic relationships, the model may not be able to fully capture context information, resulting in low-quality generated text or inaccurate semantic understanding. In addition, when facing large-scale datasets, the computational cost of the training and inference processes of existing models is relatively high, which limits the scope of their practical applications. Summary of the Invention

[0004] The present invention provides a semantic understanding and generation method in natural language processing based on the self-attention mechanism. By constructing a multi-layer self-attention mechanism, it can effectively capture long-range dependencies in the input text, thereby improving the accuracy of semantic understanding; combining the multi-layer self-attention mechanism and semantic vectors to generate output text related to the semantics of the input text, and being able to generate smooth text that conforms to the semantics of the input text; the multi-layer self-attention mechanism has high computational efficiency and can significantly reduce the computational cost.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] The present invention provides a semantic understanding and generation method in natural language processing based on the self-attention mechanism, including:

[0007] Segment the input text to obtain word or sub-word units.

[0008] Use a pre-trained word embedding model to map the word or the sub-word unit to a first vector of a preset dimension.

[0009] Add the first vector and a position encoding vector to obtain a second vector.

[0010] Construct a multi-layer self-attention mechanism to perform a linear transformation on the second vectors at each position to obtain a sequence of vectors.

[0011] Encode the sequence of vectors and extract semantic vectors.

[0012] Combine the multi-layer self-attention mechanism and the semantic vectors to generate an output text related to the semantics of the input text.

[0013] Further, in the method for semantic understanding and generation in natural language processing based on the self-attention mechanism, constructing a multi-layer self-attention mechanism to perform a linear transformation on the second vectors at each position to obtain a sequence of vectors includes:

[0014] Convert the second vectors into query vectors, key vectors, and value vectors.

[0015] Calculate self-attention scores according to the query vectors and the key vectors.

[0016] Normalize the scaled self-attention scores to obtain self-attention weights.

[0017] Multiply the self-attention weights by the value vectors to obtain a weighted output.

[0018] Perform concatenation and linear transformation on the weighted output to obtain the sequence of vectors.

[0019] Further, in the method for semantic understanding and generation in natural language processing based on the self-attention mechanism, after combining the multi-layer self-attention mechanism and the semantic vectors to generate an output text related to the semantics of the input text, it further includes:

[0020] Measure the difference between the output text and the target text, and correspondingly adjust the parameters of the word embedding model according to the difference.

[0021] Further, in the method for semantic understanding and generation in natural language processing based on the self-attention mechanism, measuring the difference between the output text and the target text and correspondingly adjusting the parameters of the word embedding model according to the difference includes:

[0022] Use the cross-entropy loss function to measure the difference between the output text and the target text, and correspondingly adjust the parameters of the word embedding model according to the difference through the gradient descent method.

[0023] Further, in the method for semantic understanding and generation in natural language processing based on the self-attention mechanism, segmenting the input text to obtain word or sub-word units includes:

[0024] When the input text is an English text, split it using spaces to obtain words; when the input text is a Chinese text, use a word segmentation tool to segment the Chinese text into single characters or phrases to obtain sub-word units.

[0025] Further, in the semantic understanding and generation method of natural language processing based on the self-attention mechanism, the pre-trained word embedding model includes:

[0026] The pre-trained word embedding model is Word2Vec or GloVe or BERT.

[0027] Further, in the semantic understanding and generation method of natural language processing based on the self-attention mechanism, the preset dimension includes:

[0028] The preset dimension is 768 dimensions.

[0029] Further, in the semantic understanding and generation method of natural language processing based on the self-attention mechanism, encoding the vector sequence and extracting the semantic vector includes:

[0030] Encoding the vector sequence and extracting the semantic vector through a pooling operation or an attention mechanism.

[0031] Further, in the semantic understanding and generation method of natural language processing based on the self-attention mechanism, calculating the self-attention score according to the query vector and the key vector includes:

[0032] Calculating the dot product of the query vector and the key vector, and then dividing by the scaling factor to obtain the self-attention score.

[0033] Further, in the semantic understanding and generation method of natural language processing based on the self-attention mechanism, normalizing the scaled self-attention score to obtain the self-attention weight includes:

[0034] Normalizing the scaled self-attention score through the Softmax function to obtain the self-attention weight.

[0035] The present invention provides a semantic understanding and generation method in natural language processing based on self-attention mechanism, including: segmenting the input text to obtain word or sub-word units; using a pre-trained word embedding model to map the word or sub-word units into first vectors of a preset dimension; adding the first vectors to position encoding vectors to obtain second vectors; constructing a multi-layer self-attention mechanism to perform a linear transformation on the second vectors at each position to obtain a vector sequence; encoding the vector sequence and extracting semantic vectors; combining the multi-layer self-attention mechanism and the semantic vectors to generate an output text semantically related to the input text. Compared with the prior art, the present invention can effectively capture long-distance dependency relationships in the input text by constructing a multi-layer self-attention mechanism, thereby improving the accuracy of semantic understanding; combining the multi-layer self-attention mechanism and the semantic vectors to generate an output text semantically related to the input text can generate a smooth text that conforms to the semantics of the input text; the multi-layer self-attention mechanism has high computational efficiency and can significantly reduce the computational cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. The drawings are only for the purpose of showing the embodiments and are not considered to be a limitation of the present invention.

[0037] Figure 1 It is a schematic flow chart of a semantic understanding and generation method in natural language processing based on self-attention mechanism in an embodiment of the present invention;

[0038] Figure 2 It is a schematic flow chart of another semantic understanding and generation method in natural language processing based on self-attention mechanism in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0039] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the scope of protection of the present invention.

[0040] Unless otherwise defined, all technical and scientific terms used in the present invention have the same meaning as commonly understood by those skilled in the technical field to which the present invention belongs; the terms used in the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The terms "including" and "having" in the specification and claims of the present invention and any variations thereof are intended to cover non-exclusive inclusion.

[0041] In the description of the embodiments of the present invention, technical terms such as "first" and "second" are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present invention, the meaning of "a plurality of" is more than two, unless otherwise specifically defined.

[0042] In the description of the embodiments of the present invention, the term "and / or" is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this article generally represents an "or" relationship between the associated objects before and after.

[0043] In the description of the embodiments of the present invention, the term "a plurality of" refers to more than two (including two). Similarly, "a plurality of groups" refers to more than two groups (including two groups), and "a plurality of pieces" refers to more than two pieces (including two pieces).

[0044] In the description of the embodiments of the present invention, technical terms such as "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc. indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the embodiments of the present invention and simplifying the description, rather than indicating or implying that the indicated device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the embodiments of the present invention.

[0045] In the description of the embodiments of the present invention, unless otherwise clearly specified and limited, technical terms such as "installation", "connection", "connection", "fixation", etc. should be understood in a broad sense. For example, it may be a fixed connection, a detachable connection, or integrated; it may also be a mechanical connection or an electrical connection; it may be directly connected or indirectly connected through an intermediate medium, and it may be the communication inside two elements or the interaction relationship between two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the embodiments of the present invention can be understood according to specific situations.

[0046] Embodiment 1

[0047] The embodiments of the present invention provide a semantic understanding and generation method in natural language processing based on the self-attention mechanism, as Figure 1 shown, including:

[0048] S101. Segment the input text to obtain word or sub-word units.

[0049] Among them, the input text is usually a complete sentence or text. For example: "Natural language processing is an interesting research field." To enable a computer to process this text, we need to split it into smaller units, namely word or sub-word units. If the input text is in English, it can be directly split by spaces. For example, "Hello world" is split into "Hello" and "world". If the input text is in Chinese, since there are no obvious word delimiters in Chinese, a word segmentation tool (such as jieba) is usually required for segmentation. For example, "Natural language processing" is split into "Natural", "language", and "processing".

[0050] S102. Use a pre-trained word embedding model to map the word or sub-word unit to a first vector of a preset dimension.

[0051] Among them, the pre-trained word embedding model can map the word or sub-word unit into a vector space of a fixed dimension while preserving semantic information. The preset dimension refers to the dimension of the word embedding. For example, the word embedding dimension of BERT is usually 768. Through the word embedding model, the semantic similarity between words can be captured. For example, "Natural" and "language" will be closer in the vector space, while words unrelated to "car" will be farther away.

[0052] S103. Add the first vector and the position encoding vector to obtain a second vector.

[0053] Among them, the position encoding vector is used to represent the position of the word in the sentence. It is usually generated by sine and cosine functions and can embed the position information into the vector. Add the first vector of each word to the corresponding position encoding vector to obtain the second vector. For example, the position of "Natural" in the sentence is the first word, its first vector is [0.1, -0.2, 0.3,..., 0.5], and the position encoding vector is [0.2, 0.1, -0.1,..., 0.3]. Then the second vector after addition is [0.3, -0.1, 0.2,..., 0.8].

[0054] S104. Construct a multi-layer self-attention mechanism to perform a linear transformation on the second vectors at each position to obtain a vector sequence.

[0055] Among them, the self-attention mechanism dynamically focuses on the more important parts of the input text, and the multi-layer structure further enhances the ability to capture complex semantics. The linear transformation means that in the self-attention mechanism, each second vector will be converted into three vectors: a query vector, a key vector, and a value vector. After being processed by the multi-layer self-attention mechanism, the second vectors at each position will be updated to form a new vector sequence. For example, assume that the input text has 3 words. After each word is processed by the multi-layer self-attention mechanism, a new vector sequence [v 1 ', v2 ',v 3 ']。After passing through the multi-layer self-attention mechanism, it can capture the long-distance dependencies between words, such as structures like "because... so...".

[0056] S105. Encode the vector sequence and extract the semantic vector.

[0057] Among them, an encoder (such as a Transformer encoder) is usually used to encode the vector sequence. The semantic vector refers to a vector with a fixed dimension extracted from the encoded vector sequence through a pooling operation or an attention mechanism, and this vector can highly summarize the semantics of the input text.

[0058] S106. Combine the multi-layer self-attention mechanism and the semantic vector to generate an output text related to the semantics of the input text.

[0059] Among them, a decoder (such as a Transformer decoder) is usually used to gradually generate the output text. The decoder also uses the self-attention mechanism internally to process the generated text to ensure that the generated text is coherent and conforms to the semantics. The decoder generates the output text related to the semantics of the input text by generating words one by one. For example: Suppose the input text is "The weather is very good today", and the semantic vector contains the core meaning of this sentence. The decoder will gradually generate an output text like "The weather today is very sunny and suitable for going out."

[0060] The embodiment of the present invention provides a semantic understanding and generation method in natural language processing based on the self-attention mechanism, including: segmenting the input text to obtain word or sub-word units; using a pre-trained word embedding model to map the word or sub-word units into first vectors with a preset dimension; adding the first vectors to the position encoding vectors to obtain second vectors; constructing a multi-layer self-attention mechanism to perform a linear transformation on the second vectors at each position to obtain a vector sequence; encoding the vector sequence and extracting the semantic vector; combining the multi-layer self-attention mechanism and the semantic vector to generate an output text related to the semantics of the input text. Compared with the prior art, the embodiment of the present invention can effectively capture the long-distance dependencies in the input text by constructing a multi-layer self-attention mechanism, thereby improving the accuracy of semantic understanding; combining the multi-layer self-attention mechanism and the semantic vector to generate an output text related to the semantics of the input text can generate a smooth text that conforms to the semantics of the input text; the multi-layer self-attention mechanism has high computational efficiency and can significantly reduce the computational cost.

[0061] Embodiment 2

[0062] The embodiment of the present invention provides a semantic understanding and generation method in natural language processing based on the self-attention mechanism, as Figure 2 shown, including:

[0063] S201. Segment the input text to obtain word or sub-word units.

[0064] Specifically, when the input text is in English, use spaces to segment it to obtain words; when the input text is in Chinese, use a word segmentation tool (such as jieba) to segment the Chinese text into single characters or phrases to obtain sub-word units.

[0065] S202. Use a pre-trained word embedding model to map the word or sub-word units to first vectors of a preset dimension.

[0066] Among them, the pre-trained word embedding model is Word2Vec or GloVe or BERT. The following will introduce each word embedding model:

[0067] Word2Vec is a technique for mapping words to a continuous vector space. It generates word embeddings by analyzing large-scale input text and captures the semantic relationships between words. The core idea of Word2Vec is that if two words frequently appear in many contexts, then their semantics and usages may be similar.

[0068] GloVe (Global Vectors for Word Representation) is an unsupervised learning algorithm for generating word vectors. It maps words to a continuous vector space by analyzing the word co-occurrence statistical information in a large-scale corpus and captures the semantic and syntactic relationships between words. The core idea of GloVe is to combine global statistical information (word co-occurrence frequency) and local context information (the position of the word in the context) to generate high-quality word embeddings. Different from Word2Vec, GloVe pays more attention to the utilization of global information.

[0069] BERT (Bidirectional Encoder Representations from Transformers) can generate context-related word embeddings based on the Transformer architecture. The emergence of BERT has greatly promoted the development of the natural language processing (NLP) field and has become a benchmark model for many NLP tasks. The architecture of BERT is based on the encoder part of the Transformer and is a typical bidirectional encoding model. It is composed of multiple stacked Transformer layers, and each Transformer layer contains a multi-head self-attention mechanism and a feed-forward neural network.

[0070] Among them, the dimension of the vector refers to the size of the vector. In this embodiment, the preset dimension is 768 dimensions.

[0071] S203. Add the first vector to the position encoding vector to obtain a second vector.

[0072] S204. Construct a multi-layer self-attention mechanism to perform a linear transformation on the second vector at each position to obtain a vector sequence.

[0073] S2041. Convert the second vector into a query vector, a key vector, and a value vector.

[0074] Among them, converting the second vector into a query vector, a key vector, and a value vector is the core part of the multi-layer self-attention mechanism. The second vector at each position is converted into a query vector, a key vector, and a value vector through a linear transformation. Among them, the query vector is used to find relevant information; the key vector is used to represent the position of the information; the value vector is used to store the actual information.

[0075] S2042. Calculate the self-attention scores based on the query vector and the key vector.

[0076] Specifically, calculate the dot product of the query vector and the key vector, and then divide it by the scaling factor to obtain the self-attention scores. Among them, the scaling factor is a numerical value used to describe the ratio at which the size of an object is enlarged or reduced in a scaling transformation. Dot product

[0077] S2043. Normalize the scaled self-attention scores to obtain self-attention weights.

[0078] Specifically, normalize the scaled self-attention scores through the Softmax function to obtain self-attention weights. Among them, the Softmax function is a commonly used activation function in machine learning and deep learning, which is used to convert a vector or a set of real numbers into a probability distribution. The Softmax function converts the scores into probability values, making the sum of the attention weights of each word to other words equal to 1.

[0079] S2044. Multiply the self-attention weights by the value vector to obtain a weighted output.

[0080] Previously, the self-attention weights have been calculated based on the query vector and the key vector. These weights represent the "attention degree" of each position (such as each word in a sentence) to other positions. Next, we need to use these weights to extract information from the value vector. Among them, the self-attention weights determine the importance of the information at each position in the final output, and the value vector stores the actual information at each position. By multiplying the self-attention weights by the value vector, it is possible to perform a weighted sum on the value vector according to the weights, thereby generating an output vector that synthesizes context information. This output vector can dynamically reflect the importance of different positions in the input sequence.

[0081] S2045. Concatenate and linearly transform the weighted output to obtain a vector sequence.

[0082] Concatenating and linearly transforming the weighted output to obtain a vector sequence can be divided into two main parts: concatenation and linear transformation. Among them, concatenation is to concatenate the outputs of all heads together to form a wider vector, and linear transformation is to transform the concatenated vector through a linear layer to integrate information and adjust the dimensions.

[0083] S205. Encode the vector sequence and extract the semantic vector.

[0084] Specifically, encode the vector sequence and extract the semantic vector through a pooling operation or an attention mechanism. Among them, the pooling operation is a commonly used dimensionality reduction technique in neural networks, mainly used to reduce the spatial dimension of data while retaining important feature information. It is usually applied to convolutional neural networks (CNNs) and certain natural language processing (NLP) tasks to extract local or global features of data. In natural language processing, the pooling operation is usually used to process sequence data (such as text or time series) with the aim of extracting key information from the sequence while reducing the computational complexity.

[0085] S206. Combine the multi-layer self-attention mechanism and the semantic vector to generate an output text related to the semantics of the input text.

[0086] The core of combining the multi-layer self-attention mechanism and the semantic vector to generate an output text related to the semantics of the input text lies in using the self-attention mechanism to capture the context information in the input text and guiding the generation process through the semantic vector, thereby generating an output text related to the semantics of the input text.

[0087] S207. Use the cross-entropy loss function to measure the difference between the output text and the target text, and correspondingly adjust the parameters of the word embedding model according to the difference through the gradient descent method.

[0088] Among them, the cross-entropy loss function is a loss function widely used in machine learning and deep learning, especially suitable for classification tasks. It can measure the difference between the probability distribution predicted by the model and the probability distribution of the true labels. In natural language processing (NLP), the cross-entropy loss function is often used in tasks such as text classification, language model training, and text generation.

[0089] The gradient descent method is an iterative algorithm for optimizing the objective function, widely used in machine learning and deep learning to minimize the loss function. It calculates the gradient of the loss function with respect to the model parameters and updates the parameters along the opposite direction of the gradient, thereby gradually reducing the loss.

[0090] It should be noted here that: for the detailed description of each step in this embodiment, reference can be made to other embodiments correspondingly, and details will not be elaborated here.

[0091] The embodiment of the present invention provides a semantic understanding and generation method in natural language processing based on the self-attention mechanism, including: segmenting the input text to obtain word or sub-word units; using a pre-trained word embedding model to map the word or sub-word units into first vectors of a preset dimension; adding the first vectors and position encoding vectors to obtain second vectors; constructing a multi-layer self-attention mechanism to perform a linear transformation on the second vectors at each position to obtain a vector sequence; encoding the vector sequence and extracting semantic vectors; combining the multi-layer self-attention mechanism and the semantic vectors to generate an output text semantically related to the input text. Compared with the prior art, the embodiment of the present invention can effectively capture long-distance dependency relationships in the input text by constructing a multi-layer self-attention mechanism, thereby improving the accuracy of semantic understanding; combining the multi-layer self-attention mechanism and the semantic vectors to generate an output text semantically related to the input text can generate a fluent text that conforms to the semantics of the input text; the multi-layer self-attention mechanism has high computational efficiency and can significantly reduce the computational cost.

[0092] Meanwhile, this embodiment uses the cross-entropy loss function to measure the difference between the output text and the target text, and correspondingly adjusts the parameters of the word embedding model by the gradient descent method, further improving the accuracy of semantic understanding.

[0093] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered by the scope of the claims and the description of the present invention. In particular, as long as there is no structural conflict, the technical features mentioned in each embodiment can be combined in any way. The present invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions that fall within the scope of the claims.

Claims

1. A semantic understanding and generation method in natural language processing based on self-attention mechanism, characterized by: include: Segment the input text to obtain word or subword units; Mapping the word or the subword unit into a first vector of a preset dimension using a pre-trained word embedding model; Adding the first vector to the position encoding vector to obtain a second vector; Construct a multi-layer self-attention mechanism, perform linear transformation on the second vector at each position, and obtain a vector sequence; Encoding the vector sequence and extracting a semantic vector; The multi-layer self-attention mechanism and the semantic vector are combined to generate output text that is semantically related to the input text.

2. The method for semantic understanding and generation in natural language processing based on self-attention mechanism according to claim 1, characterized in that: A multi-layer self-attention mechanism is constructed to perform a linear transformation on the second vector at each position to obtain a vector sequence, including: converting the second vector into a query vector, a key vector, and a value vector; Calculating a self-attention score based on the query vector and the key vector; Normalizing the scaled self-attention score to obtain a self-attention weight; Multiplying the self-attention weight by the value vector to obtain a weighted output; The weighted outputs are concatenated and linearly transformed to obtain the vector sequence.

3. The method for semantic understanding and generation in natural language processing based on self-attention mechanism according to claim 1, characterized in that: After combining the multi-layer self-attention mechanism and the semantic vector to generate an output text semantically related to the input text, the method further includes: The difference between the output text and the target text is measured, and the parameters of the word embedding model are adjusted according to the difference.

4. The method for semantic understanding and generation in natural language processing based on self-attention mechanism according to claim 3, characterized in that: Measuring the difference between the output text and the target text, and adjusting the parameters of the word embedding model according to the difference includes: A cross entropy loss function is used to measure the difference between the output text and the target text, and the parameters of the word embedding model are adjusted accordingly through a gradient descent method according to the difference.

5. The method for semantic understanding and generation in natural language processing based on self-attention mechanism according to claim 1, characterized in that: Segment the input text to obtain word or subword units, including: When the input text is an English text, spaces are used for segmentation to obtain words; when the input text is a Chinese text, a word segmentation tool is used to segment the Chinese text into single words or phrases to obtain sub-word units.

6. The method for semantic understanding and generation in natural language processing based on self-attention mechanism according to claim 1, characterized in that: The pre-trained word embedding model includes: The pre-trained word embedding model is Word2Vec or GloVe or BERT.

7. The method for semantic understanding and generation in natural language processing based on self-attention mechanism according to claim 1, characterized in that: The preset dimensions include: The preset dimension is 768 dimensions.

8. The method for semantic understanding and generation in natural language processing based on self-attention mechanism according to claim 1, characterized in that: Encoding the vector sequence and extracting the semantic vector includes: The vector sequence is encoded and a semantic vector is extracted through a pooling operation or an attention mechanism.

9. The method for semantic understanding and generation in natural language processing based on self-attention mechanism according to claim 2, characterized in that: Calculating a self-attention score according to the query vector and the key vector includes: The dot product of the query vector and the key vector is calculated and then divided by a scaling factor to obtain the self-attention score.

10. The method for semantic understanding and generation in natural language processing based on self-attention mechanism according to claim 2, characterized in that: The scaled self-attention score is normalized to obtain the self-attention weight, including: The scaled self-attention score is normalized by the Softmax function to obtain the self-attention weight.

Images