A text event extraction method for overlapping and nested features

Abstract

This invention discloses a method for extracting text events with overlapping and nested features, comprising: performing attention interaction on the original text word embeddings and event type word embeddings through a symmetric bidirectional attention mechanism to obtain corresponding semantically enhanced word embeddings; extracting important text from the original text word embeddings and semantically enhanced original text word embeddings by weight extraction, and integrating the important text to obtain optimized text word embeddings; concatenating the original text word embeddings, optimized text word embeddings, and corresponding semantically enhanced event type word embeddings to obtain concatenated text word embeddings, and obtaining triggers based on the concatenated text word embeddings; obtaining the start and end probabilities of each word in the concatenated text word embeddings in the question-answering task, and obtaining the event elements of the current question role through a position decoding algorithm to complete the event extraction process. This invention can better understand and represent event type information in complex structures of overlapping and nested events.

Description

Technical Field

[0001] This invention relates to the field of text event extraction technology, and more particularly to a method for extracting text events with overlapping and nested features. Background Technology

[0002] Event extraction is a crucial step in information extraction tasks. Given a sentence as input, the task of event extraction is to identify and extract its relevant parameters: triggers, event types, and parameters. However, the parameters in real-world events are often complex, frequently exhibiting overlap and nesting, which is an unavoidable problem for event extraction. Overlap and nesting refer to the fact that a single sentence may contain multiple intertwined or nested events, where some words (tokens) may simultaneously belong to multiple event types and have multiple labels. Handling these issues requires the model to identify and distinguish these complex relationships, ensuring accurate extraction of key information for each event. These challenges are particularly pronounced when dealing with tokens with multiple meanings, demanding more sophisticated mechanisms to understand and parse the context.

[0003] Most existing event extraction methods target flat events, assuming no overlap or nesting between events, thus ignoring unconventional event extraction scenarios. These methods typically use a unified decoder to extract triggers and parameters through traditional joint methods, usually only labeling once. However, the nature of overlapping and nested events leads to label conflicts, and only one label is assigned during the prediction phase. Other methods use cascaded decoding to handle overlapping events; while effective in handling overlap, this strategy still falls short for nested events. Furthermore, traditional pipeline methods, while addressing overlap, fail to effectively utilize information between subtasks and do not fully leverage sentence and label information. Recent research has improved event extraction through reading comprehension and by formulating event extraction as question-and-answer questions, but these methods are relatively fixed in design and lack contextual understanding. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a method for extracting text events with overlapping and nested features, thereby solving the technical problems of inaccurate extraction of overlapping and nested events, inability to adapt to tokens with multiple semantics, and inability to understand the positional relationships of tokens in the context.

[0005] This invention provides a method for extracting text events with overlapping and nested features, comprising the following steps:

[0006] Step 1: Obtain the word embeddings of the original text by passing it through a pre-trained encoder, with preset event type word embeddings;

[0007] Step 2: Perform attention interaction between the original text word embedding and the event type word embedding through a symmetric bidirectional attention mechanism on the original input vector to obtain the semantically enhanced original text word embedding and the semantically enhanced event type word embedding;

[0008] Step 3: Set extraction weights, extract important text from the original text word embeddings and the semantically enhanced original text word embeddings through extraction weights, and integrate the extracted important text to obtain the optimized text word embeddings;

[0009] Step 4: Concatenate the original text word embedding, the optimized text word embedding, and the corresponding semantically enhanced event type word embedding to obtain the concatenated text word embedding. Obtain the trigger corresponding to the event type based on the concatenated text word embedding.

[0010] Step 5: Construct a corresponding question-and-answer task based on the preset question roles, obtain the start and end probabilities of each word embedded in the concatenated text in the question-and-answer task, and obtain the event elements of the current question role through a position decoding algorithm. Specifically, the position decoding algorithm is as follows:

[0011] When the start and end probabilities of the current word are both greater than the start and end probabilities of all separators, and the current word is not in the question statement of the question-answering task, calculate the sum of the start and end probabilities of the current word, and take the word with the largest sum of probabilities among all words as the event element of the current question role.

[0012] Step 6: After obtaining the event elements for all problem roles, construct the event extraction results to complete the event extraction process.

[0013] Furthermore, the symmetric bidirectional attention mechanism in step 2 is expressed as follows:

[0014]

[0015] In the formula, d h Q is the scaling factor; Q, K, and V are the query, key, and value tensors, respectively.

[0016] Furthermore, in step 3, the specific formula for extracting the important text embedded by each word by extracting weights is as follows:

[0017]

[0018] In the formula, g is the extraction weight; m and n represent the two input quantities respectively.

[0019] Furthermore, the formula for setting the extraction weights is as follows:

[0020] g=σ(W g [m;n]+b g )

[0021] In the formula, σ(·) represents the sigmoid activation function; Indicates element-wise multiplication; [;] indicates cascading operation; W g and b g These are all training parameters within the model.

[0022] Furthermore, the specific process of step 3 is as follows:

[0023] Step 31: Obtain extraction weights from the original text word embeddings and the semantically enhanced original text word embeddings through a gating equation. Extract important texts from the original text word embeddings and the semantically enhanced original text word embeddings according to the extraction weights. Integrate the extracted important texts to obtain the transitional text word embeddings.

[0024] Step 32: Obtain extraction weights from the transitional text word embeddings and the semantically enhanced event type word embeddings through a gating equation. Based on the extraction weights, extract the important text from the transitional text word embeddings and the corresponding semantically enhanced event type word embeddings. Integrate the extracted important texts to obtain the optimized text word embeddings.

[0025] Furthermore, in step 4, the trigger is determined and obtained by embedding the concatenated text words into binary tags.

[0026] Furthermore, in step 5, the specific method for obtaining the start and end probabilities of each word in the concatenated text word embedding in the question-answering task is as follows:

[0027] First, the RoBERTa model is used to convert each input word into a corresponding embedding vector. Then, the softmax function is used to normalize the obtained embedding vectors to obtain the start and end probabilities of each word.

[0028] Furthermore, the embedding vector includes: word embedding vector, position embedding vector, and separator embedding vector.

[0029] The beneficial effects of this invention are:

[0030] This invention uses a symmetric bidirectional attention mechanism to enable two types of word embeddings to interact with each other, thereby enhancing the semantic interaction between event types and text. This model can effectively capture the multiple label meanings of complex tokens, especially when dealing with overlapping and nested events, and can better understand and represent event type information in these complex structures.

[0031] This invention utilizes extraction weights to effectively filter out unimportant information in text, reducing its visibility and highlighting key information. It performs exceptionally well when dealing with overlapping and nested events with multiple tags, improving the accuracy of trigger recognition. The important text extraction method provided by this invention can address situations where events overlap or one event completely encompasses another, improving the recognition and extraction capabilities of such events while maintaining the extraction accuracy of individual events.

[0032] This invention employs a positional decoding algorithm to obtain the relationship between the start and end positions of words, and further refines the consideration of the start and end positional relationships between each token. This not only enhances the model's ability to determine the start and end positions of arguments, but also better handles the uniqueness of arguments, making argument extraction more accurate in complex overlapping and nested event relationships. Attached Figure Description

[0033] The features and advantages of the invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the invention in any way. In the drawings:

[0034] Figure 1 This is a flowchart of a specific embodiment of the present invention;

[0035] Figure 2 This is a flowchart illustrating the specific process of obtaining event elements according to a particular embodiment of the present invention;

[0036] Figure 3 This is the code representation of the position decoding algorithm in a specific embodiment of the present invention. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] The present invention will be further illustrated below with reference to specific embodiments. Those skilled in the art should understand that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Modifications to the present invention in various equivalent forms all fall within the scope defined by the appended claims.

[0039] like Figure 1 As shown in the figure, a specific embodiment of the present invention provides a method for extracting text events with overlapping and nested features.

[0040] The following example, using the text about overlapping financial events: "In the context of a global economic slowdown, central banks around the world are implementing interest rate cuts. Meanwhile, banks in some countries are increasing their support for small and medium-sized enterprises by lowering financing costs in an attempt to stimulate economic recovery," illustrates the steps involved.

[0041] Step 1: Obtain the word embeddings of the original text by passing it through a pre-trained BERT encoder, with preset event type word embeddings;

[0042] BERT is pre-trained using a bidirectional Transformer encoder, enabling it to learn rich semantic representations from large amounts of text. The `transformers` library in Python is used to load and use the pre-trained BERT model and tokenizer. The tokenizer first decomposes the sentence into word fragments (tokens). The decomposition result in this example is as follows:

[0043] ["In","the","context","of","global","economic","slowdown",",","central","banks","around","the","world","are","implementing","in terest","rate","cuts",".","Meanswhile",",","banks","in","some","countries","are","increasing","their","support","for","small","a nd","medium-sized","enterprises","by","lowering","financing","costs","in","an","attempt","to","stimulate","economic","recovery","."],

[0044] Each word segment is mapped to a word vector, which is then combined with positional and segmentation encodings to form the model's input. BERT, through its bidirectional Transformer layer, analyzes and processes each word in the sentence, capturing its relationships with other words. For example, BERT can understand the causal relationship between "global economic growth slowdown" and "major central banks have adopted interest rate cuts," as well as the objective relationship between "banks in some countries have increased lending support for SMEs" and "stimulating economic recovery by reducing financing costs." Simultaneously, it also performs initial randomized word embeddings on event-type words.

[0045] After step 1, BERT will generate a high-dimensional vector representation (768-dimensional vector), resulting in the preprocessed text word embeddings:

[0046] "global"->[0.22,-0.55,...,0.19]

[0047] "economic"->[0.48,-0.12,...,0.67]

[0048] "slowdown"->[-0.34,0.77,...,-0.23]

[0049] "background"->[0.11,-0.24,...,0.88]

[0050] "central bank"->[0.65,0.12,...,-0.45]

[0051] Preprocessing results of preset event type word embedding (768-dimensional vector):

[0052] "Investment"->[-0.52,0.17,...,0.44]

[0053] "Policy"->[0.69,-0.24,...,0.66]

[0054] Step 2: Perform attention interaction between the original text word embedding and the event type word embedding through a symmetric bidirectional attention mechanism on the original input vector to obtain the semantically enhanced original text word embedding and the semantically enhanced event type word embedding;

[0055] To better understand complex events and differentiate the different effects of various event types on the same token, attention weights are calculated between the input query (Q), key (K), and value (V) of the original text word embeddings and the semantically enhanced event type word embeddings. A weighted sum is then output, specifically: the dot product QK^T is calculated using torch.matmul, normalized using softmax, and finally multiplied by the value V to generate the final attention representation. This process helps the model focus on the most relevant event types and textual information when dealing with complex semantics. The symmetric bidirectional attention can be represented as:

[0056]

[0057] In the formula, d h Q is the scaling factor; Q, K, and V are the query, key, and value tensors, respectively.

[0058] The representation of event types obtained through a pre-trained BERT model is: E = {e 1 ,e 2 ,e 3 ,…,e M}, where M represents the number of event types, and the symmetric bidirectional attention between event types and text is represented as:

[0059] E g =Attention(W qe H,W ke E,W ve E),

[0060] H g =Attention(W qh E,W kh H,W vh H),

[0061] Among them, E g and H g The output of the attention layer is equivalent to the semantically enhanced word embedding; W qe W ke W ve W qh W kh W vh These are all built-in training parameters within the model.

[0062] Step 3: Set extraction weights. Extract the important text from the original text word embeddings and the semantically enhanced original text word embeddings using the extraction weights. Integrate the extracted important text to obtain the optimized text word embeddings. The specific process is as follows:

[0063] Step 31: Obtain extraction weights from the original text word embeddings and the semantically enhanced original text word embeddings through a gating equation. Extract important texts from the original text word embeddings and the semantically enhanced original text word embeddings according to the extraction weights. Integrate the extracted important texts to obtain the transitional text word embeddings.

[0064] Step 32: Obtain extraction weights from the transitional text word embeddings and the corresponding semantically enhanced event type word embeddings through a gating equation. Based on the extraction weights, extract important text from the transitional text word embeddings and the corresponding semantically enhanced event type word embeddings respectively. Integrate the extracted important texts to obtain the optimized text word embeddings.

[0065] The extraction weight can be expressed as:

[0066] g=σ(W g [m;n]+b g )

[0067] Where g represents the extraction weights, m and n represent the input vectors; σ(·) represents the sigmoid activation function; Element-wise multiplication is represented by `np.multiply()`; `[;]` represents a concatenation operation, implemented using `np.concatenate()`; W g and b g These are all training parameters.

[0068] The extracted weights are generated through a fully connected layer with a sigmoid function, adaptively controlling the input information.

[0069] The extracted weights are fused from two different sources (original text word embeddings and semantically enhanced original text word embeddings, transitional text word embeddings and corresponding semantically enhanced event type word embeddings) using the following formula, while controlling the flow of information based on the extracted weights g:

[0070]

[0071] Where g represents the extraction weight, and m and n represent the input vector; This represents element-wise multiplication, achieved using np.multiply().

[0072] The model's ability to handle multi-label tokens is improved by adaptively deciding which information to retain and which to ignore. Vector g is generated through a fully connected layer with a sigmoid function, adaptively controlling the input information. The formulas for the filtering nodes are shown in equations (3) and (4), generating the final output from two input vectors, m and n.

[0073] In an example of a specific embodiment of the present invention, event type information (such as "interest rate cut policy") and text information (the context of the sentence) are extracted and weighted to generate a more expressive representation. The original text is represented as H. g The text representation after symmetrical bidirectional attention is E. g The target event type corresponding to this text is represented as e. t The extraction process can then be represented as:

[0074] S g =Node(H g E g ),

[0075] V t =Node(S) g ,e t ),

[0076] Among them, e t ∈E represents the target event type embedding, V t ={v1,v2,v3,…,v L The word represents the final perceived event type.

[0077] Step 4: Concatenate the original text word embedding, the optimized text word embedding, and the semantically enhanced event type word embedding to obtain the concatenated text word embedding. The trigger is determined and obtained by classifying the concatenated text word embedding into binary labels.

[0078] For the concatenated representation, the information of the original text word embedding, the optimized text word embedding, and the semantically enhanced event type word embedding are concatenated using `combined_context = torch.cat([sentence_context, event_context.replace(sentence_context.size(1),1).unsqueeze(0)], dim = -1)`. The probability value of each word is obtained through a classifier, which represents the likelihood of triggering a given token. Finally, binary labels are generated based on the model's output probability using `trigger_labels = (trigger_probabilistic ties > 0.5).int()`. Words with a probability value greater than 0.5 are marked as triggers (label 1), otherwise they are marked as non-triggers (label 0).

[0079] For the provided case, one of the triggers is clearly interest rate cuts, and the corresponding role is Impact on investment. Therefore, the question template can be used.<Interrogative words> is the <role>in <trigger>We can design a specific problem, that is, express the problem as: What is the impact of interest rate cuts on investment? This problem can be interpreted as: interest rate cuts are the triggering factor, and we are asking about its impact on investment.

[0080] To distinguish it from the text, this paper adds [CLS] and [SEP] as separators. The standard input of the model will then be expressed as: [CLS] What is the impact of interest rate cuts on investment? [SEP] In the context of global economic slowdown, central banks around the world are implementing interest rate cuts. Meanwhile, banks in some countries are increasing their support for small and medium-sized enterprises by lowering financing costs in an attempt to stimulate economic recovery. [SEP]

[0081] like Figure 2 As shown, step 5: Construct the corresponding question-answering task based on the preset question roles. First, use the RoBERTa model to convert each input word into a corresponding embedding vector. Then, use the softmax function to normalize the obtained embedding vectors to obtain the start and end probabilities of each word.

[0082] The RoBERTa model transforms each input token into a corresponding embedding vector, including three types of embeddings: word embeddings (simply mapping each token to a vector representation), positional embeddings (introducing positional information for each token, allowing the model to know its location within the sequence), and separator embeddings (RoBERTa distinguishes different parts using [SEP] boundaries). The embedded text is then fed into RoBERTa's Transformer encoder. RoBERTa employs a multi-layered self-attention mechanism, updating the relevance representation between each token and other tokens in the input sequence at each layer. Finally, a softmax function is used to obtain the start and end probabilities of each token.

[0083] Load the pre-trained RoBERTa model and tokenizer using the Transformers library. Pass the normalized input to the RoBERTa model to obtain the tokenized embeddings, specifically represented as: c1, c2, c3, ..., c m =RoBERTa(a1,a2,a3,…,a m Two new parameter matrices W are introduced. start and W end , for c i W start and c i W end Apply softmax normalization to obtain probabilities, calculate the start and end probabilities for each tag, i.e., through softmax(c i W start ) and softmax(c ui W end The operation obtains the start and end probabilities.

[0084] The event elements of the current problem character are obtained through a position decoding algorithm, where the position decoding algorithm is specifically as follows:

[0085] When the start and end probabilities of the current word are both greater than the start and end probabilities of all separators, and the current word is not in the question statement of the question-answering task, calculate the sum of the start and end probabilities of the current word, and take the word with the largest sum of probabilities among all words as the event element of the current question role.

[0086] like Figure 3 The code shown, for position decoding, takes a start position probability vector and an end position probability vector as input, and outputs a candidate group of trigger words, with [i,j,score] representing the start, end, and score respectively (the start and end positions are closed intervals). The first value I of the start position probability vector... start (1) (This position corresponds to the [SEP] special separator in the original token) is the lowest value of the starting score, and the first value I of the ending position probability vector is used. end (1) The lowest score is used as the termination score to filter and obtain the argument candidate group. The member with the highest score in the obtained candidate group is the answer for argument extraction.

[0087] For this case, the results for candidates are as follows:

[0088] [[2,10,0.85],

[0089] [3,12,0.87],

[0090] [5,15,0.90],

[0091] [7,20,0.88],

[0092] [10,22,0.84]].

[0093] Step 6: After obtaining the event elements for all problem roles, construct the event extraction results to complete the event extraction process.

[0094] Finally, taking the case study as an example, the result of event extraction is as follows:

[0095] Event 1

[0096] Event Type: Investment

[0097] Event Trigger: "increasing"

[0098] Event Arguments:

[0099] Banks - Role: Agent

[0100] Small and Medium-Sized Enterprises (SMEs) - Role: Beneficiary

[0101] Loan Support - Role: Resource

[0102] Elemental Role (Argument Role)

[0103] Implementer (Agent): "Bank"

[0104] Beneficiary: Small and Medium Enterprises

[0105] Funds (Resource): "Loan Support"

[0106] Event 2

[0107] Event Type: Policy

[0108] Event Trigger: "interest rate cut"

[0109] Event Arguments:

[0110] Central Banks - Role: Agent

[0111] Policy - Role: Policy Measure

[0112] Elemental Role (Argument Role)

[0113] Implementer (Agent): "Central Bank"

[0114] Policy Measure: "interest rate cut"

[0115] The following is a comparison of the performance of the present invention and the prior art under four indicators:

[0116]

[0117] Table 1.

[0118] As shown in Table 1, our invention achieved F1 scores of 89.0, 80.0, 76.4, and 73.5 in the four F1 metrics: TI (%), TC (%), AI (%), and AC (%). In terms of overall performance, our invention achieved F1 scores of 89.0% and 80.0% in the TI and TC tasks, respectively, comparable to OneEE and CasEE, and superior to most BERT series models, particularly excelling in trigger word recognition and classification tasks. In terms of fine-grained performance, in the AI ​​task, our invention achieved an F1 score of 76.4%, slightly higher than OneEE (76.2%) and significantly higher than PLMEE (70.6%), indicating strong argument recognition performance. In the AC task, our invention achieved an F1 score of 73.5%, superior to CasEE (71.4%), demonstrating excellent performance in argument classification. Compared to classic models: Compared to the BERT series, our invention exhibits better recall and F1 scores across multiple tasks. Although PLMEE has a slight advantage in TI tasks, it performs worse than this invention in TC and AI tasks.

[0119] The following is a comparison of the performance of this invention and existing technologies under the Argument ID+Classification metric:

[0120]

[0121] Table 2

[0122] As shown in Table 2, compared with Random NE, the present invention (Ours) achieves an F1 score of 66.13%, which is 40.47% higher than Random NE. Furthermore, its precision and recall are significantly improved, indicating that the present invention can effectively identify and classify arguments in events, rather than relying on random operations. Compared with BERT_QA_Arg w / Role name and BERT_QA_Arg w / Type+Role, the present invention shows a slight improvement. This is because the present invention better utilizes input information, thereby improving the overall performance of argument recognition and classification. This demonstrates that the present invention is more accurate in identifying event arguments, especially robust in tasks requiring precise differentiation of roles and types.

[0123] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.< / trigger> < / role>

Claims

1. A method for extracting text events with overlapping and nested features, characterized in that, Includes the following steps: Step 1: Obtain the word embeddings of the original text by passing it through a pre-trained encoder, with preset event type word embeddings; Step 2: Perform attention interaction between the original text word embedding and the event type word embedding through a symmetric bidirectional attention mechanism on the original input vector to obtain the semantically enhanced original text word embedding and the semantically enhanced event type word embedding; Step 3: Set extraction weights, extract important text from the original text word embeddings and the semantically enhanced original text word embeddings through extraction weights, and integrate the extracted important text to obtain the optimized text word embeddings; Step 4: Concatenate the original text word embedding, the optimized text word embedding, and the corresponding semantically enhanced event type word embedding to obtain the concatenated text word embedding. Obtain the trigger corresponding to the event type based on the concatenated text word embedding. Step 5: Construct a corresponding question-and-answer task based on the preset question roles, obtain the start and end probabilities of each word embedded in the concatenated text in the question-and-answer task, and obtain the event elements of the current question role through a position decoding algorithm. Specifically, the position decoding algorithm is as follows: When the start and end probabilities of the current word are both greater than the start and end probabilities of all separators, and the current word is not in the question statement of the question-answering task, calculate the sum of the start and end probabilities of the current word, and take the word with the largest sum of probabilities among all words as the event element of the current question role. Step 6: After obtaining the event elements for all problem roles, construct the event extraction results to complete the event extraction process.

2. The method for extracting text events with overlapping and nested features as described in claim 1, characterized in that, The symmetric bidirectional attention mechanism in step 2 is represented as follows: In the formula, d h Q is the scaling factor; Q, K, and V are the query, key, and value tensors, respectively.

3. The method for extracting text events with overlapping and nested features as described in claim 1, characterized in that, In step 3, the specific formula for extracting the important text embedded in each word by extracting weights is as follows: In the formula, g is the extraction weight; m and n represent the two input quantities respectively.

4. The method for extracting text events with overlapping and nested features as described in claim 3, characterized in that, The formula for setting the extraction weight is: g=σ(W g [m;n]+b g ) In the formula, σ(·) represents the sigmoid activation function; Indicates element-wise multiplication; [;] indicates cascading operation; W g and b g These are all training parameters within the model.

5. The method for extracting text events with overlapping and nested features as described in claim 1, characterized in that, The specific process of step 3 is as follows: Step 31: Obtain extraction weights from the original text word embeddings and the semantically enhanced original text word embeddings through a gating equation. Extract important texts from the original text word embeddings and the semantically enhanced original text word embeddings according to the extraction weights. Integrate the extracted important texts to obtain the transitional text word embeddings. Step 32: Obtain extraction weights from the transitional text word embeddings and the semantically enhanced event type word embeddings through a gating equation. Based on the extraction weights, extract the important text from the transitional text word embeddings and the corresponding semantically enhanced event type word embeddings. Integrate the extracted important texts to obtain the optimized text word embeddings.

6. The method for extracting text events with overlapping and nested features as described in claim 1, characterized in that, In step 4, the trigger is determined and obtained by embedding the concatenated text words into binary tags.

7. The method for extracting text events with overlapping and nested features as described in claim 1, characterized in that, In step 5, the specific method for obtaining the start and end probabilities of each word in the concatenated text word embedding in the question-answering task is as follows: First, the RoBERTa model is used to convert each input word into a corresponding embedding vector. Then, the softmax function is used to normalize the obtained embedding vectors to obtain the start and end probabilities of each word.

8. The method for extracting text events with overlapping and nested features as described in claim 7, characterized in that, The embedding vectors include: word embedding vectors, position embedding vectors, and separator embedding vectors.

Images