A cross-language summarization method based on cross-attention alignment and multi-language mutual attention

Abstract

The application belongs to the technical field of natural language processing, and particularly relates to a cross-language abstracting method based on cross-attention alignment and multi-language mutual attention. In order to improve the performance of cross-language abstracting, an end-to-end neural network model based on multi-task learning is used to perform cross-language abstracting on a text, including the following steps: original texts and abstracts of monolingual abstracting data sets are crawled from data set websites on the Internet, and the data is processed through a round-trip translation strategy and a discourse implication strategy to construct a cross-language abstracting data set; the source texts and abstracts in the data set are segmented, word embedding vector representations are obtained, and encoding and preliminary decoding are performed; a cross-attention alignment mechanism is used to optimize the attention distribution of cross-language abstracting; a multi-language mutual attention mechanism is used to optimize bilingual hidden state representations; and a cross-language abstracting model is trained to generate cross-language abstracts.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing technology, specifically relating to a cross-lingual summarization method based on cross-attention alignment and multilingual mutual attention. Background Technology

[0002] In recent years, with the development of economic globalization, computer-assisted rapid reading of foreign language documents has become a key challenge in the field of artificial intelligence. Due to the complexity of foreign language content and language barriers, ordinary readers often face difficulties. Therefore, research on cross-linguistic summarization methods is particularly important.

[0003] Compared to monolingual summarization tasks, cross-lingual summarization tasks often underperform due to data scarcity and cross-lingual characteristics. To address this challenge, recent research has focused on cross-lingual summarization tasks, with multi-task learning emerging as a popular solution. This approach integrates auxiliary tasks such as monolingual summarization to improve performance. However, while these methods have improved performance to some extent, they neglect the inherent connection between monolingual and cross-lingual summarization in the summarization process, failing to effectively utilize the cross-attention information and hidden information generated during monolingual summarization. Summary of the Invention

[0004] This invention addresses the aforementioned problems by providing a cross-lingual summarization method based on cross-attention alignment and multilingual mutual attention.

[0005] To achieve the above objectives, the present invention employs the following technical solution:

[0006] A cross-lingual summarization method based on cross-attention alignment and multilingual mutual attention includes the following steps:

[0007] Step 1: Obtain the original text and summary of the monolingual summary dataset, process the data, and construct a cross-lingual summary dataset.

[0008] Step 2: Input the data from the cross-language summarization dataset into the pre-trained multi-task cross-language summarization model and generate cross-language summaries. The multi-task cross-language summarization model includes: 1) segmenting the source language documents, source language summaries, and target language summaries in the cross-language summarization dataset into words, obtaining word embedding vector representations, and performing encoding and preliminary decoding; 2) optimizing the attention distribution of cross-language summaries using a cross-attention alignment mechanism; and 3) optimizing the bilingual hidden state representation using a multilingual mutual attention mechanism.

[0009] Furthermore, step 1 specifically includes:

[0010] Step 101: Use web scraping technology to scrape text data of monolingual articles and their summaries from dataset websites on the Internet.

[0011] Step 102: Process the data using round-trip translation and discourse implication strategies to construct a cross-linguistic summary dataset.

[0012] Furthermore, step 102 specifically includes:

[0013] First, the text and corresponding summaries in the monolingual summarization dataset crawled in step S101 are processed through round-trip translation. A high-performance machine translation API is used to translate the summaries in the source language text-summary pairs into the target language, and then the results are translated back into the source language. ROUGE is calculated for the text-summary pairs consisting of the source language text and the translated summaries. If the ROUGE value is greater than a set threshold, the corresponding source language text-target language summary pair is retained; otherwise, the data is filtered. Subsequently, the data filtered through the round-trip translation strategy is further filtered using a discourse entailment strategy. An entailment score is calculated between the translated summary and the original summary using an entailment model. If the entailment score exceeds a predefined threshold, the corresponding source language text-target language summary pair is retained; otherwise, the data is also filtered. Through these steps, a high-quality cross-lingual summarization dataset is finally constructed.

[0014] Furthermore, in step 2, the source language documents, source language summaries, and target language summaries in the cross-language summarization dataset are segmented into words to obtain word embedding vector representations, which are then encoded and preliminarily decoded. Specifically:

[0015] 1.1) Word-level segmentation is performed on sentences in the source language documents, source language summaries, and target language summaries in the cross-language summarization dataset; for English text, spaces are used directly for word segmentation; for Chinese text, the jieba library is used for word segmentation.

[0016] 1.2) Preprocess the segmented data: Load the BERT tokenizer and set some special tokens and their corresponding vocab IDs; convert the sentences in each sentence list into space-separated strings, and filter out sentences whose length is greater than the specified minimum length; truncate these excessively long sentences so that their length does not exceed the specified maximum length, and ensure that the final number of sentences does not exceed the set maximum number of sentences; use the BERT tokenizer to convert the filtered sentences into subwords, add "[CLS]" at the beginning of each sentence and "[SEP]" at the end, and convert the segmentation results into their corresponding vocab IDs; add "[PAD]" to the end of the sequence to pad shorter sequences to the same length as the longest sequence.

[0017] 1.3) Based on the results of 1.2), source and target sequences are constructed and transformed into embedding vector representations for encoding and preliminary decoding. The source language documents processed in 1.2) are directly input into the encoder as source sequences. For the decoder under the teacher-enforced policy, the source language summary and target language summary processed in 1.2) are used as target sequences for monolingual and cross-lingual summaries, respectively. When inputting into the decoder, the target sequences need to be shifted one position to the right at each time step, so that the decoder can receive the next label of the standard summary as input at each time step. At the same time, the unshifted target sequences are used to calculate the loss for the decoder's prediction. The source and target sequences are converted into tensors in the PyTorch deep learning framework using the torch.tensor() function. These tensors are then used as input to the neural network model. The word embedding layer that receives the input tensors converts them into corresponding word embedding vectors. The word embedding vectors of the source sequences are fed into the encoder for encoding, generating a context vector H={h1,h2,…,h... n The word embedding vectors of the offset target sequence are fed into the self-attention layer of the cross-lingual and monolingual decoders for initial decoding, thereby generating the hidden state h of the target language decoder. t The hidden state h of the source language decoder s .

[0018] Furthermore, in step 2, a cross-attention alignment mechanism is used to optimize the attention distribution of cross-language summarization, specifically as follows:

[0019] 2.1) Using a cross-attention mechanism, calculate the attention distribution of the source text for both cross-lingual and monolingual summaries: [This involves] plotting the hidden states h of the cross-lingual decoder... t The hidden state h of the source language decoder s Using these as query vectors, the relevance between the query vector and the source dialogue context vector H is calculated, thus obtaining the cross-language decoder's cross-attention weight matrix for the original text. Cross-attention weight matrix of the original text and monolingual decoder As shown in equations (1) and (2),

[0020] (1)

[0021] (2)

[0022] in, and These represent the query and key in the cross-language attention process, respectively. It is the dimension of query and key in the cross-language attention process. and These represent the query and key in the monolingual attention process, respectively. It is the dimension of query and key in the monolingual attention process.

[0023] use and These two attention distributions are weighted and summed over the source dialogue context vector H to obtain the context attention result of the cross-language decoder on the original text. The representation of contextual attention results of the original text by the monolingual decoder. As shown in equations (3) and (4):

[0024] (3)

[0025] (4)

[0026] in, and These represent the value matrices in the cross-lingual attention and monolingual attention processes, respectively. The two contextual results will participate in the generation of cross-lingual summaries and monolingual summaries, respectively.

[0027] 2.2) Optimize the attention distribution of cross-lingual summarization through a cross-attention alignment mechanism: Use cross-attention between the encoder and decoder of the same language to guide the cross-attention of cross-lingual summarization, as shown in Equation (5):

[0028] (5)

[0029] in, It is the attention distribution of the source language summarization decoder on the hidden states of the source language text, i.e., the attention distribution of monolingual summarization. It is the attention distribution of the target language summarization decoder on the hidden state of the source language text, i.e., the attention distribution of cross-language summarization.

[0030] Furthermore, in step 2, a multilingual mutual attention mechanism is used to optimize the bilingual hidden state representation, specifically as follows:

[0031] 3.1) Optimize the bilingual hidden state representation by utilizing a multilingual mutual attention mechanism: At the p-th time step of the cross-lingual summarization decoder, the hidden state of the target language decoder is... As the query vector Q, the last p hidden states of the source language decoder are computed. , ,..., The correlation between the p hidden states is weighted and summed using a multilingual mutual attention module to obtain the contextual attention result representation of the target language decoder hidden state on the source language decoder hidden state. As shown in equation (6):

[0032] (6)

[0033] in, This refers to the hidden state of the target language summarization decoder at time step p. It refers to all the hidden states of the source language digest decoder in the first p time steps.

[0034] Similarly, at the p-th time step of the source language summarization decoder, the hidden state of the source language decoder is... As the query vector Q, the last p hidden states of the target language decoder are computed. , ,..., The correlation between the p hidden states is weighted and summed using a multilingual mutual attention module to obtain the contextual attention result representation of the source language decoder hidden state on the target language decoder hidden state. As shown in equation (7):

[0035] (7)

[0036] in, This refers to the hidden state of the source language summarization decoder at time step p. It refers to all hidden states of the target language summarization decoder in the first p time steps.

[0037] 3.2) Representing the generated contextual attention results The contextual attention result of the generated target language decoder hidden state to the source language decoder hidden state. Generate a summary: At time p steps, the probability of the next word in the cross-linguistic summary is calculated as shown in equation (8):

[0038] (8)

[0039] Wherein, function F includes the merged and spliced ​​components. and The system consists of an MLP layer for information, a feedforward layer for obtaining intermediate vector representations, a linear layer for learning complex patterns and features of intermediate vectors, and a softmax layer for predicting word probability distributions.

[0040] Furthermore, it also includes training a multi-task cross-language summarization model, specifically:

[0041] After the monolingual decoder and cross-lingual decoder predict their corresponding summaries in multi-task learning, the negative log-likelihood loss is used to optimize both tasks simultaneously. The loss for a single example is:

[0042] (9)

[0043] in, Represents the source document. Represents a summary of the source language. α represents the target language summary; α is a hyperparameter used to balance the weights of monolingual summarization tasks and cross-lingual summarization tasks. This represents the source language summary at the i-th time step. Let represent the target language summary at the i-th time step. The decoding result of the monolingual decoder is the source language summary, and the decoding result of the cross-lingual decoder is the target language summary.

[0044] In addition, the attention distribution for cross-linguistic summarization is constrained by minimizing the KL divergence loss, and the total loss is calculated as follows:

[0045] (10)

[0046] Here, β is a hyperparameter used to balance the weights of different loss functions.

[0047] Compared with the prior art, the present invention has the following advantages:

[0048] (1) The present invention uses an end-to-end neural network model based on multi-task learning to perform cross-language summarization of text. In this end-to-end framework, encoder parameters, attention weights and decoder parameters are learned together.

[0049] (2) This invention constructs a cross-attention alignment mechanism. By introducing KL divergence loss into the cross-attention module of the neural network model, the cross-language summarization model can learn the attention weights of the source text of the monolingual summarization model, thereby optimizing the attention distribution of cross-language summarization.

[0050] (3) The present invention also constructs a multilingual mutual attention mechanism. By adding a multilingual mutual attention module to the neural network model, the hidden states of the cross-lingual summarization model and the monolingual summarization model can learn from each other, thereby optimizing the representation of the bilingual hidden states.

[0051] (4) In the model constructed in this invention, the attention result representation of the cross attention module and the multilingual mutual attention module are comprehensively considered to summarize the source text, thereby improving the recall and accuracy of the summary to a certain extent.

[0052] (5) The method proposed in this invention can provide high-tech products and services for Chinese researchers, students and other groups who need to study foreign language documents, generating direct practical value and economic benefits. Attached Figure Description

[0053] Figure 1This is a flowchart of a cross-lingual summarization method based on cross-attention alignment and multilingual mutual attention;

[0054] Figure 2 This is a flowchart of the process for constructing a cross-language summarization dataset;

[0055] Figure 3 It is a flowchart of the process of segmenting data into words, obtaining word embedding vector representations, encoding and preliminary decoding;

[0056] Figure 4 It is a flowchart of optimizing the attention distribution of cross-language summarization using a cross-attention alignment mechanism;

[0057] Figure 5 It is a flowchart of optimizing bilingual hidden state representation by utilizing a multilingual mutual attention mechanism;

[0058] Figure 6 It is a flowchart of the process of training a cross-language summarization model and generating cross-language summaries;

[0059] Figure 7 This is a structural diagram of the multi-task cross-lingual summarization model based on cross-attention alignment and multilingual mutual attention constructed in this invention. Detailed Implementation

[0060] To further illustrate the technical solution of the present invention, the present invention will be further described below through embodiments.

[0061] like Figure 1 As shown, this embodiment of a cross-lingual summarization method based on cross-attention alignment and multilingual mutual attention includes the following steps:

[0062] Step 1: Obtain the original text and summary of the monolingual summary dataset, process the data, and construct a cross-lingual summary dataset.

[0063] Step 2: Input the data from the cross-language summarization dataset into the pre-trained multi-task cross-language summarization model (e.g., ...). Figure 7 (as shown) and generate cross-language summaries. The multi-task cross-language summarization model includes: 1) segmenting the source language documents, source language summaries and target language summaries in the cross-language summarization dataset, obtaining word embedding vector representations, and performing encoding and preliminary decoding; 2) optimizing the attention distribution of cross-language summaries using a cross-attention alignment mechanism; 3) optimizing the bilingual hidden state representation using a multilingual mutual attention mechanism.

[0064] And the steps for training a multi-task cross-language summarization model.

[0065] like Figure 2 As shown, the specific steps for constructing the cross-language summarization dataset in step 1 are as follows:

[0066] Step 101: Use web scraping technology to scrape text data of monolingual articles and their summaries from dataset websites on the Internet.

[0067] This embodiment utilizes the Scrapy web crawler to scrape text and corresponding summaries from various English monolingual summary datasets on internet dataset websites. Examples of text and corresponding summaries from English monolingual summary datasets are shown in Table 1:

[0068] Table 1

[0069]

[0070] Step 102: Process the data using round-trip translation and discourse implication strategies to construct a cross-linguistic summary dataset.

[0071] First, the text and corresponding summaries in the monolingual summary dataset crawled in step S101 are subjected to round-trip translation. A high-performance machine translation API is used to translate the summaries in the source language text-summary pairs into the target language, and then the results are translated back into the source language. ROUGE is calculated for the text-summary pairs consisting of the source language text and the translated summaries. If the ROUGE value is greater than a set threshold, the corresponding source language text-target language summary pair is retained; otherwise, the data is filtered out. The intermediate results of the round-trip translation and the retained translated source language text-target summary pairs are shown in Table 2.

[0072]

[0073] Subsequently, the data selected through the round-trip translation strategy was further filtered using a discourse entailment strategy. An entailment score was calculated between the translated and original summaries using an entailment model. If the entailment score exceeded a predefined threshold, the corresponding source language text-target language summary pair was retained; otherwise, the data was also filtered. Through these steps, a high-quality cross-lingual summarization dataset was finally constructed. The intermediate results of the discourse entailment strategy and the retained translated source language text-target summary pairs are shown in Table 3.

[0074] Table 3

[0075]

[0076] like Figure 3 As shown, word segmentation is performed on the source language documents, source language summaries, and target language summaries in the cross-language summarization dataset to obtain word embedding vector representations. Encoding and preliminary decoding are then performed. The specific steps are as follows:

[0077] 1.1) Word-level segmentation is performed on sentences in the source language documents, source language summaries, and target language summaries in the cross-language summarization dataset. Given that English text is naturally separated by spaces, spaces can be used directly to segment English text. However, for Chinese text, due to the lack of clear word boundaries, the jieba library is used for segmentation.

[0078] 1.2) Perform data preprocessing on the segmented data to convert the original text data into a format that the model can understand. Load the BERT tokenizer and set some special tokens (such as [SEP], [CLS], [PAD], etc.) and their corresponding vocab IDs; convert the sentences in each sentence list into space-separated strings, and filter out sentences whose length is greater than the specified minimum length (min_src_ntokens_per_sent); truncate these excessively long sentences so that their length does not exceed the specified maximum length (max_src_ntokens_per_sent), and ensure that the final number of sentences does not exceed the set maximum number of sentences (max_src_nsents); use the BERT tokenizer to convert the filtered sentences into subtokens, add the "[CLS]" tag at the beginning of each sentence and the "[SEP]" tag at the end, and convert the tokenization results into their corresponding vocab IDs; since there are sequences of different lengths in the same batch, in order to facilitate batch processing, it is also necessary to add the "[PAD]" tag to the end of the sequence, and fill the shorter sequences to the same length as the longest sequence (here, the sequence refers to the sentence with the added tags).

[0079] 1.3) Based on the results of 1.2), source and target sequences are constructed and transformed into embedding vector representations for encoding and preliminary decoding. The source language documents processed in 1.2) are directly input into the encoder as source sequences. For the decoder under the teacher-enforced policy, the source language summary and target language summary processed in 1.2) are used as target sequences for monolingual and cross-lingual summaries, respectively. When inputting into the decoder, the target sequences need to be shifted one position to the right at each time step, so that the decoder can receive the next label of the standard summary as input at each time step. At the same time, the unshifted target sequences are used to calculate the loss for the decoder's prediction. The source and target sequences are converted into tensors in the PyTorch deep learning framework using the torch.tensor() function. These tensors are then used as input to the neural network model. The word embedding layer that receives the input tensors converts them into corresponding word embedding vectors. The word embedding vectors of the source sequences are fed into the encoder for encoding, generating a context vector H={h1,h2,…,h... nThe word embedding vectors of the offset target sequence are fed into the self-attention layer of the cross-lingual and monolingual decoders for initial decoding, thereby generating the hidden state h of the target language decoder. t The hidden state h of the source language decoder s .

[0080] like Figure 4 As shown, the attention distribution of cross-language summarization is optimized using a cross-attention alignment mechanism. The specific steps are as follows:

[0081] 2.1) Using a cross-attention mechanism, calculate the attention distribution of the source text for both cross-lingual and monolingual summarization:

[0082] To calculate the attention distribution of the summary over the source text, an attention mechanism is typically used. This is a method in neural network models, especially in natural language processing tasks, to make the model "pay attention" to specific parts of the input sequence. Since attention weights are not directly provided for a given text and summary, a trained model is needed to generate these weights.

[0083] Based on the content of the summary and the source text, a hypothetical analysis is conducted to determine which parts are likely to receive more attention. Table 4 shows an example analyzing the attention distribution of the source text in monolingual and cross-lingual summaries:

[0084] Table 4

[0085]

[0086] The analysis in Table 4 shows that the key information in both cross-lingual and monolingual summarization results includes: (1) a description of a witness seeing a 36-year-old man attacking his dog in Penrith. (2) an emphasis on the man continuing to attack despite the dog barking loudly. (3) a mention that the man was accused of animal cruelty and that the dog was taken away. The key attention objects in both cross-lingual and monolingual attention distributions include: (1) event description: Since the summary explicitly mentions the abuse incident and the accusation, the event description in the source text is likely to receive high attention. (2) specific behavior: The summary emphasizes that the man continued to attack despite the dog barking loudly. This indicates that the model gives high attention to the part of the source text describing specific abusive behavior when generating the summary. (3) the reactions of the witness and the dog: Although the summary does not directly mention the witness's intervention or the dog's resistance, this information is important for constructing a complete event context and may therefore also receive some attention. It should be noted that the above analysis is based on assumptions, and the actual attention distribution will depend on the specific model structure and training data.

[0087] The hidden state h of the cross-language decoder t The hidden state h of the source language decoder s Using these as query vectors, the relevance between the query vector and the source dialogue context vector H is calculated, thus obtaining the cross-language decoder's cross-attention weight matrix for the original text. Cross-attention weight matrix of the original text and monolingual decoder As shown in equations (1) and (2),

[0088] (1)

[0089] (2)

[0090] in, and These represent the query and key in the cross-language attention process, respectively. It is the dimension of query and key in the cross-language attention process. and These represent the query and key in the monolingual attention process, respectively. It is the dimension of query and key in the monolingual attention process.

[0091] use and These two attention distributions are weighted and summed over the source dialogue context vector H to obtain the context attention result of the cross-language decoder on the original text. The representation of contextual attention results of the original text by the monolingual decoder. As shown in equations (3) and (4):

[0092] (3)

[0093] (4)

[0094] in, and These represent the value matrices in the cross-lingual attention and monolingual attention processes, respectively. The two contextual results will participate in the generation of cross-lingual summaries and monolingual summaries, respectively.

[0095] 2.2) Optimize the attention distribution of cross-linguistic summarization through a cross-attention alignment mechanism (e.g., Figure 7 As shown in “cross-attention alignment” in the text): Since the content to be summarized in different languages ​​in the same text is consistent, the cumulative cross-attention distribution of the cross-language decoder and the monolingual decoder for the source text should also be similar. Therefore, the cross-attention between the same language summarizing encoder and decoder is used to guide the cross-attention of cross-language summarization, as shown in Equation (5):

[0096] (5)

[0097] in, It is the attention distribution of the source language summarization decoder on the hidden states of the source language text, i.e., the attention distribution of monolingual summarization. This refers to the attention distribution of the target language summarizing decoder on the hidden states of the source language text, i.e., the attention distribution for cross-language summarizing. By minimizing the KL divergence loss, it is possible to... Attention distribution closer to that of the same language This makes the cross-language decoder behave more like a monolingual decoder when focusing on source text information.

[0098] like Figure 5 As shown, the bilingual hidden state representation is optimized using a multilingual mutual attention mechanism. The specific steps are as follows:

[0099] 3.1) Optimize bilingual hidden state representation by utilizing multilingual mutual attention mechanisms (e.g., Figure 7 As shown in the "Multilingual Attention Layer" (in the diagram), the multilingual mutual attention mechanism is an interactive decoding mechanism. Its core idea is that during decoding, it not only pays attention to the information contained in the hidden state of the current language decoder for generating the summary, but also simultaneously focuses on relevant information in the hidden state of another language decoder. Specifically, at the decoder end, the generation of a language summary depends not only on previously generated outputs in the same language, but also on previously generated outputs in a different language from the other language. For example, at the p-th time step of the cross-language summarization decoder, the hidden state of the target language decoder is... As the query vector Q, the last p hidden states of the source language decoder are computed. , ,..., The correlation between the p hidden states is weighted and summed using a multilingual mutual attention module to obtain the contextual attention result representation of the target language decoder hidden state on the source language decoder hidden state. It participates in the process of cross-language summary generation, as shown in equation (6):

[0100] (6)

[0101] in, This refers to the hidden state of the target language summarization decoder at time step p. It refers to all the hidden states of the source language digest decoder in the first p time steps.

[0102] Equation (6) illustrates the computation process of multilingual mutual attention, where the MultiLingualAttn function is calculated. and{ , ,..., The correlation of} and the hidden state { , ,..., The weighted summation yields the context-aware result representation. The calculation process is consistent with equations (1) and (3).

[0103] Similarly, at the p-th time step of the source language summarization decoder, the hidden state of the source language decoder is... As the query vector Q, the last p hidden states of the target language decoder are computed. , ,..., The correlation between the p hidden states is weighted and summed using a multilingual mutual attention module to obtain the contextual attention result representation of the source language decoder hidden state on the target language decoder hidden state. As shown in equation (7):

[0104] (7)

[0105] in, This refers to the hidden state of the source language summarization decoder at time step p. This refers to all hidden states of the target language summarization decoder in the first p time steps; the decoder of monolingual summarization, like cross-lingual summarization, depends not only on the previously generated monolingual output but also on the previously generated cross-lingual output during the decoding process.

[0106] Table 5 shows an example illustrating the hidden states generated during the decoding process when performing cross-lingual and monolingual summarization on the source text in Table 3:

[0107] Table 5

[0108] Step (n) Hidden state of a monolingual decoder Hidden state of cross-language decoders 1 A member of One 2 the public public 3 saw In Penris 4 a See 5 36-year-old One 6 man 36 years old … … …

[0109] As shown in Table 5, the generation processes of different language summaries can help each other: when the cross-language summary decoder decodes the Chinese word "36-year-old" at step n = 6, the English word "36-year-old" with the same meaning that has been predicted by the monolingual summary decoder at step n = 5 can provide semantic assistance for the Chinese decoding, and the English generated before step 6 also has the same information as the Chinese summary. The reason for this is that in languages with different grammatical structures, the order of appearance of the same word is different, but their semantics are the same. Therefore, in the decoding process, the mutual attention of the hidden states of the two languages helps to improve the quality of the finally generated summary.

[0110] 3.2) Use the generated context attention result representation and the context attention result representation of the generated target language decoder hidden state on the source language decoder hidden state Generate a summary: At time step p, the probability of the next word in the cross-language summary is calculated as shown in Equation (8):

[0111] (8)

[0112] where the function F includes an MLP layer that fuses the concatenated and information, a feed-forward layer that obtains the intermediate vector representation, a linear layer that learns the complex patterns and features of the intermediate vector, and a softmax layer that predicts the word probability distribution.

[0113] As Figure 6 shown, establish a loss function and train the cross-language summary model. The specific steps are as follows:

[0114] After the monolingual decoder and the cross-language decoder in multi-task learning predict their corresponding summaries, the negative log-likelihood loss is used to optimize both tasks simultaneously. The loss of a single example is:

[0115] (9)

[0116] where represents the source document, represents the source language summary, represents the target language summary; α is a hyperparameter used to balance the weights of the monolingual summary task and the cross-language summary task; represents the source language summary at the i-th time step, represents the target language summary at the i-th time step. The decoding result of the monolingual decoder is the source language summary, and the decoding result of the cross-language decoder is the target language summary. It should be noted that different from other multi-task models, the model of the present invention uses the decoding information of another language in the decoding process.

[0117] In addition, the attention distribution for cross-linguistic summarization is constrained by minimizing the KL divergence loss, and the total loss is calculated as follows:

[0118] (10)

[0119] Here, β is a hyperparameter used to balance the weights of different loss functions.

[0120] A new text material is fed into a trained multi-task cross-lingual summarization model. The model generates a predicted summary, which is then pushed to the user as the answer.

[0121] The foregoing has shown and described the main features and advantages of the present invention. It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, it is intended that all variations falling within the meaning and scope of equivalents of the claims be included within the present invention.

[0122] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A cross-lingual summarization method based on cross-attention alignment and multilingual mutual attention, characterized in that, Includes the following steps: Step 1: Obtain the original text and summary of the monolingual summary dataset, process the data, and construct a cross-lingual summary dataset; Step 2: Input the data from the cross-language summarization dataset into the pre-trained multi-task cross-language summarization model and generate cross-language summaries. The multi-task cross-language summarization model includes: 2.1) segmenting the source language documents, source language summaries, and target language summaries in the cross-language summarization dataset into words, obtaining word embedding vector representations, and performing encoding and preliminary decoding; 2.2) optimizing the attention distribution of cross-language summarization using a cross-attention alignment mechanism; 2.3) optimizing the bilingual hidden state representation using a multilingual mutual attention mechanism. The method of optimizing the attention distribution of cross-language summarization using the cross-attention alignment mechanism is as follows: 2.2.1) Calculate the attention distribution of cross-language summarization and monolingual summarization on the source text respectively through the cross-attention mechanism; 2.2.2) Optimize the attention distribution of cross-language summarization through the cross-attention alignment mechanism: Use the cross-attention between the encoder and decoder of the same language to guide the cross-attention of cross-language summarization, as shown in Equation (5): （5） in, It is the attention distribution of the source language summarization decoder on the hidden states of the source language text, i.e., the attention distribution of monolingual summarization. It is the attention distribution of the target language summarization decoder to the hidden state of the source language text, i.e., the attention distribution of cross-language summarization; The optimization of bilingual hidden state representation using a multilingual mutual attention mechanism is specifically as follows: 2.3.1) Optimize the bilingual hidden state representation by utilizing a multilingual mutual attention mechanism: At the p-th time step of the target language summarization decoder, the hidden state of the target language decoder is... As the query vector Q, the last p hidden states of the source language decoder are computed. , ,..., The correlation between the p hidden states is weighted and summed using a multilingual mutual attention module to obtain the contextual attention result representation of the target language decoder hidden state on the source language decoder hidden state. As shown in equation (6): （6） in, This refers to the hidden state of the target language summarization decoder at time step p. This refers to all hidden states of the source language digest decoder in the first p time steps; Similarly, at the p-th time step of the source language summarization decoder, the hidden state of the source language decoder is... As the query vector Q, the last p hidden states of the target language decoder are computed. , ,..., The correlation between the p hidden states is weighted and summed using a multilingual mutual attention module to obtain the contextual attention result representation of the source language decoder hidden state on the target language decoder hidden state. As shown in equation (7): （7） in, This refers to the hidden state of the source language summarization decoder at time step p. This refers to all hidden states of the target language summarization decoder in the first p time steps; 2.3.2) Representing the generated contextual attention results The contextual attention result of the generated target language decoder hidden state to the source language decoder hidden state. Generate a summary: At time p steps, the probability of the next word in the cross-linguistic summary is calculated as shown in equation (8): （8） Wherein, function F includes the merged and spliced ​​components. and The system consists of an MLP layer for information, a feedforward layer for obtaining intermediate vector representations, a linear layer for learning complex patterns and features of intermediate vectors, and a softmax layer for predicting word probability distributions.

2. The cross-lingual summarization method based on cross-attention alignment and multilingual mutual attention as described in claim 1, characterized in that, Step 1 specifically involves: Step 101: Use web scraping technology to scrape text data of monolingual articles and their summaries from dataset websites on the Internet; Step 102: Process the data using round-trip translation and discourse implication strategies to construct a cross-linguistic summary dataset.

3. The cross-lingual summarization method based on cross-attention alignment and multilingual mutual attention as described in claim 2, characterized in that, Step 102 specifically involves: First, the text and corresponding summaries in the monolingual summarization dataset crawled in step 101 are processed through round-trip translation. A high-performance machine translation API is used to translate the summaries in the source language text-summary pairs into the target language, and then translate the results back into the source language. ROUGE is calculated for the text-summary pairs consisting of the source language text and the translated summaries. If the ROUGE value is greater than a set threshold, the corresponding source language text-target language summary pair is retained; otherwise, the source language text-target language summary pair is filtered. Subsequently, the data filtered through the round-trip translation strategy is further filtered using a discourse entailment strategy. An entailment score is calculated between the translated summary and the original summary using an entailment model. If the entailment score exceeds a predefined threshold, the corresponding source language text-target language summary pair is retained; otherwise, the source language text-target language summary pair is also filtered. Through these steps, a cross-lingual summarization dataset is finally constructed.

4. A cross-lingual summarization method based on cross-attention alignment and multilingual mutual attention as described in claim 2, characterized in that, Step 2.1 involves segmenting the source language documents, source language summaries, and target language summaries in the cross-language summarization dataset into words, obtaining word embedding vector representations, and performing encoding and preliminary decoding, specifically as follows: 2.1.1) Word-level segmentation is performed on sentences in the source language documents, source language summaries, and target language summaries in the cross-language summarization dataset; for English text, spaces are used directly for word segmentation; for Chinese text, the jieba library is used for word segmentation. 2.1.2) Preprocess the segmented data: Load the BERT tokenizer and set some tokens and their corresponding vocab IDs; convert the sentences in each sentence list into space-separated strings, and filter out sentences whose length is greater than the specified minimum length; truncate these excessively long sentences so that their length does not exceed the specified maximum length, and ensure that the final number of sentences does not exceed the set maximum number of sentences; use the BERT tokenizer to convert the filtered sentences into words, add "[CLS]" at the beginning of each sentence and "[SEP]" at the end, and convert the segmentation results into their corresponding vocab IDs; add "[PAD]" to the end of the sequence to pad shorter sequences to the same length as the longest sequence; 2.1.3) Based on the results of 2.1.2), the source and target sequences are constructed and transformed into embedding vector representations for encoding and preliminary decoding. The source language documents processed in 2.1.2) are directly input into the encoder as source sequences. For the decoder under the teacher-enforced policy, the source language summary and target language summary processed in 2.1.2) are used as target sequences for monolingual and cross-lingual summaries, respectively. When input into the decoder, the target sequences need to be shifted one position to the right at each time step, so that the decoder can receive the next label of the standard summary as input at each time step. At the same time, the unshifted target sequences are used to calculate the loss for the decoder's prediction. The source and target sequences are converted into tensors in the PyTorch deep learning framework using the torch.tensor() function. These tensors are then used as input to the neural network model. The word embedding layer that receives the input tensors converts them into corresponding word embedding vectors. The word embedding vectors of the source sequences are fed into the encoder for encoding, generating a context vector H={h1,h2,…,h... n The word embedding vectors of the offset target sequence are fed into the self-attention layer of the cross-lingual and monolingual decoders for initial decoding, thereby generating the hidden state h of the cross-lingual decoder. t The hidden state h of the source language decoder s .

5. A cross-lingual summarization method based on cross-attention alignment and multilingual mutual attention as described in claim 4, characterized in that, In step 2.2.1, the attention distribution of the source text for cross-lingual and monolingual summaries is calculated using a cross-attention mechanism, specifically as follows: The hidden state h of the cross-language decoder t The hidden state h of the source language decoder s Using these as query vectors, the relevance between the query vector and the source dialogue context vector H is calculated, thus obtaining the cross-language decoder's cross-attention weight matrix for the original text. Cross-attention weight matrix of the original text and monolingual decoder As shown in equations (1) and (2), （1） （2） in, and These represent the query and key in the cross-language attention process, respectively. It is the dimension of query and key in the cross-language attention process. and These represent the query and key in the monolingual attention process, respectively. It refers to the query and key dimensions in the monolingual attention process; use and These two attention distributions are weighted and summed over the source dialogue context vector H to obtain the context attention result of the cross-language decoder on the original text. The representation of contextual attention results of the original text by the monolingual decoder. As shown in equations (3) and (4): （3） （4） in, and These represent the value matrices in the cross-lingual attention and monolingual attention processes, respectively. The two contextual results will participate in the generation of cross-lingual summaries and monolingual summaries, respectively.

6. A cross-lingual summarization method based on cross-attention alignment and multilingual mutual attention as described in claim 1, characterized in that, This also includes training multi-task cross-language summarization models, specifically: After the monolingual decoder and cross-lingual decoder predict their corresponding summaries in multi-task learning, the negative log-likelihood loss is used to optimize both tasks simultaneously. The loss for a single example is: （9） in, Represents the source document. Represents a summary of the source language. α represents the target language summary; α is a hyperparameter used to balance the weights of monolingual summarization tasks and cross-lingual summarization tasks. This represents the source language summary at the i-th time step. Let represent the target language summary at the i-th time step. The decoding result of the monolingual decoder is the source language summary, and the decoding result of the cross-lingual decoder is the target language summary. In addition, the attention distribution for cross-linguistic summarization is constrained by minimizing the KL divergence loss, and the total loss is calculated as follows: （10） Here, β is a hyperparameter used to balance the weights of different loss functions.

Images