Cross-language abstract method based on cross attention alignment and multi-language mutual attention

Abstract

The invention belongs to the technical field of natural language processing, and particularly relates to a cross-language abstract method based on cross attention alignment and multi-language mutual attention. In order to improve the performance of cross-language abstracts, an end-to-end neural network model based on multi-task learning is adopted to carry out cross-language abstracts on texts, and the method comprises the following steps that original texts and abstracts of a monolingual abstract data set are crawled from a data set website of the Internet, and data are processed through a round-trip translation strategy and a discourse implication strategy; constructing a cross-language abstract data set; word segmentation is carried out on the source text and the abstract in the data set, word embedding vector representation is obtained, and coding and preliminary decoding are carried out; the attention distribution of the cross-language abstract is optimized by using a cross attention alignment mechanism; utilizing a multi-language mutual attention mechanism to optimize bilingual hidden state representation; and training the cross-language abstract model and generating a cross-language abstract.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and in particular relates to a cross-language summarization method based on cross-attention alignment and multi-language mutual attention. Background Art

[0002] In recent years, with the development of economic globalization, computer-assisted rapid reading of foreign literature has become a key challenge in the field of artificial intelligence. Due to the complexity of foreign articles and language barriers, ordinary readers often face difficulties in reading. Therefore, it is particularly important to study cross-language summarization methods.

[0003] Compared with monolingual summarization tasks, cross-lingual summarization tasks often perform worse than monolingual summarization due to data scarcity and cross-lingual characteristics. In order to meet this challenge, many studies have focused on cross-lingual summarization tasks in recent years. Among them, multi-task learning, as a popular solution, aims to improve the performance of cross-lingual summarization by integrating auxiliary tasks such as monolingual summarization. However, although these methods have improved the performance of cross-lingual summarization to a certain extent, they ignore the intrinsic connection between monolingual summarization and cross-lingual summarization in the summary generation process, and fail to effectively utilize the cross-attention information of monolingual summarization and the hidden information generated during the generation process. Summary of the invention

[0004] In view of the above problems, the present invention provides a cross-language summarization method based on cross-attention alignment and multi-language mutual attention.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] A cross-language summarization method based on cross-attention alignment and multi-language mutual attention includes the following steps:

[0007] Step 1: Obtain the original text and summary of the monolingual summary dataset, process the data, and construct a cross-language summary dataset.

[0008] Step 2, input the data in the cross-language summary dataset into the trained multi-task cross-language summary model and generate a cross-language summary, the multi-task cross-language summary model includes: 1) segmenting the source language documents, source language summaries and target language summaries in the cross-language summary dataset, obtaining word embedding vector representations, encoding and preliminary decoding; 2) using the cross-attention alignment mechanism to optimize the attention distribution of the cross-language summary; 3) using the multi-language mutual attention mechanism to optimize the bilingual hidden state representation.

[0009] Further, the step 1 is specifically as follows:

[0010] Step 101, using crawler technology, crawling text data of monolingual articles and their abstracts from dataset websites on the Internet.

[0011] Step 102 , processing the data through round-trip translation strategy and discourse entailment strategy to construct a cross-language summary dataset.

[0012] Furthermore, the step 102 is specifically as follows:

[0013] First, the text and the corresponding abstracts in the monolingual abstract dataset crawled in step S101 are round-trip translated; using a high-performance machine translation API, the abstracts in the source language text-abstract pair are translated into the target language, and then the results are translated back to the source language; ROUGE calculation is performed on the text-abstract pair consisting of the source language text and the translated abstract; if the ROUGE value is greater than the set threshold, the corresponding source language text-target language abstract pair is retained, otherwise, the data will be filtered; then, the data filtered out by the round-trip translation strategy is further filtered by applying the discourse entailment strategy; the entailment model is used to calculate the entailment score between the translated abstract and the original abstract; if the entailment score exceeds the predefined threshold, the corresponding source language text-target language abstract pair is retained; otherwise, the data will also be filtered; through the above steps, a high-quality cross-language abstract dataset is finally constructed.

[0014] Furthermore, in step 2, the source language documents, source language summaries, and target language summaries in the cross-language summary dataset are segmented, word embedding vector representations are obtained, and encoding and preliminary decoding are performed, specifically:

[0015] 1.1) Perform word-level segmentation on the sentences of the source language documents, source language summaries, and target language summaries in the cross-language summary dataset; for English text, directly use spaces to segment the English text; for Chinese text, use the jieba library for word segmentation.

[0016] 1.2) Preprocess the data after word segmentation: load the BERT word segmenter and set some special tokens and their corresponding vocab IDs; convert the sentences in each sentence list into space-delimited strings and filter out sentences whose length is greater than the specified minimum length; truncate these overlong sentences so that their length does not exceed the specified maximum length and ensure that the final number of sentences does not exceed the set maximum number of sentences; use the BERT word segmenter to convert the filtered sentences into subwords, add a "[CLS]" tag at the beginning of each sentence and a "[SEP]" tag at the end, and convert the word segmentation results into their corresponding vocab IDs; add a "[PAD]" tag at the end of the sequence to pad the shorter sequence to the same length as the longest sequence.

[0017] 1.3) Based on the results of 1.2), construct the source sequence and target sequence, and convert them into embedded vector representations for encoding and preliminary decoding; the source language document processed by 1.2) is directly input into the encoder as the source sequence; for the decoder under the teacher-forcing strategy, the source language summary and target language summary processed by 1.2) are used as the target sequences of the single-language summary and the cross-language summary respectively. When input into the decoder, the target sequence needs to be offset to the right by one position at each time step, so that the decoder can receive the next token of the standard summary as input at each time step; at the same time, the unshifted target sequence is used to calculate the loss with the decoder's prediction; the source sequence and target sequence are converted into tensors in the PyTorch deep learning framework through the torch.tensor() function, which are then used as input to the neural network model; the word embedding layer that receives the input tensor converts it into the corresponding word embedding vector; the word embedding vector of the source sequence is sent to the encoder for encoding to generate the context vector H = {h1,h2,…,h n}; The word embedding vector of the offset target sequence is fed into the self-attention layer of the cross-language and single-language decoders for preliminary decoding, thereby generating the hidden state h of the target language decoder t and the hidden state h of the source language decoder s .

[0018] Furthermore, in step 2, the cross-attention alignment mechanism is used to optimize the attention distribution of the cross-language summary, specifically:

[0019] 2.1) Through the cross-attention mechanism, the attention distribution of the cross-language summary and the single-language summary on the source text is calculated separately: the hidden state h of the cross-language decoder is t and the hidden state h of the source language decoder s As query vectors, calculate the correlation with the source dialogue context vector H, and then obtain the cross-language decoder's cross-attention weight matrix for the original text. And the cross attention weight matrix of the single language decoder for the original text As shown in formula (1) and formula (2),

[0020]

[0021] in, and represent the query and key in the cross-lingual attention process, are the dimensions of query and key in the cross-lingual attention process, and Represent the query and key in the monolingual attention process, is the dimension of query and key in monolingual attention process.

[0022] use and These two attention distributions perform weighted summation on the source dialogue context vector H to obtain the context attention result representation of the cross-language decoder on the original text And the single language decoder's contextual attention result representation of the original text As shown in formula (3) and formula (4):

[0023]

[0024] in, and Represent the value matrices in the cross-lingual attention and monolingual attention processes respectively. The two context results will participate in the process of generating cross-lingual summaries and monolingual summaries respectively.

[0025] 2.2) Optimize the attention distribution of cross-lingual summaries through the cross-attention alignment mechanism: Use the cross-attention between the same-language summary encoder and decoder to guide the cross-lingual summary, as shown in formula (5):

[0026]

[0027] in, is the attention distribution of the source language summary decoder on the hidden state of the source language text, that is, the attention distribution of the monolingual summary, is the attention distribution of the target language summary decoder on the hidden state of the source language text, that is, the attention distribution of the cross-language summary.

[0028] Furthermore, in step 2, the multilingual mutual attention mechanism is used to optimize the bilingual hidden state representation, specifically:

[0029] 3.1) Optimize the bilingual hidden state representation using the multilingual mutual attention mechanism: At the pth time step of the cross-lingual summary decoder, the hidden state of the target language decoder is As the query vector Q, calculate the last p hidden states of the source language decoder The correlation between them is obtained by weighting and summing these p hidden states through the multi-language mutual attention module to obtain the contextual attention result representation of the target language decoder hidden state to the source language decoder hidden state As shown in formula (6):

[0030]

[0031] in, refers to the hidden state of the target language summary decoder at the pth time step, refers to all hidden states of the source language summary decoder in the previous p time steps.

[0032] Similarly, at the pth time step of the source language summary decoder, the hidden state of the source language decoder is As the query vector Q, calculate the last p hidden states of the target language decoder The correlation between them is calculated by weighting the p hidden states through the multi-language mutual attention module to obtain the contextual attention result representation of the source language decoder hidden state to the target language decoder hidden state. As shown in formula (7):

[0033]

[0034] in, refers to the hidden state of the source language summary decoder at the pth time step, refers to all hidden states of the target language summary decoder in the previous p time steps.

[0035] 3.2) Using the generated context to pay attention to the result representation The resulting contextual attention representation of the target language decoder hidden state to the source language decoder hidden state Generate summary: At time step p, the probability of the next word of the cross-language summary is calculated as shown in formula (8):

[0036]

[0037] Among them, function F includes the fusion and splicing and The MLP layer extracts information, the feed-forward layer obtains the intermediate vector representation, the linear layer learns the complex patterns and features of the intermediate vectors, and the softmax layer predicts the probability distribution of words.

[0038] Furthermore, it also includes training a multi-task cross-language summary model, specifically:

[0039] After the single-language decoder and the cross-language decoder of multi-task learning predict their corresponding summaries, the two tasks are optimized simultaneously using the negative log-likelihood loss. The loss of a single example is:

[0040]

[0041] Among them, X s Represents the source document, y s Represents the source language summary, y t represents the target language summary; α is a hyperparameter used to balance the weights of the monolingual summary task and the cross-lingual summary task; represents the source language summary in the i-th time step, represents the target language summary in the i-th time step. The decoding result of the single-language decoder is the source language summary, and the decoding result of the cross-language decoder is the target language summary.

[0042] In addition, the KL divergence loss is minimized to constrain the attention distribution of the cross-language summary, and the total loss is calculated as:

[0043] L=βL KL +L nll (10)

[0044] Among them, β is a hyperparameter used to balance the weights of different loss functions.

[0045] Compared with the prior art, the present invention has the following advantages:

[0046] (1) The present invention adopts an end-to-end neural network model based on multi-task learning to perform cross-language summarization of text. In this end-to-end framework, encoder parameters, attention weights and decoder parameters are learned together.

[0047] (2) The present invention constructs a cross-attention alignment mechanism, which introduces the KL divergence loss in the cross-attention module of the neural network model, so that the cross-language summary model can learn the attention weight of the single-language summary model on the source text, thereby optimizing the attention distribution of the cross-language summary.

[0048] (3) The present invention also constructs a multilingual mutual attention mechanism. By adding a multilingual mutual attention module to the neural network model, the hidden states of the cross-lingual summary model and the monolingual summary model can learn from each other, thereby optimizing the representation of the bilingual hidden states.

[0049] (4) In the model constructed by the present invention, the attention result representations of the cross-attention module and the multilingual mutual attention module are comprehensively considered to summarize the source text, thereby improving the recall rate and accuracy of the summary to a certain extent.

[0050] (5) The method proposed in the present invention can provide high-tech products and services for Chinese researchers, students and other groups who need to study foreign literature, thereby generating direct practical value and economic benefits. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a flowchart of a cross-language summarization method based on cross-attention alignment and multi-language mutual attention;

[0052] Figure 2 This is a flowchart for building a cross-language summary dataset;

[0053] Figure 3 This is a flowchart for word segmentation, word embedding vector representation, encoding and preliminary decoding of data;

[0054] Figure 4 It is a flowchart of optimizing the attention distribution of cross-language summarization using the cross-attention alignment mechanism;

[0055] Figure 5 It is a flowchart of optimizing the bilingual hidden state representation using the multilingual mutual attention mechanism;

[0056] Figure 6 It is a flowchart of training a cross-language summary model and generating a cross-language summary;

[0057] Figure 7 It is a structural diagram of the multi-task cross-language summarization model based on cross-attention alignment and multi-language mutual attention constructed by the present invention. DETAILED DESCRIPTION

[0058] In order to further illustrate the technical solution of the present invention, the present invention is further described below through embodiments.

[0059] like Figure 1 As shown, a cross-language summarization method based on cross-attention alignment and multi-language mutual attention in this embodiment includes the following steps:

[0060] Step 1: Obtain the original text and summary of the monolingual summary dataset, process the data, and construct a cross-language summary dataset.

[0061] Step 2: Input the data in the cross-language summary dataset into the trained multi-task cross-language summary model (such as Figure 7 As shown in the figure, a cross-language summary is generated, and the multi-task cross-language summary model includes: 1) performing word segmentation on the source language documents, source language summaries and target language summaries in the cross-language summary dataset, obtaining word embedding vector representations, encoding and preliminary decoding; 2) optimizing the attention distribution of the cross-language summary by using the cross-attention alignment mechanism; 3) optimizing the bilingual hidden state representation by using the multi-language mutual attention mechanism.

[0062] And the steps for training a multi-task cross-lingual summarization model.

[0063] like Figure 2 As shown in Figure 1, the specific steps for constructing a cross-language summary dataset in step 1 are as follows:

[0064] Step 101, using crawler technology, crawling text data of monolingual articles and their abstracts from dataset websites on the Internet.

[0065] This embodiment uses the Scrapy crawler tool to crawl the text of each English monolingual summary dataset and its corresponding abstract from the dataset website on the Internet. The text of the English monolingual summary dataset and its corresponding abstract examples are shown in Table 1:

[0066] Table 1

[0067]

[0068]

[0069] Step 102 , processing the data through round-trip translation strategy and discourse entailment strategy to construct a cross-language summary dataset.

[0070] First, the text and the corresponding abstract in the crawled monolingual abstract dataset in step S101 are round-trip translated; the abstract in the source language text-abstract pair is translated into the target language using a high-performance machine translation API, and the result is then translated back to the source language; the text-abstract pair consisting of the source language text and the translated abstract is subjected to ROUGE calculation; if the ROUGE value is greater than the set threshold, the corresponding source language text-target language abstract pair is retained, otherwise, the data will be filtered. The intermediate results of the round-trip translation and the retained translated source language text-target abstract pairs are shown in Table 2:

[0071]

[0072]

[0073] Subsequently, the data filtered by the round-trip translation strategy is further filtered by applying the discourse entailment strategy; the entailment score between the translated summary and the original summary is calculated using the entailment model; if the entailment score exceeds the predefined threshold, the corresponding source language text-target language summary pair is retained; otherwise, the data is also filtered; through the above steps, a high-quality cross-language summary dataset is finally constructed. The intermediate results of the discourse entailment strategy and the retained translated source language text-target summary pairs are shown in Table 3:

[0074] Table 3

[0075]

[0076]

[0077] like Figure 3 As shown, the source language documents, source language summaries, and target language summaries in the cross-language summary dataset are segmented, word embedding vector representations are obtained, and encoding and preliminary decoding are performed. The specific steps are as follows:

[0078] 1.1) Perform word-level segmentation on the sentences of the source language documents, source language summaries, and target language summaries in the cross-language summary dataset. Given that English text is naturally separated by spaces, spaces can be used directly to segment English text. However, for Chinese text, due to the lack of clear word boundaries, the jieba library is used for word segmentation.

[0079] 1.2) Perform data preprocessing on the data after word segmentation in order to convert the original text data into a format that the model can understand. Load the BERT tokenizer and set some special tokens (such as [SEP], [CLS], [PAD], etc.) and their corresponding vocab IDs; convert the sentences in each sentence list into space-delimited strings, and filter out sentences whose length is greater than the specified minimum length (min_src_ntokens_per_sent); truncate these overlong sentences so that their length does not exceed the specified maximum length (max_src_ntokens_per_sent), and ensure that the final number of sentences does not exceed the set maximum number of sentences (max_src_nsents); use the BERT tokenizer to convert the filtered sentences into subtokens, add a "[CLS]" token at the beginning of each sentence and a "[SEP]" token at the end, and convert the tokenization results into their corresponding vocab IDs; since there are sequences of different lengths in the same batch, in order to facilitate batch processing, it is also necessary to add a "[PAD]" token at the end of the sequence to pad the shorter sequence to the same length as the longest sequence (the sequence here refers to the sentence with the token added).

[0080] 1.3) Based on the results of 1.2), construct the source sequence and target sequence, and convert them into embedded vector representations for encoding and preliminary decoding; the source language document processed by 1.2) is directly input into the encoder as the source sequence; for the decoder under the teacher-forcing strategy, the source language summary and target language summary processed by 1.2) are used as the target sequences of the single-language summary and the cross-language summary respectively. When input into the decoder, the target sequence needs to be offset to the right by one position at each time step, so that the decoder can receive the next token of the standard summary as input at each time step; at the same time, the unshifted target sequence is used to calculate the loss with the decoder's prediction; the source sequence and target sequence are converted into tensors in the PyTorch deep learning framework through the torch.tensor() function, which are then used as input to the neural network model; the word embedding layer that receives the input tensor converts it into the corresponding word embedding vector; the word embedding vector of the source sequence is sent to the encoder for encoding to generate the context vector H = {h1,h2,…,h n}; The word embedding vector of the offset target sequence is fed into the self-attention layer of the cross-language and single-language decoders for preliminary decoding, thereby generating the hidden state h of the target language decoder t and the hidden state h of the source language decoder s .

[0081] like Figure 4 As shown in Figure 1, the cross-attention alignment mechanism is used to optimize the attention distribution of cross-language summaries. The specific steps are as follows:

[0082] 2.1) Through the cross-attention mechanism, the attention distribution of the cross-language summary and the single-language summary on the source text is calculated separately:

[0083] To calculate the attention distribution of the summary to the source text, the attention mechanism is usually used, which is a method in neural network models, especially in natural language processing tasks, to let the model "pay attention" to specific parts of the input sequence. For a given text and summary, the attention weights are not directly provided, so a trained model is needed to generate these weights.

[0084] Based on the content of the summary and the source text, we hypothetically analyze which parts may receive more attention. Table 4 shows an example of analyzing the attention distribution of single-language and cross-language summaries on the source text:

[0085] Table 4

[0086]

[0087]

[0088] From the analysis in Table 4, we can see that the key information in both the cross-language summary and the monolingual summary results includes: (1) Description of the witness seeing a 36-year-old man attacking his dog in Penrith. (2) Emphasis on the fact that the man continued to attack despite the dog’s loud barking. (3) Mention that the man was accused of animal cruelty and the dog was taken away. The key attention objects of both the cross-language attention distribution and the monolingual attention distribution include: (1) Event description: Since the summary explicitly mentions the abuse incident and the accusation, the event description part in the source text is likely to receive higher attention. (2) Specific behavior: The summary emphasizes that despite the dog’s loud barking, the man continued to attack. This shows that when generating the summary, the model paid higher attention to the part of the source text that describes the specific abuse behavior. (3) Reactions of the witness and the dog: Although the summary does not directly mention the witness’ intervention or the dog’s resistance, this information is important for constructing a complete context of the incident and may therefore also receive a certain amount of attention. It should be noted that the above analysis is based on assumptions, and the actual attention distribution will depend on the specific model structure and training data.

[0089] The hidden state h of the cross-language decoder t and the hidden state h of the source language decoder s As query vectors, calculate the correlation with the source dialogue context vector H, and then obtain the cross-language decoder's cross-attention weight matrix for the original text. And the cross attention weight matrix of the single language decoder for the original text As shown in formula (1) and formula (2),

[0090]

[0091] in, and represent the query and key in the cross-lingual attention process, are the dimensions of query and key in the cross-lingual attention process, and Represent the query and key in the monolingual attention process, is the dimension of query and key in monolingual attention process.

[0092] use and These two attention distributions perform weighted summation on the source dialogue context vector H to obtain the context attention result representation of the cross-language decoder on the original text And the single language decoder's contextual attention result representation of the original text As shown in formula (3) and formula (4):

[0093]

[0094] in, and Represent the value matrices in the cross-lingual attention and monolingual attention processes respectively. The two context results will participate in the process of generating cross-lingual summaries and monolingual summaries respectively.

[0095] 2.2) Optimize the attention distribution of cross-language summaries through the cross-attention alignment mechanism (e.g. Figure 7 Since the content of the summary in different languages ​​is consistent in the same text content, the cumulative cross-attention distribution of the source text by the cross-language decoder and the single-language decoder should also be similar. To this end, the cross-attention between the same-language summary encoder and decoder is used to guide the cross-attention of the cross-language summary, as shown in formula (5):

[0096]

[0097] in, is the attention distribution of the source language summary decoder on the hidden state of the source language text, that is, the attention distribution of the monolingual summary, is the attention distribution of the target language summary decoder to the hidden state of the source language text, that is, the attention distribution of the cross-language summary. By minimizing the KL divergence loss, we can make Closer to the attention distribution of the same language This makes the cross-language decoder behave more like a single-language decoder when focusing on the source text information.

[0098] like Figure 5 As shown in Figure 1, the multilingual mutual attention mechanism is used to optimize the bilingual hidden state representation. The specific steps are as follows:

[0099] 3.1) Using the multi-language mutual attention mechanism to optimize the bilingual hidden state representation (e.g. Figure 7 The multilingual attention mechanism is an interactive decoding mechanism. Its core idea is that when decoding, it not only pays attention to the information contained in the hidden state of the current language decoder for generating summaries, but also pays attention to the relevant information in the hidden state of the other language decoder. Specifically, on the decoder side, the generation of summaries in one language depends not only on the previously generated outputs in the same language, but also on the previously generated outputs in another language in a different language. For example, at the pth time step of the cross-language summary decoder, the hidden state of the target language decoder is As the query vector Q, calculate the last p hidden states of the source language decoder The correlation between them is obtained by weighting and summing these p hidden states through the multi-language mutual attention module to obtain the contextual attention result representation of the target language decoder hidden state to the source language decoder hidden state , which participates in the process of cross-language summary generation, as shown in formula (6):

[0100]

[0101] in, refers to the hidden state of the target language summary decoder at the pth time step, refers to all hidden states of the source language summary decoder in the previous p time steps.

[0102] Formula (6) shows the calculation process of multilingual mutual attention, where the MultiLingualAttn function calculates and The correlation of hidden state Perform weighted summation to get the context attention result representation The calculation process is consistent with equation (1) and equation (3).

[0103] Similarly, at the pth time step of the source language summary decoder, the hidden state of the source language decoder is As the query vector Q, calculate the last p hidden states of the target language decoder The correlation between them is calculated by weighting the p hidden states through the multi-language mutual attention module to obtain the contextual attention result representation of the source language decoder hidden state to the target language decoder hidden state. As shown in formula (7):

[0104]

[0105] in, refers to the hidden state of the source language summary decoder at the pth time step, refers to all hidden states of the target language summary decoder in the previous p time steps; the decoder side of the monolingual summary, like the cross-lingual summary, depends not only on the previously generated monolingual output but also on the previously generated cross-lingual output during the decoding process.

[0106] Table 5 shows an example of the hidden states generated during the decoding process when performing cross-language summarization and monolingual summarization on the source text in Table 3:

[0107] Table 5

[0108] Step(n) Hidden states of a single language decoder Hidden states of the cross-lingual decoder 1 A member of One 2 thepublic public 3 saw In Penrith 4 a See 5 36-year-old One 6 man 36 years old … … …

[0109] As shown in Table 5, the generation process of summaries in different languages ​​can help each other: when the cross-language summary decoder decodes the Chinese word "36 years old" in step n=6, the single-language summary decoder has predicted the English word "36-year-old" with the same meaning in step n=5, which can provide semantic help for Chinese decoding, and the English generated before step 6 also has the same information as the Chinese summary. The reason for this is that in languages ​​with different grammatical structures, the order of appearance of the same word is different, but their semantics are consistent. Therefore, in the decoding process, the mutual attention of the hidden states of the two languages ​​helps to improve the quality of the final generated summary.

[0110] 3.2) Using the generated context to pay attention to the result representation The resulting contextual attention representation of the target language decoder hidden state to the source language decoder hidden state Generate summary: At time step p, the probability of the next word of the cross-language summary is calculated as shown in formula (8):

[0111]

[0112] Among them, function F includes the fusion and splicing and The MLP layer extracts information, the feed-forward layer obtains the intermediate vector representation, the linear layer learns the complex patterns and features of the intermediate vectors, and the softmax layer predicts the probability distribution of words.

[0113] like Figure 6 As shown, a loss function is established to train a cross-language summary model. The specific steps are as follows:

[0114] After the single-language decoder and the cross-language decoder of multi-task learning predict their corresponding summaries, the two tasks are optimized simultaneously using the negative log-likelihood loss. The loss of a single example is:

[0115]

[0116] Among them, X s Represents the source document, y s Represents the source language summary, y t represents the target language summary; α is a hyperparameter used to balance the weights of the monolingual summary task and the cross-lingual summary task; represents the source language summary in the i-th time step, represents the target language summary in the i-th time step, the decoding result of the single language decoder is the source language summary, and the decoding result of the cross-language decoder is the target language summary. It should be noted that, unlike other multi-task models, the model of the present invention uses the decoding information of another language in the decoding process.

[0117] In addition, the KL divergence loss is minimized to constrain the attention distribution of the cross-language summary, and the total loss is calculated as:

[0118] L=βL KL +L nll (10)

[0119] Among them, β is a hyperparameter used to balance the weights of different loss functions.

[0120] A new text material is used as an input sample and fed into the trained multi-task cross-language summary model. The model will generate a predicted summary and finally push this summary to the user as the answer. The input material and the final result are shown in Table 6.

[0121] Table 6

[0122]

[0123] The above shows and describes the main features and advantages of the present invention. It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive. The scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes that fall within the meaning and scope of the equivalent elements of the claims are included in the present invention.

[0124] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A cross-language summarization method based on cross-attention alignment and multi-language mutual attention, characterized in that: The following steps are involved: Step 1: Obtain the original text and summary of the monolingual summary dataset, process the data, and construct a cross-language summary dataset. Step 2, input the data in the cross-language summary dataset into the trained multi-task cross-language summary model and generate a cross-language summary, the multi-task cross-language summary model includes: 1) segmenting the source language documents, source language summaries and target language summaries in the cross-language summary dataset, obtaining word embedding vector representations, encoding and preliminary decoding; 2) using the cross-attention alignment mechanism to optimize the attention distribution of the cross-language summary; 3) using the multi-language mutual attention mechanism to optimize the bilingual hidden state representation.

2. A cross-language summarization method based on cross-attention alignment and multi-language mutual attention according to claim 1, characterized in that: The step 1 is specifically as follows: Step 101, using crawler technology to crawl text data of monolingual articles and their abstracts from a dataset website on the Internet; Step 102 , processing the data through round-trip translation strategy and discourse entailment strategy to construct a cross-language summary dataset.

3. A cross-language summarization method based on cross-attention alignment and multi-language mutual attention according to claim 2, characterized in that: The step 102 is specifically as follows: First, the text and the corresponding abstract in the monolingual abstract dataset crawled in step S101 are round-trip translated; the abstract in the source language text-abstract pair is translated into the target language using a high-performance machine translation API, and the result is then translated back to the source language; the text-abstract pair consisting of the source language text and the translated abstract is subjected to ROUGE calculation; if the ROUGE value is greater than a set threshold, the corresponding source language text-target language abstract pair is retained, otherwise, the data is filtered; then, the data filtered out by the round-trip translation strategy is further filtered by applying the discourse entailment strategy; the entailment score between the translated abstract and the original abstract is calculated using the entailment model; If the implication score exceeds the predefined threshold, the corresponding source language text-target language summary pair is retained; otherwise, the data will also be filtered; through the above steps, a cross-language summary dataset is finally constructed.

4. A cross-language summarization method based on cross-attention alignment and multi-language mutual attention according to claim 2, characterized in that: In step 2, the source language documents, source language summaries, and target language summaries in the cross-language summary dataset are segmented, word embedding vector representations are obtained, and encoding and preliminary decoding are performed, specifically: 1.1) Perform word-level segmentation on the source language documents, source language summaries, and target language summaries in the cross-language summary dataset; for English text, directly use spaces to segment the English text; for Chinese text, use the jieba library for word segmentation; 1.2) Perform data preprocessing on the data after word segmentation: load the BERT word segmenter and set some tokens and their corresponding vocab IDs; convert the sentences in each sentence list into space-delimited strings and filter out sentences whose length is greater than the specified minimum length; truncate these overlong sentences so that their length does not exceed the specified maximum length and ensure that the final number of sentences does not exceed the set maximum number of sentences; use the BERT word segmenter to convert the filtered sentences into subwords, add a "[CLS]" tag at the beginning of each sentence and a "[SEP]" tag at the end, and convert the word segmentation results into their corresponding vocab IDs; add a "[PAD]" tag at the end of the sequence to pad the shorter sequence to the same length as the longest sequence; 1.3) Based on the results of 1.2), construct the source sequence and target sequence, and convert them into embedded vector representations for encoding and preliminary decoding; the source language document processed by 1.2) is directly input into the encoder as the source sequence; for the decoder under the teacher-forcing strategy, the source language summary and target language summary processed by 1.2) are used as the target sequences of the single-language summary and the cross-language summary respectively. When input into the decoder, the target sequence needs to be offset to the right by one position at each time step, so that the decoder can receive the next token of the standard summary as input at each time step; at the same time, the unshifted target sequence is used to calculate the loss with the decoder's prediction; through the torch.tensor() function, the source sequence and target sequence are converted into tensors in the PyTorch deep learning framework, which are then used as input to the neural network model; the word embedding layer that receives the input tensor converts it into the corresponding word embedding vector; the word embedding vector of the source sequence is sent to the encoder for encoding to generate the context vector H = {h1,h2,…,h n }; The word embedding vector of the offset target sequence is fed into the self-attention layer of the cross-language and single-language decoders for preliminary decoding, thereby generating the hidden state h of the target language decoder t and the hidden state h of the source language decoder s .

5. A cross-language summarization method based on cross-attention alignment and multi-language mutual attention according to claim 4, characterized in that: In step 2, the cross-attention alignment mechanism is used to optimize the attention distribution of the cross-language summary, specifically: 2.1) Through the cross-attention mechanism, the attention distribution of the cross-language summary and the single-language summary on the source text is calculated separately: the hidden state h of the cross-language decoder is t and the hidden state h of the source language decoder s As query vectors, calculate the correlation with the source dialogue context vector H, and then obtain the cross-language decoder's cross-attention weight matrix for the original text. And the cross attention weight matrix of the single language decoder for the original text As shown in formula (1) and formula (2), in, and represent the query and key in the cross-lingual attention process, are the dimensions of query and key in the cross-lingual attention process, and Represent the query and key in the monolingual attention process, is the dimension of inquiry and key in monolingual attention processes; use and These two attention distributions perform weighted summation on the source dialogue context vector H to obtain the context attention result representation of the cross-language decoder on the original text And the single language decoder's contextual attention result representation of the original text As shown in formula (3) and formula (4): in, and Represent the value matrices in the cross-lingual attention and monolingual attention processes respectively. The two context results will participate in the process of generating cross-lingual summaries and monolingual summaries respectively; 2.2) Optimize the attention distribution of cross-lingual summaries through the cross-attention alignment mechanism: Use the cross-attention between the same-language summary encoder and decoder to guide the cross-lingual summary, as shown in formula (5): in, is the attention distribution of the source language summary decoder on the hidden state of the source language text, that is, the attention distribution of the monolingual summary, is the attention distribution of the target language summary decoder on the hidden state of the source language text, that is, the attention distribution of the cross-language summary.

6. A cross-language summarization method based on cross-attention alignment and multi-language mutual attention according to claim 5, characterized in that: In step 2, a multilingual mutual attention mechanism is used to optimize the bilingual hidden state representation, specifically: 3.1) Optimize the bilingual hidden state representation using the multilingual mutual attention mechanism: At the pth time step of the target language summary decoder, the hidden state of the target language decoder is As the query vector Q, calculate the last p hidden states of the source language decoder The correlation between them is obtained by weighting and summing these p hidden states through the multi-language mutual attention module to obtain the contextual attention result representation of the target language decoder hidden state to the source language decoder hidden state As shown in formula (6): in, refers to the hidden state of the target language summary decoder at the pth time step, refers to all hidden states of the source language summary decoder in the previous p time steps; Similarly, at the pth time step of the source language summary decoder, the hidden state of the source language decoder is As the query vector Q, calculate the last p hidden states of the target language decoder The correlation between them is calculated by weighting the p hidden states through the multi-language mutual attention module to obtain the contextual attention result representation of the source language decoder hidden state to the target language decoder hidden state. As shown in formula (7): in, refers to the hidden state of the source language summary decoder at the pth time step, refers to all hidden states of the target language summary decoder in the previous p time steps; 3.2) Using the generated context to pay attention to the result representation The resulting contextual attention representation of the target language decoder hidden state to the source language decoder hidden state Generate summary: At time step p, the probability of the next word of the cross-language summary is calculated as shown in formula (8): Among them, function F includes the fusion and splicing and The MLP layer extracts information, the feed-forward layer obtains the intermediate vector representation, the linear layer learns the complex patterns and features of the intermediate vectors, and the softmax layer predicts the probability distribution of words.

7. A cross-language summarization method based on cross-attention alignment and multi-language mutual attention according to claim 6, characterized in that: It also includes training a multi-task cross-lingual summarization model, specifically: After the single-language decoder and the cross-language decoder of multi-task learning predict their corresponding summaries, the two tasks are optimized simultaneously using the negative log-likelihood loss. The loss of a single example is: Among them, X s Represents the source document, y s Represents the source language summary, y t represents the target language summary; α is a hyperparameter used to balance the weights of the monolingual summary task and the cross-lingual summary task; represents the source language summary in the i-th time step, represents the target language summary in the i-th time step. The decoding result of the single-language decoder is the source language summary, and the decoding result of the cross-language decoder is the target language summary; In addition, the KL divergence loss is minimized to constrain the attention distribution of the cross-language summary, and the total loss is calculated as: L=βL KL +L nll (10) Among them, β is a hyperparameter used to balance the weights of different loss functions.

Images