Method and device for constructing bilateral relationship knowledge base based on multi-task deep learning

Abstract

The present application relates to a method and device for constructing a bilateral relations knowledge base based on multi-task deep learning. The method allows the four subtasks of constructing a bilateral relations knowledge base of countries to share a multi-task deep learning model, thereby realizing the joint learning of the four tasks in the same model, so that the news text can be simultaneously subjected to entity extraction, summary generation, news classification and sentiment analysis, and output complete information including bilateral countries, time, news summaries, news types, and sentiment types, and finally efficiently construct a large-scale bilateral relations knowledge base covering multiple levels of bilateral relations between countries. The adoption of this method effectively solves the problems of independent training of each subtask in the construction of the existing bilateral relations knowledge base, lack of collaborative optimization and data annotation, and high model deployment costs, and realizes the efficient construction of news text understanding and national bilateral relations database, with significant technical advantages and application value.

Description

Technical Field

[0001] The present application relates to the field of natural language processing technology, and in particular to a method and device for constructing a bilateral relationship knowledge base based on multi-task deep learning. Background Art

[0002] Building a knowledge base on bilateral relations between countries based on news text data typically requires completing the following four subtasks: extracting key entities from news text, such as country (sponsor, recipient), time, and location; compressing the original long news text into a concise summary; categorizing news text into specific fields, such as politics, economy, and military; and performing sentiment analysis on news text to determine the emotional tendency conveyed by the events, such as positive, neutral, or negative. By completing training for each subtask, predictions are made on large-scale news text, outputting information such as bilateral countries, time, news summary, news type, and sentiment, thereby constructing a knowledge base describing bilateral relations between countries.

[0003] Although the existing technologies used in each subtask have made significant progress in their respective fields, they have the following shortcomings when applied in combination to construct a knowledge base on bilateral relations between countries:

[0004] (1) Independent training of each task. In existing technologies, each task is usually trained independently, and each task needs to process the same input text separately, which results in repeated word vector representation and feature learning, wasting computing resources.

[0005] (2) Lack of collaborative optimization between tasks. These independently trained tasks cannot share information, making it difficult to utilize the potential correlation between different tasks to improve overall performance.

[0006] (3) Complex deployment and maintenance. Different tasks require the preparation of multiple data sets, as well as the training, deployment, and maintenance of multiple independent models, which increases the complexity and maintenance cost of the system. Summary of the Invention

[0007] Based on this, it is necessary to provide a method and device for constructing a bilateral relationship knowledge base based on multi-task deep learning to address the above technical problems.

[0008] A method for constructing a bilateral relationship knowledge base based on multi-task deep learning, the method comprising:

[0009] A multi-source news text dataset describing bilateral relations between countries was collected and split into training, validation, and test sets according to a set ratio. The multi-source news text dataset included news text and annotations of corresponding entities, news summaries, news types, and news sentiment types.

[0010] Construct a multi-task deep learning model, including an input layer, an embedding layer, a shared encoding layer, a task-specific layer, an output layer, and a result processing layer. The task-specific layer includes a task feature learning module, a cross-task information interaction module, and a feature fusion module. The output layer includes a news classification layer, a sentiment analysis layer, a news summary layer, and an entity extraction layer.

[0011] The training set is input into the multi-task deep learning model for training, and the validation set and test set are used to evaluate and verify the multi-task deep learning model during the training process until a well-trained multi-task deep learning model is obtained through iteration;

[0012] The news text to be processed is input into the trained multi-task deep learning model to extract bilateral relationship information, and the extracted information is stored and constructed into a bilateral relationship knowledge base; specifically, the news text to be processed is first preprocessed by inserting markers according to the input layer; then, the preprocessed news text is embedded and attention-weighted encoded in turn according to the embedding layer and the shared encoding layer to generate embedding representations for news classification, sentiment analysis, summary generation and entity extraction tasks respectively; then, the embedding representations of each task are interactively calculated according to the specific task layer to obtain the final embedding representation of each task; then, the final embedding representation of the news classification task is obtained according to the news classification layer for news type classification, the final embedding representation of the sentiment analysis task is obtained according to the sentiment analysis layer for news sentiment classification, the final embedding representation of the summary generation task is obtained according to the news summary layer for summary generation, and the final embedding representation of the entity extraction task is obtained according to the entity extraction layer for entity extraction; finally, the output results of the news classification layer, sentiment analysis layer, news summary layer and entity extraction layer are converted into natural language according to the result processing layer, and the converted information is stored and constructed into a bilateral relationship knowledge base.

[0013] In one embodiment, a training set is input into a multi-task deep learning model for training, and a validation set and a test set are used to evaluate and verify the multi-task deep learning model during the training process until a trained multi-task deep learning model is obtained through iteration, including:

[0014] The training set is fed into a multi-task deep learning model for entity extraction, summary generation, news classification, and sentiment analysis. The cross-entropy loss function is used to weight the losses of the four tasks to obtain the total loss function for model training. The AdamW optimizer is used to perform backpropagation on the multi-task deep learning model based on the total loss function to implement gradient update training.

[0015] During the training process, the validation and test sets were used to evaluate the entity extraction, summary generation, news classification, and sentiment analysis tasks of the multi-task deep learning model. A ten-fold cross-validation algorithm was used to verify the generalization performance of the evaluation model until a well-trained multi-task deep learning model was obtained through iteration.

[0016] Among them, the evaluation indicators of the entity extraction task are precision, recall and F1 score;

[0017] The evaluation metric for the summary generation task is the ROUGE score;

[0018] The evaluation indicators of news classification and sentiment analysis tasks are precision, recall and F1 score.

[0019] In one embodiment, performing marker insertion preprocessing on the news text to be processed according to the input layer includes:

[0020] The news text to be processed consists of several sentences. In the input layer, a [SEP] marker is inserted at the end of each sentence to indicate the end of the sentence; a [CLS] marker is inserted at the beginning of each sentence to indicate the embedded information of the sentence; and a [CLS] marker is inserted at the beginning of the entire news text to indicate the embedded information of the entire news text.

[0021] In one embodiment, the pre-processed news text is embedded and attention-weighted encoded in sequence according to the embedding layer and the shared encoding layer to generate embedding representations for news classification, sentiment analysis, summary generation, and entity extraction tasks, respectively, including:

[0022] The preprocessed news text is embedded in words, sentences, and positions using the embedding layer to obtain word vectors, sentence vectors, and position vectors corresponding to each word in the news text. Word vectors are used to capture the semantic relationship between words, sentence vectors are used to distinguish adjacent sentences in the news text, and position vectors are used to capture the relative position relationship between words.

[0023] According to the shared encoding layer, a pre-trained language model based on the Transformer architecture is used to perform attention-weighted encoding on word vectors, sentence vectors, and the position vector corresponding to each word to generate an embedded representation of the complete news text for news classification and sentiment analysis tasks, an embedded representation of each sentence in the news text for the summary generation task, and an embedded representation of each word in the news text for the entity extraction task. Among them, pre-trained language models based on the Transformer architecture include but are not limited to BERT, GPT, and XLNet.

[0024] In one embodiment, interactively calculating the embedding representation of each task according to a specific task layer to obtain the final embedding representation of each task includes:

[0025] According to the task feature learning module, the embedded representations of the shared encoding layer output for news classification, sentiment analysis, summary generation, and entity extraction tasks are obtained respectively. The multi-head self-attention mechanism is used to learn the feature representations related to the corresponding tasks in each embedded representation, and the updated embedded representations of each task are output. The calculation process of the task feature learning module is expressed as follows:

[0026] ;

[0027] ;

[0028] ;

[0029] ;

[0030] in, Indicates that it is used for f The embedding representation of each task, Indicates the f The updated embedding representation of each task is represent query vector, key vector and value vector respectively, 、 、 and represents the trainable parameters, Indicates the dimension of the trainable parameters, softmax represents the normalized exponential function, and the superscript T Represents the transposition operation, and Concat represents the concatenation operation of the multi-head attention matrix Attention;

[0031] According to the cross-task information interaction module, the updated embedding representation of each task is interactively calculated with the updated embedding representation of other tasks, and the updated embedding representation of each task is output; wherein, the cross-task information interaction module is composed of several layers of cross-task multi-head attention mechanism, and the calculation process of the cross-task information interaction module is expressed as

[0032] ;

[0033] ;

[0034] ;

[0035] ;

[0036] in, Indicates the fThe query vector corresponding to each task, and Respectively represent The key vector and value vector corresponding to each task, Indicates that it is used for The embedding representation of each task, Indicates the f Task and The embedded representation updated after the interaction of the tasks;

[0037] According to the feature fusion module, the outputs of the task feature learning module and the cross-task information interaction module are obtained and feature fusion is performed to obtain the final embedding representation of each task.

[0038] In one embodiment, obtaining the final embedding representation of the news classification task according to the news classification layer to classify the news type, and obtaining the final embedding representation of the sentiment analysis task according to the sentiment analysis layer to classify the news sentiment, includes:

[0039] The news classification layer and sentiment analysis layer are both composed of fully connected layers and softmax classifiers;

[0040] The news classification layer classifies the final embedded representation of the news classification task into news types and outputs the news type score of the complete news text. News types include politics, economy, military, and others.

[0041] The final embedded representation of sentiment analysis is classified into news sentiment types according to the sentiment analysis layer, and the news sentiment type score of the complete news text is output; among them, the news sentiment types include negative, neutral and positive.

[0042] In one embodiment, obtaining a final embedding representation of the summary generation task according to the news summary layer to generate a summary includes:

[0043] The news summary layer consists of a Transformer layer and a sigmoid classifier;

[0044] The final embedding representation of the summary generation task is binary classified according to the news summary layer, and the category score of each sentence in the complete news text is output; a sentence category score of 0 indicates that the sentence is not included in the news summary, and a sentence category score of 1 indicates that the sentence is included in the news summary.

[0045] In one embodiment, obtaining a final embedding representation of the entity extraction task according to the entity extraction layer to perform entity extraction includes:

[0046] The entity extraction layer consists of a fully connected layer, a conditional random field, and a softmax classifier;

[0047] The final embedding representation of the entity extraction task is classified according to the entity extraction layer, and the entity type score and corresponding entity label of each word in the complete news text are output; among them, entity types include country, person and time, and entity labels include B-CNT, I-CNT, B-PER, I-PER, B-TIM, I-TIM and O, among which CNT represents country, PER represents person, TIM represents time, B represents the beginning of the entity, I represents the continuation of the entity, and O represents non-entity.

[0048] In one embodiment, the result processing layer converts the outputs of the news classification layer, sentiment analysis layer, news summary layer, and entity extraction layer into natural language, and stores the converted information to construct a bilateral relationship knowledge base, including:

[0049] According to the result processing layer, the news type score, news sentiment type score, category score of each sentence, and entity type score of each word output by the news classification layer, sentiment analysis layer, news summary layer, and entity extraction layer are converted into natural language respectively. The bilateral country, time, news summary, news type, and news sentiment type fields in the converted information are integrated, and the bilateral relationship database is constructed and stored in the format of each field as a column.

[0050] A device for constructing a bilateral relationship knowledge base based on multi-task deep learning, the device comprising:

[0051] The data collection module is used to collect a multi-source news text dataset describing bilateral relations between countries and split it into a training set, a validation set, and a test set according to a set ratio. The multi-source news text dataset includes news text and the corresponding entities, news summaries, news types, and news sentiment types.

[0052] The model building module is used to build a multi-task deep learning model, including an input layer, an embedding layer, a shared encoding layer, a specific task layer, an output layer, and a result processing layer. The specific task layer includes a task feature learning module, a cross-task information interaction module, and a feature fusion module. The output layer includes a news classification layer, a sentiment analysis layer, a news summary layer, and an entity extraction layer.

[0053] The model training module is used to input the training set into the multi-task deep learning model for training, and use the validation set and test set to evaluate and verify the multi-task deep learning model during the training process until a well-trained multi-task deep learning model is obtained through iteration;

[0054] The knowledge base construction module is used to input the news text to be processed into the trained multi-task deep learning model to extract bilateral relationship information, and store the extracted information to construct a bilateral relationship knowledge base; specifically, first, the news text to be processed is preprocessed by inserting markers according to the input layer; then, the preprocessed news text is embedded and attention-weighted encoded in turn according to the embedding layer and the shared encoding layer to generate embedding representations for news classification, sentiment analysis, summary generation and entity extraction tasks respectively; then, the embedding representation of each task is interactively calculated according to the specific task layer to obtain the final embedding representation of each task; then, the final embedding representation of the news classification task is obtained according to the news classification layer for news type classification, the final embedding representation of the sentiment analysis task is obtained according to the sentiment analysis layer for news sentiment classification, the final embedding representation of the summary generation task is obtained according to the news summary layer for summary generation, and the final embedding representation of the entity extraction task is obtained according to the entity extraction layer for entity extraction; finally, the results output by the news classification layer, sentiment analysis layer, news summary layer and entity extraction layer are converted into natural language according to the result processing layer, and the converted information is stored to construct a bilateral relationship knowledge base.

[0055] The above-mentioned method and device for constructing a bilateral relations knowledge base based on multi-task deep learning, by sharing a multi-task deep learning model for the four subtasks of constructing a bilateral relations knowledge base, achieves joint learning of the four tasks in the same model, thereby being able to simultaneously perform entity extraction, summary generation, news classification and sentiment analysis on news texts, and output complete information including bilateral countries, time, news summaries, news types, and sentiment types, ultimately efficiently constructing a large-scale bilateral relations knowledge base covering multiple levels of bilateral relations between countries. Compared with the existing technology, this application has the following technical effects:

[0056] (1) Automatic construction of a multi-domain bilateral relationship knowledge base. This application introduces a news classification task to automatically identify the domain to which a news text belongs. As a result, the constructed bilateral relationship knowledge base covers multiple domains. Compared with existing domain-specific knowledge base construction solutions, this greatly expands the coverage and application scope of the bilateral relationship knowledge base.

[0057] (2) End-to-end multi-task joint learning. This application integrates multiple tasks such as entity extraction, summary generation, news classification, and sentiment analysis into the same deep learning model for joint training, rather than traditional independent or pipeline execution. This highly integrated end-to-end multi-task deep learning architecture not only reduces repeated calculations, but also, through personalized task feature learning, cross-task information interaction, and feature fusion, can fully explore the correlation between tasks, learn from each other, and achieve closer task collaborative learning, further improving the performance of each task, thereby improving the overall quality of the bilateral relationship knowledge base. In addition, the task-specific layer organically integrates intra-task feature learning and inter-task information interaction, achieving closer task collaborative learning, which not only helps to enhance the model's understanding and reasoning capabilities of multi-domain news texts, but also reduces the redundancy of model design and improves scalability in practical applications.

[0058] (3) Simplified data labeling and system deployment. This application defines unified data labeling standards and processes to enable simultaneous labeling of entities, news summaries, news types, and sentiment information in the same dataset, simplifying the data preparation process and improving data labeling consistency. Through a multi-task deep learning model framework, only one unified model needs to be deployed and maintained, reducing the number of independent models that need to be managed and maintained, simplifying the model system architecture, and improving the reliability and maintainability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 1 is a flowchart of a method for constructing a bilateral relationship knowledge base based on multi-task deep learning in one embodiment;

[0060] Figure 2 Schematic diagram of the structure of a multi-task deep learning model in one embodiment. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0062] In one embodiment, Figure 1 As shown, a method for constructing a bilateral relationship knowledge base based on multi-task deep learning is provided, which includes the following steps:

[0063] Step S1: Collect and obtain a multi-source news text dataset used to describe bilateral relations between countries, and split it into a training set, a validation set, and a test set according to a set ratio; wherein the multi-source news text dataset includes news text and corresponding entities, news summaries, news types, and news sentiment type annotations.

[0064] Specifically, a multi-source news text dataset describing bilateral relations between countries was collected from various news websites. 3,000 (or more) news texts were randomly selected from the dataset, and after annotating entities, news summaries, news types, and news sentiment types, they were divided into training set, validation set, and test set in a ratio of 7:2:1.

[0065] Step S2, constructing a multi-task deep learning model, including an input layer, an embedding layer, a shared encoding layer, a specific task layer, an output layer and a result processing layer; wherein the specific task layer includes a task feature learning module, a cross-task information interaction module and a feature fusion module; the output layer includes a news classification layer, a sentiment analysis layer, a news summary layer and an entity extraction layer.

[0066] Specifically, multi-task deep learning models such as Figure 2 This model uses a pre-trained language model based on the Transformer architecture (BERT in this example) as the base model for processing text feature representation.

[0067] In step S3, the training set is input into the multi-task deep learning model for training, and the validation set and test set are used to evaluate and verify the multi-task deep learning model during the training process until a trained multi-task deep learning model is obtained through iteration.

[0068] Specifically, the training set was fed into a multi-task deep learning model for entity extraction, summary generation, news classification, and sentiment analysis. The losses of these four tasks were weighted using the cross-entropy loss function to obtain the total loss function for model training. The AdamW optimizer was then used to perform backpropagation on the multi-task deep learning model based on this total loss function to implement gradient update training. By setting weights, the loss weights for each task can be dynamically adjusted, ensuring a balance between tasks and improving the model's overall performance when handling complex multi-task scenarios.

[0069] Among them, the total loss function L Expressed as

[0070] ;

[0071] ;

[0072] Where, 、 、 and Represent the weights of entity extraction, summary generation, news classification, and sentiment analysis tasks respectively, and their values ​​are calculated based on the GradNorm (gradient normalization) algorithm; 、 、 and They represent the loss functions for entity extraction, summary generation, news classification, and sentiment analysis tasks, respectively, and all use the cross entropy loss function; N is the number of categories, Indicates that the sample belongs to the category i The probability of (between 0 and 1); is a symbolic function, if the true category of the sample is equal to i Takes 1, otherwise takes 0.

[0073] During the training process, the validation set and test set are used to evaluate the entity extraction, summary generation, news classification, and sentiment analysis tasks of the multi-task deep learning model respectively, and the ten-fold cross-validation algorithm is used to verify the generalization performance of the evaluation model until a well-trained multi-task deep learning model is obtained through iteration.

[0074] Among them, the evaluation indicators of the entity extraction task are precision, recall and F1 score;

[0075] The evaluation indicator of the summary generation task is the ROUGE score, which is used to measure the word overlap rate between two word sequences;

[0076] The evaluation indicators of news classification and sentiment analysis tasks are precision, recall and F1 score.

[0077] In step S4, the news text to be processed is input into the trained multi-task deep learning model to extract bilateral relationship information, and the extracted information is stored and constructed into a bilateral relationship knowledge base.

[0078] Specifically, if Figure 2 As shown, step S4 includes the following steps:

[0079] Step S4.1, perform marker insertion preprocessing on the news text to be processed according to the input layer.

[0080] Specifically, the news text to be processed consists of several sentences, represented as [sent1, sent2, ..., sent m ], among which sent m represents the mth sentence in the news text;

[0081] In the input layer, a [SEP] tag is inserted at the end of each sentence to indicate the end of the sentence; a [CLS] tag is inserted at the beginning of each sentence to indicate the embedded information of the sentence; and a [CLS] tag is inserted at the beginning of the entire news text to indicate the embedded information of the entire news text.

[0082] In step S4.2, the preprocessed news text is subjected to word embedding, sentence embedding, and position embedding according to the embedding layer to obtain the word vector, sentence vector, and position vector corresponding to each word in the news text.

[0083] Among them, the main function of word embedding is to map words or phrases from the vocabulary into continuous vector representations. Used to capture semantic relationships between words.

[0084] The main function of sentence embedding is to distinguish different sentence structures. It is used to distinguish adjacent sentences in the news text, that is, to determine the sentence vector E according to whether it is an odd sentence or an even sentence. A or E B .

[0085] Position embedding is used to encode the position information of each word into a vector, position vector Used to capture the relative position relationship between words.

[0086] Step S4.3: Based on the shared encoding layer, a pre-trained language model based on the Transformer architecture is used to perform attention-weighted encoding on the word vector, sentence vector, and position vector corresponding to each word to generate an embedded representation of the complete news text for news classification and sentiment analysis tasks (see Figure 2 T-shirt with light grey background [CLS] ), the embedding representation of each sentence in the news text for the summary generation task (see Figure 2 T with blank background [CLS] ), and the embedding representation of each word in the news text for entity extraction tasks (see Figure 2 Middle T sent 、T [SEP] wait).

[0087] Among them, pre-trained language models based on the Transformer architecture include but are not limited to BERT, GPT, and XLNet. In this embodiment, the shared encoding layer uses the BERT model, which is based on a multi-head self-attention mechanism and assigns different attention weights to different parts of the input text sequence to better capture semantic relationships. The multi-head self-attention mechanism is expressed as

[0088] ;

[0089] ;

[0090] ;

[0091] ;

[0092] in, is the embedding representation of the input text sequence, , and represent query vector, key vector and value vector respectively, , 、 and represents the trainable parameters, represents the normalized exponential function, the superscript T represents the transpose operation, is the dimension of the trainable parameters, Concat represents the concatenation operation of the multi-head attention matrix Attention, H Represents the new embedding representation after weighted summation.

[0093] In step S4.4, the embedding representation of each task is interactively calculated according to the specific task layer to obtain the final embedding representation of each task.

[0094] Specifically, first, the task feature learning module obtains the embedded representations of the shared encoding layer output for news classification, sentiment analysis, summary generation, and entity extraction tasks, and uses the multi-head self-attention mechanism to learn the feature representations related to the corresponding tasks in each embedded representation, and outputs the updated embedded representations of each task; the calculation process of the task feature learning module is expressed as follows:

[0095] ;

[0096] ;

[0097] ;

[0098] ;

[0099] in, Indicates that it is used for f The embedding representation of each task, Indicates the f The updated embedding representation of each task is represent query vector, key vector and value vector respectively, 、 、 and represents the trainable parameters, Indicates the dimension of the trainable parameters, softmax represents the normalized exponential function, and the superscript T Represents the transposition operation, and Concat represents the concatenation operation of the multi-head attention matrix Attention.

[0100] Then, according to the cross-task information interaction module, the updated embedding representation of each task is interactively calculated with the updated embedding representation of other tasks, and the updated embedding representation of each task is output; wherein, the cross-task information interaction module is composed of several layers of cross-task multi-head attention mechanism, and the calculation process of the cross-task information interaction module is expressed as

[0101] ;

[0102] ;

[0103] ;

[0104] ;

[0105] in, Indicates the f The query vector corresponding to each task, and Respectively represent The key vector and value vector corresponding to each task, Indicates that it is used for The embedding representation of each task, Indicates the f Task and The updated embedding representation of the task interactions.

[0106] Through the cross-task information interaction module, information transmission and fusion between tasks can be achieved.

[0107] Finally, the feature fusion module obtains the output of the task feature learning module and the cross-task information interaction module and performs feature fusion to obtain the final embedding representation of each task. Feature fusion methods include splicing, averaging, or attention mechanism. When splicing is selected, the expression of the output final embedding representation is

[0108] ;

[0109] in, Indicates the f The final embedding representation of the task is and splicing.

[0110] In step S4.5, the final embedding representation of the news classification task is obtained according to the news classification layer to perform news type classification, the final embedding representation of the sentiment analysis task is obtained according to the sentiment analysis layer to perform news sentiment classification, the final embedding representation of the summary generation task is obtained according to the news summary layer to perform summary generation, and the final embedding representation of the entity extraction task is obtained according to the entity extraction layer to perform entity extraction.

[0111] Specifically, the news classification layer obtains the final embedding representation of the news classification task to classify news types, and the sentiment analysis layer obtains the final embedding representation of the sentiment analysis task to classify news sentiment, including:

[0112] The news classification layer and sentiment analysis layer are both composed of fully connected layers and softmax classifiers;

[0113] The news classification layer classifies the final embedded representation of the news classification task into news types, and outputs the news type score of the complete news text; among which, news types include politics, economy, military, and others; the sentiment analysis layer classifies the final embedded representation of sentiment analysis into news sentiment types, and outputs the news sentiment type score of the complete news text; among which, news sentiment types include negative, neutral, and positive. The calculation formula of the softmax classifier is expressed as

[0114] ;

[0115] in, represents the final embedding representation.

[0116] Specifically, the final embedding representation of the summary generation task is obtained according to the news summary layer to generate a summary, including:

[0117] The news summary layer consists of a Transformer layer and a sigmoid (S-shaped growth curve) classifier. The news summary layer performs binary classification on the final embedding representation of the summary generation task and outputs the category score of each sentence in the complete news text. A sentence category score of 0 indicates that the sentence is not included in the news summary, and a sentence category score of 1 indicates that the sentence is included in the news summary. The calculation formula of the sigmoid classifier is expressed as follows:

[0118] .

[0119] Specifically, the final embedding representation of the entity extraction task is obtained according to the entity extraction layer to perform entity extraction, including:

[0120] The entity extraction layer consists of a fully connected layer, a conditional random field, and a softmax classifier. The entity extraction layer performs entity classification on the final embedding representation of the entity extraction task, and outputs the entity type score and corresponding entity label of each word in the complete news text. Among them, entity types include country, person, and time, and entity labels include B-CNT, I-CNT, B-PER, I-PER, B-TIM, I-TIM, and O. Among them, CNT represents country, PER represents person, TIM represents time, B represents the beginning of the entity, I represents the continuation of the entity, and O represents non-entity.

[0121] Step S4.6, according to the result processing layer, the news type score, news sentiment type score, category score of each sentence and entity type score of each word output by the news classification layer, sentiment analysis layer, news summary layer and entity extraction layer are converted into natural language respectively, and the bilateral country, time, news summary, news type and news sentiment type fields in the converted information are integrated, and each field is constructed and stored as a bilateral relationship database in the format of a column.

[0122] In summary, this application effectively solves the problems of independent training of each subtask, lack of collaborative optimization, data annotation and high model deployment costs in the construction of the existing bilateral relations knowledge base by introducing a multi-task deep learning model, and realizes the efficient construction of news text understanding and national bilateral relations database, with significant technical advantages and application value.

[0123] In one embodiment, a device for constructing a bilateral relationship knowledge base based on multi-task deep learning is provided, comprising:

[0124] The data collection module is used to collect a multi-source news text dataset describing bilateral relations between countries and split it into a training set, a validation set, and a test set according to a set ratio. The multi-source news text dataset includes news text and the corresponding entities, news summaries, news types, and news sentiment types.

[0125] The model building module is used to build a multi-task deep learning model, including an input layer, an embedding layer, a shared encoding layer, a specific task layer, an output layer, and a result processing layer. The specific task layer includes a task feature learning module, a cross-task information interaction module, and a feature fusion module. The output layer includes a news classification layer, a sentiment analysis layer, a news summary layer, and an entity extraction layer.

[0126] The model training module is used to input the training set into the multi-task deep learning model for training, and use the validation set and test set to evaluate and verify the multi-task deep learning model during the training process until a well-trained multi-task deep learning model is obtained through iteration;

[0127] The knowledge base construction module is used to input the news text to be processed into the trained multi-task deep learning model to extract bilateral relationship information, and store the extracted information to construct a bilateral relationship knowledge base; specifically, first, the news text to be processed is preprocessed by inserting markers according to the input layer; then, the preprocessed news text is embedded and attention-weighted encoded in turn according to the embedding layer and the shared encoding layer to generate embedding representations for news classification, sentiment analysis, summary generation and entity extraction tasks respectively; then, the embedding representation of each task is interactively calculated according to the specific task layer to obtain the final embedding representation of each task; then, the final embedding representation of the news classification task is obtained according to the news classification layer for news type classification, the final embedding representation of the sentiment analysis task is obtained according to the sentiment analysis layer for news sentiment classification, the final embedding representation of the summary generation task is obtained according to the news summary layer for summary generation, and the final embedding representation of the entity extraction task is obtained according to the entity extraction layer for entity extraction; finally, the results output by the news classification layer, sentiment analysis layer, news summary layer and entity extraction layer are converted into natural language according to the result processing layer, and the converted information is stored to construct a bilateral relationship knowledge base.

[0128] Regarding the specific limitations of the bilateral relationship knowledge base construction device based on multi-task deep learning, please refer to the limitations of the bilateral relationship knowledge base construction method based on multi-task deep learning above, which will not be repeated here. The various modules in the above-mentioned bilateral relationship knowledge base construction device based on multi-task deep learning can be implemented in whole or in part by software, hardware, and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0129] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0130] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A method for constructing a bilateral relationship knowledge base based on multi-task deep learning, characterized in that: The method comprises: A multi-source news text dataset describing bilateral relations between countries is collected and split into a training set, a validation set, and a test set according to a set ratio; wherein the multi-source news text dataset includes news text and corresponding entities, news summaries, news types, and news sentiment types. Construct a multi-task deep learning model, including an input layer, an embedding layer, a shared encoding layer, a specific task layer, an output layer, and a result processing layer; wherein the specific task layer includes a task feature learning module, a cross-task information interaction module, and a feature fusion module; the output layer includes a news classification layer, a sentiment analysis layer, a news summary layer, and an entity extraction layer; Input the training set into the multi-task deep learning model for training, and use the validation set and the test set to evaluate and verify the multi-task deep learning model during the training process until a trained multi-task deep learning model is obtained through iteration; The news text to be processed is input into the trained multi-task deep learning model to extract bilateral relationship information, and the extracted information is stored and constructed into a bilateral relationship knowledge base; specifically, the news text to be processed is first preprocessed by inserting markers according to the input layer; then, the preprocessed news text is embedded and attention-weighted encoded in sequence according to the embedding layer and the shared encoding layer to generate embedding representations for news classification, sentiment analysis, summary generation and entity extraction tasks respectively; then, the embedding representations of each task are interactively calculated according to the specific task layer to obtain the final embedding representation of each task; thereafter, the final embedding representation of the news classification task is obtained according to the news classification layer to classify the news type, the final embedding representation of the sentiment analysis task is obtained according to the sentiment analysis layer to classify the news sentiment, the final embedding representation of the summary generation task is obtained according to the news summary layer to generate the summary, and the final embedding representation of the entity extraction task is obtained according to the entity extraction layer to extract the entity; finally, the results output by the news classification layer, the sentiment analysis layer, the news summary layer and the entity extraction layer are converted into natural language according to the result processing layer, and the converted information is stored and constructed into a bilateral relationship knowledge base.

2. The method according to claim 1, characterized in that Inputting the training set into the multi-task deep learning model for training, and evaluating and verifying the multi-task deep learning model using the validation set and the test set during the training process until a trained multi-task deep learning model is obtained through iteration, including: The training set is input into the multi-task deep learning model to perform entity extraction, summary generation, news classification, and sentiment analysis respectively. The losses of the four tasks are weighted using the cross-entropy loss function to obtain the total loss function of the model training. The AdamW optimizer is used to perform backpropagation on the multi-task deep learning model based on the total loss function to achieve gradient update training; During the training process, the validation set and test set are used to evaluate the entity extraction, summary generation, news classification, and sentiment analysis tasks of the multi-task deep learning model, and a ten-fold cross-validation algorithm is used to verify the generalization performance of the evaluation model until a well-trained multi-task deep learning model is obtained through iteration. Among them, the evaluation indicators of the entity extraction task are precision, recall and F1 score; The evaluation metric for the summary generation task is the ROUGE score; The evaluation indicators of news classification and sentiment analysis tasks are precision, recall and F1 score.

3. The method according to claim 1, characterized in that Performing marker insertion preprocessing on the news text to be processed according to the input layer includes: The news text to be processed consists of several sentences. In the input layer, a [SEP] marker is inserted at the end of each sentence to indicate the end of the sentence; a [CLS] marker is inserted at the beginning of each sentence to indicate the embedded information of the sentence; and a [CLS] marker is inserted at the beginning of the entire news text to indicate the embedded information of the entire news text.

4. The method according to claim 1, wherein The pre-processed news text is embedded and attention-weighted encoded in sequence according to the embedding layer and shared encoding layer to generate embedding representations for news classification, sentiment analysis, summary generation, and entity extraction tasks, including: The pre-processed news text is subjected to word embedding, sentence embedding, and position embedding according to the embedding layer, thereby obtaining word vectors, sentence vectors, and position vectors corresponding to each word in the news text. The word vectors are used to capture the semantic relationship between words, the sentence vectors are used to distinguish adjacent sentences in the news text, and the position vectors are used to capture the relative position relationship between words. According to the shared coding layer, a pre-trained language model based on the Transformer architecture is used to perform attention-weighted encoding on the word vector, sentence vector, and the position vector corresponding to each word to generate an embedded representation of the complete news text for news classification and sentiment analysis tasks, an embedded representation of each sentence in the news text for the summary generation task, and an embedded representation of each word in the news text for the entity extraction task; wherein, the pre-trained language model based on the Transformer architecture includes but is not limited to BERT, GPT, and XLNet.

5. The method according to claim 4, wherein The embedding representations of each task are interactively calculated according to the specific task layer to obtain the final embedding representation of each task, including: According to the task feature learning module, the embedded representations output by the shared encoding layer for news classification, sentiment analysis, summary generation, and entity extraction tasks are obtained, and the multi-head self-attention mechanism is used to learn the feature representations related to the corresponding tasks in each embedded representation, and the updated embedded representations of each task are output; wherein, the calculation process of the task feature learning module is expressed as follows ； ； ； ； in, Indicates that it is used for f The embedding representation of each task, Indicates the f The updated embedding representation of each task is represent query vector, key vector and value vector respectively, 、 、 and represents the trainable parameters, Indicates the dimension of the trainable parameters, softmax represents the normalized exponential function, and the superscript T Represents the transposition operation, and Concat represents the concatenation operation of the multi-head attention matrix Attention; According to the cross-task information interaction module, the updated embedding representation of each task is interactively calculated with the updated embedding representation of other tasks, and the updated embedding representation of each task is output; wherein, the cross-task information interaction module is composed of several layers of cross-task multi-head attention mechanism, and the calculation process of the cross-task information interaction module is expressed as ； ； ； ； in, Indicates the f The query vector corresponding to each task, and Respectively represent The key vector and value vector corresponding to each task, Indicates that it is used for The embedding representation of each task, Indicates the f Task and The embedded representation updated after the interaction of the tasks; The feature fusion module obtains the outputs of the task feature learning module and the cross-task information interaction module and performs feature fusion to obtain the final embedding representation of each task.

6. The method according to claim 1, characterized in that The final embedding representation of the news classification task is obtained according to the news classification layer to classify news types. The final embedding representation of the sentiment analysis task is obtained according to the sentiment analysis layer to classify news sentiments, including: The news classification layer and sentiment analysis layer are both composed of a fully connected layer and a softmax classifier; Classify the final embedded representation of the news classification task into news types according to the news classification layer, and output a news type score for the complete news text; where news types include politics, economy, military, and others; The final embedded representation of the sentiment analysis is classified into news sentiment types according to the sentiment analysis layer, and a news sentiment type score of the complete news text is output; wherein the news sentiment types include negative, neutral and positive.

7. The method according to claim 1, characterized in that The final embedding representation of the summary generation task is obtained according to the news summary layer to generate a summary, including: The news summary layer consists of a Transformer layer and a sigmoid classifier; The final embedding representation of the summary generation task is binary classified according to the news summary layer, and the category score of each sentence in the complete news text is output; a sentence category score of 0 indicates that the sentence is not included in the news summary, and a sentence category score of 1 indicates that the sentence is included in the news summary.

8. The method according to claim 1, characterized in that The final embedding representation of the entity extraction task is obtained according to the entity extraction layer to perform entity extraction, including: The entity extraction layer consists of a fully connected layer, a conditional random field and a softmax classifier; The final embedding representation of the entity extraction task is classified according to the entity extraction layer, and the entity type score and corresponding entity label of each word in the complete news text are output; among them, entity types include country, person and time, and entity labels include B-CNT, I-CNT, B-PER, I-PER, B-TIM, I-TIM and O, among which CNT represents country, PER represents person, TIM represents time, B represents the beginning of the entity, I represents the continuation of the entity, and O represents non-entity.

9. The method according to claim 1, characterized in that The result processing layer converts the output of the news classification layer, sentiment analysis layer, news summary layer, and entity extraction layer into natural language, and stores the converted information into a bilateral relationship knowledge base, including: According to the result processing layer, the news type score, news sentiment type score, category score of each sentence, and entity type score of each word output by the news classification layer, sentiment analysis layer, news summary layer, and entity extraction layer are converted into natural language respectively. The bilateral country, time, news summary, news type, and news sentiment type fields in the converted information are integrated, and the bilateral relationship database is constructed and stored in the format of each field as a column.

10. A device for constructing a bilateral relationship knowledge base based on multi-task deep learning, characterized in that: The device comprises: A data collection module is used to collect a multi-source news text dataset describing bilateral relations between countries and split it into a training set, a validation set, and a test set according to a set ratio; wherein the multi-source news text dataset includes news text and corresponding entities, news summaries, news types, and news sentiment types. A model building module, for building a multi-task deep learning model, including an input layer, an embedding layer, a shared encoding layer, a specific task layer, an output layer, and a result processing layer; wherein the specific task layer includes a task feature learning module, a cross-task information interaction module, and a feature fusion module; the output layer includes a news classification layer, a sentiment analysis layer, a news summary layer, and an entity extraction layer; A model training module is used to input the training set into the multi-task deep learning model for training, and use the validation set and test set to evaluate and verify the multi-task deep learning model during the training process until a trained multi-task deep learning model is obtained through iteration; The knowledge base construction module is used to input the news text to be processed into the trained multi-task deep learning model to extract bilateral relationship information, and store the extracted information to construct a bilateral relationship knowledge base; specifically, first, the news text to be processed is preprocessed by inserting markers according to the input layer; then, the preprocessed news text is embedded and attention-weighted encoded in sequence according to the embedding layer and the shared encoding layer to generate embedding representations for news classification, sentiment analysis, summary generation and entity extraction tasks respectively; then, the embedding representation of each task is interactively calculated according to the specific task layer to obtain the final embedding representation of each task; thereafter, the final embedding representation of the news classification task is obtained according to the news classification layer to classify the news type, the final embedding representation of the sentiment analysis task is obtained according to the sentiment analysis layer to classify the news sentiment, the final embedding representation of the summary generation task is obtained according to the news summary layer to generate the summary, and the final embedding representation of the entity extraction task is obtained according to the entity extraction layer to extract the entity; finally, the results output by the news classification layer, the sentiment analysis layer, the news summary layer and the entity extraction layer are converted into natural language according to the result processing layer, and the converted information is stored to construct a bilateral relationship knowledge base.

Images