Model training for event extraction based on continual learning, method for event extraction

Abstract

The present disclosure provides a model training method and device for event extraction based on continuous learning, an event extraction method and device, and equipment, and belongs to the technical field of natural language processing. The method comprises: obtaining a text stream data set; inputting the text stream data set into a pre-trained language model and a self-attention mechanism for processing to obtain a first fusion feature vector corresponding to the text of the text stream data set; using a knowledge distillation method, training a student network and a teacher network of an event extraction model based on continuous learning to be trained using the first fusion feature vector to obtain the event extraction model based on continuous learning, realizing knowledge transfer learning through a knowledge distillation framework, using a historical enhancement feature transfer network to enable the event extraction model to learn new event knowledge while not forgetting the historical event knowledge that has been learned, to detect new event types while still being able to detect historical event types, and to realize continuous event learning and detection.

Description

TECHNICAL FIELD

[0001] The present disclosure belongs to the technical field of natural language processing, and particularly relates to a model training method and an event extraction method based on continuous learning. BACKGROUND

[0002] With the development of the Internet in the big data era, emerging media such as social media, new media and self-media have developed rapidly, generating a large amount of text data. News is more commonly carried by text, and text is a relatively easy way to obtain valuable information. News sources are relatively diverse, and news information is generated in various ways, resulting in disorganized formats and information contained in news texts. The amount of news information generated is relatively large, and it is difficult to rely entirely on manual work to detect news events.

[0003] Although the event detection model can solve certain problems for dynamically added event types, it also has certain disadvantages, for example: the newly added text data and historical data need to be integrated, and the model needs to be retrained, but this method requires a long time and occupies a lot of resources, and cannot correctly detect and classify newly added events to be extracted.

[0004] Therefore, training an event extraction model with continuous learning ability to realize the ability to continuously extract new events has important research and application value. SUMMARY

[0005] In view of the above problems, the present disclosure provides a model training method and an event extraction method based on continuous learning.

[0006] According to a first aspect of the present disclosure, a model training method based on continuous learning of event extraction is provided, comprising:

[0007] Obtaining a text stream data set, the text stream data set including historical text stream data already memorized by the 0th to the (t-1)th model and newly added text stream data to be identified by the tth to the nth model, wherein t≥1, n≥t;

[0008] Inputting the text stream data set into a pre-trained language model and a self-attention mechanism for processing to obtain a first fusion feature vector corresponding to the text of the text stream data set;

[0009] Using a knowledge distillation method, training a student network and a teacher network of the event extraction model based on continuous learning to be trained using the first fusion feature vector to obtain the event extraction model based on continuous learning, comprising:

[0010] input the tth first fusion feature vector into a student network of the event extraction model based on the lifelong learning to be trained to obtain a t-1th first feature vector;

[0011] map the tth first feature vector to the t-1th first feature vector through the history enhanced feature transfer network to obtain a t-1th second feature vector;

[0012] input the first t-1 first fusion feature vectors into a teacher network of the event extraction model based on the lifelong learning to be trained to obtain first t-1 third feature vectors;

[0013] obtain a first loss value according to the tth first feature vector and the labeled text stream data;

[0014] obtain a second loss value according to the t-1th second feature vector and the first t-1 third feature vectors;

[0015] add the first loss value and the second loss value to obtain a total loss value;

[0016] train the student network and the teacher network of the event extraction model based on the lifelong learning to be trained by using the total loss value to obtain the event extraction model based on the lifelong learning.

[0017] According to an embodiment of the present disclosure, a training stream data set is obtained, comprising:

[0018] obtain text stream data;

[0019] clean the obtained text stream data to obtain unlabeled text stream data;

[0020] label the unlabeled text stream data to obtain labeled text stream data;

[0021] fuse the labeled text stream data and the unlabeled text stream data to obtain a text stream data set.

[0022] According to an embodiment of the present disclosure, inputting the text stream data set into a pre-trained language model and a self-attention mechanism to obtain a first fusion feature vector corresponding to a text in the text stream data set comprises:

[0023] input each text stream data in the text stream data set into the pre-trained language model to output a text stream feature vector corresponding to the text stream data, and obtain n text stream feature vectors, each of which includes a first feature sub-vector corresponding to different clauses;

[0024] for each text stream feature vector, fuse the first feature sub-vector in the text stream feature vector by using the self-attention mechanism to obtain a first fusion feature vector corresponding to a text in the text stream data set.

[0025] According to an embodiment of the present disclosure, the history-enhanced feature transfer network is composed of a fully connected layer neural network and a nonlinear activation layer.

[0026] According to an embodiment of the present disclosure, the model training method for event extraction based on continuous learning further comprises:

[0027] The event extraction model based on continuous learning is updated after training, and the updating comprises:

[0028] The student network of the event extraction model based on continuous learning after training is taken as a teacher network, the (t+1)th first fusion feature vector is input into the student network of the event extraction model based on continuous learning to be trained for a new round of event extraction model training based on continuous learning, and an updated event extraction model based on continuous learning is obtained.

[0029] A second aspect of the present disclosure provides an event extraction method based on continuous learning, comprising:

[0030] Obtaining text stream data to be extracted, wherein the text stream data to be extracted is new text data to be identified;

[0031] Inputting the text stream data to be extracted into a pre-trained language model and a self-attention mechanism to obtain a second fusion feature vector corresponding to the text to be extracted in the text stream data to be extracted;

[0032] Inputting the second fusion feature vector into the event extraction model based on continuous learning to obtain an event extraction result, wherein the event extraction model based on continuous learning is trained by the model training method for event extraction based on continuous learning in the above embodiments;

[0033] Inputting the event extraction result into a sequence classification model to obtain an event extraction category;

[0034] The inputting the text stream data to be extracted into the pre-trained language model and the self-attention mechanism to obtain the second fusion feature vector corresponding to the text to be extracted in the text stream data to be extracted comprises:

[0035] Inputting the text stream data to be extracted into the pre-trained language model to obtain a text stream feature vector of the text to be extracted, the text stream feature vector comprising second feature sub-vectors corresponding to different clauses respectively;

[0036] For the text stream feature vector to be extracted, the second feature sub-vectors in the text stream feature vector to be extracted are fused by using the self-attention mechanism to obtain the second fusion feature vector corresponding to the text to be extracted in the text stream data to be extracted.

[0037] According to an embodiment of the present disclosure, the text stream data to be extracted is obtained by the following way:

[0038] acquire text stream data to be extracted;

[0039] clean the acquired text stream data to be extracted to obtain untagged text stream data to be extracted;

[0040] label the untagged text stream data to be extracted to obtain tagged text stream data to be extracted,

[0041] fuse the tagged text stream data to be extracted and the untagged text stream data to be extracted to obtain text stream data to be extracted.

[0042] In a third aspect, the present disclosure provides a model training device for event extraction based on continuous learning, comprising:

[0043] a first acquisition module configured to acquire a text stream data set, the text stream data set comprising historical stream data memorized by a 0th to (t-1)th model and new stream data to be identified by a tth to nth model, wherein t is greater than or equal to 1, and n is greater than or equal to t;

[0044] a first processing module configured to input the text stream data set into a pre-trained language model and a self-attention mechanism for processing to obtain a first fusion feature vector corresponding to the text of the text stream data set;

[0045] a model training module configured to train a student network and a teacher network of a continuous learning-based event extraction model to be trained using the first fusion feature vector in a knowledge distillation manner to obtain the continuous learning-based event extraction model, comprising:

[0046] inputting a tth first fusion feature vector into the student network of the continuous learning-based event extraction model to be trained to obtain a tth first feature vector;

[0047] mapping the tth first feature vector to a (t-1)th first feature vector through a historical enhancement feature transfer network to obtain a (t-1)th second feature vector;

[0048] inputting the first t-1 first fusion feature vectors into the teacher network of the continuous learning-based event extraction model to be trained to obtain the first t-1 third feature vectors;

[0049] obtaining a first loss value according to the tth first feature vector and the tagged text stream data;

[0050] obtaining a second loss value according to the (t-1)th second feature vector and the first t-1 third feature vectors;

[0051] adding the first loss value and the second loss value to obtain a total loss value;

[0052] The student network and the teacher network of the event extraction model based on continuous learning to be trained are trained by using the total loss value, so as to obtain the event extraction model based on continuous learning.

[0053] In a fourth aspect of the present disclosure, an event extraction device based on continuous learning is provided, and the device comprises:

[0054] A second acquisition module is configured to acquire text stream data to be extracted, wherein the text stream data to be extracted is new text data to be identified.

[0055] A second processing module is configured to input the text stream data to be extracted into a pre-trained language model and a self-attention mechanism, so as to obtain a second fusion feature vector corresponding to text to be extracted in the text stream data to be extracted.

[0056] An event extraction module is configured to input the second fusion feature vector into an event extraction model based on continuous learning, so as to obtain an event extraction result, wherein the event extraction model based on continuous learning is trained by using the event extraction model training method based on continuous learning in the above embodiments.

[0057] A classification module is configured to input the event extraction result into a sequence classification model, so as to obtain an event extraction category.

[0058] In a fifth aspect of the present disclosure, an electronic device is provided, and the device comprises:

[0059] One or more processors;

[0060] A storage device is configured to store one or more programs,

[0061] When the one or more programs are executed by the one or more processors, the one or more processors are caused to execute the event extraction model training method based on continuous learning and the event extraction method described above.

[0062] In a sixth aspect of the present disclosure, a computer readable storage medium is provided, and the medium stores executable instructions, which are executed by a processor to cause the processor to execute the event extraction model based on continuous learning and the event extraction method described above.

[0063] According to an embodiment of the present disclosure, a text stream data set is obtained, each text stream data in the text stream data set is input into a pre-trained language model and a self-attention mechanism for processing, the text stream data is converted into a text stream feature vector by the pre-trained language model, wherein n text stream data correspond to n text stream feature vectors, and n > t. Each text stream feature vector includes a first feature vector corresponding to each sub-clause, and then the self-attention mechanism is used to fuse the first feature sub-vector in the text stream feature vector, thereby constructing the context relationship between each sub-clause and other sub-clauses in each text stream, and obtaining a first fused feature vector corresponding to the text of the text stream data set.

[0064] Then, in a knowledge distillation manner, the first fused feature vector is used to learn a student network and a teacher network in a to-be-trained event extraction model based on continuous learning. The tthfirst fused feature vector is input into the student network of the to-be-trained event extraction model based on continuous learning, the student network is used to learn the newly added stream data, and the tthfirst feature vector is obtained. The first fused feature vectors before the t-1thare input into the teacher network of the to-be-trained event extraction model based on continuous learning, the teacher network is used to learn the first t-1 data, and the first t-1 third feature vectors are obtained. The tthfirst feature vector is mapped to the t-1thfirst feature vector through a historical enhanced feature transfer network, that is, the historical enhanced feature transfer network is used to map the newly added text stream feature vector to be identified to the feature dimension of the historical text stream feature vector, and the t-1thsecond feature vector is obtained. The historical enhanced feature transfer network can retain and adjust the feature distribution of the newly added text stream data and the historical text stream data by reconstructing the features in the historical text stream data. The feature space of the reconstructed historical text stream data is responsible for saving the historical feature data, and the feature space of the newly added text stream data is only focused on learning the feature classification of the newly added text stream data, so as to realize that the model learns the features in the newly added text stream data while not forgetting the features in the historical text stream data.

[0065] Finally, a first loss value is obtained according to the tthfirst feature vector and the labeled text stream data, a second loss value is obtained according to the t-1thsecond feature vector and the first t-1 third feature vectors, the first loss value and the second loss value are added to obtain a total loss value, and the student network and the teacher network of the to-be-trained event extraction model based on continuous learning are trained and optimized by using the total loss value, thereby obtaining the event extraction model based on continuous learning. The model has good memory ability and can retain the feature distribution information in the historical text stream data after each training of the feature information in the newly added text stream data. BRIEF DESCRIPTION OF DRAWINGS

[0066] The above and other objects, features and advantages of the present disclosure will become more apparent from the following description when taken in conjunction with the accompanying drawings, in which:

[0067] Figure 1 A detection schematic diagram of a conventional event extraction model is schematically shown;

[0068] Figure 2 A system architecture diagram of a model training method, an event extraction method and device based on continual learning of event extraction according to embodiments of the present disclosure is schematically shown;

[0069] Figure 3 A flowchart of a model training method based on continual learning of event extraction according to embodiments of the present disclosure is schematically shown;

[0070] Figure 4 An architecture schematic diagram of a feature transfer network in a model training method based on continual learning of event extraction according to embodiments of the present disclosure is schematically shown;

[0071] Figure 5 A model architecture diagram of a model training method based on continual learning of event extraction according to embodiments of the present disclosure is schematically shown;

[0072] Figure 6 A flowchart of a model training method based on continual learning of event extraction according to another embodiment of the present disclosure is schematically shown;

[0073] Figure 7 A flowchart of an event extraction method based on continual learning according to embodiments of the present disclosure is schematically shown;

[0074] Figure 8 A structural block diagram of a model training device based on continual learning of event extraction according to embodiments of the present disclosure is schematically shown;

[0075] Figure 9 A structural block diagram of a device based on continual learning of event extraction according to embodiments of the present disclosure is schematically shown; and

[0076] Figure 10 A block diagram of an electronic device suitable for implementing a model and an event extraction method based on continual learning of event extraction according to embodiments of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0077] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood, however, that the description is merely exemplary and is not intended to limit the scope of the present disclosure. In the following detailed description of the embodiments, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present disclosure. However, it would be apparent to those skilled in the art that the embodiments, or one or more of the embodiments, can be practiced without these specific details. In other instances, well-known structures and

[0078] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present disclosure. As used herein, the term "including" "comprising" and the like are meant to be inclusive, but not limiting to the components, steps, operations and / or features that were listed.

[0079] All terms used herein (including technical and scientific terms) have the meanings commonly understood by one of ordinary skill in the art unless otherwise defined. It should be noted that the terms used herein are to be interpreted as having a meaning that is consistent with the understanding of a person of ordinary skill in the art, and should not be interpreted in an idealized or overly formal manner.

[0080] In the case of using expressions similar to "at least one of A, B, and C, etc.", it should generally be interpreted to include any of one, all, or a combination thereof. For example, "a system having at least one of A, B, and C" should be interpreted to include a system having A alone, a system having B alone, a system having C alone, a system having A and B together, a system having A and C together, a system having B and C together, and / or a system having A, B, and C together, etc.

[0081] Event extraction includes event type identification and event argument extraction, event type extraction refers to extracting the event trigger and judging the event type from the given text data, and event argument extraction refers to extracting the arguments of the event from the given text and identifying the type of the participants.

[0082] The conventional event extraction detection model is generally subject to independent distribution of all data during model training and testing, and the event extraction model trained by this method is too idealized. Because new event types will continue to appear and form new and different classifications, the idealized model is not suitable for practical applications for real-time streaming data. In order to solve this problem, a relatively simple method is to fine-tune the newly added event types on the trained conventional event extraction model, for example: Figure 1As shown, the event extraction model is trained using Meet event data (meeting events or other event types), and a trained event extraction model on Meet events is obtained, which can initially correctly classify the categories of Meet events. After fine-tuning the Meet event extraction model trained on Meet events using Elect events (election events), the output feature space of the Meet event type is mixed with the output feature space of the Elect event type. When the Meet event data is input into the fine-tuned event extraction model (the Meet event extraction model trained on the Elect event type) again, the fine-tuned event extraction model incorrectly classifies the Meet event type as the Elect event type, which is called catastrophic forgetting.

[0083] To solve this problem, a traditional event extraction model based on a knowledge distillation framework is proposed. The knowledge distillation framework is a teacher-student network structure, which usually provides historical event type knowledge with a trained teacher network, and the student network acquires the knowledge in the teacher network through knowledge distillation training. It can transfer the feature knowledge learned from the complex teacher network to the simple student network at the cost of small performance loss, and the student network can continue to learn new event knowledge (new event type feature knowledge). Knowledge distillation also has the advantage of not requiring additional tools or data storage during training, reducing resource usage.

[0084] However, the traditional event extraction model based on the knowledge distillation algorithm also has some main problems. For example, during the training process, the model cannot simultaneously achieve the preservation of historical event type features and the preservation and adjustment conflict between the output features shared by the model in the learning of new event types. That is, the feature space of the traditional event extraction model in the learning of historical event types and the feature space in the learning of new event types may exist to a certain extent. Cross and mixing may cause the model to lose a certain degree of feature knowledge. In addition, during the model training process, the event extraction model fails to fully utilize the feature knowledge of historical event types, which easily causes the event extraction model to forget the knowledge it has learned, thus failing to effectively utilize the feature knowledge of historical event types, thereby affecting the learning effect of the model on new event types and historical event types, and thus the knowledge distillation algorithm cannot be directly applied to the traditional event extraction model to realize continuous event detection.

[0085] To this end, the present disclosure provides a model training method for event extraction based on a knowledge distillation framework and a historical enhancement feature transfer network, which can continuously extract events, and the extracted events can be related to real-time economic news, entertainment news, novel text content, etc. The event extraction based on continuous learning of the present disclosure is an extended task of event extraction, which can continuously learn new event type feature knowledge from real-time new text events, and can save most of the historical event type knowledge that has been learned. The present disclosure utilizes the knowledge distillation framework, trains another simple student network by using the output of the pre-trained teacher network as a supervision signal, and through the guidance of the teacher network, the student network can better retain the feature information in the teacher network. In addition, the event extraction model based on continuous learning provided by the present disclosure does not need to re-establish a knowledge base or train a text stream data set when facing a new event type that needs to be identified, but directly trains the new event based on the original event extraction model based on continuous learning, that is, the detection of the new event type can be realized. At the same time, by utilizing the historical enhancement feature transfer network, the new event type feature distribution is supplemented to the feature space of the historical event as much as possible without affecting the historical event type feature distribution, and the feature distribution of the new event and the historical event is separated and reserved by reconstructing the feature distribution of the historical event type. The feature space of the reconstructed historical event is responsible for saving historical feature data, and the feature space of the new event focuses only on learning the feature distribution of the new event, so that the event extraction model based on continuous learning has good memory ability, which ensures that the model can effectively distinguish the feature distribution of different event types, and at the same time of learning the new event type knowledge, it will not forget the historical event type knowledge learned before, and still can realize the detection of the historical event.

[0086] According to an embodiment of the present disclosure, a model training method for event extraction based on continuous learning is provided, comprising:

[0087] Obtaining a text stream data set, the text stream data set comprising historical text stream data memorized by the 0th to the (t-1)th models and new text stream data to be identified by the tth to the nth models, wherein t≥1, n≥t;

[0088] Inputting the text stream data set into a pre-trained language model and a self-attention mechanism for processing to obtain a first fusion feature vector corresponding to the text of the text stream data set;

[0089] Training a student network and a teacher network of the event extraction model based on continuous learning to be trained by using the first fusion feature vector in a knowledge distillation manner to obtain the event extraction model based on continuous learning, comprising:

[0090] The t-th first fusion feature vector is input into the student network of the event extraction model based on continuous learning to be trained, and the t-th first feature vector is obtained.

[0091] By using a history-enhanced feature transfer network, the t-th first feature vector is mapped to the (t-1)-th first feature vector to obtain the (t-1)-th second feature vector;

[0092] The first t-1 first fused feature vectors are input into the teacher network of the event extraction model based on continuous learning to be trained, and the first t-1 third feature vectors are obtained.

[0093] The first loss value is obtained based on the t-th first feature vector and the labeled text stream data;

[0094] The second loss value is obtained based on the (t-1)th second feature vector and the first (t-1)th third feature vector;

[0095] The first loss value and the second loss value are summed to obtain the total loss value;

[0096] The total loss value is used to train the student network and teacher network of the event extraction model based on continuous learning, thus obtaining the event extraction model based on continuous learning.

[0097] Figure 2 The diagram illustrates a system architecture of a model training, event extraction method, apparatus, and device for event extraction based on continuous learning, according to embodiments of the present disclosure.

[0098] like Figure 2 As shown, application scenario 200 according to this embodiment may include a first terminal device 201, a second terminal device 202, and a third terminal device 203. Network 204 serves as a medium for providing communication links between the first terminal device 201, the second terminal device 202, the third terminal device 203, and the server 205. Network 204 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0099] Users can use the first terminal device 201, the second terminal device 202, and the third terminal device 203 to interact with the server 205 via the network 204 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 201, the second terminal device 202, and the third terminal device 203, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0100] The first terminal device 201, the second terminal device 202, and the third terminal device 203 can be various electronic devices with a display screen and supporting web browsing, including but not limited to a smartphone, a tablet computer, a laptop computer, a desktop computer, and the like.

[0101] The server 205 can be a server providing various services, for example, a background management server supporting a website browsed by a user using the first terminal device 201, the second terminal device 202, and the third terminal device 203 (only as an example). The background management server can perform analysis and the like on received user requests and the like, and feed back the processing result (for example, a webpage, information, or data, or the like obtained or generated according to a user request) to a terminal device.

[0102] It should be noted that the model for event extraction based on continuous learning and the method for event extraction provided in the embodiments of the present disclosure can generally be executed by the server 205. Accordingly, the model for event extraction based on continuous learning and the apparatus for event extraction provided in the embodiments of the present disclosure can generally be arranged in the server 205. The model for event extraction based on continuous learning and the method for event extraction provided in the embodiments of the present disclosure can also be executed by a server or a server cluster different from the server 205 and capable of communicating with the first terminal device 201, the second terminal device 202, the third terminal device 203, and / or the server 205. Accordingly, the model for event extraction based on continuous learning and the apparatus for event extraction provided in the embodiments of the present disclosure can also be arranged in a server or a server cluster different from the server 205 and capable of communicating with the first terminal device 201, the second terminal device 202, the third terminal device 203, and / or the server 205.

[0103] The model for event extraction based on continuous learning and the method for event extraction provided in the embodiments of the present disclosure can also be executed by the first terminal device 201, the second terminal device 202, and the third terminal device 203. Accordingly, the model for event extraction based on continuous learning and the apparatus for event extraction provided in the embodiments of the present disclosure can also be arranged in the first terminal device 201, the second terminal device 202, and the third terminal device 203. The model for event extraction based on continuous learning and the method for event extraction provided in the embodiments of the present disclosure can also be executed by other terminals different from the first terminal device 201, the second terminal device 202, and the third terminal device 203. Accordingly, the model for event extraction based on continuous learning and the apparatus for event extraction provided in the embodiments of the present disclosure can also be arranged in other terminals different from the first terminal device 201, the second terminal device 202, and the third terminal device 203.

[0104] It should be understood that, Figure 2The number of terminal devices, networks and servers in the figure is only illustrative. According to the implementation needs, there can be any number of terminal devices, networks and servers.

[0105] The following will be based on Figure 2 The described scenario, by Figures 3-7 The model training and event extraction method of the disclosed embodiment based on continuous learning are described in detail.

[0106] Figure 3 The flowchart of the model training method of the continuous learning-based event extraction according to the embodiments of the present disclosure is schematically shown.

[0107] As Figure 3 The model training method of the continuous learning-based event extraction of the embodiment includes operation S310 to operation S330.

[0108] In operation S310, a text stream data set is obtained, and the text stream data set includes historical stream data memorized by the 0th to the (t-1)th model and new stream data to be identified from the tth to the nth, where t≥1 and n≥t.

[0109] In operation S320, the text stream data set is input into the pre-trained language model and the self-attention mechanism for processing to obtain a first fusion feature vector corresponding to the text of the text stream data set.

[0110] In operation S330, the first fusion feature vector is used to train the student network and the teacher network of the continuous learning-based event extraction model to be trained in a knowledge distillation manner to obtain the continuous learning-based event extraction model, including:

[0111] The tth first fusion feature vector is input into the student network of the continuous learning-based event extraction model to be trained to obtain the tth first feature vector;

[0112] The tth first feature vector is mapped to the (t-1)th first feature vector through the historical enhancement feature transfer network to obtain the (t-1)th second feature vector;

[0113] The first t-1 first fusion feature vectors are input into the teacher network of the continuous learning-based event extraction model to be trained to obtain the first t-1 third feature vectors;

[0114] The first loss value is obtained according to the tth first feature vector and the labeled text stream data;

[0115] The second loss value is obtained according to the (t-1)th second feature vector and the first t-1 third feature vectors;

[0116] The first loss value and the second loss value are added to obtain a total loss value;

[0117] The student network and the teacher network of the event extraction model based on the continuous learning to be trained are trained by using the total loss value, to obtain the event extraction model based on the continuous learning.

[0118] According to the embodiments of the present disclosure, in step S310, the obtained text stream data set has n streams, and the n streams can be n sentences. For the n text stream data, the 0th to the (t-1)th are historical text stream data memorized by the model, and the tth to the nth are new text stream data to be identified, where t>1 and n>t.

[0119] According to the embodiments of the present disclosure, obtaining the text stream data set comprises:

[0120] Obtaining text stream data;

[0121] Cleaning the obtained text stream data to obtain unlabeled text stream data;

[0122] Labeling the unlabeled text stream data to obtain labeled text stream data;

[0123] Fusing the labeled text stream data and the unlabeled text stream data to obtain the text stream data set.

[0124] According to the embodiments of the present disclosure, according to the demand, the relevant text stream data can be real-time news, novel text information and the like, which are obtained from the web page by using the web crawler technology. The so-called web crawler is a program or script that automatically crawls the information of the World Wide Web according to certain rules. Specifically, it first obtains the response of the target server through a web request, then parses the corresponding returned by using the relevant script, and obtains the important resources therein.

[0125] According to the embodiments of the present disclosure, because the data crawled by the network has problems such as garbled code and repetition, the relevant regular expression rules need to be used to clean the data, remove the repeated and garbled text data in the text stream data, and correct the wrong text data, so that the finally output text can be as smooth as possible and the text meaning can be understood by reading, that is, to obtain the unlabeled text stream data.

[0126] According to an embodiment of the present disclosure, the text stream data obtained through web crawling and data cleaning is generally not labeled, and therefore, the training label of the event extraction model based on continuous learning needs to be provided by labeling the text stream data through a manual labeling method. After labeling the real-time unlabeled text stream data, labeled text stream data is obtained. The labeling method in the embodiment of the present disclosure can be BIO labeling, BIOES labeling, IOB labeling, or other labeling methods, where B (begin) represents the beginning, I (intermediate) represents the intermediate, O (other) represents other (non-entity), E (end) represents the end, and S (single) represents a single character.

[0127] For example, the text content of “The student-developed mobile phone noise reduction technology won the second prize in the national competition” is labeled, the event type is “innovation: technology research and development”, and the label is “{O, O, O, O, O, O, O, O, O, O, B-research, I-research, O, O, O, O, O, O, O, O}”.

[0128] According to an embodiment of the present disclosure, the text data in the text stream data set includes labeled text stream data and unlabeled text stream data. For example:

[0129] T = {(x1, y1), (x2, y2),..., (x n , y n )};

[0130] wherein, i = 1, 2,..., n, x i is the input unlabeled text stream data sequence; y i is the corresponding label text data sequence, and n is the number of text stream data, which has different values for different text stream data. The label y i is a binary classification label (B-, I-, or O-) for each word in the text stream data, B- represents that the word does not belong to the beginning of a certain class, I- represents that the word belongs to a certain class, and the meanings of B, I, and O are as explained in the above embodiment.

[0131] According to an embodiment of the present disclosure, in step S320, the text stream data set is input into the pre-trained language model and the self-attention mechanism to obtain the first fusion feature vector corresponding to the text of the text stream data set, which includes:

[0132] Each text stream data in the input text stream data set is output as a stream feature vector corresponding to the text stream data by using the pre-trained language model, and n text stream data sets are obtained, n text stream feature vectors, each of which includes a first feature subvector corresponding to different clauses.

[0133] text stream data x t For example, the specific implementation is as follows:

[0134]

[0135] wherein x t is text stream data, which can be a sentence; t represents the tthtext stream event; w represents a token (character) generated by sentence segmentation; and m represents the number of tokens, which can be the number of words in a sentence.

[0136] Specifically, the text stream data x t is converted into the text stream feature vector H t corresponding to the text stream data x t , that is, the hidden layer vector H t is obtained, for example, a sentence of text is converted into the text stream feature vector corresponding to the sentence.

[0137] In the embodiment of the present disclosure, the text stream data x t is converted into the stream feature vector H t corresponding to the text stream data x t , and the specific implementation includes:

[0138] The text stream data is input into the pre-trained language model, the pre-trained language model looks up the embedding matrix corresponding to each word in the text to obtain an embedding vector, and each row in the embedding matrix is a word embedding vector, which can represent the semantic information of each word / term in the text. In addition, because the semantics carried by the words / term appearing in different positions of the text are different, the pre-trained language model cannot obtain the relative position of each word, so position encoding information needs to be introduced when the model is input, which is used to indicate the order of the words / term in the text, for example, the position information of the first word / term is 1, the position information of the second word / term is 2, the position information of the mthword / term is m, and so on, and then the position information is mapped into a position vector through a position vector matrix. Finally, the position vector and the word embedding vector are added to obtain the first feature sub-vector h1 corresponding to each clause (word / sentence) in the text stream data.

[0139] The hidden layer vector H t is obtained, and the formula involved is shown in formula (1):

[0140]

[0141] wherein, H t represents a hidden layer vector; h1 represents a first feature sub-vector; m represents the number of different clauses included in the text stream feature vector, i.e., the number of first feature vectors generating tokens.

[0142] According to an embodiment of the present disclosure, for each text stream feature vector, the first feature sub-vectors in the text stream feature vector are fused by using a self-attention mechanism to obtain a first fused feature vector corresponding to the text of the text stream data set.

[0143] For example, taking the kth hidden layer vector and the jth hidden layer as an example for fusion, the formula involved is as follows:

[0144]

[0145]

[0146] e k，j = a(h k ·h j ) = V T tanh(Wh k + Uh j ); Equation (4)

[0147] wherein, a(·) is a nonlinear alignment function, V, W, U are hyperparameters of corresponding dimensions, which are simultaneously learned and adjusted when training the model, h j is the jth first feature sub-vector, h k is the kth first feature sub-vector, e k，j represents the similarity between the kth hidden layer vector and the jth hidden layer.

[0148] According to an embodiment of the present disclosure, the first feature vectors in the tth text stream feature vector are fused by using a self-attention mechanism, which can enable deep interaction between the input first feature sub-vectors, learn sentence-level feature expression, and thus obtain a first fused feature vector corresponding to the text of the tth text stream data Specifically as follows:

[0149]

[0150] wherein, represents the first first fused feature vector, represents the mth first fused feature vector.

[0151] According to an embodiment of the present disclosure, in operation S330, a student network of the event extraction model to be trained is trained by using the first fusion feature vector in a manner of knowledge distillation, to obtain an event extraction model, including operations S3301-S3304.

[0152] In operation S3301, the tth first fusion feature vector is input into the student network of the event extraction model to be trained based on continuous learning, to obtain a tth first feature vector.

[0153] In operation S3302, the tth first feature vector is mapped to a (t-1)th first feature vector by using a historical enhanced feature transfer network, to obtain a (t-1)th second feature vector.

[0154] In operation S3303, the first fusion feature vectors before the t-1th are input into the teacher network of the event extraction model to be trained based on continuous learning, to obtain third feature vectors before the t-1th. It should be noted that the first fusion feature vectors before the t-1th are obtained in the same manner as the tth first fusion feature vector, which will not be described here.

[0155] In operation S3304, a first loss value is obtained according to the tth first feature vector and the labeled text stream data; a second loss value is obtained according to the (t-1)th second feature vector and the third feature vectors before the t-1th; the first loss value and the second loss value are added to obtain a total loss value; the student network and the teacher network of the event extraction model to be trained based on continuous learning are trained by using the total loss value, to obtain the event extraction model based on continuous learning, which has good memory capability and can retain the feature distribution information in the historical text stream data after each training of the feature information in the newly added text stream data is completed.

[0156] According to an embodiment of the present disclosure, in order to alleviate the problem of the conflict between retention and adjustment of newly added events and historical events existing in the existing knowledge distillation method, a historical enhanced feature transfer network is used to restore the feature space of historical events. When learning the newly added events, the historical enhanced feature transfer network can map the feature space of the newly added events to be identified to the feature space of the historical events, while avoiding the problems of cross mixing and forgetting of different types of event feature distributions in a high-dimensional feature space.

[0157] Figure 4 An architecture schematic diagram of a feature transfer network in a model training method of event extraction based on continuous learning according to an embodiment of the present disclosure is schematically shown.

[0158] As Figure 4As shown, the historical enhancement feature transfer network adopts a multi-layer feedback neural network, a nonlinear activation layer and a multi-layer feedforward neural network, and its core idea is to map the features of the newly added event to the feature space of the historical event, realize the mutual transfer learning of the features between the newly added event and the historical event. Assuming that the event extraction model based on continuous learning has been trained on the first t-1 events, the training of the tth event is to be performed. At this time, the D-dimensional features of the first fusion feature vector of the tth event are projected to a smaller dimension H by using a multi-layer feedback neural network, and then the H-dimensional feature vector is projected back to the D-dimensional feature vector by a nonlinear activation layer and a multi-layer feedforward neural network, and the first fusion feature vector of the tth event and the feature vector projected back to the D-dimensional feature vector by the multi-layer feedforward neural network are added, so as to realize the mapping of the first feature vector of the tth event to the first feature vector of the t-1th event by the historical enhancement feature transfer network, and obtain the second feature vector of the t-1th event. That is, the feature space distribution of the newly added event can be remapped to the feature space distribution of the historical event without affecting the distribution of the historical event feature space, and the cross mixing of different types of event feature vectors in a high-dimensional feature space is avoided. At the same time, the model can effectively distinguish the feature distribution of different event types, realize the mutual transfer learning of the knowledge between the newly added event and the historical event, and thus enable the event extraction model based on continuous learning to perform continuous event detection.

[0159] In the knowledge distillation framework, in order to enable the teacher network to remember more historical event knowledge, the original text enhancement playback mechanism is adopted in the embodiment of the disclosure to store the text stream data belonging to the historical event type as memory data, that is, the text stream data of each historical event type is evenly divided to ensure that there are historical text streams that have been memorized and newly added text streams that need to be identified during the training process. In other words, the event extraction model based on continuous learning is trained by combining the memory data and the newly added text stream data. In the knowledge distillation framework, the knowledge in the teacher network with historical memory can be transferred to the student network to realize knowledge distillation. Based on the knowledge transfer of the teacher network, the student network can continue to learn the continuous learning of the newly added event, and at the same time, due to the existence of the historical enhancement feature transfer network, the student network will not forget the historical knowledge that has been learned.

[0160] Figure 5 The model architecture diagram of the model training method of the event extraction based on continuous learning according to the embodiment of the disclosure is schematically shown.

[0161] As Figure 5As shown, after the input text stream data is encoded by the pre-trained language model and the feature fusion is performed by the self-attention mechanism, the tth first fusion feature vector is input into the student network of the event extraction model based on the continuous learning to be trained to obtain the tth first feature vector. Then, the tth first feature vector is mapped to the (t-1)th first feature vector through the historical enhancement feature transfer network to obtain the (t-1)th second feature vector. At the same time, the first t-1 fusion feature vectors are also input into the teacher network based on the continuous learning to be trained to obtain the first t-1 third feature vectors.

[0162] According to the tth first feature vector and the labeled text stream data, a first loss value is obtained; according to the (t-1)th second feature vector and the first t-1 third feature vectors, a second loss value is obtained; the first loss value and the second loss value are added to obtain a total loss value; the event extraction model is trained by using the total loss value to obtain the event extraction model.

[0163] According to an embodiment of the present disclosure, according to the tth first feature vector and the labeled text stream data, a first loss value L cross-entropy is obtained, which is specifically represented as:

[0164] The t first fusion feature vectors are input into the student network of the event extraction model based on the continuous learning to be trained, and an activation function is used to realize the judgment of the tth new text event type, and the formula involved is shown as formula (5):

[0165]

[0166] Wherein, P (t) represents the event category output by the student network; W t represents the student network parameter.

[0167] In order to evaluate the accuracy of the student network output, a minimum cross-entropy loss function is designed to constrain the error between the student network output result and the real label, so as to improve the learning performance of the student network in the event extraction model based on the continuous learning to be trained, and the specific loss function involved is shown as formula (6):

[0168] L cross-entropy = F BCE (P (t) , Y (t) ); formula (6)

[0169] Wherein, Y (t) is the real label category of the text stream.

[0170] In combination with Figure 4In order to enable the output of the new event to be identified to be fused with the output of the historical event, the feature transfer network is adopted to remap the feature vector space of the new event to be identified to the feature vector space of the historical event, so as to realize the fusion of the features and avoid the problem of cross mixing of different types of event feature vectors in a high-dimensional feature space.

[0171] According to an embodiment of the present disclosure, the t-th first feature vector is mapped to the t-1-th first feature vector by the historical enhancement feature transfer network to obtain the t-1-th second feature vector, wherein the calculation formula involved is shown as formula (7):

[0172]

[0173] wherein, denotes the t-1-th second feature vector; FTM t，t-1 The subscript in the formula indicates that the feature vector is transferred from the t-th event to the t-1-th event.

[0174] Next, the knowledge distillation mechanism is adopted to transfer the knowledge from the teacher network to the student network by using the teacher-student framework, which is specifically represented as follows: the cosine similarity loss function is adopted to measure the feature difference between the outputs of the teacher network and the student network, and the consistency of the student network learning from the teacher network is strengthened by minimizing the loss function, that is, the second loss value is obtained according to the t-1-th second feature vector and the first t-1-th third feature vector, wherein the specific formula involved in calculating the second loss value is shown as formula (8):

[0175]

[0176] wherein, the second loss value L distill ; denotes the first t-1-th third feature vector output by the teacher network based on the first t-1-th event; F cs (·) refers to the cosine similarity function.

[0177] According to an embodiment of the present disclosure, the event extraction model to be trained is trained by using the total loss value to obtain the event extraction model, which is specifically represented as follows:

[0178] By combining the cross-entropy classification loss function and the cosine similarity distillation loss function, the overall loss function is formed to obtain the total loss value, and the specific calculation formula involved is shown as formula (9):

[0179] L total = L cross-entropy + λ·L distill ; formula (9)

[0180] wherein, L totaldenotes the total loss value; λ is a balance parameter, controlling the influence of each loss function, and the value range is [0, 1].

[0181] The objective function of the event extraction model based on continuous learning is to minimize L total The random gradient descent method is used to iteratively optimize and update the parameters of the event extraction model based on continuous learning. In addition, each event type has a classifier, and the first t-1 event classifiers can identify the probability of the first t-1 events belonging to each category, and the tth event classifier can identify the probability of the tth event category.

[0182] For example, the teacher network in the event extraction model has completed the training of event features of various types such as "personnel: qualification obtained", "life: population born", and "movement: logistics transportation". The student network learns from the teacher network through knowledge distillation, and the current student network can continue to learn the event feature knowledge of the "innovation: technology research and development" type.

[0183] The learning process of the event extraction model based on continuous learning includes a basic learning stage and an incremental learning stage. In the basic learning stage, the event extraction model based on continuous learning learns a batch of types of events to form a teacher network. In the incremental learning stage, the event extraction model based on continuous learning faces new events that need to be identified, and uses a knowledge distillation algorithm to jointly train the teacher network and the student network, so that the student network can quickly learn the new events that need to be identified and has the ability to detect new events, thereby achieving the purpose of continuous evolution and detection. When the current event extraction model based on continuous learning is trained, the student network that has completed learning becomes a teacher network in the next iteration, which can guide the training of a new student network, and the training work is repeated to enable the event extraction model to continuously learn different event types.

[0184] Figure 6 A flowchart of a model training method for event extraction based on continuous learning according to another embodiment of the present disclosure is schematically shown.

[0185] As Figure 6 shown, according to an embodiment of the present disclosure, the model training method for event extraction based on continuous learning further includes updating the event extraction model based on continuous learning that has completed training, and the updating includes:

[0186] The event extraction model based on continuous learning that has completed training is used as a teacher network, and the t+1th first fusion feature vector is input into the student network of the event extraction model based on continuous learning to be trained for a new round of event extraction model based on continuous learning training, thereby obtaining an updated event extraction model based on continuous learning.

[0187] For example, the student network and the teacher network of the event extraction model based on continuous learning to be trained are trained by using the tth first feature vector, the (t-1)th second feature vector, the first t-1 third feature vectors, and the labeled text stream data, to obtain the trained event extraction model based on continuous learning, that is, the event extraction model based on continuous learning about the tth event. Then, when the (t+1)th event type training (an event type that the model has not learned) is performed, the (t+1)th first fusion feature vector is input into the student network of the event extraction model based on continuous learning to perform a new round of event extraction model based on continuous learning training, to obtain an updated event extraction model based on continuous learning, that is, the (t+1)th event extraction model.

[0188] In the embodiments of the present disclosure, the event extraction model can continuously learn new event type knowledge from the new text stream data through continuous updating, that is, the event extraction model can learn new event knowledge based on the original event extraction model when facing the new text stream data, while retaining most of the historical event type knowledge that the model has learned, and the detection of the historical event type can also be realized.

[0189] Figure 7 A flowchart of the event extraction method based on continuous learning according to the embodiments of the present disclosure is schematically shown.

[0190] As shown in Figure 7 The event extraction method based on continuous learning in the embodiments includes operations S710-S740.

[0191] In operation S710, the text stream data to be extracted is obtained, wherein the text stream data to be extracted is new text data to be identified.

[0192] In the embodiments of the present disclosure, in operation S710, the text stream data to be extracted obtained is new text data to be identified, and the new text stream data can be real-time text information related to economic news, social news, entertainment news, novels, etc.

[0193] In operation S720, the text stream data to be extracted is input into the pre-trained language model and the self-attention mechanism to obtain a second fusion feature vector corresponding to the text to be extracted in the text stream data to be extracted.

[0194] According to the embodiments of the present disclosure, in operation S720, obtaining the second fusion feature vector corresponding to the text to be extracted in the text stream data to be extracted includes:

[0195] The pre-trained language model is input with the to-be-extracted text stream data to obtain a to-be-extracted text stream feature vector, and the to-be-extracted text stream feature vector includes second feature sub-vectors corresponding to different clauses.

[0196] The self-attention mechanism is used to fuse the second feature sub-vectors in the to-be-extracted text stream feature vector, to obtain a second fusion feature vector corresponding to the to-be-extracted text in the to-be-extracted text stream data.

[0197] In the embodiment of the present disclosure, the to-be-extracted text stream feature vector corresponds to the to-be-extracted text stream data, and the to-be-extracted text feature vector includes position information, facilitating subsequent recognition and detection of the to-be-extracted text. Then the self-attention mechanism is used to fuse the second feature sub-vectors in the to-be-extracted text stream feature vector, to construct the context relationship between the second feature sub-vectors, and obtain the second fusion feature vector corresponding to the to-be-extracted text.

[0198] In operation S730, the second fusion feature vector is input into the event extraction model based on continuous learning to obtain an event extraction result, wherein the event extraction model is trained by the model training method based on continuous learning of event extraction.

[0199] In the embodiment of the present disclosure, because the event extraction model based on continuous learning has learned the event type of the to-be-extracted text, when the to-be-extracted text is input into the event extraction model, the model can determine the prediction result (i.e., the event extraction result) of each character in the to-be-extracted text.

[0200] In operation S740, the event extraction result is input into a sequence classification model to obtain an event extraction category.

[0201] In the embodiment of the present disclosure, according to the prediction result of each character / word in the to-be-extracted text, it is necessary to combine which category each character belongs to, and finally obtain the prediction result in word units, so as to determine which event category the to-be-extracted text belongs to.

[0202] For example, the input news "After professional guidance, and through the corresponding examination, qualified persons can obtain the qualification of human resources management position", the event extraction model based on continuous learning provided by the present disclosure can accurately judge that it is a "personnel: obtain qualification" type event.

[0203] According to the embodiment of the present disclosure, the to-be-extracted text stream data is obtained in the following way:

[0204] Obtaining to-be-extracted text stream data;

[0205] Cleaning the obtained to-be-extracted text stream data to obtain unlabeled to-be-extracted text stream data;

[0206] annotating the unlabeled text stream data to be extracted to obtain labeled text stream data to be extracted,

[0207] fusing the labeled text stream data to be extracted and the unlabeled text stream data to be extracted to obtain text stream data to be extracted.

[0208] In an embodiment of the present disclosure, the text stream data to be extracted is obtained by a web crawler, and the obtained data is cleaned to remove repeated, garbled and other text data in the text stream data to be extracted, and the erroneous text data is corrected, so that the finally output text is as smooth as possible, thereby obtaining clean unlabeled text stream data to be extracted. Then, the unlabeled text stream data to be extracted is labeled by manual labeling to obtain labeled text stream data to be extracted. Finally, the labeled text stream data to be extracted and the unlabeled text stream data to be extracted are fused to obtain text stream data to be extracted. The manual labeling method used in the embodiment of the present disclosure can be BIO labeling, BIOES labeling, IOB labeling or other labeling methods. Since the foregoing has been described, no further limitation will be made here.

[0209] Based on the above-mentioned model training method for event extraction based on continuous learning, the present disclosure further provides a model training device for event extraction based on continuous learning. The following will be described in combination with Figure 8 The device will be described in detail.

[0210] Figure 8 The structure block diagram of the model training device for event extraction based on continuous learning according to the embodiment of the present disclosure is schematically shown.

[0211] As Figure 8 shown, the model training device for event extraction based on continuous learning 800 of the embodiment includes a first acquisition module 810, a first processing module 820 and a model training module 830.

[0212] The first acquisition module 810 is configured to acquire a text stream data set, the text stream data set including historical text stream data already memorized by the 0th to (t-1)th models and new text stream data to be identified by the tth to nth models, where t≥1 and n≥t.

[0213] In an embodiment, the first acquisition module 810 includes a first text acquisition unit, a first cleaning unit, a first labeling unit and a first fusion unit.

[0214] The first text acquisition unit is configured to obtain text stream data by a web crawler.

[0215] The first cleaning unit is configured to clean the obtained text stream data to obtain no-label text stream data, clean the data by using a relevant regular expression rule, remove repeated and garbled text data in the text stream data, and correct the error text data, so that the finally output text can be as smooth as possible and the text meaning can be understood by reading, that is, the no-label text stream data is obtained.

[0216] The first labeling unit is configured to label the no-label text stream data to obtain labeled text stream data.

[0217] The first fusion unit is configured to fuse the labeled text stream data and the no-label text stream data to obtain a text stream data set.

[0218] The first processing module 820 is configured to input the text stream data set into a pre-trained language model and a self-attention mechanism for processing to obtain a first fusion feature vector corresponding to the text of the text stream data set. The first processing module includes a first feature encoding unit and a first feature fusion unit.

[0219] The first feature encoding unit is configured to input each text stream data in the text stream data set into the pre-trained language model to output a text stream feature vector corresponding to the text stream data, and obtain n text stream feature vectors. Each text stream feature vector includes a first feature sub-vector corresponding to a different clause.

[0220] The first feature fusion unit is configured to fuse the first feature sub-vectors in the text stream feature vector by using the self-attention mechanism for each text stream feature vector to obtain a first fusion feature vector corresponding to the text stream feature vector.

[0221] The model training module 830 is configured to train a student network and a teacher network of the event extraction model based on continuous learning to be trained by using the first fusion feature vector in a knowledge distillation manner to obtain the event extraction model based on continuous learning, including:

[0222] The tth first fusion feature vector is input into the student network of the event extraction model based on continuous learning to be trained to obtain the tth first feature vector.

[0223] The tth first feature vector is mapped to the (t-1)th first feature vector by the historical enhancement feature transfer network to obtain the (t-1)th second feature vector.

[0224] The first fusion feature vector is input into the teacher network of the event extraction model based on continuous learning to be trained to obtain the third feature vector.

[0225] According to the tth first feature vector and the labeled text stream data, a first loss value is obtained;

[0226] According to the t-1th second feature vector and the first t-1th third feature vector, a second loss value is obtained.

[0227] The first loss value and the second loss value are added to obtain a total loss value.

[0228] The total loss value is used to train a student network and a teacher network of the event extraction model based on continuous learning to be trained to obtain the event extraction model based on continuous learning.

[0229] According to an embodiment of the present disclosure, the model training device for event extraction based on continuous learning further comprises an updating module for updating the trained event extraction model, and the updating module comprises:

[0230] The trained event extraction model based on continuous learning is used as the teacher network, and the t+1th first fusion feature vector is input into the student network of the event extraction model based on continuous learning to be trained for a new round of event extraction model training based on continuous learning to obtain an updated event extraction model based on continuous learning.

[0231] According to an embodiment of the present disclosure, any of the first acquisition module 810, the first processing module 820 and the model training module 830 can be combined in one module for implementation, or any of the modules can be split into multiple modules. Alternatively, at least part of the function of one or more of the modules can be combined with at least part of the function of the other modules, and implemented in one module. According to an embodiment of the present disclosure, at least one of the first acquisition module 810, the first processing module 820 and the model training module 830 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on board, a system on package, an application specific integrated circuit (ASIC), or any other reasonable way of integrating or packaging a circuit, etc. hardware or firmware, or in any one of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, at least one of the first acquisition module 810, the first processing module 820 and the model training module 830 can be at least partially implemented as a computer program module which can perform corresponding functions when the computer program module is run.

[0232] Based on the above dynamic event extraction method, the present disclosure further provides a model training device for event extraction based on continuous learning. The following will be described in detail in combination with Figure 9 The device is described in detail.

[0233] Figure 9 A structural block diagram of the event extraction device based on continuous learning is shown.

[0234] As shown in the embodiment, the model training device 900 for event extraction based on continuous learning includes a second acquisition module 910, a second processing module 920, an event extraction module 930, and a classification module 940. Figure 9

[0235] The second acquisition module 910 is configured to acquire text stream data to be extracted, wherein the text stream data to be extracted is new text data to be identified.

[0236] In an embodiment, the second acquisition module 910 includes a second text acquisition unit, a second cleaning unit, a second labeling unit, and a second fusion unit.

[0237] The second text acquisition unit is configured to acquire the text data to be extracted by a web crawler.

[0238] The second cleaning unit is configured to clean the acquired text stream data to be extracted to obtain unlabeled text stream data to be extracted.

[0239] The second labeling unit is configured to label the unlabeled text stream data to be extracted to obtain labeled text stream data to be extracted.

[0240] The second fusion unit is configured to fuse the labeled text stream data to be extracted and the unlabeled text stream data to be extracted to obtain the text stream data to be extracted.

[0241] The second processing module 920 is configured to input the text stream data to be extracted into a pre-trained language model and a self-attention mechanism to obtain a second fusion feature vector corresponding to the text to be extracted in the text stream data to be extracted. The second processing module includes a second feature encoding unit and a second fusion unit.

[0242] The second feature encoding module is configured to input the text stream data to be extracted into the pre-trained language model to obtain a text stream feature vector of the text to be extracted, wherein the text stream feature vector includes second feature sub-vectors corresponding to different clauses.

[0243] The second fusion module is configured to fuse the second feature sub-vectors in the text stream feature vector of the text to be extracted by using the self-attention mechanism to obtain a second fusion feature vector corresponding to the text stream feature vector of the text to be extracted.

[0244] The event extraction module 930 is configured to input the second fusion feature vector into an event extraction model based on continuous learning to obtain an event extraction result, wherein the event extraction model based on continuous learning is trained by the model training method for event extraction based on continuous learning.​

[0245] The classification module 940 is configured to input the event extraction result into a sequence classification model to obtain an event extraction category.

[0246] According to an embodiment of the present disclosure, any one or more of the second acquisition module 910, the second processing module 920, the event extraction module 930, and the classification module 940 can be combined in one module, or any one of them can be split into multiple modules. Alternatively, at least part of the function of one or more of these modules can be combined with at least part of the function of the other modules, and implemented in one module. According to an embodiment of the present disclosure, at least one of the second acquisition module 910, the second processing module 920, the event extraction module 930, and the classification module 940 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on board, a system on package, an application specific integrated circuit (ASIC), or any other reasonable way of hardware or firmware that can be integrated or packaged with a circuit, or any one of software, hardware and firmware or any appropriate combination of any of them. Alternatively, at least one of the second acquisition module 910, the second processing module 920, the event extraction module 930, and the classification module 940 can be at least partially implemented as a computer program module that can perform corresponding functions when the computer program module is run.

[0247] Figure 10 The block diagram of the electronic device suitable for implementing the model for event extraction based on continuous learning and the method of event extraction according to an embodiment of the present disclosure is schematically shown.

[0248] As shown in Figure 10 The electronic device 1000 according to an embodiment of the present disclosure includes a processor 1001 that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 1002 or loaded from a storage portion 1008 into a random access memory (RAM) 1003. The processor 1001 can include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor, and / or a related chipset, and / or a special-purpose microprocessor (such as an application specific integrated circuit (ASIC)), and the like. The processor 1001 can also include an on-board memory for cache use. The processor 1001 can include a single processing unit or a plurality of processing units for performing different actions of the method processes according to an embodiment of the present disclosure.

[0249] In the RAM 1003, various programs and data required by the electronic device 1000 to operate are stored. The processor 1001, the ROM 1002, and the RAM 1003 are connected to each other via the bus 1004. The processor 1001 performs various operations of the method flow according to the embodiments of the present disclosure by executing the programs in the ROM 1002 and / or the RAM 1003. It needs to be noted that the programs can also be stored in one or more memories other than the ROM 1002 and the RAM 1003. The processor 1001 can also perform various operations of the method flow according to the embodiments of the present disclosure by executing the programs stored in the one or more memories.

[0250] According to the embodiments of the present disclosure, the electronic device 1000 can further include an input / output (I / O) interface 1005, which is also connected to the bus 1004. The electronic device 1000 can further include one or more of the following components connected to the I / O interface 1005: an input part 1006 including a keyboard, a mouse, etc.; an output part 1007 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage part 1008 including a hard disk, etc.; and a communication part 1009 including a network interface card such as a LAN card, a modem, etc. The communication part 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as necessary. A removable medium 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 1010 as necessary, so that a computer program read out therefrom is installed in the storage part 1008 as necessary.

[0251] The present disclosure also provides a computer readable storage medium, which can be included in the device / apparatus / system described in the above embodiments; or can exist separately without being assembled into the device / apparatus / system. The above computer readable storage medium carries one or more programs, when the one or more programs are executed, the method according to the embodiments of the present disclosure is implemented.

[0252] According to an embodiment of the present disclosure, the computer readable storage medium can be a nonvolatile computer readable storage medium, for example, can include, but is not limited to, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any appropriate combination thereof. In the present disclosure, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present disclosure, the computer readable storage medium can include one or more memories such as the ROM 1002 and / or the RAM 1003 described above and / or one or more memory chips other than the ROM 1002 and the RAM 1003.

[0253] Embodiments of the present disclosure also include a computer program product including a computer program containing program codes for executing the methods shown in the flowcharts. When the computer program product is run in a computer system, the program codes are used to make the computer system implement the item recommendation method provided by the embodiments of the present disclosure.

[0254] The above-described functions defined in the system / device of the embodiments of the present disclosure are performed when the computer program is executed by the processor 1001. According to an embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by computer program modules.

[0255] In one embodiment, the computer program can rely on a tangible storage medium such as an optical storage device, a magnetic storage device, etc. In another embodiment, the computer program can also be transmitted, distributed, and downloaded in the form of a signal on a network medium and installed and downloaded through the communication part 1009 and / or installed from the detachable medium 1011. The program codes contained in the computer program can be transmitted by any appropriate network medium, including but not limited to wireless, wired, etc., or any appropriate combination thereof.

[0256] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1009 and / or installed from the detachable medium 1011. When the computer program is executed by the processor 1001, the above-described functions defined in the system of the embodiments of the present disclosure are performed. According to an embodiment of the present disclosure, the system, device, apparatus, module, unit, etc. described above can be implemented by computer program modules.

[0257] According to embodiments of the present disclosure, program code of the computer program for performing the methods provided by the embodiments of the present disclosure can be written in any combination of one or more programming languages, and can be implemented in a computer program product. Specifically, the computer program can be implemented in a high-level procedural and / or object-oriented programming language, and / or in assembly / machine language. The programming language includes, but is not limited to, Java, C++, python, “C” language, or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's device, and partly on a remote computing device, or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider (ISP).

[0258] The computer program product of the present disclosure can be a computer program product, which is a machine-readable medium (or computer readable medium) having stored therein a sequence of instructions executable by a machine such as a personal digital assistant (PDA), a laptop, a desktop computer, or a server. Alternatively, the computer program product can be a propagated signal per se generated by using the program code, and the program code can be stored in a computer readable medium in a server, a database, or the like. The service provider of the mobile communication system can provide the program code of the computer program product directly to the user equipment, or the user equipment can acquire the program code of the computer program product from the server or database etc. either through manual installation or automatically, i.e. “over the air” (OTA) installation. The program code embodied on the computer readable medium or propagated signal can cause a computer, or other processing systems, to perform the methods of the present disclosure.

[0259] Those skilled in the art will appreciate that features of the various embodiments and / or claims of the present disclosure can be combined or / and integrated with one another, even though such combinations or integrations are not expressly disclosed in the present disclosure. In particular, the features of the various embodiments and / or claims of the present disclosure can be combined or / and integrated with one another in any combination, without departing from the spirit and scope of the present disclosure. All such combinations and / or integrations are within the scope of the present disclosure.

[0260] The above describes embodiments of the present disclosure. However, these embodiments are merely for illustrative purposes, and are not intended to limit the scope of the present disclosure. Although each embodiment is described above separately, this does not mean that the measures in each embodiment cannot be used advantageously in combination. The scope of the present disclosure is defined by the appended claims and their equivalents. Those skilled in the art can make various substitutions and modifications without departing from the scope of the present disclosure, and these substitutions and modifications should all fall within the scope of the present disclosure.

Claims

1. A model training method for event extraction based on continuous learning, comprising: obtaining a text stream data set, the text stream data set comprising historical text stream data memorized by a 0th to (t-1)th model and new text stream data from a tth to nth to be identified, wherein t≥1 and n≥t; inputting the text stream data set into a pre-trained language model and a self-attention mechanism for processing to obtain a first fusion feature vector corresponding to the text of the text stream data set; training a student network and a teacher network of a to-be-trained event extraction model based on continuous learning using the first fusion feature vector in a knowledge distillation manner to obtain an event extraction model based on continuous learning, comprising: inputting a tth first fusion feature vector into the student network of the to-be-trained event extraction model based on continuous learning to obtain a tth first feature vector; mapping the tth first feature vector to a (t-1)th first feature vector through a historical enhanced feature transfer network to obtain a (t-1)th second feature vector, the historical enhanced feature transfer network being composed of a multi-layer feedback neural network, a nonlinear activation layer and a multi-layer feedforward neural network, the D-dimensional feature of the tth first feature vector being projected onto a smaller dimension H through the multi-layer feedback neural network, and then the H-dimensional feature vector being projected back onto the D dimension through the nonlinear activation layer and the multi-layer feedforward neural network, while the tth first feature vector and the feature vector projected back onto the D dimension through the multi-layer feedforward neural network are added together, so as to realize mapping of the tth first feature vector to the (t-1)th first feature vector through the historical enhanced feature transfer network to obtain the (t-1)th second feature vector; inputting the first fusion feature vectors from the 0th to the (t-1)th into the teacher network of the to-be-trained event extraction model based on continuous learning to obtain third feature vectors from the 0th to the (t-1)th; obtaining a first loss value according to the tth first feature vector and labeled text stream data; obtaining a second loss value according to the (t-1)th second feature vector and the third feature vectors from the 0th to the (t-1)th; adding the first loss value and the second loss value to obtain a total loss value; training the student network and the teacher network of the to-be-trained event extraction model based on continuous learning using the total loss value to obtain an event extraction model based on continuous learning.

2. The method of claim 1, wherein, obtaining the text stream data set, comprising: obtaining text stream data; cleaning the obtained text stream data to obtain unlabeled text stream data; labeling the unlabeled text stream data to obtain labeled text stream data; fusing the labeled text stream data and the unlabeled text stream data to obtain a text stream data set.

3. The method of claim 1, wherein, the inputting the text stream data set into a pre-trained language model and a self-attention mechanism to obtain a first fusion feature vector corresponding to the text of the text stream data set comprises: input each text stream data in the text stream data set into a pre-trained language model, output a text stream feature vector corresponding to the text stream data, and obtain n text stream feature vectors, each of which includes first feature sub-vectors corresponding to different clauses respectively; for each text stream feature vector, a self-attention mechanism is used to fuse the first feature sub-vectors in the text stream feature vector, and a first fused feature vector corresponding to the text stream data set text is obtained.

4. The method of claim 1, further comprising: updating the trained event extraction model based on continuous learning, the updating comprising: using the student network of the trained event extraction model based on continuous learning as a teacher network; inputting the first fused feature vector into the student network of the event extraction model based on continuous learning to be trained for a new round of event extraction model training based on continuous learning, and obtaining an updated event extraction model based on continuous learning.

5. An event extraction method based on continuous learning, comprising: obtaining text stream data to be extracted, wherein the text stream data to be extracted is new text data to be identified; inputting the text stream data to be extracted into a pre-trained language model and a self-attention mechanism to obtain a second fused feature vector corresponding to the text to be extracted in the text stream data to be extracted; inputting the second fused feature vector into an event extraction model based on continuous learning to obtain an event extraction result, wherein the event extraction model based on continuous learning is trained by the model training method of event extraction based on continuous learning of any one of claims 1-4; inputting the event extraction result into a sequence classification model to obtain an event extraction category; wherein the inputting the text stream data to be extracted into a pre-trained language model and a self-attention mechanism to obtain a second fused feature vector corresponding to the text to be extracted in the text stream data to be extracted comprises: inputting the text stream data to be extracted into a pre-trained language model to obtain a text stream feature vector, the text stream feature vector including second feature sub-vectors corresponding to different clauses respectively; for the text stream feature vector to be extracted, a self-attention mechanism is used to fuse the second feature sub-vectors in the text stream feature vector to be extracted, and a second fused feature vector corresponding to the text to be extracted in the text stream data to be extracted is obtained.

6. The method of claim 5, wherein, The text stream data to be extracted is obtained by the following method: obtaining text stream data to be extracted; cleaning the obtained text stream data to be extracted to obtain unlabeled text stream data to be extracted; annotating the unlabeled text stream data to be extracted to obtain labeled text stream data to be extracted, fusing the labeled text stream data to be extracted and the unlabeled text stream data to be extracted to obtain text stream data to be extracted.

7. An event extraction model training device based on continuous learning, comprising: The first obtaining module is configured to obtain a text stream data set, the text stream data set comprising historical text stream data memorized by a 0th to (t-1)th model and new text stream data from a tth to nth event to be identified, where t≥1 and n≥t. The first processing module is configured to input the text stream data set into a pre-trained language model and a self-attention mechanism for processing to obtain a first fusion feature vector corresponding to text in the text stream data set. The model training module is configured to train a student network and a teacher network of a to-be-trained event extraction model based on continuous learning using the first fusion feature vector in a knowledge distillation manner to obtain the event extraction model based on continuous learning. The tth first fusion feature vector is input into the student network of the to-be-trained event extraction model based on continuous learning to obtain a tth first feature vector. The tth first feature vector is mapped to a (t-1)th first feature vector through a historical enhancement feature transfer network to obtain a (t-1)th second feature vector, the historical enhancement feature transfer network comprising a multi-layer feedback neural network, a nonlinear activation layer and a multi-layer feedforward neural network, the multi-layer feedback neural network being configured to project D-dimensional features of the tth first feature vector onto a smaller dimension H, and then the H-dimensional feature vector is projected back onto the D dimension through the nonlinear activation layer and the multi-layer feedforward neural network, and the tth first feature vector and the feature vector projected back onto the D dimension through the multi-layer feedforward neural network are added together, so that the tth first feature vector is mapped to the (t-1)th first feature vector through the historical enhancement feature transfer network to obtain the (t-1)th second feature vector. The first t-1 first fusion feature vectors are input into the teacher network of the to-be-trained event extraction model based on continuous learning to obtain first t-1 third feature vectors. A first loss value is obtained according to the tth first feature vector and the labeled text stream data. A second loss value is obtained according to the (t-1)th second feature vector and the first t-1 third feature vectors. The first loss value and the second loss value are added together to obtain a total loss value. The student network and the teacher network of the to-be-trained event extraction model based on continuous learning are trained using the total loss value to obtain the event extraction model based on continuous learning.

8. An event extraction apparatus based on continuous learning, comprising: The second obtaining module is configured to obtain to-be-extracted text stream data, the to-be-extracted text stream data being new text data to be identified. The second processing module is configured to input the to-be-extracted text stream data into a pre-trained language model and a self-attention mechanism to obtain a second fusion feature vector corresponding to to-be-extracted text in the to-be-extracted text stream data. The event extraction module is configured to input the second fusion feature vector into an event extraction model based on continuous learning to obtain an event extraction result, the event extraction model based on continuous learning being trained by the model training method for event extraction based on continuous learning according to any one of claims 1-4. A classification module is configured to input the event extraction result into a sequence classification model to obtain an event extraction category.

9. An electronic device, comprising: one or more processors; storage configured to store one or more programs, wherein the one or more programs that, when executed by the one or more processors, cause the one or more processors to perform the methods recited in any one of claims 1-6.

Images