Methods for training information extraction models, information extraction methods and devices

Abstract

This application discloses a method, method, and apparatus for training an information extraction model. The main technical solution includes: acquiring a third dataset, which includes multiple unlabeled third text samples; using a pre-trained first information extraction model to extract target information of a preset type from the third text samples in the third dataset, as a second target sample; inputting the second target sample into a pre-trained text generation model to obtain text generated by the text generation model using the second target sample as a second text sample; forming a second sample pair by combining the second target sample and its corresponding second text sample; obtaining a second dataset using multiple second sample pairs; and training a second information extraction model using the second dataset. This application can improve the information extraction effect of the information extraction model while reducing manual costs and time consumption.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and natural language processing technology, and in particular to a method, method and apparatus for training an information extraction model. Background Technology

[0002] Information extraction (IE) refers to the process of using information extraction models to extract specific types of targets from text, thereby automatically classifying, extracting, and reconstructing massive amounts of content, such as named entity recognition, entity relationship extraction, event extraction, and sentiment extraction.

[0003] The training of information extraction models typically depends on the quality of the labeled dataset. In some data domains, insufficient labeled samples can severely impact the information extraction performance of the model. Manually labeling samples is both costly and time-consuming. Summary of the Invention

[0004] In view of this, this application provides a method, an information extraction method, and an apparatus for training an information extraction model, thereby improving the information extraction effect of the information extraction model while reducing labor costs and time consumption.

[0005] This application provides the following solution:

[0006] Firstly, a method for training an information extraction model is provided, the method comprising:

[0007] Obtain a third dataset, which includes multiple unlabeled third text samples;

[0008] Using a pre-trained first information extraction model, target information of a preset type is extracted from the third text samples of the third dataset and used as the second target sample;

[0009] The second target sample is input into a pre-trained text generation model, and the text generated by the text generation model using the second target sample is obtained as the second text sample.

[0010] The second target sample and the second text sample corresponding to the second target sample are combined to form a second sample pair, and multiple second sample pairs are used to obtain a second dataset.

[0011] The second information extraction model is trained using the second dataset, wherein the second text sample in the second sample pair is used as the input of the second information extraction model, and the difference between the target information of the preset type extracted by the second information extraction model from the input second text sample and the second target sample in the second sample pair is minimized.

[0012] According to one achievable method in the embodiments of this application, the method further includes:

[0013] Obtain a first dataset, which contains multiple first sample pairs. Each first sample pair includes a first text sample and a first target sample corresponding to the first text sample. The first target sample includes target information of a preset type that is labeled for the first text sample.

[0014] A first information extraction model and a text generation model are trained using the first dataset. The first text sample in the first sample pair is used as the input of the first information extraction model to minimize the difference between the target information of a preset type extracted by the first information extraction model from the input first text sample and the first target sample in the first sample pair. The first target sample in the first sample pair is used as the input of the text generation model to minimize the difference between the text generated by the text generation model using the input first target sample and the first text sample in the first sample pair.

[0015] According to one achievable method in the embodiments of this application, the first dataset and the third dataset correspond to different data domains.

[0016] According to one achievable method in an embodiment of this application, before training the second information extraction model using the second dataset, the method further includes:

[0017] Filter out second sample pairs from the second dataset whose target information is inconsistent with that of the second text sample and the second target sample; and / or,

[0018] Filter out second sample pairs from the second dataset that do not meet the preset format requirements.

[0019] According to one feasible method in an embodiment of this application, training a second information extraction model using the second dataset includes:

[0020] The first dataset and the second dataset are merged, and the merged dataset is used to train the second information extraction model, wherein the second extraction model is initialized using the model parameters of the first information extraction model.

[0021] According to one achievable method in the embodiments of this application, both the first information extraction model and the second information extraction model adopt a sequence-to-sequence model;

[0022] The sequences output by the first information extraction model and the second information extraction model include target information and the type identifier corresponding to the target information.

[0023] Secondly, an information extraction method is provided, the method comprising:

[0024] Get the text to be extracted;

[0025] The text to be extracted is input into the second information extraction model to obtain target information of a preset type extracted from the text to be extracted by the second information extraction model.

[0026] The second information extraction model is pre-trained using the method described in any one of the first aspects above.

[0027] Thirdly, an information extraction method is provided, executed by a cloud server, the method comprising:

[0028] Obtain the text to be extracted from the user terminal;

[0029] The text to be extracted is input into the second information extraction model to obtain target information of a preset type extracted from the text to be extracted by the second information extraction model.

[0030] The extracted target information is returned to the user terminal;

[0031] The second information extraction model is pre-trained using the method described in any one of the first aspects above.

[0032] Fourthly, a method for training an emotion extraction model is provided, the method comprising:

[0033] Obtain a third dataset, which includes multiple unlabeled third text samples;

[0034] Using a pre-trained first information extraction model, a preset type of sentiment element is extracted from the third text sample of the third dataset as a second sentiment element sample. The preset type of sentiment element includes at least one of sentiment object, sentiment word, sentiment polarity, and emotional information.

[0035] The second sentiment element sample is input into the pre-trained text generation model, and the text generated by the text generation model using the second sentiment element sample is used as the second text sample.

[0036] The second sentiment element sample and the second text sample corresponding to the second sentiment element sample are used to form a second sample pair, and the second sample pair is used to obtain the second dataset.

[0037] The second information extraction model is trained using the second dataset, wherein the second text sample in the second sample pair is used as the input of the second information extraction model, and the difference between the preset type of sentiment element extracted by the second information extraction model from the input second text sample and the second sentiment element sample in the second sample pair is minimized.

[0038] Fifthly, an emotion extraction method is provided, the method comprising:

[0039] Get the text to be extracted;

[0040] The text to be extracted is input into the second information extraction model to obtain the preset type of emotional elements extracted by the second information extraction model from the text to be extracted. The preset type of emotional elements includes at least one of emotional object, emotional word, emotional polarity and emotional information.

[0041] The second information extraction model is pre-trained using the method described in the fourth aspect above.

[0042] Sixthly, an apparatus for training an information extraction model is provided, the apparatus comprising:

[0043] The second acquisition unit is configured to acquire a third dataset, which includes multiple unlabeled third text samples;

[0044] The sample expansion unit is configured to extract target information of a preset type from the third text samples of the third dataset using a pre-trained first information extraction model, as a second target sample; input the second target sample into a pre-trained text generation model, and obtain the text generated by the text generation model using the second target sample as the second text sample;

[0045] The third acquisition unit is configured to form a second sample pair by combining the second target sample and the second text sample corresponding to the second target sample, and to obtain a second dataset by using multiple second sample pairs.

[0046] The third training unit is configured to train a second information extraction model using the second dataset, wherein the second text sample in the second sample pair is used as the input of the second information extraction model, and the difference between the target information of a preset type extracted by the second information extraction model from the input second text sample and the second target sample in the second sample pair is minimized.

[0047] In a seventh aspect, an information extraction device is provided, the device comprising:

[0048] The text acquisition unit is configured to acquire the text to be extracted.

[0049] The information extraction unit is configured to input the text to be extracted into a second information extraction model and obtain target information of a preset type extracted by the second information extraction model from the text to be extracted.

[0050] The second information extraction model is pre-trained by the device described in the sixth aspect above.

[0051] According to an eighth aspect, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps of the method described in any one of the first to fifth aspects.

[0052] According to the ninth aspect, an electronic device is provided, comprising:

[0053] One or more processors; and

[0054] A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any one of the first to fifth aspects.

[0055] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0056] 1) This application adopts a bidirectional generative framework. First, a trained first information extraction model generates target information from unlabeled third text samples. Then, a text generation model reverse-engineers the target information to generate text. The generated target information and text are used as a second sample pair to construct a new dataset, i.e., the third dataset, for training the second text information extraction model. This approach can utilize unlabeled sample data to expand a large amount of new sample data for training the text information extraction model, improving the extraction capability of the model and reducing the need for labeled data, thereby reducing the cost of manual labeling.

[0057] 2) This application first uses the labeled dataset, i.e. the first dataset, to pre-train the first information extraction model and the text generation model, and then uses the trained first information extraction model and text generation model to expand the unlabeled third dataset to obtain a newly labeled third dataset. This method can make reasonable and full use of the existing labeled data and improve the expanded dataset.

[0058] 3) The method provided in this application can be flexibly applied to different data domains. By utilizing the first information extraction model and text generation model that have been trained on the existing data domain, the samples on the new data domain can be expanded, thereby realizing the cross-domain transfer of the information extraction model at low cost and low time consumption.

[0059] 4) The information extraction model of this application adopts a generative sequence-to-sequence model, which transforms the information extraction task into the task of predicting text sequences that include target information and the type identifiers corresponding to the target information. It does not require special model design for different information extraction tasks, which improves the versatility of the information extraction model and makes full use of the powerful generative capabilities of generative models.

[0060] Of course, any product implementing this application does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 This is a system architecture diagram applicable to the embodiments of this application;

[0063] Figure 2 A flowchart illustrating the method for extracting training information from a model provided in this application embodiment;

[0064] Figure 3 A flowchart of the information extraction method provided in the embodiments of this application;

[0065] Figure 4 A schematic diagram illustrating the principle of the training information extraction model provided in the embodiments of this application;

[0066] Figure 5 A schematic block diagram of an apparatus for extracting training information from a model provided in an embodiment of this application;

[0067] Figure 6 A schematic block diagram of an information extraction device provided in the embodiments of this application;

[0068] Figure 7 A schematic block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0069] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0070] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0071] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0072] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0073] To facilitate understanding of this application, a brief description of the system architecture to which this application applies will be given first. Figure 1 An exemplary system architecture that can be applied to embodiments of this application is shown, such as Figure 1 As shown, the system architecture includes a model training device that builds an information extraction model offline, and an information extraction device that performs information extraction online.

[0074] In this embodiment, after acquiring training data, the model training device pre-trains the information extraction model using the method provided in this application.

[0075] The information extraction model is used to extract target information of a specific target type from the input text to be extracted. The target may correspond to different content in different application scenarios, which will be detailed in subsequent embodiments. The information extraction device can use the above-described information extraction model to obtain target information of a specific target type extracted from the text to be extracted.

[0076] The model training device and the information extraction device can be configured as separate servers, or they can be configured on the same server or server group, or they can be configured on a separate or the same cloud server. A cloud server, also known as a cloud computing server or cloud host, is a host product in the cloud computing service system, designed to address the shortcomings of traditional physical hosts and Virtual Private Servers (VPS) services, such as high management difficulty and weak service scalability. The model training device or information extraction device can also be configured on a computer terminal with strong computing capabilities.

[0077] In one common scenario, users can use a user terminal to interact with an information extraction device set up on the server. Various applications can be installed on the user terminal, such as voice interaction applications, web browser applications, and communication applications.

[0078] User terminals can be various electronic devices, including those with screens and those without. These include, but are not limited to, smartphones, tablets, smart speakers, smart TVs, PCs (Personal Computers), wearable devices, and so on.

[0079] Users can input text to be extracted through their user terminals. This text is sent over the network to an information extraction device on the server. The information extraction device then extracts target information of a specific target type from the text and returns the target information to the user terminal. The specific target type can be a user-specified target type or a default target type.

[0080] It should be understood that Figure 1 The number of model training devices, information extraction devices, user terminals, and information extraction models shown in the diagram is merely illustrative. Depending on implementation needs, any number of model training devices, information extraction devices, user terminals, and information extraction models can be included.

[0081] Figure 2 This is a flowchart of a method for extracting training information from a model provided in an embodiment of this application. The method can be described by… Figure 1 The model training device in the system architecture shown is executed. For example... Figure 2 As shown, the method may include the following steps:

[0082] Step 202: Obtain the third dataset, which includes multiple unlabeled third text samples.

[0083] Step 204: Use the pre-trained first information extraction model to extract target information of a preset type from the third text samples of the third dataset, and use it as the second target sample.

[0084] Step 206: Input the second target sample into the pre-trained text generation model, and obtain the text generated by the text generation model using the second target sample as the second text sample.

[0085] Step 208: Combine the second target sample and the corresponding second text sample to form a second sample pair, and use the second sample pair to obtain the second dataset.

[0086] Step 210: Train a second information extraction model using the second dataset, wherein the second text sample in the second sample pair is used as the input of the second information extraction model, and the difference between the target information of the preset target type extracted by the second information extraction model from the input second text sample and the second target sample in the second sample pair is minimized.

[0087] As can be seen from the above process, this application adopts a bidirectional generative framework. First, a trained first information extraction model generates target information from unlabeled third text samples. Then, a text generation model reverse-engineers the target information to generate text. The generated target information and text are used as a second sample pair to construct a new dataset, i.e., the third dataset, which is then used to train the second text information extraction model. This approach can utilize unlabeled sample data to expand a large amount of new sample data for training the text information extraction model, improving the extraction capability of the model and reducing the need for labeled data, thereby reducing the cost of manual labeling.

[0088] It should be noted that the terms "first," "second," etc., used in this disclosure do not imply any limitation in size, order, or quantity; they are merely used for descriptive convenience to differentiate between the three datasets. For example, "first dataset," "second dataset," and "third dataset" are used to distinguish between three datasets. Similarly, "first information extraction model" and "second information extraction model" are used to distinguish between two information extraction models. And so on.

[0089] The following describes each step in the above method flow. First, step 202, namely "obtaining a third dataset, which includes multiple unlabeled third text samples," will be described in detail with reference to the embodiment.

[0090] With the explosive growth of network data, text data is readily available, and large amounts of text data can be obtained in any data domain. However, manually labeling specific target information within these texts to generate target samples would be very costly. In this embodiment, after obtaining a third dataset containing multiple unlabeled third text samples, manual labeling is not used; instead, subsequent steps are performed to automatically generate labeled data.

[0091] In real-world scenarios, the third dataset may be a dataset belonging to a new data domain that differs from the data domain of the existing information extraction model (the first information extraction model in this embodiment). A data domain is a collection of closely related data topics, typically categorized based on multiple dimensions such as industry, service category, data source, and data purpose.

[0092] The following describes in detail step 204, namely, "using the pre-trained first information extraction model to extract target information of a preset type from the third text sample of the third dataset as the second target sample", with reference to the embodiments.

[0093] The first information extraction model involved in this application embodiment is an information extraction model that has been trained using a first dataset. The first dataset is a labeled dataset, meaning that the text samples in the first dataset are labeled with corresponding target information. Specifically, the first dataset may contain multiple first sample pairs, and each first sample pair includes a first text sample and a first target sample corresponding to the first text sample. The first target sample includes target information of a preset type labeled for the first text sample, which is commonly referred to as a "label".

[0094] Different application scenarios correspond to different types of target information. For example, in entity recognition applications, the target information is entity words of a preset type, which may include, but is not limited to, one or more combinations of: person names, place names, organization names, work names, brand names, addresses, dates, etc. As another example, in emotion recognition applications, the target information is emotion elements of a preset type, which may include, but is not limited to, at least one of: emotion objects, emotion words, emotion polarity, and emotional information. A more detailed description will follow using emotion recognition applications as an example.

[0095] The main research direction of information extraction models is to solve various extraction tasks within a specific data domain. However, in practical applications, multiple data domains often exist. When applying an existing information extraction model to a new data domain, the information extraction accuracy of the model is poor due to differences in information representation between different data domains. It is necessary to transfer the information extraction model after obtaining labeled samples in the new data domain. However, manual labeling is costly and time-consuming, posing a significant challenge to cross-domain transfer of information extraction models. In this embodiment, the first information extraction model can be considered as an information extraction model already trained on an existing data domain, aiming to transfer the information extraction task to a new data domain. In this case, the first dataset and the third dataset correspond to different data domains.

[0096] The training process of the first information extraction model is briefly described below. The first information extraction model is trained using the first dataset, where the first text sample in the first sample pair is used as the input of the first information extraction model. The first information extraction model extracts target information of a preset type from the first text sample. The training objective is to minimize the difference between the target information of the preset type extracted by the first information extraction model from the input first text sample and the first target sample in the first sample pair.

[0097] The information extraction models (including a first information extraction model and a second information extraction model) involved in this application embodiment can adopt a generative seq2seq (Sequence to Sequence) model. The information extraction model takes a text sequence as input and outputs a text sequence. When the first information extraction model is input with a first text sample, it outputs target information of a preset type and the corresponding type identifier in the first text sample. The type identifier can be in the form of text or an identifier. The format of the output sequence of the first information extraction model can be predefined, for example, it can be in the format "<type identifier 1> target information 1; <type identifier 2> target information 2; ...". Correspondingly, when labeling the first text sample with its corresponding first target sample, the first target sample also adopts the same format of text sequence.

[0098] The first information extraction model can include an encoder network and a decoder network.

[0099] The encoding network is used to encode the features of each token (element) in the input text sequence (which is the first text sample at this time) to obtain the feature representation of each token.

[0100] The encoding network can first perform embedding processing on each token in the input text sequence. This embedding process can include at least word embedding and position embedding. Word embedding involves encoding each token into word vectors to obtain word vector representations. Position embedding involves encoding the position of each token in the input text sequence to obtain position representations.

[0101] Each token in a text sequence refers to an element that constitutes the text sequence. For a text sequence, if the text sequence is divided into character or word sequences, then the characters or words in the text sequence, as well as the start character and the separator, are all tokens.

[0102] The decoding network is used to perform decoding processing using the feature representation of each token. The output is also a text sequence, which is the target information of the preset type and the type identifier corresponding to the target information extracted from the input text sequence.

[0103] The information extraction model described in this embodiment can be trained based on a pre-trained language model. It employs pre-trained language models such as T5 (Transfer Text-to-Text Transformer), BERT (Bidirectional Encoder Representation from Transformers), XLNet (an autoregressive model that uses permutations of language models to achieve bidirectional contextual information), and GPT (Generative Pre-Training) as its model framework, and then performs further training on top of these. BERT is a bidirectional pre-trained language model that uses a Transformer Encoder as its model structure, and it effectively utilizes contextual information for feature learning. XLNet is a model similar to BERT, a more generalized autoregressive pre-trained model. GPT uses a Transformer Decoder structure and retains only masked multi-head attention within the Transformer Decoder. T5 is still an encoder-decoder structure consisting of stacked Transformer layers, but the Decoder has a standard attention layer after the self-attention layer. This standard attention layer will take the encoder's output into the attention calculation. The last layer of the Decoder outputs the probability of each token belonging to each preset type through the Softmax classifier.

[0104] The aforementioned first information extraction model can be viewed as a "text-to-label" model. During training, it can construct a loss function using the training objective, such as the maximum likelihood loss function. In each iteration, based on the value of the loss function, the model parameters of the first information extraction model are updated using methods such as gradient descent until a preset training termination condition is met. This termination condition may include, for example, the loss function value being less than or equal to a preset loss function threshold, or the number of iterations reaching a preset threshold.

[0105] In this step, the first information extraction model that has been trained can be used to extract information from the unlabeled third text samples in the third dataset, and the extracted target information of a preset type can be used as the second target sample.

[0106] The following describes step 206, namely, "inputting the second target sample into the pre-trained text generation model and obtaining the text generated by the text generation model using the second target sample as the second text sample," in conjunction with an embodiment.

[0107] If the third dataset and the first dataset come from different data domains, the target information extracted by the first information extraction model from the third dataset will be inaccurate, meaning the second target sample will be inaccurate relative to the third text sample. Alternatively, if the first and third datasets come from the same data domain, but the first dataset has a smaller data volume, the target information extracted by the first information extraction model from the third dataset will also be inaccurate, meaning the second target sample will be inaccurate relative to the third text sample. Therefore, in step 206 of this embodiment, a text generation model is further used to perform a "from label to text" conversion. This involves generating text based on the second target sample (i.e., target information of a preset type). This text can be understood as text containing the target information from the second target sample. Using this text as the second text sample and forming a sample pair with the second target sample is more accurate than forming a sample pair with the third text sample and the second target sample.

[0108] The training process of the text generation model is briefly described below. To ensure the accuracy of the text generation model, it can still be trained on the first dataset. The first target sample in the first sample pair is used as the input to the text generation model. The model generates text using the first target sample, and the training objective is to minimize the difference between the text generated by the model using the first target sample and the first text sample in the first sample pair. In other words, the processing of the text generation model can be seen as the inverse process of the first information extraction model.

[0109] The text generation model involved in this embodiment can also be a generative seq2seq (sequence-to-sequence) model. The text generation model takes a text sequence as input and outputs a text sequence. When the text generation model is input with a first target sample, it outputs text containing the target information in the first target sample, and this target information still corresponds to a preset type in the text.

[0110] Text generation models can include an encoder network and a decoder network.

[0111] The encoding network is used to encode the features of each element Token in the input text sequence (which is the first target sample at this time) to obtain the feature representation of each Token.

[0112] The encoding network can first perform embedding processing on each token in the input text sequence. This embedding process can include at least word embedding and position embedding. Word embedding involves encoding each token into word vectors to obtain word vector representations. Position embedding involves encoding the position of each token in the input text sequence to obtain position representations.

[0113] Each token in a text sequence refers to an element that constitutes the text sequence. For a text sequence, if the text sequence is divided into character or word sequences, then the characters or words in the text sequence, as well as the start character and the separator, are all tokens.

[0114] The decoding network is used to perform decoding processing using the characteristic representation of each token, and the output is also a text sequence, which is the text containing the target information in the input text sequence.

[0115] The text generation model described in this application embodiment can be trained based on a pre-trained language model, using pre-trained language models such as T5, BERT, XLNet, and GPT as the model framework, and then further trained on this basis.

[0116] The text generation model described above can be viewed as a "label-to-text" model. During training, a loss function can be constructed using the aforementioned training objective, such as the maximum likelihood loss function. In each iteration, the model parameters are updated using methods such as gradient descent based on the value of the loss function, until a preset training termination condition is met. This termination condition may include, for example, the loss function value being less than or equal to a preset loss function threshold, or the number of iterations reaching a preset threshold.

[0117] It should be noted that, in the above embodiments of this application, the first dataset is preferably used to train the text generation model. However, in addition, an extra dataset can be constructed to train the text generation model, which will not be described in detail here.

[0118] The following describes step 208, namely, "forming a second sample pair by combining the second target sample and the second text sample corresponding to the second target sample, and using the second sample pair to obtain the second dataset," in conjunction with an embodiment.

[0119] In this step, the second target sample input to the text generation model and the second text sample output by the text generation model can be combined to form a second sample pair. Multiple such second sample pairs constitute the second dataset. This second dataset is actually generated from the third dataset; it is a new dataset obtained by expanding the unlabeled third text samples in the third dataset after processing them in steps 204 and 206 above. This third dataset can be used to train the second information extraction model.

[0120] The second text sample generated by the text generation model can be seen as natural language formed by adding context to the target information in the second target sample. However, some factors may affect the quality of the generated text during the generation process. Therefore, the second dataset can be filtered, and this filtering process can include, but is not limited to, the following:

[0121] The first filtering process: Filter out second sample pairs from the second dataset whose target information is inconsistent with that of the second text sample and the second target sample.

[0122] Under normal circumstances, the target information in the second target sample and the target information in the second text sample should be consistent in the sample pair used to train the second information extraction model. The second text sample should not contain any target information other than the target information in the second target sample, nor should it have any omissions or inconsistent target information. Therefore, second sample pairs with these conditions should be filtered out to improve the quality of the second information extraction model.

[0123] The second filtering process: filter out second sample pairs from the second dataset that do not meet the preset format requirements.

[0124] As mentioned in the above embodiments, the format of the output sequence of the first information extraction model in this application embodiment is predefined, and the target information needs to be output according to the defined format. For example, the format is "<type identifier 1> target information 1; <type identifier 2> target information 2;...". For second target samples that do not meet the format requirements, the second sample pair they belong to can be deleted.

[0125] The following describes step 210, namely "training the second information extraction model using the second dataset", in detail with reference to the embodiments.

[0126] One feasible approach is to use a second dataset to train the second information extraction model. The second text sample from the second sample pair is used as input to the second information extraction model, minimizing the difference between the target information of a preset type extracted by the second information extraction model from the input second text sample and the second target sample in the second sample pair.

[0127] The structure of the second information extraction model is the same as that of the first information extraction model described above, and will not be repeated here.

[0128] As another feasible approach, the first and second datasets can be merged, and the merged dataset can be used to train the second information extraction model. This involves sampling sample pairs (which could be either the first or second sample pair) from the merged dataset, using the text samples from these sample pairs as input to the second information extraction model, and minimizing the difference between the target information of a preset type extracted by the second information extraction model from the input text samples and the target samples in the sample pair.

[0129] The aforementioned second information extraction model is also a "text-to-label" model. During training, it can construct a loss function using the training objective, such as the maximum likelihood loss function. In each iteration, based on the value of the loss function, the model parameters of the second information extraction model are updated using methods such as gradient descent until a preset training termination condition is met. This termination condition may include, for example, the loss function value being less than or equal to a preset loss function threshold, or the number of iterations reaching a preset threshold.

[0130] Before training, the model parameters of the second information extraction model can be initialized randomly or with preset parameter values. However, as a preferred implementation, the model parameters of the first information extraction model can be used for initialization. That is, the second information extraction model is fine-tuned based on the first information extraction model, thereby realizing knowledge transfer from the first information extraction model to the second information extraction model.

[0131] After training and obtaining the second information extraction model, it can be used for information extraction. For example... Figure 3 As shown, the information extraction process may include the following steps:

[0132] Step 302: Obtain the text to be extracted.

[0133] The text to be extracted can be text belonging to the same data domain as the third dataset, or text belonging to the same data domain as the first dataset.

[0134] Step 304: Input the text to be extracted into the second information extraction model to obtain the target information of the preset type extracted by the second information extraction model from the text to be extracted.

[0135] In this step, the encoding network in the second information extraction model performs feature encoding on each token in the input text to be extracted, and obtains the feature representation of each token.

[0136] The encoding network can first perform embedding processing on each token in the input text to be extracted. This embedding processing can include at least word embedding and position embedding. Word embedding involves encoding each token into word vectors to obtain word vector representations. Position embedding involves encoding the position of each token in the text to be extracted to obtain position representations.

[0137] The decoding network is used to perform decoding processing using the characteristic representation of each token. The output is a text sequence, which is the target information of the preset type extracted from the text to be extracted, as well as the type identifier corresponding to the target information.

[0138] The type identifier can be in the form of text or an identifier. The text sequence output by the second information extraction model adopts a predefined format, such as "<type identifier 1> target information 1; <type identifier 2> target information 2; ...".

[0139] The methods provided in this application embodiment can be applied to various application scenarios, such as entity extraction, sentiment extraction, etc. The following description uses sentiment extraction as an example to illustrate the methods provided in this application embodiment.

[0140] The emotion extraction task is a task that analyzes human emotions, that is, extracting emotion elements of a pre-defined type from text. The pre-defined emotion elements include at least one of the following: emotion object, emotion word, emotion polarity, and emotional information.

[0141] The emotional object, also known as the emotional aspect, refers to the object to which the emotion is directed. The emotional object can be a transaction or an attribute of a transaction. For example, in the text "This dress doesn't look good," the emotional object is the transaction "dress." Similarly, in the text "This dress is too small," the emotional object is an attribute of a transaction, namely "the size of the dress."

[0142] Emotional words, also known as evaluative words or opinion words, refer to words that express emotions, evaluations, or opinions. For example, in the sentence "The apple is so sweet," the emotional word is "sweet."

[0143] Emotion refers to a user's views or opinions on things, which are usually formed over a long period of time. Emotional polarity refers to the discrete values ​​that divide emotions, such as positive, negative, and neutral.

[0144] Emotions refer to the reactions formed by changes in a user's perceptions over a short period of time, including emotions such as joy, sadness, fear, anger, surprise, and disgust.

[0145] Sentiment extraction tasks can extract at least one of the aforementioned sentiment elements. For example, the aspect term extraction (ATE) task extracts sentiment objects from text. Another example is the sentiment triple extraction (ASTE) task, which extracts sentiment objects, sentiment words, and sentiment polarities from text.

[0146] The main research direction in sentiment extraction is to solve various extraction tasks within a specific data domain. However, in practical applications, multiple data domains often exist. When applying existing information extraction models to new data domains, the expression of sentiment elements differs between different data domains. The accuracy of sentiment extraction in the new data domain is poor, requiring the acquisition of labeled samples in the new domain for model transfer. However, the high cost and time-consuming nature of manual labeling pose significant challenges to cross-domain transfer of information extraction models.

[0147] In this embodiment, it is assumed that a labeled dataset for a certain data domain already exists. This data domain is called the source domain, and the labeled dataset is called the first dataset, denoted as D. S D S The first sample pair is denoted as {x,y}, where x is the first text sample and y is the first target sample. The first target sample includes sentiment elements of a preset type in x.

[0148] Taking the task of extracting emotional triples as an example, y can be in the following format:

[0149] “ <aspect>XXX; <opinion>YYY; <sentiment>ZZZ”

[0150] in," <aspect> ”、" <opinion> ”、" <sentiment>"XXX", "YYY", and "ZZZ" are the type identifiers for the emotion object, emotion word, and emotion polarity, respectively.

[0151] Using D S The first information extraction model is trained, wherein after x is input into the first information extraction model, the output sequence of the first information extraction model can include sentiment elements of a preset type and the type identifier of the sentiment elements. The training objective is to minimize the difference between the output sequence and the corresponding y.

[0152] Furthermore, using D S The text generation model is trained by inputting y into it, and generating a text sequence containing the sentiment elements of y. The training objective is to minimize the difference between the generated text sequence and its corresponding x. The text generation model can be viewed as the reverse process of information extraction performed by the first information extraction model.

[0153] The first information extraction model trained performs better in the source domain for sentiment extraction, but its accuracy is poor in a new data domain called the target domain. A large amount of unlabeled text in the target domain can be easily obtained as a third dataset, denoted as D. T The unannotated text is represented as x. T .

[0154] like Figure 4 As shown, the first information extraction model is used for x. T Information extraction is performed to obtain the output sequence. Includes from x T The extracted sentiment elements and their corresponding type identifiers. Because the first information extraction model performs poorly in the target domain, therefore... This is inaccurate, therefore the text generation model can continue to be used.

[0155] like Figure 4 As shown, Input text generation model, output text generation model containing Text sequences containing emotional elements Using this process, a new dataset D can be obtained. g D g The text pairs in the text are represented as These sample pairs are all data from the target domain. It can be seen that the text generation model acts as a "mediator," resulting in pseudo-sample pairs... Minimized The noise present in the process.

[0156] For D g Further filtering can be performed if the sample pairs... middle, and The emotional elements contained therein are inconsistent, for example Emotional elements included It is not in the middle, or Emotional elements included It is not in the middle, or and If the same emotional element is expressed in different ways, then the sample pair is deleted. If the sample pair... middle If the format does not meet the requirements, the sample pair will be deleted.

[0157] like Figure 4 As shown, the filtered D g and D S By merging the data, the model parameters of the first information extraction model are used to further train the model on the merged dataset, resulting in a second information extraction model. This second information extraction model achieves excellent sentiment extraction performance in both the source and target domains.

[0158] For example, when inputting the text "This apple is so sweet!" into the second information extraction model, it can extract pre-defined sentiment elements from the text and output a sequence:

[0159] " <aspect>apple; <opinion>So sweet; <sentiment>positive".

[0160] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0161] According to another embodiment, an apparatus for training an information extraction model is provided. Figure 5 A schematic block diagram of an apparatus for extracting training information from a model according to one embodiment is shown. Figure 1 The model training device in the architecture shown. Figure 5 As shown, the device 500 includes: a second acquisition unit 501, a sample expansion unit 502, a third acquisition unit 503, and a third training unit 504, and may further include: a first acquisition unit 505, a first training unit 506, a second training unit 507, and a data filtering unit 508. The main functions of each component are as follows:

[0162] The second acquisition unit 501 is configured to acquire a third dataset, which includes multiple unlabeled third text samples.

[0163] The sample expansion unit 502 is configured to extract target information of a preset type from the third text sample of the third dataset using a pre-trained first information extraction model as a second target sample; input the second target sample into a pre-trained text generation model to obtain the text generated by the text generation model using the second target sample as the second text sample.

[0164] The third acquisition unit 503 is configured to form a second sample pair by combining the second target sample and the second text sample corresponding to the second target sample, and to obtain a second dataset by using multiple second sample pairs.

[0165] The third training unit 504 is configured to train a second information extraction model using a second dataset, wherein the second text sample in the second sample pair is used as the input of the second information extraction model, and the difference between the target information of a preset type extracted by the second information extraction model from the input second text sample and the second target sample in the second sample pair is minimized.

[0166] The first acquisition unit 505 is configured to acquire a first dataset, which contains multiple first sample pairs. Each first sample pair includes a first text sample and a first target sample corresponding to the first text sample. The first target sample includes target information of a preset type labeled for the first text sample.

[0167] The first training unit 506 is configured to train a first information extraction model using a first dataset, wherein the first text sample in the first sample pair is used as the input of the first information extraction model, and the difference between the target information of a preset type extracted by the first information extraction model from the input first text sample and the first target sample in the first sample pair is minimized.

[0168] The second training unit 507 is configured to train a text generation model using the first dataset, wherein the first target sample in the first sample pair is used as the input of the text generation model, and the difference between the text generated by the text generation model using the input first target sample and the first text sample in the first sample pair is minimized.

[0169] As one possible approach, the first and third datasets mentioned above correspond to different data domains.

[0170] Furthermore, the data filtering unit 508 can be configured to filter out second sample pairs from the second dataset where the target information of the second text sample and the second target sample is inconsistent; or, to filter out second sample pairs from the second dataset where the second target sample does not meet the preset format requirements.

[0171] The third training unit 504 uses the second dataset filtered by the data filtering unit 508 to train the second information extraction model.

[0172] As one possible approach, the third training unit 504 can merge the first and second datasets and use the merged dataset to train a second information extraction model, wherein the second extraction model is initialized using the model parameters of the first information extraction model.

[0173] As one possible approach, both the first and second information extraction models described above employ a sequence-to-sequence model. The sequences output by both models include target information and the type identifier corresponding to that target information.

[0174] According to another embodiment, an information extraction device is provided. Figure 6 A schematic block diagram of an information extraction apparatus according to one embodiment is shown. Figure 6 As shown, the device 600 includes a text acquisition unit 601 and an information extraction unit 602. The main functions of each component are as follows:

[0175] The text acquisition unit 601 is configured to acquire the text to be extracted.

[0176] The information extraction unit 602 is configured to input the text to be extracted into the second information extraction model and obtain target information of a preset type extracted from the text to be extracted by the second information extraction model.

[0177] Among them, the aforementioned second information extraction model is composed of Figure 5 The device shown was pre-trained.

[0178] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0179] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0180] In addition, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0181] And an electronic device, comprising:

[0182] One or more processors; and

[0183] A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any of the foregoing method embodiments.

[0184] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0185] in, Figure 7 An exemplary architecture of an electronic device is shown, which may include a processor 710, a video display adapter 711, a disk drive 712, an input / output interface 713, a network interface 714, and a memory 720. The processor 710, video display adapter 711, disk drive 712, input / output interface 713, network interface 714, and memory 720 can communicate with each other via a communication bus 730.

[0186] The processor 710 can be implemented using a general-purpose CPU, microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits to execute relevant programs in order to implement the technical solution provided in this application.

[0187] The memory 720 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 720 can store the operating system 721 for controlling the operation of the electronic device 700, and the basic input / output system (BIOS) 722 for controlling the low-level operations of the electronic device 700. Additionally, it can store a web browser 723, a data storage management system 724, and a model training device / information extraction device 725, etc. The aforementioned model training device / information extraction device 725 can be the application program that specifically implements the aforementioned steps in this embodiment. In summary, when the technical solution provided in this application is implemented through software or firmware, the relevant program code is stored in the memory 720 and is called and executed by the processor 710.

[0188] Input / output interface 713 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0189] Network interface 714 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0190] Bus 730 includes a pathway for transmitting information between various components of the device, such as processor 710, video display adapter 711, disk drive 712, input / output interface 713, network interface 714, and memory 720.

[0191] It should be noted that although the above-described device only shows the processor 710, video display adapter 711, disk drive 712, input / output interface 713, network interface 714, memory 720, bus 730, etc., in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the solution of this application, and does not necessarily include all the components shown in the figures.

[0192] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer program product. This computer program product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0193] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.< / sentiment> < / opinion> < / aspect> < / sentiment> < / opinion> < / aspect> < / sentiment> < / opinion> < / aspect>

Claims

1. A method of training an information extraction model, the method comprising: The method includes: Obtain a first dataset, which contains multiple first sample pairs. Each first sample pair includes a first text sample and a first target sample corresponding to the first text sample. The first target sample includes target information of a preset type that is labeled for the first text sample. A first information extraction model and a text generation model are trained using the first dataset. The first text sample from the first sample pair is used as input to the first information extraction model to minimize the difference between the target information of a preset type extracted by the first information extraction model from the input first text sample and the first target sample in the first sample pair. The first target sample from the first sample pair is used as input to the text generation model to minimize the difference between the text generated by the text generation model using the input first target sample and the first text sample in the first sample pair. Obtain a third dataset, which includes multiple unlabeled third text samples; A bidirectional generative framework is adopted, and a pre-trained first information extraction model is used to extract target information of a preset type from the third text samples of the third dataset as the second target sample. The second target sample is input into a pre-trained text generation model, and the text generated by the text generation model using the second target sample is used as the second text sample. The second target sample and the second text sample corresponding to the second target sample are combined to form a second sample pair, and multiple second sample pairs are used to obtain a second dataset. The second information extraction model is trained using the second dataset, wherein the second text sample in the second sample pair is used as the input of the second information extraction model, and the difference between the target information of the preset type extracted by the second information extraction model from the input second text sample and the second target sample in the second sample pair is minimized.

2. The method of claim 1, wherein, The first dataset and the third dataset correspond to different data domains.

3. The method according to claim 1, characterized in that, Before training the second information extraction model using the second dataset, the following steps are also included: Filter out second sample pairs from the second dataset whose target information is inconsistent with that of the second text sample and the second target sample; and / or, Filter out second sample pairs from the second dataset that do not meet the preset format requirements.

4. The method according to claim 1, characterized in that, Training the second information extraction model using the second dataset includes: The first dataset and the second dataset are merged, and the merged dataset is used to train the second information extraction model, wherein the second extraction model is initialized using the model parameters of the first information extraction model.

5. The method according to any one of claims 1 to 4, characterized in that, Both the first information extraction model and the second information extraction model adopt a sequence-to-sequence model; The sequences output by the first information extraction model and the second information extraction model include target information and the type identifier corresponding to the target information.

6. An information extraction method, characterized in that, The method includes: Get the text to be extracted; The text to be extracted is input into the second information extraction model to obtain target information of a preset type extracted from the text to be extracted by the second information extraction model. The second information extraction model is pre-trained using the method described in any one of claims 1 to 5.

7. An information extraction method, executed by a cloud server, the method comprising: Obtain the text to be extracted from the user terminal; The text to be extracted is input into the second information extraction model to obtain target information of a preset type extracted from the text to be extracted by the second information extraction model. The extracted target information is returned to the user terminal; The second information extraction model is pre-trained using the method described in any one of claims 1 to 5.

8. A method for training an emotion extraction model, characterized in that, The method includes: Obtain a first dataset, which contains multiple first sample pairs. Each first sample pair includes a first text sample and a first target sample corresponding to the first text sample. The first target sample includes target information of a preset type that is labeled for the first text sample. A first information extraction model and a text generation model are trained using the first dataset. The first text sample from the first sample pair is used as input to the first information extraction model to minimize the difference between the target information of a preset type extracted by the first information extraction model from the input first text sample and the first target sample in the first sample pair. The first target sample from the first sample pair is used as input to the text generation model to minimize the difference between the text generated by the text generation model using the input first target sample and the first text sample in the first sample pair. Obtain a third dataset, which includes multiple unlabeled third text samples; A bidirectional generative framework is adopted, and a first information extraction model that has been pre-trained is used to extract a preset type of sentiment element from the third text sample of the third dataset as a second sentiment element sample. The preset type of sentiment element includes at least one of sentiment object, sentiment word, sentiment polarity and emotional information. The second sentiment element sample is input into the pre-trained text generation model, and the text generated by the text generation model using the second sentiment element sample is used as the second text sample. The second sentiment element sample and the second text sample corresponding to the second sentiment element sample are used to form a second sample pair, and the second sample pair is used to obtain the second dataset. The second information extraction model is trained using the second dataset, wherein the second text sample in the second sample pair is used as the input of the second information extraction model, and the difference between the preset type of sentiment element extracted by the second information extraction model from the input second text sample and the second sentiment element sample in the second sample pair is minimized.

9. An emotion extraction method, characterized in that, The method includes: Get the text to be extracted; The text to be extracted is input into the second information extraction model to obtain the preset type of emotional elements extracted by the second information extraction model from the text to be extracted. The preset type of emotional elements includes at least one of emotional object, emotional word, emotional polarity and emotional information. The second information extraction model is pre-trained using the method described in claim 8.

10. An apparatus for training an information extraction model, characterized in that, The device includes: The first acquisition unit is configured to acquire a first dataset, the first dataset containing multiple first sample pairs, the first sample pair including a first text sample and a first target sample corresponding to the first text sample, the first target sample including target information of a preset type annotated for the first text sample; The first training unit is configured to train a first information extraction model and a text generation model using the first dataset, respectively; wherein the first text sample in the first sample pair is used as the input of the first information extraction model to minimize the difference between the target information of a preset type extracted by the first information extraction model from the input first text sample and the first target sample in the first sample pair; the first target sample in the first sample pair is used as the input of the text generation model to minimize the difference between the text generated by the text generation model using the input first target sample and the first text sample in the first sample pair. The second acquisition unit is configured to acquire a third dataset, which includes multiple unlabeled third text samples; The sample expansion unit is configured to use a bidirectional generation framework to extract target information of a preset type from the third text samples of the third dataset using a pre-trained first information extraction model as a second target sample; input the second target sample into a pre-trained text generation model to obtain the text generated by the text generation model in reverse using the second target sample as a second text sample. The third acquisition unit is configured to form a second sample pair by combining the second target sample and the second text sample corresponding to the second target sample, and to obtain a second dataset by using multiple second sample pairs. The third training unit is configured to train a second information extraction model using the second dataset, wherein the second text sample in the second sample pair is used as the input of the second information extraction model, and the difference between the target information of a preset type extracted by the second information extraction model from the input second text sample and the second target sample in the second sample pair is minimized.

11. An information extraction device, characterized in that, The device includes: The text acquisition unit is configured to acquire the text to be extracted. The information extraction unit is configured to input the text to be extracted into a second information extraction model and obtain target information of a preset type extracted by the second information extraction model from the text to be extracted. The second information extraction model is pre-trained by the device described in claim 10.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 9.

13. An electronic device, characterized in that, include: One or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method according to any one of claims 1 to 9.

Images