Method for training information extraction model, information extraction method and corresponding device

Abstract

Embodiments of the present application disclose a method for training an information extraction model, an information extraction method and corresponding devices, and relate to the technical field of artificial intelligence. The method comprises: obtaining a training sample set, wherein the information category labels of some elements of the samples in the training sample set are not correctly labeled; training an information extraction model using the training sample set; sampling samples from the training sample set, wherein the sampling rate of the samples corresponding to each category label is determined according to the accuracy and recall of the information extraction model obtained by the current training on the corresponding category; identifying the information category of the sampled samples using the information extraction model obtained by the current training, and labeling the information category labels of the sampled samples using the identification results; and returning the sampled samples with the labeled information category labels to the training sample set, and proceeding to the step of training the information extraction model using the training sample set. The present application can improve the information extraction effect of the information extraction model.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method for training an information extraction model, an information extraction method, and a corresponding apparatus. Background Technology

[0002] Information extraction (IE) originally referred to extracting specific categories of information from text, thereby automatically classifying, extracting, and reconstructing massive amounts of content, such as named entity recognition, entity relationship extraction, and event extraction. With the continuous development of artificial intelligence technology, information extraction technology has gradually expanded to extracting specific categories of information from images, videos, etc., such as object detection and the recognition of relationships between objects.

[0003] The construction of information extraction models relies on training sample sets, which contain a large number of samples labeled with information category tags. However, in real-world scenarios, training sample sets often suffer from severe incomplete labeling issues. That is, some elements in samples that should have been labeled with information category tags are not correctly labeled, including being unlabeled or incorrectly labeled. In this case, these elements become negative examples of the information category tags during training, introducing a lot of noise into the model training process and thus affecting the performance of the information extraction model. Summary of the Invention

[0004] In view of this, this application provides a method for training an information extraction model, an information extraction method, and a corresponding apparatus to improve the effectiveness of information extraction.

[0005] This application provides the following solution:

[0006] Firstly, a method for training an information extraction model is provided, the method comprising:

[0007] Obtain a training sample set, in which some samples have elements whose information category labels are not correctly labeled;

[0008] The information extraction model is trained using the training sample set.

[0009] Samples are sampled from the training sample set, wherein the sampling rate of samples corresponding to each category label is determined based on the accuracy and recall of the currently trained information extraction model in the corresponding category;

[0010] Using the currently trained information extraction model, the information category of the sampled samples is identified, and the identification results are used to label the sampled samples with information category labels.

[0011] The sampled samples with the labeled category information are put back into the training sample set, and the process proceeds to the step of training the information extraction model using the training sample set.

[0012] According to one achievable method in the embodiments of this application, the sampling rate of samples corresponding to category labels is positively correlated with the accuracy of the currently trained information extraction model in the corresponding category, and negatively correlated with the recall rate of the currently trained information extraction model in the corresponding category.

[0013] According to one achievable method in the embodiments of this application, the sampling rate of samples corresponding to each category label is determined based on the accuracy and recall of the currently trained information extraction model in the corresponding category, including:

[0014] For category i, using μ i =[P i *(1-R i )] β Determine the sampling rate μ of the samples corresponding to this category label. i , where P i R represents the accuracy of the currently trained information extraction model on category i. i β represents the recall rate of the currently trained information extraction model in category i, and β is a hyperparameter.

[0015] According to one feasible method in an embodiment of this application, training an information extraction model using the training sample set includes:

[0016] The information extraction model is used to input the sample data from the training sample set, and the information extraction model outputs the information category of the elements in the sample.

[0017] In each iteration, the model parameters of the information extraction model are updated using the value of the loss function until the preset training termination condition is met. The loss function is pre-constructed based on the training objective, which includes minimizing the difference between the information category and the corresponding information category label of the element in the sample output by the information extraction model.

[0018] According to one feasible embodiment of this application, before sampling samples from the training sample set, the method further includes:

[0019] Obtain the F1 score of the model obtained from the current training;

[0020] If the F1 score meets the preset criteria, the currently trained information extraction model will be used as the final information extraction model; otherwise, the step of sampling a preset number of samples from the training sample set will continue.

[0021] According to one achievable method in an embodiment of this application, before sampling a preset number of samples from the training sample set, the method further includes:

[0022] Obtain a verification sample set, wherein all elements of the samples in the verification sample set are labeled with the correct information category labels;

[0023] The information extraction model trained so far is used to identify the information categories of samples in the validation sample set.

[0024] By using the identification results and the information category labels of the samples in the validation sample set, the accuracy and recall of the information extraction model in each category are obtained.

[0025] According to one achievable method in an embodiment of this application, the sample is a text sample, the element is an entity pair, and the information category is an entity relationship category; or...

[0026] The sample is a text sample, the element is a character or word in the text sample, and the information category is an entity category; or...

[0027] The sample is an image sample, the element is an image patch in the image sample, and the information category includes people and object categories.

[0028] Secondly, an information extraction method is provided, the method comprising:

[0029] Obtain resource objects, including text or images;

[0030] The resource object is input into the information extraction model to obtain the information category of the elements in the resource object output by the information extraction model;

[0031] The information extraction model is pre-trained using the method described in any one of the first aspects above.

[0032] Thirdly, a method for extracting entity relations is provided, the method comprising:

[0033] Get the text;

[0034] The text input information extraction model is used to obtain the relationship categories between entity pairs in the text output by the information extraction model;

[0035] The information extraction model is pre-trained using the method described in any one of the first aspects above.

[0036] Fourthly, an apparatus for training an information extraction model is provided, the apparatus comprising:

[0037] The sample acquisition unit is configured to acquire a training sample set, wherein the information category labels of some samples are not correctly labeled.

[0038] The model training unit is configured to extract a model using training information from the training sample set.

[0039] The sample sampling unit is configured to sample samples from the training sample set, wherein the sampling rate of samples corresponding to each category label is determined based on the accuracy and recall of the currently trained information extraction model in the corresponding category;

[0040] The labeling unit is configured to use the currently trained information extraction model to identify the information category of the sampled samples, and use the identification result to label the sampled samples with information category labels; put the sampled samples with labeled information category labels back into the training sample set, and trigger the model training unit to train the information extraction model using the training sample set.

[0041] Fifthly, an information extraction device is provided, the device comprising:

[0042] An object acquisition unit is configured to acquire resource objects, including text or images;

[0043] The information extraction unit is configured to input the resource object into the information extraction model and obtain the information category of the elements in the resource object output by the information extraction model.

[0044] The information extraction model is pre-trained using the apparatus described in the fourth aspect above.

[0045] According to a fifth aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any one of the first aspects.

[0046] According to a sixth aspect, an electronic device is provided, comprising:

[0047] One or more processors; and

[0048] A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any one of the first aspects above.

[0049] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0050] 1) This application draws on the idea of ​​self-training, uses the accuracy and recall of the information extraction model in the corresponding category to sample samples, and puts the information category labels (i.e., "pseudo-labels") generated by the information extraction model after identifying the information category of the sampled samples back into the training sample set to train the information extraction model, thereby reducing the impact of the information category labels that are not correctly labeled, and thus improving the information extraction effect of the information extraction model.

[0051] 2) The self-training method of the information extraction model provided in this application can still produce an information extraction model with excellent performance on a training sample set with incomplete labeling, avoiding the problem of needing to manually supplement the training sample set and greatly reducing labor costs.

[0052] 3) In this application, the sampling rate of samples corresponding to category labels is positively correlated with the accuracy of the currently trained information extraction model in the corresponding category and negatively correlated with the recall of the currently trained information extraction model in the corresponding category. This allows for the priority sampling of samples corresponding to categories with high accuracy but low recall. By gradually adding "pseudo-labels" to the sampled samples, the recall of these categories is improved while ensuring accuracy.

[0053] 4) In this application, the self-training loop is controlled by the F1 index value of the information extraction model, so that the self-training process of the information extraction model can be executed cyclically until the information extraction model meets the preset index conditions.

[0054] Of course, any product implementing this application does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a system architecture diagram applicable to the embodiments of this application;

[0057] Figure 2 A flowchart illustrating the method for extracting training information from a model provided in this application embodiment;

[0058] Figure 3 A structural block diagram of the entity relationship extraction model provided in the embodiments of this application;

[0059] Figure 4 A schematic block diagram of a training information extraction model provided in an embodiment of this application;

[0060] Figure 5 A schematic block diagram of an information extraction device provided in the embodiments of this application;

[0061] Figure 6 A schematic block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0062] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0063] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0064] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0065] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0066] In information extraction scenarios, the problem of incomplete labeling in the training sample set is significant. Adding information category labels to the samples in the training sample set is labor-intensive. In some scenarios, the number of samples in the training sample set is in the tens of thousands, or even millions, making it virtually impossible to add information category labels.

[0067] In view of this, this application draws on the idea of ​​model self-training and provides a novel training method for information extraction models. To facilitate understanding of this application, the system architecture to which this application is applied will be briefly described first. Figure 1 An exemplary system architecture that can be applied to embodiments of this application is shown, such as Figure 1 As shown, the system architecture includes a model training device that builds an information extraction model offline, and an information extraction device that extracts information from resource objects online.

[0068] After acquiring the training sample set, the model training device can use the method provided in the embodiments of this application to train the model and obtain the information extraction model.

[0069] The information extraction device uses an established information extraction model to extract information from resource objects, obtaining the information categories of elements within the resource objects. These resource objects can be resource categories such as text or images. For example, entity recognition is performed on text to obtain entity category information. Another example is entity relation extraction from text to obtain relation category information for entity pairs. Yet another example is object detection on images to obtain category information for people and objects within the image.

[0070] The model training device and information extraction device can be configured as independent servers, or they can be configured on the same server or server cluster, or they can be configured on a separate or the same cloud server. A cloud server, also known as a cloud computing server or cloud host, is a host product in the cloud computing service system, designed to address the shortcomings of traditional physical hosts and Virtual Private Servers (VPs) services, such as high management difficulty and weak service scalability. The model training device and prediction device can also be configured on computer terminals with strong computing capabilities.

[0071] It should be noted that, in addition to online information extraction, the aforementioned information extraction device can also perform information extraction offline, such as extracting information from batches of resource objects separately.

[0072] It should be understood that Figure 1 The number of model training devices, information extraction devices, and information extraction models shown in the diagram is merely illustrative. Depending on the implementation requirements, any number of model training devices, information extraction devices, and information extraction models can be included.

[0073] Figure 2 This is a flowchart of a method for extracting training information from a model provided in an embodiment of this application. This method can be performed by... Figure 1 The model training device in the system architecture shown is executed. For example... Figure 2 As shown, the method may include the following steps:

[0074] Step 202: Obtain the training sample set. In the training sample set, some sample elements have not been correctly labeled with their category information.

[0075] Step 204: Train the information extraction model using the training sample set.

[0076] Step 206: Sample samples from the training sample set, wherein the sampling rate of samples corresponding to each category label is determined based on the accuracy and recall of the information extraction model trained at the current level in the corresponding category.

[0077] Step 208: Using the information extraction model trained now, identify the information category of the sampled samples, and use the identification results to label the sampled samples with information category labels.

[0078] Step 210: Put the sampled samples with the labeled information category tags back into the training sample set, and proceed to the step of training the information extraction model using the training sample set.

[0079] As can be seen from the above process, this application draws on the idea of ​​self-training, uses the accuracy and recall of the information extraction model in the corresponding category to sample samples, and puts the "pseudo-labels" generated by the information extraction model after identifying the information category of the sampled samples back into the training sample set to train the information extraction model, thereby reducing the impact of the information category labels that are not correctly labeled, and thus improving the information extraction effect of the information extraction model.

[0080] The following describes each step in the above method flow. First, step 202, "obtaining the training sample set," will be described in detail with reference to the embodiment.

[0081] The training sample set obtained in this step refers to a sample set with incomplete information category labeling issues. That is, some elements in the samples that should have been labeled with information category labels were not correctly labeled. Incorrect labeling includes not being labeled at all or being labeled incorrectly. Not being labeled means that an element in the sample that should have been labeled with information category label A was not labeled with that category label. Incorrect labeling means that an element in the sample that should have been labeled with information category label A was labeled with information category B.

[0082] The training sample set can contain text samples, image samples, etc. Image samples can be pictures or video frames. Different elements in the samples are labeled with information categories for different application scenarios. For example, in entity recognition, the training sample set labels characters or words in text samples with entity categories. In entity relation extraction, the training sample set labels entity pairs in text samples with entity relation categories. In object detection, the training sample set labels image patches in image samples with people and object categories.

[0083] In general, correctly labeled elements are the majority in the training sample set, while incorrectly labeled elements are the minority. However, these incorrectly labeled elements become negative examples of the correct information category labels, introducing noise during model training. Taking entity relationship extraction as an example, two datasets are currently widely used as training sample sets when training entity relationship extraction models: TACRED and DocRED. However, these two datasets suffer from serious labeling incompleteness. For instance, suppose Zhang San and Li Si are both employees of Company A, but the text in the training sample set labels Zhang San and Li Si incorrectly as "no relationship" or "family relationship," when in fact they should be labeled as "colleagues." The training dataset obtained in this application can be directly based on these two training sample sets without additional manual labeling.

[0084] The following describes step 204, namely "training the information extraction model using the training sample set", in detail with reference to the embodiments.

[0085] The model training process in this step is supervised training. Specifically, the process of training the extraction model involves inputting samples from the training sample set into the extraction model, which then outputs the information category of each element in the sample. In each iteration, the model parameters of the extraction model are updated using the value of the loss function until the preset training termination condition is met.

[0086] The loss function can be pre-constructed based on the training objective, which includes minimizing the difference between the information category and the corresponding information category label of an element in the sample output by the information extraction model. The loss function can be, for example, the cross-entropy loss function.

[0087] The training termination conditions mentioned above may include, for example, the value of the loss function being less than or equal to a preset loss function threshold, or the number of iterations reaching a preset threshold.

[0088] This application does not limit the specific structure of the information extraction model; information extraction models of any structure are applicable. This application does not improve the information extraction model itself. Taking the relation extraction model as an example, such as... Figure 3 As shown, the relation extraction model can include a feature extraction network and a classification network.

[0089] A text input feature extraction network is used, where the text already contains labeled entity pairs. This network extracts features from the input text, obtaining feature representations of the entity pairs. A classification network then performs classification based on these feature representations, mapping them to entity relation categories to determine the corresponding entity relation category.

[0090] The feature extraction network first performs embedding processing on each token in the input text. Tokens in the text can include characters, delimiters, etc. In different languages, tokens may also include words. The embedding process can include at least word embedding and position embedding. Word embedding involves encoding each token into word vectors to obtain word vector representations. Position embedding involves encoding the position of each token in the text sequence to be predicted to obtain position representations.

[0091] Based on the Embedding processing results of each Token, the feature extraction network can extract features from entity pairs using methods such as convolutional networks and pre-trained language models.

[0092] The aforementioned classification network can be implemented using networks such as the Softmax network for multi-class classification.

[0093] The following describes in detail, with reference to the embodiments, step 206, namely "sampling samples from the training sample set", and step 208, namely "using the currently trained information extraction model to identify the information category of the sampled samples, and using the identification results to label the sampled samples with information category labels".

[0094] Experiments have shown that when there are a large number of incompletely labeled samples in the training sample set, these incompletely labeled samples will become false "negative examples" of the corresponding information category, introducing noise during the model training process. The information extraction results of the information extraction model trained based on this have the following characteristics: high accuracy but very low recall.

[0095] This application borrows the concept of self-training, sampling a portion of samples from unlabeled samples. An information extraction model is then used to label these sampled samples with information category labels, or pseudo-labels. These pseudo-labels are positive labels with high confidence, and are then used together with correctly labeled samples to further train the information extraction model. Through this process, the number of positive labels gradually increases, leading to a gradual improvement in the recall rate of the information extraction model. However, traditional self-training methods introduce confirmation bias, where the pseudo-labels assigned by the information extraction model tend to concentrate on high-frequency categories, and this bias is continuously reinforced in subsequent self-training iterations. Therefore, this application improves the self-training method to suit information extraction scenarios, proposing a category-adaptive self-training approach. The sampling rate for each category is determined based on the accuracy and recall of the information extraction model across each category.

[0096] First, it is necessary to calculate the accuracy and recall of the information extraction model across each category. One feasible approach is to determine the accuracy and recall of the information extraction model across each category by analyzing its performance on a validation set. This involves using the currently trained information extraction model to identify the information categories of samples in the validation set; then, by comparing the identification results with the information category labels assigned to the samples in the validation set, the accuracy and recall of the information extraction model across each category can be obtained.

[0097] The validation sample set contains all elements correctly labeled with their information category. This validation sample set can be a small dataset labeled with all information categories, specifically used to validate the accuracy and recall of the information extraction model. Given the small size of the validation sample set, manual labeling of the information category labels can be used to ensure the correctness of each category label.

[0098] Among them, the accuracy P of the information extraction model on category i i This refers to the ratio of the number of samples correctly identified in category i by the information extraction model to the total number of samples in that category, which can be determined using the following formula:

[0099]

[0100] Among them, TP i To determine the number of correctly identified samples in category i by the information extraction model, FP i This represents the number of samples that the information extraction model incorrectly identifies in category i.

[0101] The recall R of the information extraction model in category i i This refers to the proportion of information of category i that the information extraction model identifies in samples of category i, which can be determined by the following formula:

[0102]

[0103] Among them, FN i This represents the proportion of samples in category i where the information extraction model fails to identify that information category.

[0104] During the sampling process, it is desirable to prioritize sampling categories with high precision but low recall, thereby maximizing recall for those categories while maintaining precision. Therefore, when designing the sampling rate for category i, it is desirable to ensure that the sampling rate is positively correlated with the precision of the information extraction model on category i and negatively correlated with the recall of the information extraction model on category i.

[0105] As one possible approach, the sampling rate μ for samples of category i is... i The following formula can be used to determine it:

[0106] μ i =[P i *(1-R i )] β (3)

[0107] Where β is a hyperparameter used to control the sampling smoothness, and can be an empirical or experimental value.

[0108] When sampling samples from the training sample set, a preset number of samples can be sampled.

[0109] For samples sampled from the training sample set according to the sampling rate of each category, the currently trained information extraction model is used to identify the information category of the sampled samples, and the identification results are used to label the sampled samples with information category labels, i.e., assigning "pseudo-labels". For the sampled samples, if an element that was not originally labeled with an information category is identified by the information extraction model, then the element is labeled with an information type according to the identification results of the information extraction model. If an element that was originally labeled with an information category is identified by the information extraction model as having a different information type, then the element's label is changed according to the identification results of the information extraction model to the information type label identified by the information extraction model. If an element that was originally labeled with an information category is identified by the information extraction model as having the same information type, then the element's information type label remains unchanged.

[0110] After labeling the sampled samples based on the recognition results of the information extraction model, in step 210, the labeled sampled samples are put back into the training sample set. This increases the number of correctly labeled information category labels in the training sample set. The information extraction model is then trained again using the training sample set, which is the self-training cycle. This gradually improves the recognition effect of the information extraction model, ensuring both accuracy and recall in each category.

[0111] The termination condition of the aforementioned self-training loop can be controlled by the recognition performance of the information extraction model. One possible approach is to obtain the F1 score of the currently trained information extraction model before step 206. If the F1 score meets a preset condition, the currently trained information extraction model is taken as the final information extraction model, thus ending the self-training loop. Figure 2 Follow the process shown; otherwise, continue with step 206.

[0112] The F1 score is a commonly used metric in classification problems. For example, macro-F1, which is the arithmetic mean of the F1 scores for each category, can be used. Another example is weighted-F1, which uses the number of samples in each category as a weight to sum the F1 scores for each category.

[0113] For class i, the F1 score represents the average of the precision and recall of the information extraction model on class i. It can be expressed by the following formula:

[0114]

[0115] Similarly, when calculating the F1 score of the information extraction model, the precision and recall of each category are determined by the performance of the information extraction model on the validation sample set, which will not be elaborated here.

[0116] After training to obtain the final information extraction model, information can be extracted using this model. First, resource objects are acquired, which can include text or images. Then, the resource objects are input into the information extraction model, and the model outputs the information categories of the elements within the resource objects.

[0117] Taking entity relation extraction as an example, we obtain text and aim to extract the relationships between entities from it. We can pre-annotate the text with entity information or entity pair information. Then, we input this text into the finally trained information extraction model (in this example, the entity relation extraction model). The model outputs the relationship categories between entity pairs in the text, thus extracting the entity relation information from the text.

[0118] Taking entity recognition as an example, the goal is to extract entities from text and determine their categories. This text can be input into a pre-trained information extraction model (in this case, an entity recognition model). The model outputs entity category labels for each token in the text, thus revealing the entities and their categories within the text.

[0119] Taking object detection as an example, the goal is to extract the object category and location information from an image. This image can be input into a finally trained information extraction model (in this example, an object detection model). The model outputs the object category labels (e.g., individual pixels or image blocks of a preset size) for each image patch, thus obtaining the object and its location information from the image.

[0120] In addition to the three application scenarios mentioned above, the methods provided in this application embodiment can also be applied to other application scenarios, which will not be listed here.

[0121] This application conducts experiments on entity relation extraction scenarios. Using the DocRED dataset as the training sample set, entity relation extraction model 1 is trained directly on the training sample set. Entity relation extraction model 2 is trained using a traditional self-training method. Entity relation extraction model 3 is trained using the method provided in this application's embodiments. Experiments comparing the F1 scores of each entity relation extraction model show that entity relation extraction model 3 consistently outperforms the others, improving recall while maintaining high accuracy. Furthermore, entity relation extraction model 3 demonstrates superior performance in both document-level and sentence-level relation extraction.

[0122] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0123] According to another embodiment, an apparatus for training an information extraction model is provided. Figure 4 A schematic block diagram of an apparatus for extracting training information from a model according to one embodiment is shown. Figure 1 The model training device in the architecture shown. Figure 4 As shown, the device 400 includes: a sample acquisition unit 401, a model training unit 402, a sample sampling unit 403, and a label annotation unit 404. It may further include a loop control unit 405 and a model validation unit 406. The main functions of each component are as follows:

[0124] The sample acquisition unit 401 is configured to acquire a training sample set in which the information category labels of some sample elements are not correctly labeled.

[0125] Model training unit 402 is configured to extract the model using training sample set training information.

[0126] The sample sampling unit 403 is configured to sample samples from the training sample set, wherein the sampling rate of samples corresponding to each category label is determined based on the accuracy and recall of the currently trained information extraction model in the corresponding category.

[0127] The labeling unit 404 is configured to use the currently trained information extraction model to identify the information category of the sampled samples, and use the identification result to label the sampled samples with information category labels; put the sampled samples with labeled information category labels back into the training sample set, and trigger the model training unit 402 to train the information extraction model using the training sample set.

[0128] As one possible approach, the sampling rate of samples corresponding to category labels is positively correlated with the accuracy of the currently trained information extraction model in the corresponding category, and negatively correlated with the recall rate of the currently trained information extraction model in the corresponding category.

[0129] As one possible approach, for category i, sample sampling unit 403 utilizes μ i =[P i *(1-R i )] β Determine the sampling rate μ of the samples corresponding to this category label. i , where P i R represents the accuracy of the currently trained information extraction model on category i. i β represents the recall rate of the currently trained information extraction model in category i, and β is a hyperparameter.

[0130] As one possible implementation, the model training unit 402 can be configured to input the information extraction model from the training sample set, and output the information category of the elements in the sample; in each iteration, the model parameters of the information extraction model are updated using the value of the loss function until the preset training termination condition is met, wherein the loss function is pre-constructed according to the training objective, and the training objective includes: minimizing the difference between the information category of the elements in the sample output by the information extraction model and the corresponding information category label.

[0131] Furthermore, the loop control unit 405 is configured to obtain the F1 index value of the currently trained information extraction model; if the F1 index value meets the preset index conditions, the information extraction model currently trained by the model training unit 402 is used as the final information extraction model; otherwise, the sample sampling unit 403 is triggered to perform the processing of sampling a preset number of samples from the training sample set.

[0132] The F1 score is a metric used to measure the performance of classification problems. For example, macro-F1, which is the arithmetic mean of the F1 scores for each category, can be used. Another example is weighted-F1, which uses the number of samples in each category as a weight to sum the F1 scores for each category.

[0133] Furthermore, the model validation unit 406 is configured to acquire a validation sample set, wherein all elements of the samples in the validation sample set are labeled with the correct information category labels; to use the currently trained information extraction model to identify the information category of the samples in the validation sample set; and to use the identification results and the information category labels labeled on the samples in the validation sample set to obtain the accuracy and recall of the information extraction model in each category and provide them to the sample sampling unit 403 and the loop control unit 405.

[0134] As one application scenario, the above samples are text samples, the elements are entity pairs, and the information categories are entity relationship categories, which are used to train the entity relationship extraction model.

[0135] As another application scenario, the above samples are text samples, the elements are characters or words in the text samples, and the information category is the entity category, which is used to train the entity recognition model.

[0136] As another application scenario, the above samples are image samples, the elements are image patches in the image samples, and the information categories include people and object categories, which are used to train the target detection model.

[0137] Figure 5 A schematic block diagram of an information extraction apparatus according to one embodiment is shown, such as Figure 5 As shown, the device 500 includes an object acquisition unit 501 and an information extraction unit 502. The main functions of each component are as follows:

[0138] The object acquisition unit 501 is configured to acquire resource objects, which may include text or images, etc.

[0139] The information extraction unit 502 is configured to input a resource object into an information extraction model and obtain the information category of the elements in the resource object output by the information extraction model.

[0140] The information extraction model utilizes, for example, Figure 4 The device shown is pre-trained.

[0141] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0142] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0143] In addition, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0144] And an electronic device, comprising:

[0145] One or more processors; and

[0146] A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any of the foregoing method embodiments.

[0147] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0148] in, Figure 6An exemplary architecture of an electronic device is shown, which may include a processor 610, a video display adapter 611, a disk drive 612, an input / output interface 613, a network interface 614, and a memory 620. The processor 610, video display adapter 611, disk drive 612, input / output interface 613, network interface 614, and memory 620 can communicate with each other via a communication bus 630.

[0149] The processor 610 can be implemented using a general-purpose CPU, microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits to execute relevant programs and implement the technical solution provided in this application.

[0150] The memory 620 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 620 can store the operating system 621 for controlling the operation of the electronic device 600, and the basic input / output system (BIOS) 622 for controlling the low-level operations of the electronic device 600. Additionally, it can store a web browser 623, a data storage management system 624, and an information extraction device / model training device 625, etc. The aforementioned information extraction device / model training device 625 can be the application program that specifically implements the aforementioned steps in this embodiment. In summary, when the technical solution provided in this application is implemented through software or firmware, the relevant program code is stored in the memory 620 and is called and executed by the processor 610.

[0151] Input / output interface 613 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0152] Network interface 614 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0153] Bus 630 includes a pathway for transmitting information between various components of the device, such as processor 610, video display adapter 611, disk drive 612, input / output interface 613, network interface 614, and memory 620.

[0154] It should be noted that although the above-described device only shows the processor 610, video display adapter 611, disk drive 612, input / output interface 613, network interface 614, memory 620, bus 630, etc., in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the solution of this application, and does not necessarily include all the components shown in the figures.

[0155] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer program product. This computer program product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0156] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for training an information extraction model, characterized in that, The method includes: Obtain a training sample set, wherein some samples in the training sample set have elements whose information category labels are not labeled or are not correctly labeled; the training sample set includes text samples and / or image samples, and the information category labels not being labeled or not correctly labeled includes: in entity recognition scenarios, the entity category of characters or words in text samples in the training sample set is not labeled or is not correctly labeled; or in entity relation extraction scenarios, the relationship category between entity pairs in text samples in the training sample set is not labeled or is not correctly labeled; or in object detection scenarios, the category of people or objects in image blocks in image samples in the training sample set is not labeled or is not correctly labeled. The information extraction model is trained using the training sample set. The information extraction model trained in the current test set is used to identify the information category of the samples in the validation test set. The accuracy and recall of the information extraction model trained in the current test set are obtained by comparing the identification results with the information category labels of the samples in the validation test set. All elements of the samples in the validation test set are labeled with the correct information category labels. Samples are sampled from the training sample set, wherein the sampling rate of samples corresponding to each category label is positively correlated with the accuracy of the currently trained information extraction model in the corresponding category, and negatively correlated with the recall rate in the corresponding category; Using the currently trained information extraction model, the sampled samples are identified in terms of information category, and the identification results are used to label the sampled samples with information category labels. Specifically, for the sampled samples, those that were not originally labeled with information category labels are labeled with information category labels according to the identification results of the currently trained information extraction model. For samples that were originally labeled with information category labels, after being identified as having different information categories by the currently trained information extraction model, the information category labels of the sampled samples are changed according to the identification results of the currently trained information extraction model. The sampled samples with labeled information category labels are then put back into the training sample set, and the process proceeds to the step of training the information extraction model using the training sample set.

2. The method according to claim 1, characterized in that, The sampling rate of samples corresponding to each category label is positively correlated with the accuracy of the currently trained information extraction model in the corresponding category, and negatively correlated with the recall rate in the corresponding category, including: For category ,use Determine the sampling rate of the samples corresponding to this category label. ,in For the information extraction model trained now, in categories The accuracy rate For the information extraction model trained now, in categories Recall rate on This is a hyperparameter.

3. The method according to claim 1, characterized in that, Training the information extraction model using the training sample set includes: The information extraction model is used to input the sample data from the training sample set, and the information extraction model outputs the information category of the elements in the sample. In each iteration, the model parameters of the information extraction model are updated using the value of the loss function until the preset training termination condition is met. The loss function is pre-constructed based on the training objective, which includes minimizing the difference between the information category and the corresponding information category label of the element in the sample output by the information extraction model.

4. The method according to claim 1, characterized in that, Before sampling samples from the training sample set, the method further includes: Obtain the F1 score of the model obtained from the current training; If the F1 score meets the preset criteria, the currently trained information extraction model will be used as the final information extraction model; otherwise, the step of sampling a preset number of samples from the training sample set will continue.

5. An information extraction method, characterized in that, The method includes: Obtain resource objects, including text or images; The resource object is input into the information extraction model to obtain the information category of the elements in the resource object output by the information extraction model; The information extraction model is pre-trained using the method described in any one of claims 1 to 4.

6. A method for extracting entity relations, characterized in that, The method includes: Get the text; The text input information extraction model is used to obtain the relationship categories between entity pairs in the text output by the information extraction model; The information extraction model is pre-trained using the method described in any one of claims 1 to 4.

7. An apparatus for training an information extraction model, characterized in that, The device includes: The sample acquisition unit is configured to acquire a training sample set, wherein some samples in the training sample set have elements whose information category labels are not labeled or are not correctly labeled; the training sample set includes text samples and / or image samples, and the information category labels not being labeled or not correctly labeled includes: in an entity recognition scenario, the entity category of characters or words in the text samples in the training sample set is not labeled or is not correctly labeled; or in an entity relation extraction scenario, the relation category between entity pairs in the text samples in the training sample set is not labeled or is not correctly labeled; or in an object detection scenario, the category of people or objects in the image blocks in the image samples in the training sample set is not labeled or is not correctly labeled. The model training unit is configured to extract a model using training information from the training sample set. The model validation unit is configured to use the currently trained information extraction model to identify the information category of the samples in the validation sample set, and to use the identification results and the information category labels labeled on the samples in the validation sample set to obtain the accuracy and recall of the currently trained information extraction model in each category; wherein, all elements of the samples in the validation sample set are labeled with the correct information category labels. The sample sampling unit is configured to sample samples from the training sample set, wherein the sampling rate of samples corresponding to each category label is positively correlated with the accuracy of the currently trained information extraction model in the corresponding category and negatively correlated with the recall rate in the corresponding category. The labeling unit is configured to use the currently trained information extraction model to identify the information category of the sampled samples, and to label the sampled samples with information category tags based on the identification results. Specifically, for the sampled samples, samples that were not originally labeled with information category tags are labeled with information category tags according to the identification results of the currently trained information extraction model. Samples that were originally labeled with information category tags are identified as having different information categories by the currently trained information extraction model, and their information category tags are changed according to the identification results of the currently trained information extraction model. The sampled samples with labeled information category tags are then put back into the training sample set, triggering the model training unit to train the information extraction model using the training sample set.

8. An information extraction device, characterized in that, The device includes: An object acquisition unit is configured to acquire resource objects, including text or images; The information extraction unit is configured to input the resource object into the information extraction model and obtain the information category of the elements in the resource object output by the information extraction model. The information extraction model is pre-trained using the apparatus as described in claim 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 6.

10. An electronic device, characterized in that, include: One or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method according to any one of claims 1 to 6.

Images