Information extraction model training method, information extraction method, and device

Abstract

The present disclosure provides a training method of an information extraction model, an information extraction method and device, and relates to the technical field of artificial intelligence, in particular to the field of deep learning and natural language processing. The specific implementation scheme is: performing N times of iteration operations based on an initialized target model to obtain an information extraction model; wherein the i-th iteration operation in the N times of iteration operations comprises: based on a current industry data proportion, extracting general data and / or industry data of at least one industry from a first data set to obtain a second data set, the current industry data proportion is greater than the industry data proportion in the i-1th iteration operation, i is a positive integer not greater than N; based on the second data set, training the i-1th updated target model to obtain the i-th updated target model; in the case of meeting a first preset condition, taking the i-th updated target model as the information extraction model. The present disclosure can reduce the artificial cost and resources of data labeling.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to the fields of deep learning and natural language processing. Background Technology

[0002] Currently, to achieve information extraction, a fully supervised approach can be used to train information extraction models, enabling them to extract information from text. Generally, to improve model performance, a large amount of labeled data needs to be acquired to train the model until convergence. Summary of the Invention

[0003] This disclosure provides a training method for an information extraction model, an information extraction method, and an apparatus.

[0004] According to one aspect of this disclosure, a method for training an information extraction model is provided, comprising:

[0005] Based on the initialized target model, perform N iterations to obtain the information extraction model; where N is an integer greater than 1.

[0006] The i-th iteration operation in the N iteration operations includes:

[0007] Based on the current industry data proportion, extract general data and / or industry data of at least one industry from the first dataset to obtain the second dataset; wherein, the current industry data proportion is greater than the industry data proportion in the (i-1)th iteration operation, and i is a positive integer not greater than N;

[0008] Based on the second dataset, train the target model for the (i-1)th update to obtain the target model for the i-th update;

[0009] If the first preset condition is met, the target model updated in the i-th time is used as the information extraction model.

[0010] According to another aspect of this disclosure, an information extraction method is provided, comprising:

[0011] The information extraction model is used to extract target information from the text to be processed; wherein the information extraction model is trained based on any model training method in the embodiments of this disclosure.

[0012] According to another aspect of this disclosure, a training apparatus for an information extraction model is provided, comprising:

[0013] The first iteration module is used to perform N iterations based on the initialized target model to obtain the information extraction model; where N is a positive integer.

[0014] The first iteration module includes:

[0015] The data determination unit is used to extract general data and / or industry data of at least one industry from the first dataset based on the current industry data proportion to obtain the second dataset; wherein the current industry data proportion is greater than the industry data proportion in the (i-1)th iteration operation, and i is a positive integer not greater than N;

[0016] The first training unit is used to train the target model updated for the (i-1)th time based on the second dataset, so as to obtain the target model updated for the ith time.

[0017] The first confirmation unit is used to use the target model updated in the i-th time as the information extraction model if the first preset condition is met.

[0018] According to another aspect of this disclosure, an information extraction apparatus is provided, comprising:

[0019] The model processing module is used to extract target information from the text to be processed using an information extraction model; wherein the information extraction model is trained based on any model training device in the embodiments of this disclosure.

[0020] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0021] At least one processor; and

[0022] The memory is communicatively connected to the at least one processor; wherein,

[0023] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform any of the methods described in the present disclosure.

[0024] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform any of the methods according to embodiments of this disclosure.

[0025] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the methods according to embodiments of this disclosure.

[0026] According to the technical solutions of the embodiments of this disclosure, the manual costs and resources for data annotation can be reduced.

[0027] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0028] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0029] Figure 1 This is a flowchart illustrating a training method for an information extraction model provided in an embodiment of this disclosure;

[0030] Figure 2 This is a schematic diagram of a pre-training overall scheme in an application example according to an embodiment of the present disclosure;

[0031] Figure 3 This is a schematic diagram illustrating a pre-training scheme in an application example according to an embodiment of the present disclosure;

[0032] Figure 4 This is a schematic flowchart of an information extraction method provided in an embodiment of this disclosure;

[0033] Figure 5 This is a schematic diagram illustrating an application example of the information extraction model in the embodiments of this disclosure;

[0034] Figure 6 This is a schematic block diagram of a training apparatus for an information extraction model provided in an embodiment of this disclosure;

[0035] Figure 7 This is a schematic block diagram of a training apparatus for an information extraction model provided in another embodiment of this disclosure;

[0036] Figure 8 This is a schematic block diagram of an information extraction device provided in an embodiment of the present disclosure;

[0037] Figure 9 This is a block diagram of an electronic device used to implement the methods of the embodiments of this disclosure. Detailed Implementation

[0038] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0039] The basic concepts involved in the embodiments of this disclosure are briefly described below. It should be understood that the basic concepts described below do not limit the embodiments of this disclosure.

[0040] 1. SPO (Subject-Predicate-Object) Triples: SPO triples are knowledge triple data, which include the subject (e.g., entity), the predicate (e.g., entity attribute / relationship between entities), and the object (e.g., entity attribute value / related entity).

[0041] 2. Entity: In a knowledge graph, an entity generally refers to a real-world object, but without a clearly defined boundary. The most commonly used entity in practical applications is the named entity, which is the name or symbol of a specific type of object. It generally refers to real-world objects that can be identified by names, such as person names, place names, and organization names. In a broader sense, named entities also include time expressions, numeric expressions, addresses, etc.

[0042] 3. Attributes: The collective term for entity characteristics and relationships between entities.

[0043] 4. Slot: A slot is a unit of information or a filling unit, such as the unit of a subject (S) and an object (O). S and O may contain multiple slots. It's important to note that missing slots are allowed in the actual extraction results. However, both S and O must have at least one slot. If both S and O contain only a single slot, the extracted SPO triples are simple knowledge triples. If either S or O contains multiple slots, the extracted SPO triples are multi-dimensional knowledge triples.

[0044] To facilitate understanding of the training method and information extraction method of the information extraction model provided in the embodiments of this disclosure, the relevant technologies of the embodiments of this disclosure are described below. The following relevant technologies are optional solutions and can be arbitrarily combined with the technical solutions of the embodiments of this disclosure, and they all fall within the protection scope of the embodiments of this disclosure.

[0045] In this disclosure, information extraction may include, but is not limited to, SPO triple extraction. In related technologies, the SPO triple extraction method is implemented in a pipeline manner, that is, SPO triple extraction is split into two sub-tasks: entity recognition and entity relation classification. First, the entity recognition task typically uses a sequence labeling model to identify entity pairs in the text. Then, for each entity pair, a relation classification model is used to determine the relationship between the two entities in the pair. This method ignores the correlation between the two sub-tasks, resulting in the feature information of the two sub-tasks not being able to interact. Furthermore, errors in the entity recognition module inevitably propagate to the relation classification module, thus affecting the overall extraction model's performance.

[0046] Alternatively, a pre-trained language model-based SPO triple extraction scheme can be adopted, which involves building an end-to-end SPO triple extraction model using a pre-trained language model and a multi-pointer classification module. However, this scheme requires a large amount of labeled data for model training to converge, resulting in significant manpower costs.

[0047] In practical applications, different models can be designed to solve different SPO triple extraction tasks, such as named entity recognition, entity relation extraction, and event attribute extraction. However, designing multiple models requires significant human and resource investment and hinders interactive learning between different tasks.

[0048] The technical solutions of the embodiments disclosed herein are used to solve at least one of the above-mentioned problems. Figure 1 A flowchart illustrating a training method for an information extraction model according to an embodiment of this disclosure is shown. This method can be applied to a training device for an information extraction model, which can be deployed in an electronic device. The electronic device can be a single-machine or multi-machine terminal, server, or other processing device. The terminal can be a mobile device, a personal digital assistant (PDA), a handheld device, a computing device, an in-vehicle device, a wearable device, or other user equipment (UE). In some possible implementations, the method can also be implemented by a processor calling computer-readable instructions stored in memory. Figure 1 As shown, the method may include:

[0049] Step S110: Perform N iterations based on the initialized target model to obtain the information extraction model; where N is an integer greater than 1.

[0050] The i-th iteration operation in the N iteration operations includes:

[0051] Based on the current industry data proportion, extract general data and / or industry data of at least one industry from the first dataset to obtain the second dataset; wherein, the current industry data proportion is greater than the industry data proportion in the (i-1)th iteration operation, and i is a positive integer not greater than N;

[0052] Based on the second dataset, train the target model for the (i-1)th update to obtain the target model for the i-th update;

[0053] If the first preset condition is met, the target model updated in the i-th time is used as the information extraction model.

[0054] For example, in this embodiment of the disclosure, step S110 can be applied to the pre-training stage of the model. Specifically, the training of the target model updated for the (i-1)th time in step S110 can refer to continued pre-training. Optionally, after continuing pre-training of the model, the information extraction model can also be fine-tuned.

[0055] Optionally, the number of iterations N can be preset or determined based on the model's performance. In one example, the number of iterations is preset, and the first preset condition in step S110 above could be that the current iteration number i equals N. In another example, the number of iterations is determined based on the model's performance, and the first preset condition in step S110 above could be that the model's performance meets convergence requirements, such as the model's prediction accuracy in zero-sample / small-sample scenarios reaching a preset threshold or no longer increasing. In practical applications, the model's performance can be validated using industry data corresponding to downstream tasks after each iteration of the training process.

[0056] For example, in this embodiment of the disclosure, the first dataset may include industry data from at least one industry and general data. The amount of industry data and the amount of general data are not limited. General data can be used to enable the model to learn the basic paradigm of information extraction through context. Industry data can improve the model's performance in downstream zero-shot / few-shot scenarios. At least one industry may include power, healthcare, finance, etc., and correspondingly, industry data for at least one industry may include power-related data, healthcare-related data, financial-related data, etc.

[0057] In the above embodiments, the industry data proportion can be the ratio between the quantity of industry data and the quantity of general data, or it can be the ratio between the quantity of industry data and the total quantity of data in the second dataset. The specific ratio can be set according to actual needs. It can be understood that extracting general data and / or at least one industry data from the first dataset based on the industry data proportion means extracting a certain amount of general data and / or industry data from the first dataset such that the quantity of industry data meets the quantity requirement corresponding to the industry data proportion.

[0058] Optionally, in this embodiment of the disclosure, the proportion of industry data in each iteration operation can be preset. For example, the proportion of industry data in N iteration operations can be preset to 0, 1 / 10, 1 / 5, etc.

[0059] Optionally, in this embodiment of the disclosure, the current industry data proportion can also be determined in the i-th iteration operation based on the industry data proportion in the (i-1)-th iteration operation and the prediction effect of the target model updated in the (i-1)-th iteration operation. The specific determination method can be set according to actual needs.

[0060] Optionally, the industry data percentage can be zero or infinite. For example, during the first iteration, the industry data percentage can be zero, and correspondingly, general data is extracted from the first dataset to obtain the second dataset used in the first iteration. Subsequently, the industry data percentage gradually increases, and industry data and general data are extracted from the first dataset to obtain the second dataset for each iteration. During the final iteration, the industry data percentage can be infinite, and correspondingly, industry data is extracted from the first dataset to obtain the second dataset used in the final iteration.

[0061] Based on the above method, during the update of the target model using a first dataset containing both general and industry data, the training process involves iteratively increasing the proportion of industry data. This allows the model to learn semantic knowledge from massive amounts of information extraction data using general data, while gradually increasing its learning from industry data. Since the amount of industry data is much smaller than that of general data in real-world scenarios, this method is beneficial for improving the learning effect of industry data. This allows for the use of a small amount of industry data and a large amount of general data, significantly improving the model's predictive ability and transferability across industry sectors. When supporting downstream tasks, only a small number of labeled samples are needed to achieve good results, thus realizing the effectiveness of information extraction models in zero-sample / small-sample scenarios and significantly reducing manual costs.

[0062] In some exemplary embodiments of this disclosure, the i-th iteration operation may further include: adjusting the target model for the i-th update based on industry data of the target industry in at least one industry.

[0063] The target industry can be industry data from downstream tasks. For example, if the downstream task is news event attribute extraction, then after each iteration of model training, industry data from the news industry can be used to fine-tune the currently updated target model.

[0064] According to this implementation method, it is beneficial to improve the model's prediction performance in downstream tasks related to the target industry, that is, the model's capabilities can be improved in a targeted manner based on downstream tasks.

[0065] In one exemplary implementation, the i-th iteration operation may further include: obtaining the prediction result of the target model for the first dataset in the i-th update; determining the irrelevant industry data subset in at least one industry data subset in the first dataset based on the prediction result; and deleting the irrelevant industry data subset in the first dataset.

[0066] For example, the prediction result may include target information output by the target model for each data point (e.g., each text) in the first dataset.

[0067] In practical applications, the industry data (i.e., at least one subset of industry data) in the first dataset may contain industry data that is irrelevant to the information extraction task. According to the above implementation method, the prediction results of the target model on the original data after each training iteration can be obtained. Based on these prediction results, irrelevant industry data subsets can be identified. By deleting these irrelevant industry data subsets, the quality of the first dataset is improved, allowing it to gradually play a more positive role during model training and thus improving the model training effect.

[0068] In some exemplary implementations, based on the prediction results, determining an irrelevant industry data subset within at least one industry data subset in the first dataset includes: determining sample data in the first dataset that does not contain target information based on the prediction results; and determining an irrelevant industry data subset within at least one industry data subset in the first dataset based on the sample data that does not contain target information.

[0069] For example, if the pre-designed information extraction model is a medical entity recognition model, and the first dataset includes industry data subsets from the financial or entertainment industries, the target model may be unable to extract target information from the data in these training dataset subsets during training. Based on the prediction results, sample data lacking target information can be summarized, and thus, irrelevant industry data subsets can be determined based on the industry data subsets to which these sample data belong.

[0070] According to the above implementation method, based on sample data without target information, a subset of data from irrelevant industries can be accurately determined, which is conducive to steadily improving the quality of the first dataset and thus improving the model training effect.

[0071] To facilitate understanding of the training method for the information extraction model described above, a specific application example is provided below.

[0072] The overall goal of the pre-training process is to enable the model to fully learn semantic knowledge from massive amounts of data through data-driven approaches, thereby improving the model's performance in zero-shot / few-shot scenarios. Specifically, the selection of an information extraction pre-training scheme needs to meet the following basic principles: the selected scheme should significantly improve the performance of downstream business datasets in zero-shot / few-shot scenarios; and it should possess strong domain-specific and low-resource generalization capabilities.

[0073] Figure 2This diagram illustrates the overall pre-training scheme in this application example. Because datasets with different accuracies have different effects on model pre-training, particularly, higher-accuracy data has a more positive impact on the model. Based on this, as... Figure 2 As shown, the training steps of the pre-training scheme are as follows:

[0074] First, using the initial model 201 and the initial information extraction pre-training data 202, including industry data (such as data from the power, military, and medical industries) and general data (such as encyclopedia information extraction data), we continue to pre-train to obtain the first stage model 203 for information extraction pre-training.

[0075] Then, the first-stage information extraction pre-training model 203 is used to predict the initial information extraction pre-training data 202, filtering out samples that the model failed to predict. Sampling is performed on industry data and general data for each domain. If a sample in a domain results in a null prediction due to sample quality issues, the corresponding sample is discarded. By discarding samples corresponding to that domain from the original data, the second-stage information extraction pre-training data 204 is obtained. This second-stage data is then used to further pre-train the first-stage information extraction pre-training model 203, resulting in the second-stage information extraction pre-training model 205.

[0076] Similarly, after multiple pre-training sessions, we obtain the third-stage information extraction pre-training model 206, the nth-stage information extraction pre-training model 207, and so on, until the validation performance of the information extraction model on the downstream dataset becomes stable.

[0077] Figure 3 This diagram illustrates the continued pre-training scheme in this application example. The continued pre-training scheme is the continued pre-training process within the overall pre-training scheme described above.

[0078] In practical applications, using general-domain data, models can learn the basic paradigm of information extraction through context. Furthermore, for zero-shot / few-shot performance on downstream industry data, models trained on industry-specific data outperform those trained on general-domain data. Based on this, such as... Figure 3 As shown, the training steps for the continued pre-training scheme are as follows:

[0079] First, a certain amount (around 10 million) of general domain data is used to obtain dataset 301. The initial model and dataset 301 are used to train the first stage model for information extraction pre-training.

[0080] In each subsequent training phase, industry data will be incorporated, with the proportion of industry data gradually increasing until only industry data is used in the final stage. For example... Figure 3As shown, in datasets 302 to 304, the ratio of industry data to general data increases from 1:10 to 1:5, and then increases further based on 1:5, until finally only industry data is used.

[0081] The model obtained in each training phase will be validated with zero samples using downstream industry data or fine-tuned with downstream industry data to verify the model's performance in small sample scenarios and record the model's performance in that phase.

[0082] It is understood that the models in the embodiments of this disclosure, including the target model, the information extraction model, and the stage models in the application examples, may have the same model structure, and the parameters under this model structure are optimized during training. Some embodiments of this disclosure also provide model structure design schemes to enable the model to adapt to different information extraction tasks.

[0083] For example, an information extraction model may include:

[0084] The prompt layer is used to obtain prompt information based on the extraction type and extracted information structure of the information extraction task;

[0085] A pre-trained language model is used to obtain feature representations of the input information based on prompts and the text to be processed;

[0086] An information extraction layer is used to obtain at least one set of target information in the text to be processed based on the feature representation of the input information; wherein each set of target information includes at least one piece of information that conforms to the extraction type and has an extracted information structure.

[0087] The above model structure will be illustrated below through the application of an information extraction model.

[0088] Figure 4 A flowchart illustrating an embodiment of the information extraction method provided in this disclosure is shown. This method can be applied to an information extraction device, which can be deployed in an electronic device. The electronic device may be a single-machine or multi-machine terminal, server, or other processing device. The terminal may be a mobile device, a personal digital assistant (PDA), a handheld device, a computing device, an in-vehicle device, a wearable device, or other user equipment (UE). In some possible implementations, the method can also be implemented by a processor calling computer-readable instructions stored in memory. Figure 4 As shown, the method may include:

[0089] Step S410: Extract target information from the text to be processed using an information extraction model; wherein the information extraction model is trained based on the training method in the aforementioned embodiments.

[0090] Because the information extraction model in this method is trained through multiple iterations by gradually increasing the proportion of industry data, the model has good predictive ability and transferability in the industry field, and can achieve accurate information extraction in zero-sample / small-sample scenarios.

[0091] For example, the text to be processed can be the text from which target information is to be extracted. Optionally, the text to be processed can include path-level text or non-path-level text.

[0092] The path-level text can include corpus that structures the hierarchical relationships between statements. For example, the path-level text is corpus containing directory information. For instance, a path-level text is as follows: "Transformer Technical Specifications; 1. General Provisions; 2. Operating Scope; 3. Operating Conditions; 3.1 Temperature and Humidity; 3.2 Seismic Resistance; Resonance, Sine Wave Test Method, Excitation 5 times, the safety factor of the transformer shall not be less than 1.67, and the horizontal acceleration of the equipment body shall take into account the dynamic amplification factor of the equipment support of 1.2."

[0093] Non-path-level text can include unstructured corpora. For example, a non-path-level text is as follows: "The origin of poetry can be traced back to ancient times. Not only is it historically rich, but the number and types of excellent poems from both China and abroad are countless. In order to better inherit the culture of poetry, promote corporate culture construction, showcase the vitality and style of employees, improve the cultural literacy of team members, enhance team cohesion and creativity, and build an innovative and learning organization, the 2023 Corporate Culture Season Poetry Recitation Competition was held at the Guangming Village Base."

[0094] In one exemplary implementation, extracting target information from a text to be processed using an information extraction model may include: inputting the text to be processed, the extraction type, and the extracted information structure into the information extraction model to obtain the target information in the text to be processed; wherein the target information includes at least one piece of information that satisfies the extracted information structure, and at least one piece of information satisfies the task requirements corresponding to the extraction type.

[0095] For example, the extraction type can refer to the type of information extraction task. For instance, the extraction type could be named entity recognition, entity relation extraction, or event attribute extraction.

[0096] Optionally, the extraction type may also include parameters related to the information extraction task. For example, for named entity recognition, entity relationship extraction, and event attribute extraction tasks, these parameters may be the entity type of the entity to be extracted, the relationship between a group of entities to be extracted, and the event name of the attribute to be extracted, respectively. For example, the extraction type may be used to indicate the extraction of named entities with the entity type "medical"; or to indicate the extraction of two entities with a "membership" relationship; or to indicate the extraction of various attributes of the event "meeting," where attributes may refer to event parameters such as time, location, and people.

[0097] For example, the extracted information structure can refer to the amount of information contained in the target information, and the order of different types of information within a set of target information. For instance, for named entity recognition, the extracted information structure might be: a set of target information includes one entity. Another example is entity relation extraction, where the extracted information structure might be: a set of target information includes two entities with a predetermined relationship, namely S and O in the SPO triplet data, where S precedes O in the model output. Yet another example is event attribute extraction, where the extracted information structure might be: a set of target information includes an unlimited number of event attributes.

[0098] In practical applications, the above-mentioned information extraction structure can be represented by slots. For example, if the information extraction structure is a set of target information including M pieces of information, it can be represented as an information extraction structure containing M slots.

[0099] According to the above implementation method, by improving the form of the input and corresponding output information of the information extraction model, a single information extraction model can handle multiple different types of tasks. That is, different tasks are uniformly treated as slot extraction tasks. Based on this, it is unnecessary to design different models to solve different information extraction tasks, which can significantly reduce labor costs and resources, and facilitate interactive learning between different tasks, thereby improving the information extraction effect.

[0100] Optionally, inputting the text to be processed, the extraction type, and the extracted information structure into the information extraction model to obtain the target information in the text to be processed can specifically include: inputting the text to be processed, the extraction type, and the extracted information structure into the information extraction model to obtain multiple sets of target information in the text to be processed; and aligning the multiple sets of target information.

[0101] For example, in entity relation extraction tasks, a text to be processed may contain multiple sets of entities with the same relation, requiring alignment of these sets. For instance, in the task of extracting entity relations with the relation "educational qualification," the text "Zhang San's educational qualification is a bachelor's degree, and Li Si's educational qualification is a doctorate" contains two sets of SO entity pairs. Specifically, the model can extract the S entities "Zhang San" and "Li Si," and the O entities "bachelor's degree" and "educational qualification." It needs to match the two S entities and the two O entities to determine one set of target information as "Zhang San" and "bachelor's degree," and another set as "Li Si" and "doctorate." This matching process can be called alignment. In other words, different tasks can be uniformly viewed as a combination of slot extraction and alignment tasks.

[0102] In one exemplary embodiment, inputting the text to be processed, the extraction type, and the extracted information structure into an information extraction model to obtain target information in the text to be processed may include: inputting the text to be processed, the extraction type, and the extracted information structure into the information extraction model; processing the extraction type and the extracted information structure using a prompting layer in the information extraction model to obtain prompting information; processing the prompting information and the text to be processed using a pre-trained language model in the information extraction model to obtain input information feature representation; and processing the input information feature representation using an information extraction layer in the information extraction model to obtain at least one set of target information in the text to be processed.

[0103] This implementation provides an exemplary model structure. Specifically, the information extraction model includes a prompting layer, a pre-trained language model, and an information extraction layer. The prompting layer is used to obtain prompting information corresponding to the extraction type and the structure of the extracted information; the pre-trained language model is used to obtain the feature representation of the input information; and the information extraction layer is used to extract information based on the feature representation of the input information.

[0104] By setting a cue layer in the model, the model can learn the relevant knowledge of task types and information extraction structures during training. This helps to treat different tasks as slot extraction tasks, reduce labor costs and resources, and improve information extraction results.

[0105] For example, inputting the text to be processed, the extraction type, and the extracted information structure into the information extraction model to obtain the target information in the text to be processed may further include: when multiple sets of target information are obtained, aligning the multiple sets of target information using the information extraction layer in the information extraction model.

[0106] For example, the nearest neighbor mechanism can be used for alignment. For instance, in the text to be processed, "Zhang San's education is a bachelor's degree, and Li Si's education is a doctorate", the extracted S entities "Zhang San" and "Li Si", and the extracted O entities "bachelor's degree" and "education degree", based on the nearest neighbor mechanism, "bachelor's degree" which is closer to "Zhang San" is aligned with "Zhang San", and "doctorate" which is closer to "Li Si" is aligned with "Li Si".

[0107] Based on the example above, the information extraction layer is also used to align multiple sets of target information. Alignment improves the accuracy of the output.

[0108] To facilitate understanding of the above model structure, specific application examples will be used for illustration below.

[0109] Figure 5 This is a schematic diagram illustrating an application example of the information extraction model in this embodiment. In this application example, the text to be processed is "Huizhou is a famous coastal city in Guangdong", the extraction type is entity pairs with the relationship of "belonging", and the extracted information structure is "city + province".

[0110] For the example above, the model mainly includes the following processing steps:

[0111] 1. Prompt layer 501: Composed of two parts: "[Continuous prompt]][Discrete prompt]". Among them,

[0112] A continuous prompt consists of multiple prompt elements (such as...) Figure 5 In <pt-1> , <pt-2>This component is used to learn and extract task description information, i.e., to handle the extraction type. The motivation for introducing this part is to use continuous prompt parameters instead of pre-trained model parameters during training, training only the prompt parameters, thereby reducing the model's overfitting to the training data and improving the model's transfer performance on zero-shot / small-shot datasets.

[0113] Discrete prompts are used to process extracted information structures. For example... Figure 5 As shown, its structure can be: <type>[Entity / Relationship / Event Type] <slot>[Argument 1] <slot>[Argument 2]...”.

[0114] After the text to be processed passes through the prompt layer, the input to the pre-trained model is "[CLS"]. <pt-1> , <pt-2> ,..., <type>[Entity / Relationship / Event Type] <slot>[Argument 1] <slot>[argument 2]...x”.

[0115] 2. Pre-trained language model 502: Used to obtain the feature representation of each element (token) in the text to be processed. For example, it can be implemented using an ERNIE encoder.

[0116] 3. Information Extraction Layer 503: This mainly consists of slot prediction and slot alignment modules.

[0117] Slot prediction consists of a slot start position prediction matrix and a slot end position prediction matrix. Taking the slot start position prediction matrix as an example, its input includes a slot representation and a token representation. The slot representation uses the preceding values ​​from the input for each slot. <slot>The corresponding output of the label on the pre-trained model is used as the representation of the slot; the token representation uses the corresponding output of each token in the text to be processed on the pre-trained model as the representation of the token.

[0118] By concatenating these two parts and passing them through a prediction network, we can obtain the prediction result for each token regarding each slot type. It's important to note that the parameters of the prediction network are shared when predicting the start position of different slot types to handle the issue of an uncertain number of slot types to be extracted.

[0119] Slot alignment is an optional processing module. For named entity recognition tasks, since there is only one slot, alignment is not required. For relation extraction / event extraction tasks, the alignment process can be implemented using mechanisms such as nearest.

[0120] After passing through the information extraction layer, the extracted target information can be obtained.

[0121] Based on the above application examples, it can be seen that the method provided by this disclosure embodiment has the following effects:

[0122] (1) Strong industry transferability. The information extraction pre-training method for low-resource scenarios proposed in this disclosure can support end-to-end pre-training and fine-tuning of the information extraction model, thereby avoiding the problem of error propagation. Compared with the previous SPO triple extraction method, the method proposed in this disclosure has strong industry transferability and can fully learn the semantic knowledge in massive pre-training data from any number of industry domains.

[0123] (2) The information extraction pre-training technology proposed in this embodiment can fully learn the semantic knowledge in massive information extraction data. When supporting downstream business, only a small number of samples need to be labeled to achieve good results. Compared with the SPO triple extraction scheme based on pre-trained language model, it can greatly reduce the manpower cost of labeling, thereby efficiently supporting business.

[0124] (3) The information extraction model proposed in this embodiment uses the same model to model different information extraction tasks, that is, different tasks are uniformly regarded as slot extraction and alignment tasks.

[0125] According to embodiments of this disclosure, this disclosure also provides a training apparatus for an information extraction model. Figure 6 A schematic block diagram of a training apparatus for an information extraction model provided in an embodiment of this disclosure is shown, such as... Figure 6 As shown, the device includes:

[0126] The first iteration module 610 is used to perform N iteration operations based on the initialized target model to obtain the information extraction model; where N is a positive integer.

[0127] The first iteration module 610 includes:

[0128] The data determination unit 611 is used to extract general data and / or industry data of at least one industry from the first dataset based on the current industry data proportion to obtain the second dataset; wherein the current industry data proportion is greater than the industry data proportion in the (i-1)th iteration operation, and i is a positive integer not greater than N;

[0129] The first training unit 612 is used to train the target model updated for the (i-1)th time based on the second dataset, so as to obtain the target model updated for the i-th time.

[0130] The first confirmation unit 613 is used to use the target model updated in the i-th time as the information extraction model when the first preset condition is met.

[0131] Figure 7 This is a schematic diagram of a training apparatus for an information extraction model according to another embodiment of the present disclosure. The apparatus may include one or more features of the training apparatus for the information extraction model described above. In one possible implementation, the first iteration module 610 may further include:

[0132] Fine-tuning unit 701 is used to adjust the target model for the i-th update based on industry data of the target industry in at least one industry.

[0133] For example, the first iteration module 610 may further include:

[0134] Prediction unit 702 is used to obtain the prediction result of the target model for the first dataset in the i-th update;

[0135] The filtering unit 703 is used to identify an irrelevant industry data subset in at least one industry data subset in the first dataset based on the prediction results.

[0136] Deletion unit 704 is used to delete irrelevant industry data subsets from the first dataset.

[0137] For example, the filtering unit 703 is used for:

[0138] Based on the prediction results, sample data in the first dataset that do not contain target information were identified.

[0139] Based on sample data lacking target information, an irrelevant industry data subset is identified within at least one industry data subset in the first dataset.

[0140] According to embodiments of this disclosure, this disclosure also provides an information extraction device. Figure 8 A schematic block diagram of an information extraction apparatus provided in an embodiment of this disclosure is shown, such as Figure 8 As shown, the device includes:

[0141] The model processing module 810 is used to extract target information from the text to be processed using an information extraction model; wherein the information extraction model is trained based on the training device in the aforementioned embodiment.

[0142] For example, the model processing module 810 is used for:

[0143] Input the text to be processed, the extraction type, and the extraction information structure into the information extraction model to obtain the target information in the text to be processed; wherein, the target information contains at least one piece of information that satisfies the extraction information structure, and at least one piece of information satisfies the task requirements corresponding to the extraction type.

[0144] For example, the model processing module 810 is used for:

[0145] Input the text to be processed, the extraction type, and the structure of the extracted information into the information extraction model;

[0146] The extraction type and the extracted information structure are processed using the prompting layer in the information extraction model to obtain prompting information;

[0147] By using the pre-trained language model in the information extraction model, the prompt information and the text to be processed are processed to obtain the feature representation of the input information;

[0148] By utilizing the information extraction layer in the information extraction model, the feature representation of the input information is processed to obtain at least one set of target information in the text to be processed.

[0149] For example, the model processing module 810 is also used for:

[0150] Given multiple sets of target information, the information extraction layer in the information extraction model is used to align the multiple sets of target information.

[0151] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0152] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0153] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0154] Figure 9 A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0155] like Figure 9 As shown, device 900 includes a computing unit 901, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 902 or a computer program loaded from storage unit 908 into random access memory (RAM) 903. RAM 903 may also store various programs and data required for the operation of device 900. The computing unit 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904.

[0156] Multiple components in device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of monitors, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0157] The computing unit 901 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as training methods for information extraction models or information extraction methods. For example, in some embodiments, the training methods for information extraction models or information extraction methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed on device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps of the training methods for information extraction models or information extraction methods described above can be performed. Alternatively, in other embodiments, the computing unit 901 may be configured by any other suitable means (e.g., by means of firmware) to perform a training method or an information extraction method for an information extraction model.

[0158] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0159] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0160] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0161] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0162] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0163] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0164] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0165] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.< / slot> < / slot> < / slot> < / type> < / pt-2> < / pt-1> < / slot> < / slot> < / type> < / pt-1>

Claims

1. An information extraction method, comprising: An information extraction model is used to extract target information from the text to be processed; wherein, the information extraction model is trained based on the following steps: The information extraction model is obtained by performing N iterations based on the initialized target model; where N is an integer greater than 1. The i-th iteration operation in the N iteration operations includes: Based on the current industry data proportion, general data and / or industry data of at least one industry are extracted from the first dataset to obtain the second dataset; wherein, the current industry data proportion is greater than the industry data proportion in the (i-1)th iteration operation, and i is a positive integer not greater than N; Based on the second dataset, the target model updated for the (i-1)th time is trained to obtain the target model updated for the ith time. If the first preset condition is met, the target model updated in the i-th time is used as the information extraction model.

2. The method according to claim 1, wherein, The method of extracting target information from the text to be processed using the information extraction model includes: The text to be processed, the extraction type, and the extraction information structure are input into the information extraction model to obtain the target information in the text to be processed; wherein the target information includes at least one piece of information that satisfies the extraction information structure, and the at least one piece of information satisfies the task requirements corresponding to the extraction type.

3. The method according to claim 2, wherein, The step of inputting the text to be processed, the extraction type, and the extracted information structure into the information extraction model to obtain the target information in the text to be processed includes: Input the text to be processed, the extraction type, and the extracted information structure into the information extraction model; By utilizing the prompting layer in the information extraction model, the extraction type and the extracted information structure are processed to obtain prompting information; The pre-trained language model in the information extraction model is used to process the prompt information and the text to be processed to obtain the feature representation of the input information; By utilizing the information extraction layer in the information extraction model, the feature representation of the input information is processed to obtain at least one set of target information in the text to be processed.

4. The method according to claim 3, wherein, The step of inputting the text to be processed, the extraction type, and the extracted information structure into the information extraction model to obtain the target information in the text to be processed further includes: When multiple sets of target information are obtained, the information extraction layer in the information extraction model is used to align the multiple sets of target information.

5. The method according to claim 1, wherein, The i-th iteration operation also includes: The target model for the i-th update is adjusted based on industry data of the target industry in at least one of the industries.

6. The method according to claim 1 or 5, wherein, The i-th iteration operation also includes: Obtain the prediction result of the target model for the first dataset in the i-th update; Based on the prediction results, an irrelevant industry data subset is determined in at least one industry data subset in the first dataset; Remove the irrelevant industry data subset from the first dataset.

7. The method according to claim 6, wherein, Based on the prediction results, determining an irrelevant industry data subset within at least one industry data subset of the first dataset includes: Based on the prediction results, sample data in the first dataset that do not contain target information are identified; Based on the sample data lacking target information, an irrelevant industry data subset is determined from at least one industry data subset in the first dataset.

8. An information extraction device, comprising: The model processing module is used to extract target information from the text to be processed using an information extraction model; wherein, the information extraction model is trained based on the following modules: The first iteration module is used to perform N iterations based on the initialized target model to obtain the information extraction model; where N is a positive integer. The first iteration module includes: A data determination unit is used to extract general data and / or industry data of at least one industry from a first dataset based on the current industry data proportion to obtain a second dataset; wherein the current industry data proportion is greater than the industry data proportion in the (i-1)th iteration operation, and i is a positive integer not greater than N; The first training unit is used to train the target model updated for the (i-1)th time based on the second dataset to obtain the target model updated for the ith time. The first confirmation unit is used to use the target model updated in the i-th time as the information extraction model if the first preset condition is met.

9. The apparatus according to claim 8, wherein, The model processing module is used for: The text to be processed, the extraction type, and the extraction information structure are input into the information extraction model to obtain the target information in the text to be processed; wherein, the target information includes at least one piece of information that satisfies the extraction information structure, and the at least one piece of information satisfies the task requirements corresponding to the extraction type.

10. The apparatus according to claim 9, wherein, The model processing module is used for: Input the text to be processed, the extraction type, and the extracted information structure into the information extraction model; The information extraction model uses a prompting layer to process the extraction type and the extracted information structure to obtain prompting information; The pre-trained language model in the information extraction model is used to process the prompt information and the text to be processed to obtain the feature representation of the input information; By utilizing the information extraction layer in the information extraction model, the feature representation of the input information is processed to obtain at least one set of target information in the text to be processed.

11. The apparatus according to claim 10, wherein, The model processing module is also used for: When multiple sets of target information are obtained, the information extraction layer in the information extraction model is used to align the multiple sets of target information.

12. The apparatus according to claim 8, wherein, The first iteration operation also includes: The fine-tuning module is used to adjust the target model for the i-th update based on industry data of the target industry in the at least one industry.

13. The apparatus according to claim 8 or 12, wherein, The first iteration module also includes: A prediction unit is used to obtain the prediction result of the target model for the first dataset in the i-th update; A filtering unit is configured to, based on the prediction results, identify an irrelevant industry data subset from at least one industry data subset in the first dataset; The deletion unit is used to delete the irrelevant industry data subset from the first dataset.

14. The apparatus according to claim 13, wherein, The filtering unit is used for: Based on the prediction results, sample data in the first dataset that do not contain target information are identified; Based on the sample data lacking target information, an irrelevant industry data subset is determined from at least one industry data subset in the first dataset.

15. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-7.

17. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-7.

Images