Model training method, information extraction method and device, electronic device, medium

Abstract

This disclosure relates to a model training method, information extraction method and apparatus, electronic device, and storage medium, belonging to the field of computer technology. The model training method includes: acquiring training corpus and determining the triples corresponding to the training corpus, thereby identifying the triples as training data; training multiple neural network models based on the training data, and using the multiple neural network models to perform type prediction on the training data to obtain predicted triples; adjusting the weights of the predicted triples and updating the training data based on the adjusted predicted triples; and adjusting the model parameters of the neural network models based on the updated training data to obtain an information extraction model. The technical solution of this disclosure can improve the accuracy of annotation and the accuracy of the model.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically, to a model training method, a model training device, an information extraction method, an information extraction device, an electronic device, and a computer-readable storage medium. Background Technology

[0002] Information extraction is an important technique in knowledge graph construction. By extracting information, missing attributes in the graph can be completed.

[0003] There are two common information extraction schemes. One is information extraction based on extraction relation types, which extracts triples from the graph and stores them in the database using predefined extraction relation types, and then extracts triples from the training data using a trained model. The other is open-domain information extraction technology. To address the problem of low annotation efficiency, remote supervision is often used for pre-annotation.

[0004] Among the above methods, the information extraction method based on the extraction relationship type suffers from problems such as missing and mislabeled training data due to remote supervision, resulting in inaccurate labeling results; the number of labeled training data is limited, which has certain limitations; and the accuracy of the information extraction model trained based on the training data is low.

[0005] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The purpose of this disclosure is to provide a model training method, information extraction method and apparatus, electronic device, and storage medium, thereby overcoming, to at least a certain extent, the problems of inaccurate training data and low model accuracy caused by the limitations and defects of related technologies.

[0007] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0008] According to one aspect of this disclosure, a model training method is provided, comprising: acquiring training corpus and determining triples corresponding to the training corpus to determine the triples as training data; training multiple neural network models based on the training data and performing type prediction on the training data through the multiple neural network models to obtain predicted triples; adjusting the weights of the predicted triples and updating the training data based on the predicted triples with adjusted weights; and adjusting the model parameters of the neural network models based on the updated training data to obtain an information extraction model.

[0009] According to one aspect of this disclosure, an information extraction method is provided, comprising: acquiring a corpus to be processed; extracting information from the corpus to be processed using an information extraction model to obtain a prediction result corresponding to the corpus to be processed; wherein the information extraction model is trained according to any one of the model training methods described above.

[0010] According to one aspect of this disclosure, a model training apparatus is provided, comprising: a training data acquisition module, configured to acquire training corpus and determine triples corresponding to the training corpus, so as to determine the triples as training data; a prediction module, configured to train multiple neural network models based on the training data, and perform type prediction on the training data through the multiple neural network models to obtain predicted triples; a weight adjustment module, configured to adjust the weights of the predicted triples, and update the training data based on the predicted triples with adjusted weights; and a model parameter adjustment module, configured to adjust the model parameters of the neural network models based on the updated training data to obtain an information extraction model.

[0011] According to one aspect of this disclosure, an information extraction apparatus is provided, comprising: a corpus acquisition module for acquiring corpus to be processed; and a result determination module for extracting information from the corpus to be processed using an information extraction model to obtain a prediction result corresponding to the corpus to be processed; wherein the information extraction model is trained according to the model training method described in any one of the above claims.

[0012] According to one aspect of this disclosure, an electronic device is provided, comprising: a processor; and

[0013] A memory for storing executable instructions of the processor; wherein the processor is configured to execute the model training method or the information extraction method described above by executing the executable instructions.

[0014] According to one aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the model training method or the information extraction method described in any one of the preceding claims.

[0015] The model training method, model training apparatus, information extraction method, information extraction apparatus, electronic device, and computer-readable storage medium provided in this disclosure, on the one hand, train multiple neural network models using training data and obtain predicted triples by predicting the type of the training data using multiple different neural network models. This enables accurate identification of mislabeled and missing training data, improving the accuracy and comprehensiveness of labeling. On the other hand, it avoids the limitations caused by the limited number of labeled training data and the inability to address mislabeled and missing training data, thus expanding the application scope. Furthermore, by adjusting the weights of the predicted triples to update the training data, and then training the model based on the updated training data, the weights of the predicted triples in the updated training data can be adjusted bidirectionally, thereby improving the accuracy and reliability of model training.

[0016] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0018] Figure 1 A schematic diagram of a system architecture for which the model training method or information extraction method of the present disclosure embodiments can be applied is shown.

[0019] Figure 2 A schematic diagram of the structure of an electronic device suitable for implementing embodiments of the present disclosure is shown.

[0020] Figure 3 The diagram illustrates a model training method according to an embodiment of the present disclosure.

[0021] Figure 4 This illustration shows a schematic diagram of determining the triplet score in an embodiment of the present disclosure.

[0022] Figure 5 The schematic diagram illustrates the process of determining the predicted triplet in an embodiment of this disclosure.

[0023] Figure 6 The schematic diagram illustrates the process of determining candidate triples and triples to be eliminated in an embodiment of this disclosure.

[0024] Figure 7The schematic diagram illustrates the process of model training in an embodiment of this disclosure.

[0025] Figure 8 The schematic diagram illustrates the information extraction process in an embodiment of this disclosure.

[0026] Figure 9 The schematic diagram illustrates a flowchart of an information extraction method according to an embodiment of the present disclosure.

[0027] Figure 10 A block diagram of a model training apparatus according to an embodiment of the present disclosure is shown schematically.

[0028] Figure 11 A block diagram of an information extraction apparatus in an embodiment of this disclosure is shown schematically. Detailed Implementation

[0029] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0030] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0031] In related technologies, there are two common information extraction schemes. One is information extraction based on the type of the extracted relation. This involves extracting triples from the graph by predefining the target relation type schema. For example, if the schema is predefined as "wife" and "daughter," a model can be trained to extract the triples "Liu Moumou-daughter-Liu Mou, Liu Moumou-wife-Zhu Mou" from the training corpus "Liu Moumou's wife is Zhu Mou, his daughter is Liu Mou." The other is open-domain information extraction technology, which extracts triples by performing syntactic analysis and named entity recognition on the text, relying on predefined templates or models. For example, if "Liu Moumou sings song 1" matches the template: person-verb-noun, the triple "Liu Moumou-sings-song 1" can be extracted. However, open-domain entity recognition has low accuracy, and the triples are not normalized, resulting in a limited application scope.

[0032] Training information extraction models based on predefined schemas typically requires preparing and annotating large amounts of corpora, which is labor-intensive and inefficient. To address this inefficiency, a remote supervision approach is used for pre-annotation, with the following steps: preparing a general knowledge graph; identifying all triples in the text to be extracted that appear in the graph as candidates; and manually verifying the correctness of the candidate triples. However, remote supervision can lead to missed annotations. In classification problems, when positive examples are not labeled, PU learning techniques are often used to find reliable negative samples from the unlabeled samples.

[0033] For remote supervision, triples not present in the general graph cannot be identified. For example, the general graph may only contain "Liu Moumou-wife-Zhu Mou," but not "Liu Moumou-daughter-Liu Mou." During pre-labeling, the triple "Liu Moumou-daughter-Liu Mou" in the sentence "Liu Moumou's wife is Zhu Mou, his daughter is Liu Mou" cannot be identified. If the labeled corpus is directly used as training corpus in this case, this sample will negatively impact the model. In remote supervision, acquiring a large amount of unlabeled data to use as training corpus, or extracting information from newly added schemas, involves a significant amount of computation. For example, if a new schema, "husband," is added, but the corresponding triple "Zhu Mou-husband-Liu Moumou" in "Liu Moumou's wife is Zhu Mou, his daughter is Liu Mou" is not labeled during data annotation, the entire dataset must be re-labeled if the schema is added.

[0034] PU learning trains a binary classifier with only positive class and unlabeled data, and treats the unlabeled positive examples as positive examples. It does not consider that the binary classifier is actually based on existing training data, and the positive examples it produces are not 100% correct. Moreover, PU learning does not handle erroneous samples.

[0035] To address the aforementioned technical issues, this disclosure provides a model training method that can be applied to scenarios such as information extraction or sequence labeling.

[0036] Figure 1 A schematic diagram of a system architecture for a model training method and apparatus or an information extraction method and apparatus that can be applied to embodiments of the present disclosure is shown.

[0037] like Figure 1 As shown, the system architecture 100 may include a client 101, a network 102, and a server 103. The client can be a mobile device, such as a smartphone, computer, tablet, or smart speaker. The network 102 serves as the medium for providing a communication link between the client 101 and the server 103. The network 102 may include various connection types, such as wired communication links, wireless communication links, etc. In this embodiment, the network 102 between the client 101 and the server 103 can be a wired communication link, for example, provided through a serial port cable, or it can be a wireless communication link, provided through a wireless network. The server 103 can be a server with computing capabilities or a client, such as a portable computer, desktop computer, or smartphone, used to process the corpus sent by the client.

[0038] This model training method can be applied to training scenarios where information is extracted. (Reference) Figure 1 As shown, this can be specifically applied to the process where client 101 sends training corpus to server 103, and server 103 extracts information from the target object obtained on the client. The client can be various types of computing devices, such as smartphones, tablets, desktop computers, in-vehicle devices, wearable devices, etc. The target object can be various types of text information. Server 103 can use the training corpus sent by client 101 to obtain training data, and train multiple neural network models based on the training data to obtain prediction triples corresponding to the training data. Further, it updates the training data based on the adjusted weights of the prediction triples, and adjusts the model parameters of the neural network models based on the updated training data to generate an information extraction model. Therefore, when server 103 receives the corpus to be processed sent by client 101, it can use the trained information extraction model to extract information from the corpus to obtain the corresponding prediction results. The prediction results can then be sent to client 101 for display and other processing operations.

[0039] It should be noted that the model training method and information extraction method provided in this embodiment can be entirely executed by the server. Accordingly, the model training device and the information extraction device can be located in the server.

[0040] Figure 2 A schematic diagram is shown of an electronic device suitable for implementing exemplary embodiments of the present disclosure. The terminal of the present disclosure can be configured as follows: Figure 2 The electronic device shown is in the form of [the device]; however, it should be noted that... Figure 2 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0041] The electronic device disclosed herein includes at least a processor and a memory, the memory being used to store one or more programs, which, when executed by the processor, enable the processor to implement the methods of exemplary embodiments of this disclosure.

[0042] Specifically, such as Figure 2 As shown, the electronic device 200 may include: a processor 210, internal memory 221, external memory interface 222, Universal Serial Bus (USB) interface 230, charging management module 240, power management module 241, battery 242, antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, audio module 270, speaker 271, receiver 272, microphone 273, headphone jack 274, sensor module 280, display screen 290, camera module 291, indicator 292, motor 293, buttons 294, and a Subscriber Identification Module (SIM) card interface 295, etc. The sensor module 280 may include depth sensors, pressure sensors, gyroscope sensors, barometric pressure sensors, magnetic sensors, accelerometers, distance sensors, proximity sensors, fingerprint sensors, temperature sensors, touch sensors, ambient light sensors, and bone conduction sensors, etc.

[0043] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 200. In other embodiments of this application, the electronic device 200 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0044] Processor 210 may include one or more processing units, such as application processors, modem processors, graphics processors, image signal processors, controllers, video codecs, digital signal processors, baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors. Additionally, processor 210 may include memory for storing instructions and data. The model training method in this exemplary embodiment can be executed by an application processor, graphics processor, or image signal processor; when the method involves neural network-related processing, it can be executed by the NPU.

[0045] Internal memory 221 can be used to store computer executable program code, which includes instructions. Internal memory 221 may include a program storage area and a data storage area. External memory interface 222 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of electronic device 200.

[0046] The communication function of mobile terminal 200 can be implemented through a mobile communication module, antenna 1, a wireless communication module, antenna 2, a modem processor, and a baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. The mobile communication module can provide 2G, 3G, 4G, and 5G mobile communication solutions for use on mobile terminal 200. The wireless communication module can provide wireless communication solutions such as wireless LAN, Bluetooth, and near-field communication for use on mobile terminal 200.

[0047] The display screen is used to implement display functions, such as displaying the user interface, images, and videos. The camera module is used to implement shooting functions, such as capturing images and videos. The audio module is used to implement audio functions, such as playing audio and capturing voice. The power module is used to implement power management functions, such as charging the battery, supplying power to the device, and monitoring battery status.

[0048] This application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device.

[0049] Computer-readable storage media can be, for example—but not limited to—electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0050] A computer-readable storage medium can be sent, propagated, or transmitted for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0051] A computer-readable storage medium carries one or more programs that, when executed by an electronic device, cause the electronic device to perform the methods described in the following embodiments.

[0052] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0053] The units described in the embodiments of this disclosure can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the unit itself.

[0054] Next, refer to Figure 3 The model training method in the embodiments of this disclosure will be described in detail.

[0055] In step S310, training corpus is acquired, and the triples corresponding to the training corpus are determined so as to identify the triples as training data.

[0056] In this embodiment, the training corpus refers to the data before optimization, which can be of various types and can be obtained through remote supervision. Remote supervision refers to aligning text with entities in a large-scale knowledge graph and annotating the text using the existing entity relationships in the knowledge graph. This can map existing knowledge bases to rich unstructured data (such as news text), thereby generating a large amount of training data to train a relation extractor.

[0057] Triples are semantic networks representing entities and entity relationships. For example, a triple can be entity-relation-entity. For training corpora, initial triples can be obtained and filtered to determine the final triples. Specifically, all triples in the training data can be identified as initial triples based on the target relation type, and the triples to be extracted can be determined from these initial triples based on semantic information. The target relation type describes the relationship between entities in the triples to be extracted. The target relation type can be a noun representing a relationship between entities, such as "wife"; it can also be a verb representing a relationship between entities, such as "participate in an act," etc. The target relation type can be defined according to actual needs, and different types of entities correspond to different target relation types. For example, when the entity is a person, the target relation type can include, but is not limited to, husband, wife, work, participate in an act, etc.

[0058] After identifying the target relation type, the training corpus can be extracted based on this type, and all extracted triples can be designated as initial triples. Further, semantic information can be used to determine if the initial triples are to be extracted; if so, they are included in the training corpus. Semantic information can be extracted, and it can be determined whether the semantic information contains the target relation type. If it does, the initial triples are designated as to be extracted. By identifying the initial triples and filtering them based on semantic information to obtain the triples corresponding to the target relation type, these triples can be used as training data to optimize the training corpus and improve its accuracy. For example, if the training corpus is "On February 19th, 96-year-old veteran actress Hou Moumou passed away. She played the role of the grandmother in films such as 'Movie 1' and 'Movie 2,' and before her death, Grandma Hou kept saying that she loved movies and loved Zhou Moumou," the resulting training data could be "Movie 2 - Screenwriter - Zhou Moumou," and "Hou Moumou - Actress - Movie 1."

[0059] Next, in step S320, multiple neural network models are trained based on the training data, and the training data is type-predicted using the multiple neural network models to obtain predicted triples.

[0060] In this embodiment of the disclosure, the model parameters of the multiple neural network models can be different, and the multiple neural network models can be obtained based on different training data. Specifically, the training data can be divided into multiple training data subsets, and multiple different neural network models can be obtained by training the models on the multiple training data subsets respectively.

[0061] Cross-validation can be used to divide the training data into multiple subsets. Cross-validation can be n-fold cross-validation. The value of n can be set according to specific needs; for example, n can be 5 or other values. Based on this, n-fold cross-validation can be used to divide the training data into n parts, i.e., n subsets, and then n neural network models can be trained based on these n subsets. Each of the n neural network models can include a pointer network to determine the subject or object, etc.

[0062] When training multiple neural network models based on multiple subsets of training data, one subset of training data can be used as the validation set, and the remaining subsets of training data can be used as the training set. Each neural network model can then be trained separately. Specifically, for the i-th model, its validation set can be the i-th subset of training data, and all remaining subsets of training data (excluding the first subset) can be used as the training set.

[0063] During model training, each model can use its corresponding training set to predict the training data in the validation set associated with that model. The model parameters are then adjusted based on the prediction results from the validation set to train each neural network model. For each model, the highest-scoring model on its corresponding validation set is selected. Therefore, each model is trained with the goal of achieving the highest score on its corresponding validation set. That is, each model's parameters can be determined by selecting the parameters that maximize the score on the validation set, thus completing the training process for each model. The score here can be the F1 score, which is used to evaluate the model's accuracy.

[0064] For example, refer to Figure 4 As shown, when using 5-fold cross-validation, the training data is divided into 5 parts, i.e., 5 Folds (Fold1, Fold2...Fold5). The first model uses Fold1 as the validation set and Fold2, Fold3, Fold4, and Fold5 as the training set. Similarly, the fifth model uses Fold5 as the validation set and Fold1, Fold2, Fold3, and Fold4 as the training set. Each model is trained using 4 Folds and predicted using 1 Fold, resulting in 5 models. Each of the 5 models is the one with the highest score on its corresponding validation set. Therefore, the training objective for each model can be to train it with the highest score on the validation set.

[0065] After training multiple neural network models, the training data can be used to predict the corresponding predicted triples based on these models. Figure 5 The flowchart for determining the predicted triples is illustrated in the diagram. (Refer to...) Figure 5 As shown, the main steps include:

[0066] In step S510, the training data is predicted using the multiple neural network models to obtain multiple scores for multiple target triples;

[0067] In step S520, the average scores of the target triples are processed to obtain the target score of each target triple, and the predicted triple is determined based on the target score.

[0068] In this embodiment, multiple neural network models can be used to predict training data to obtain multiple target triples. A target triple refers to all triples predicted by multiple neural network models. That is, each neural network model can be used to perform a convolution operation on the training data to obtain unclassified target triples. Each neural network model can correspond to multiple target triples, and the multiple target triples obtained by multiple neural network models can be the same or different; no special limitation is made here. After obtaining multiple target triples, the score of each target triple relative to each neural network model can be calculated. Furthermore, the scores of the same target triple relative to different neural network models can be the same or different.

[0069] Furthermore, the multiple scores for each target triple can be averaged to determine the predicted triples. Averaging involves performing a summation operation and calculating the average. The target score for each target triple can be its average score. See also... Figure 5 As shown, each model selects the parameters of the model with the highest F1 score on the validation set Test, and continues to predict the training data through different models to obtain the score of each target triplet. Furthermore, the scores of the same target triplet obtained by different models are voted and summarized by averaging the scores to obtain the target score of each target triplet.

[0070] Predicted triples refer to the triples predicted from training data by multiple trained neural network models. A predicted triple is a target triple with a defined type. The type of predicted triple can be determined by combining the target score, threshold, and semantic information of the target triple, specifically including candidate triples and triples to be removed. Candidate triples can include triples that do not exist in the training data and need to be added (i.e., missing labeled triples), as well as correctly labeled triples that already exist in the training data. Triples to be removed refer to incorrectly labeled triples that need to be deleted. A first threshold is used to determine candidate triples, and a second threshold is used to determine triples to be removed.

[0071] For example, multiple neural network models are used to predict the training data "Liu Moumou and his wife Zhu Mou got married in Hong Kong." Each neural network model generates multiple target triples, and the target triples generated by each model may be repeated. Next, the score of each target triple under each neural network model is calculated, and all scores are averaged to obtain the mean score. For example, the mean score of the target triple "Liu Moumou-wife-Zhu Mou" is 0.9, and the mean score of the target triple "Liu Moumou-friend-Zhu Mou" is 0.2. Furthermore, the mean score of the triples can be compared with a first threshold or a second threshold to determine the predicted triples. For example, "Liu Moumou-friend-Zhu Mou" is marked as a triple to be removed, and the target triple "Liu Moumou-birthplace-Hong Kong" is also marked as a triple to be removed.

[0072] Figure 6 The flowchart illustrating the determination of candidate triples and triples to be eliminated is shown in the figure. (Refer to...) Figure 6 As shown, the main steps include:

[0073] In step S610, it is determined whether the target score of the target triplet is greater than the first threshold; if yes, proceed to step S620; if no, proceed to step S630.

[0074] In step S620, if the target score is greater than the first threshold, the target triplet is determined as a candidate triplet.

[0075] In step S630, it is determined whether the target score is less than the second threshold; if not, proceed to step S640; if yes, proceed to step S650.

[0076] In step S640, it is determined whether a triplet with a target score greater than a first threshold exists in the semantic information; if not, proceed to step S650. The second threshold is less than the first threshold.

[0077] In step S650, if the target score is less than the second threshold, or the target score is greater than the first threshold and the triplet does not exist in the semantic information, the target triplet is determined as a triplet to be removed.

[0078] In this embodiment, the first threshold can be an upper threshold, such as 0.6 or other values, determined according to actual needs. Here, we take 0.6 as an example for explanation. The second threshold can be a lower threshold, such as 0.3 or other values. Here, we take 0.3 as an example for explanation.

[0079] A target triplet can be identified as a candidate triplet if its target score is greater than a first threshold and it exists in the semantic information of the training data. A target triplet can be identified as a triplet to be removed if its target score is less than a second threshold. Additionally, a target triplet can be identified as a triplet to be removed if its target score is greater than the first threshold and it does not exist in the semantic information corresponding to the training data.

[0080] For example, if the target triple "Liu Moumou and his wife Zhu Mou got married in Hong Kong" has an average score of 0.9, it is considered a candidate triple. While "Liu Moumou-birthplace-Hong Kong" is factual and has a corresponding triple in the graph, it cannot be obtained from the semantic information of "Liu Moumou and his wife Zhu Mou got married in Hong Kong," so the target triple "Liu Moumou-birthplace-Hong Kong" is marked as a triple to be removed. If the target triple "Liu Moumou and his wife Zhu Mou got married in Hong Kong" has an average score of 0.2, it is also marked as a triple to be removed. In this embodiment, multiple target triples corresponding to the training data are filtered using a first threshold, a second threshold, and semantic information to obtain predicted triples. This accurately identifies incorrectly labeled and missing triples from the training data, avoiding the limitations caused by the limited number of labeled data, improving the accuracy and comprehensiveness of labeling, and enhancing the accuracy of determining predicted triples.

[0081] In addition, for new relation types representing candidate triples, only a portion of the corpus needs to be labeled as a seed set, and triples of all corpus can be obtained through iteration, avoiding the complex operation of labeling the entire training corpus and improving operational efficiency.

[0082] Continue to refer to Figure 3 As shown, in step S330, the weights of the predicted triples are adjusted, and the training data is updated based on the predicted triples with adjusted weights.

[0083] In this embodiment, since triples in the training data may have mislabeled or missed labels, directly adding candidate triples and removing triples to be removed would affect accuracy. Therefore, candidate triples and triples to be removed are labeled by adjusting their weights. Specifically, the weight of triples to be removed that appear in the training data is reduced, while the weight of candidate triples that do not appear in the training data is increased. When increasing the weight of a candidate triple, the weight can be determined according to a metric parameter. The metric parameter can be the model's accuracy or other values, set according to actual needs. For example, when the model's accuracy is 0.7, the weight of a candidate triple can be 0.7. When reducing the weight of a triple to be removed, its weight can be determined according to a score threshold. The score threshold is used to indicate whether a triple is a triple, and can be set according to actual needs, for example, it can be 0.5 or other values, as long as it is less than the weight of the candidate triple.

[0084] For example, the weight of a candidate triple is 0.7, and the weight of a triple to be removed is 0.5. For the training data "Liu Moumou and his wife Zhu Mou got married in Hong Kong, and their daughter is Liu Mou", the original triple label was Liu Moumou-birthplace-Hong Kong. During weight adjustment, the triples are set as follows: Liu Moumou-wife-Zhu Mou-0.7, Liu Moumou-birthplace-Hong Kong-0.5, and the weights of other triples are set to 1 by default. The triples with the updated weights are then used as the updated training data.

[0085] For example, if the training corpus is "On February 19th, 96-year-old veteran actress Hou Moumou passed away. She played the role of the grandmother in films such as 'Film 1' and 'Film 2,' and before her death, Grandma Hou kept saying that she loved movies and loved Zhou Moumou," the resulting training data could be "Film 2 - Screenwriter - Zhou Moumou" and "Hou Moumou - Actress - Film 1." The triple represented by "Film 2 - Screenwriter - Zhou Moumou" is a triple to be removed (label to be removed), and its weight is reduced to 0.5. The triple represented by "Hou Moumou - Actress - Film 2" is a candidate triple (candidate label), and its weight is increased to 0.7. Based on this, updated training data can be obtained. In this embodiment, by increasing the weight of candidate triples not in the training data and decreasing the weight of triples to be removed in the training data, a dual weight adjustment method is used, which can improve the accuracy of the training data and avoid the interference of directly deleting and adding triples on the training data. It should be noted that for candidate triples not in the training data, their weight before adjustment is 0; for triples to be removed that are in the training data, their weight before adjustment is 1.

[0086] Continue to refer to Figure 3As shown, in step S340, the model parameters of the neural network model are adjusted according to the updated training data to obtain the information extraction model.

[0087] In this embodiment of the disclosure, weighted updated training data can be used to adjust the model parameters of the neural network model for model training, thereby obtaining an information extraction model. Model training can be performed using updated training data based on a loss function.

[0088] Figure 7 The flowchart illustrating the model training process is shown in the image. Figure 9 As shown, the main steps include:

[0089] In step S710, the function weights of the candidate triples are adjusted, and the function weights of the triples to be eliminated are also adjusted.

[0090] In this step, function weights refer to the weights of candidate triples or triples to be eliminated in the loss function. The weights of candidate triples and triples to be eliminated in the loss function can be different. By adjusting the function weights of candidate triples and triples to be eliminated, the influence of these triples on the model can be adjusted. To improve accuracy, the function loss of candidate triples can be increased, while the function loss of triples to be eliminated can be decreased. Specifically, the function weights of candidate triples can be adjusted according to a metric parameter. This metric parameter refers to the model's accuracy, for example, 0.7. Since the model's accuracy is approximately 0.7, the expected value of a correct candidate triple is also approximately 0.7; therefore, a coefficient of 0.7 is set for the candidate triples (the coefficient for triples not originally in the training data is 0). Furthermore, the function loss of triples to be eliminated can be adjusted according to a score threshold, which is used to determine whether a triple is a true triple. For example, a score threshold of 0.5 can be used. Since the score threshold for determining whether a triplet is 0.5, the triplet to be removed is set to a coefficient of 0.5 (the coefficient of the triplet in the original training data was 1).

[0091] In step S720, the loss function is updated according to the function weights of the candidate triples and the function weights of the triples to be eliminated, and the target loss function is determined.

[0092] In this step, the loss function can be the cross-entropy loss function. When calculating the target loss function, it can be processed based on the function weights. Specifically, the function weights of candidate triples and the function weights of triples to be removed can be multiplied by the loss function to obtain the target loss function. Alternatively, the loss function can be multiplied by its function weights to adjust it to the target loss function. For example, the loss function `los` can be adjusted based on the weight `ratio` using the program `los=los*ratio`. By updating the function weights of the loss function by increasing the function weights of candidate triples according to the indicator parameters and decreasing the function weights of triples to be removed according to the score threshold, the accuracy of the target loss function can be improved, and the influence of candidate triples and triples to be removed on the loss function can be reduced.

[0093] In step S730, with the minimum target loss function as the training objective, the model parameters of the neural network model are adjusted according to the updated training data to obtain the information extraction model.

[0094] In this step, the training objective can be minimized, the updated training data can be used as input, and the triples of the updated training data (candidate triples and triples to be eliminated) can be used as fitting targets to adjust the model parameters of the neural network model until the target loss function is minimized, thus ending the training process and obtaining the information extraction model.

[0095] In this embodiment, by adjusting the weights of the loss function and training the neural network model based on updated training data to obtain an information extraction model, the problems of mislabeling and omission during information extraction in related technologies are solved, thereby improving the accuracy of labeling. Using a bidirectional weight adjustment method, the weights of unlabeled and mislabeled triples in the training data can be adjusted, thereby improving the model's accuracy and training effect. It should be noted that the neural network model can be a model based on the PU learning (Positive-unlabeled learning) algorithm. The PU learning algorithm trains a binary classifier with only positive class data and unlabeled data. Unlabeled positive examples are treated as positive examples, without considering that the binary classifier is actually based on existing training data, and the resulting positive examples are not entirely correct. Furthermore, PU learning does not handle erroneous samples. Therefore, the weights of the samples obtained from PU learning need to be reduced. The improved PU learning algorithm adjusts the weights of candidate triples and triples to be removed, and also adjusts the weights of the loss function, which can improve accuracy.

[0096] Table 1

[0097]

[0098] Table 1 illustrates the comparison results of model training. Referring to Table 1, using the improved PUlearning model, the accuracy was improved by 1.95% and the F1 score by 3.46% compared to the original model. Therefore, the accuracy of model training was improved, and the accuracy of the model itself was also improved.

[0099] Figure 8 The flowchart of information extraction is illustrated in the diagram. (See reference) Figure 8 As shown, the main steps include:

[0100] In step S810, the training corpus is prepared.

[0101] In step S820, the extraction model is trained based on remote supervised data.

[0102] In step S830, the triplet weights are adjusted.

[0103] In step S840, a weighted information extraction model is trained.

[0104] In step S850, the prediction result is obtained.

[0105] In this embodiment of the disclosure, information extraction is performed using a trained information extraction model, which can avoid the problems of mislabeling and omission, and improve the accuracy of triples.

[0106] In summary, the technical solution in this embodiment addresses the issues of insufficient training data and missing triplet labels during information extraction by using an improved PU learning information extraction process. Based on an n-fold cross-validation triplet extraction method, by training n differentiated models, it can effectively identify mislabeled and missing samples from the training data, avoiding limitations and expanding the application scope. It avoids missing or mislabeled triplet labels when the training data is limited, improving the accuracy and completeness of triplet labeling. The weights of candidate triplets and mislabeled triplets in the training data can be adjusted, using the weighted data as training data. Furthermore, the weights of the loss function for candidate triplets and triplets to be removed are adjusted using a bidirectional weight adjustment method, improving model training efficiency, accuracy, and reliability. When adding new relation types, it is no longer necessary to label the entire training data; only a small seed set needs to be labeled. Through iteration, all triplets can be determined, reducing operational steps and improving labeling efficiency.

[0107] This disclosure also provides an information extraction method, referring to... Figure 9 As shown, the main steps include:

[0108] In step S910, the corpus to be processed is obtained;

[0109] In step S920, information is extracted from the corpus to be processed using an information extraction model to obtain the prediction result corresponding to the corpus to be processed; wherein, the information extraction model is trained according to any of the model training methods described above.

[0110] In this embodiment, the corpus to be processed can be of any type, specifically the text to be extracted or the sequence to be labeled, etc. The information extraction model refers to the model that extracts triples from the corpus to be processed; it can be any type of model, such as a neural network model. Information extraction is a text processing technique that extracts factual information such as entities, relationships, and events of a specified type from natural language text and outputs structured data. The prediction result corresponding to the corpus to be processed can include all triples obtainable from the corpus. Based on this, in practical applications, it is only necessary to input the corpus to be processed into the information extraction model for convolution operation, and the output is all the triples corresponding to the corpus to be processed.

[0111] In sequence labeling scenarios, there are also models with missing or incorrect entity labels. Therefore, changing the model's task from information extraction to sequence labeling can improve the accuracy of sequence labeling.

[0112] This disclosure provides a model training apparatus, with reference to... Figure 10 As shown, the model training device 1000 may include:

[0113] The training data acquisition module 1001 is used to acquire training corpus and determine the triples corresponding to the training corpus, so as to determine the triples as training data.

[0114] Prediction module 1002 is used to train multiple neural network models based on the training data, and to perform type prediction on the training data through the multiple neural network models to obtain predicted triples;

[0115] The weight adjustment module 1003 is used to adjust the weights of the predicted triples and update the training data based on the predicted triples with adjusted weights.

[0116] The model parameter adjustment module 1004 is used to adjust the model parameters of the neural network model based on the updated training data and to obtain information extraction model.

[0117] In one exemplary embodiment of this disclosure, the training data acquisition module includes a triplet determination module, configured to determine all triplets in the training data as initial triplets according to the target relation type, and to determine the triplets to be extracted from the initial triplets according to the semantic information of the training data, thereby determining the triplets.

[0118] In one exemplary embodiment of this disclosure, the prediction module includes: a data splitting module, configured to perform cross-validation based on the training data to obtain multiple training data subsets; and a training control module, configured to use one training data subset from the multiple training data subsets as a validation set and the remaining training data subsets as a training set to train multiple neural network models.

[0119] In one exemplary embodiment of this disclosure, the training control module is configured to: use the i-th training data subset in the plurality of training data subsets as the validation set, and use the remaining training data subsets as the training set; take the highest validation set score of the i-th neural network model as the model training target, train the i-th neural network model among the plurality of neural network models, and determine the trained i-th neural network model.

[0120] In one exemplary embodiment of this disclosure, the prediction module includes: a score determination module, configured to predict the training data using the plurality of neural network models to obtain a plurality of scores for a plurality of target triples; and a prediction triple determination module, configured to average the plurality of scores of the target triples to obtain a target score for each target triple, and determine the prediction triple based on the target score.

[0121] In an exemplary embodiment of this disclosure, the predicted triplet determination module includes: a first determination module, configured to determine the triplet as a candidate triplet if the target score is greater than a first threshold; and a second determination module, configured to determine the triplet as a triplet to be removed if the target score is less than a second threshold, or if the target score is greater than the first threshold and the triplet does not exist in the semantic information corresponding to the training data; wherein the first threshold is greater than the second threshold.

[0122] In one exemplary embodiment of this disclosure, the weight adjustment module is configured to: reduce the weight of the triplet to be removed in the training data, increase the weight of the candidate triplet outside the training data, and update the training data according to the triplet with adjusted weights.

[0123] In one exemplary embodiment of this disclosure, the model parameter adjustment module includes: a weight adjustment module, used to adjust the function weights of candidate triples and the function weights of triples to be eliminated; a loss function update module, used to update the loss function according to the function weights of the candidate triples and the function weights of the triples to be eliminated, and determine a target loss function; and a model parameter update module, used to adjust the model parameters of the neural network model according to the updated training data with the minimum of the target loss function as the training objective, so as to obtain the information extraction model.

[0124] In one exemplary embodiment of this disclosure, the weight adjustment module is configured to: determine the function weight of the candidate triplet based on the index parameters, and determine the function weight of the triplet to be eliminated based on the score threshold.

[0125] It should be noted that the specific details of each module in the above-mentioned model training device have been described in detail in the corresponding model training methods, so they will not be repeated here.

[0126] This disclosure also provides an information extraction device, for reference... Figure 11 As shown, the information extraction device 1100 may include:

[0127] Corpus acquisition module 1101 is used to acquire corpus to be processed;

[0128] The result determination module 1102 is used to extract information from the corpus to be processed through an information extraction model and obtain the prediction result corresponding to the corpus to be processed; wherein the information extraction model is trained by any of the model training methods described above.

[0129] It should be noted that the specific details of each module in the above information extraction device have been described in detail in the corresponding information extraction method, so they will not be repeated here.

[0130] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0131] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0132] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0133] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims. It should be understood that this disclosure is not limited to the precise structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A model training method, characterized in that, include: Acquire training corpus and determine the triples corresponding to the training corpus, so as to determine the triples as training data; Multiple training data subsets are obtained based on the training data. One of the training data subsets is used as the validation set, and the remaining training data subsets are used as the training set. The training objective is to train multiple neural network models with the highest score on the validation set for each neural network model. The predicted triples are obtained by predicting the type of the training data through the multiple neural network models. The weights of the predicted triples are adjusted, and the training data is updated based on the predicted triples with adjusted weights. The model parameters of the neural network model are adjusted based on the updated training data to obtain information extraction models; The method of obtaining predicted triples by performing type prediction on the training data using the multiple neural network models includes: The training data is predicted using the multiple neural network models to obtain multiple scores for multiple target triples; The target score for each target triple is obtained by averaging the scores of multiple target triples, and the predicted triples are determined based on the target scores. The predicted triples include candidate triples and triples to be eliminated.

2. The model training method according to claim 1, characterized in that, Determining the triples corresponding to the training corpus includes: Based on the target relation type, the triples in the training data are determined as initial triples, and based on the semantic information of the training data, the triples to be extracted are determined from the initial triples, so as to determine the triples.

3. The model training method according to claim 1, characterized in that, The step of using one subset of training data from the plurality of training data subsets as a validation set and the remaining subset of training data as a training set to train multiple neural network models includes: The i-th training data subset in the plurality of training data subsets is used as the validation set, and the remaining training data subsets are used as the training set. The training objective is to train the i-th neural network model among multiple neural network models, with the i-th neural network model having the highest validation set score. The trained i-th neural network model is then determined.

4. The model training method according to claim 1, characterized in that, Determining the predicted triples based on the target score includes: If the target score is greater than the first threshold, the triplet is determined as a candidate triplet; If the target score is less than the second threshold, or the target score is greater than the first threshold and the triplet does not exist in the semantic information corresponding to the training data, the triplet is determined as a triplet to be removed; wherein, the first threshold is greater than the second threshold.

5. The model training method according to claim 4, characterized in that, The step of adjusting the weights of the predicted triples and updating the training data based on the adjusted predicted triples includes: The weights of the triples to be removed from the training data are reduced, the weights of the candidate triples outside the training data are increased, and the training data is updated based on the triples with adjusted weights.

6. The model training method according to claim 4, characterized in that, The step of adjusting the model parameters of the neural network model based on the updated training data to obtain the information extraction model includes: Adjust the function weights of candidate triples and adjust the function weights of triples to be eliminated; The loss function is updated based on the function weights of the candidate triples and the function weights of the triples to be eliminated, and the target loss function is determined. The training objective is to minimize the target loss function. The model parameters of the neural network model are adjusted based on the updated training data to obtain the information extraction model.

7. The model training method according to claim 6, characterized in that, The adjustment of the function weights for candidate triples and the adjustment of the function weights for triples to be eliminated include: The function weights of the candidate triples are determined based on the indicator parameters, and the function weights of the triples to be eliminated are determined based on the score threshold.

8. An information extraction method, characterized in that, include: Obtain the corpus to be processed; Information is extracted from the corpus to be processed by an information extraction model to obtain the prediction result corresponding to the corpus to be processed; wherein, the information extraction model is trained by the model training method according to any one of claims 1-7.

9. A model training device, characterized in that, include: The training data acquisition module is used to acquire training corpus and determine the triples corresponding to the training corpus, so as to identify the triples as training data. The prediction module is used to obtain multiple training data subsets based on the training data, use one training data subset as a validation set, use the remaining training data subsets as a training set, train multiple neural network models with the highest score on the validation set as the model training objective, and use the multiple neural network models to perform type prediction on the training data to obtain predicted triples. The weight adjustment module is used to adjust the weights of the predicted triples and update the training data based on the predicted triples with adjusted weights. The model parameter adjustment module is used to adjust the model parameters of the neural network model based on the updated training data and to extract information from the model. The method of obtaining predicted triples by performing type prediction on the training data using the multiple neural network models includes: The training data is predicted using the multiple neural network models to obtain multiple scores for multiple target triples; The target score for each target triple is obtained by averaging the scores of multiple target triples, and the predicted triples are determined based on the target scores. The predicted triples include candidate triples and triples to be eliminated.

10. An information extraction device, characterized in that, include: The corpus acquisition module is used to acquire the corpus to be processed. The result determination module is used to extract information from the corpus to be processed using an information extraction model to obtain the prediction result corresponding to the corpus to be processed; wherein the information extraction model is trained by the model training method according to any one of claims 1-7.

11. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the model training method of any one of claims 1-7 or the information extraction method of claim 8 by executing the executable instructions.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the model training method according to any one of claims 1-7 or the information extraction method according to claim 8.

Images