Model training method, text processing method, device and electronic equipment

Abstract

Embodiments of the present application provide a model training method, a text processing method, an apparatus and an electronic device, and relate to the field of machine learning and natural language processing. The method comprises: obtaining a first sample set and a second sample set, training an initial causal relation identification model according to the first sample set to obtain a first causal relation identification model, and determining the uncertainty degree of the prediction of the first causal relation identification model for each second sample text; screening a plurality of third sample texts according to the uncertainty degree corresponding to each second sample text; and iteratively training the first causal relation identification model based on a plurality of target sample texts until a training stop condition is met, to obtain a trained second causal relation identification model. The embodiments of the present application alleviate the problem of insufficient labeled data, and the trained second causal relation identification model has higher accuracy and robustness.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine learning and natural language processing, in particular, the present application relates to a model training method, a text processing method, a device and an electronic equipment. BACKGROUND

[0002] In today's society, various types of information are pushed in real time on the Internet. In the face of growing information, it is crucial to quickly sort out the logical relationships between events in the information.

[0003] Event causal relationship extraction takes events as the basic semantic unit to realize deep detection and extraction of event logical relationships, which plays an important role in text understanding. Due to the influence of network and social factors, the research of this technology has been increasing year by year.

[0004] Due to the complex expression mode of daily text and the difficulty of semantic understanding, when the prior art trains a causal relationship recognition model through a machine learning technology, a person needs to manually label the causal relationship of the text, which is very low in efficiency, and there is also a problem of insufficient training data. The existing method cannot utilize a large amount of unlabeled data, which is easy to cause model overfitting, and thus affects the accuracy of the model. SUMMARY

[0005] The embodiments of the present application provide a model training method, a text processing method, a device, an electronic equipment, a computer readable storage medium and a computer program product, which can solve the above problems of the prior art. The technical scheme is as follows:

[0006] According to an aspect of the embodiments of the present application, a model training method is provided, which comprises:

[0007] Obtaining a first sample set and a second sample set, the first sample set comprising a plurality of first sample texts labeled with labels, and the second sample set comprising a plurality of second sample texts without labels, the label of the first sample text representing the causal relationship between the event information contained in the first sample text;

[0008] Training an initial causal relationship recognition model according to the first sample set to obtain a first causal relationship recognition model, and determining the uncertainty degree of the first causal relationship recognition model for predicting each second sample text;

[0009] According to the uncertainty degree corresponding to each second sample text, a plurality of third sample texts are selected from each second sample text;

[0010] The each first sample text and the each third sample text with labels are taken as target sample texts, and the first cause-effect relationship identification model is iteratively trained based on the target sample texts until a training stop condition is met, to obtain a trained second cause-effect relationship identification model

[0011] According to another aspect of the embodiments of the present application, a text processing method is provided, which includes:

[0012] Obtaining a to-be-identified text;

[0013] Inputting the to-be-identified text into the trained cause-effect relationship identification model to obtain a cause-effect relationship between event information contained in the to-be-identified text;

[0014] The trained cause-effect relationship identification model is obtained by using the method.

[0015] According to another aspect of the embodiments of the present application, a model training device is provided, which includes:

[0016] A sample set obtaining module is configured to obtain a first sample set and a second sample set, the first sample set including a plurality of first sample texts labeled with labels, and the second sample set including a plurality of second sample texts without labels, the label of the first sample text representing a cause-effect relationship between event information contained in the first sample text;

[0017] An uncertainty calculating module is configured to train an initial cause-effect relationship identification model according to the first sample set to obtain a first cause-effect relationship identification model, and determine an uncertainty degree of a prediction of the first cause-effect relationship identification model for each second sample text;

[0018] A sample screening module is configured to screen a plurality of third sample texts from the second sample texts according to the uncertainty degree corresponding to each second sample text;

[0019] A retraining module is configured to take the each first sample text and the each third sample text with labels as target sample texts, iteratively train the first cause-effect relationship identification model based on the target sample texts, until a training stop condition is met, and obtain a trained second cause-effect relationship identification model.

[0020] As an optional embodiment, when the uncertainty calculating module determines the uncertainty degree of the prediction of the first cause-effect relationship identification model for each second sample text, it is configured to:

[0021] Identify a cause-effect relationship of each second sample text by using the first cause-effect relationship identification model to obtain a corresponding first identification result, and take the first identification result as a label of the corresponding second sample text;

[0022] In a case where the first matter relationship identification model is activated with dropout, the first matter relationship identification model is used to perform matter relationship identification on each second sample text for multiple times to obtain multiple second identification results of the second sample text.

[0023] According to the multiple second identification results of each second sample text, a degree of uncertainty of the first matter relationship identification model in predicting each second sample text is determined.

[0024] As an optional embodiment, the model training apparatus is further configured to: for a third sample text in the multiple target sample texts, determine a degree of dispersion of the multiple second identification results of the third sample text;

[0025] When the retraining module iteratively trains the first matter relationship identification model based on the multiple target sample texts, the retraining module is configured to:

[0026] According to the degree of dispersion corresponding to each third sample text in the multiple target sample texts, a weight corresponding to each third sample text is determined; the weight is inversely proportional to the size of the degree of dispersion;

[0027] The multiple target sample texts are input into the first matter relationship identification model to obtain prediction identification results of the multiple target sample texts respectively;

[0028] According to the prediction identification result of each first sample text in the multiple target sample texts and the corresponding label, a first target value is determined.

[0029] For each third sample text in the multiple target sample texts, an initial second target value is determined based on the prediction identification result of the third sample text and the corresponding label, and the initial second target value is weighted based on the weight corresponding to the third sample text to obtain a second target value.

[0030] According to the first target value and the second target value, a training target value of the first matter relationship identification model is obtained, and if the training target value does not meet a condition, the parameters are adjusted, and the training is continued based on the target sample texts and the adjusted parameters.

[0031] As an optional embodiment, when the retraining module determines the weight corresponding to each third sample text in the multiple target sample texts based on the degree of dispersion corresponding to each third sample text, the retraining module is configured to: for each third sample text in the multiple target sample texts, determine the weight of the third sample text based on the degree of dispersion corresponding to the third sample text.

[0032] As an optional embodiment, when the sample screening module screens the multiple third sample texts from the multiple second sample texts based on the degree of uncertainty corresponding to each second sample text, the sample screening module is configured to:

[0033] The second sample texts are sorted according to the size relationship of the uncertainty degrees corresponding to all the second sample texts, and a preset number of second sample texts are filtered as third sample texts according to the sorting result.

[0034] As an optional embodiment, when the sample filtering module filters a plurality of third sample texts from the second sample texts according to the uncertainty degrees corresponding to the second sample texts, the sample filtering module is configured to:

[0035] determine a probability of each second sample text being a third sample text according to the size relationship of the uncertainty degrees corresponding to all the second sample texts;

[0036] filter a preset number of second sample texts as third sample texts according to the probabilities corresponding to the second sample texts.

[0037] As an optional embodiment, when the sample filtering module determines the probability of each second sample text being a third sample text, the sample filtering module is configured to:

[0038] take a ratio of the uncertainty degree corresponding to the second sample text to a sum of the uncertainty degrees corresponding to all the second sample texts as the probability of the second sample text being a third sample text.

[0039] According to another aspect of the embodiments of the present application, a text processing apparatus is provided, which comprises:

[0040] a text acquisition module configured to acquire a text to be recognized;

[0041] a causal relationship analysis module configured to input the text to be recognized into a trained causal relationship recognition model to obtain a causal relationship between event information contained in the text to be recognized.

[0042] The trained causal relationship recognition model is trained by the model training apparatus.

[0043] According to another aspect of the embodiments of the present application, an electronic device is provided, which comprises a memory, a processor, and a computer program stored in the memory, and the processor executes the computer program to implement the steps of the above model training method and / or text processing method.

[0044] According to another aspect of the embodiments of the present application, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the steps of the above model training method and / or text processing method.

[0045] According to an aspect of the embodiments of the present application, a computer program product is provided, which comprises a computer program, and the computer program is executed by a processor to implement the steps of the above model training method and / or text processing method.

[0046] The technical scheme provided by the embodiments of the present application has the beneficial effects that:

[0047] By obtaining the first sample set and the second sample set, the first sample set includes a plurality of first sample texts labeled with labels, and the second sample set includes a plurality of second sample texts without labels, a first matter relationship identification model is trained based on the first sample set, the uncertainty degree of each second sample text predicted by the trained first matter relationship identification model is determined, and the third sample text with low uncertainty degree is screened out from the second sample texts according to the uncertainty degree. The number of samples can be quickly expanded, the problem of insufficient labeled data can be solved, and the time of manual labeling can be greatly saved. Of course, some second sample texts with high uncertainty degree can also be screened out, and the model is trained by using the second sample texts with high uncertainty. Compared with the second matter relationship identification model obtained by retraining the first matter relationship identification model based on the first sample text and the third sample text with low uncertainty degree, the model has higher accuracy and robustness, and is more suitable for complex text recognition. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical scheme in the embodiments of the present application, the drawings needed in the description of the embodiments of the present application will be briefly introduced.

[0049] Figure 1 A schematic diagram of an application environment provided by the embodiments of the present application;

[0050] Figure 2 A flowchart of a model training method provided by the embodiments of the present application;

[0051] Figure 3 A schematic diagram of a first matter relationship identification model provided by the embodiments of the present application, which executes T times of feedforward operation in the case of activating dropout;

[0052] Figure 4 A flowchart of a model training method provided by the embodiments of the present application, which executes T times of feedforward operation in the case of activating dropout;

[0053] Figure 5 A schematic diagram of a model training method provided by the embodiments of the present application, which executes T times of feedforward operation in the case of activating dropout;

[0054] Figure 6 A flowchart of a model training method provided by another embodiment of the embodiments of the present application;

[0055] Figure 7 A flowchart of a text processing method provided by the embodiments of the present application;

[0056] Figure 8 A structural schematic diagram of a text processing system provided for an embodiment of the present application is shown in FIG. 1.

[0057] Figure 9 A structural schematic diagram of a model training apparatus provided for an embodiment of the present application is shown in FIG. 2.

[0058] Figure 10 A structural schematic diagram of a text processing apparatus provided for an embodiment of the present application is shown in FIG. 3.

[0059] Figure 11 A structural schematic diagram of an electronic device provided for an embodiment of the present application is shown in FIG. 4. DETAILED DESCRIPTION

[0060] Embodiments of the present application will be described below in conjunction with the accompanying drawings. It should be understood that the embodiments described below in conjunction with the accompanying drawings are exemplary descriptions of the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions of the embodiments of the present application.

[0061] Those skilled in the art can understand that the singular forms "a", "an" and "the" used herein include plural forms unless specifically stated otherwise. It should be further understood that the terms "include" and "contain" used in the embodiments of the present application mean that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements and / or components, but do not exclude other features, information, data, steps, operations, elements, components and / or combinations thereof supported by the present technology. It should be understood that when we say that an element is "connected" or "coupled" to another element, the element can be directly connected or coupled to the other element, or it can mean that the element and the other element establish a connection relationship through an intermediate element. In addition, the "connection" or "coupling" used herein can include wireless connection or wireless coupling. The term "and / or" used herein indicates that at least one of the items defined by the term, for example, "A and / or B" can be implemented as "A", or as "B", or as "A and B".

[0062] Optionally, the model training method and the text processing method provided in the embodiments of the present application can be implemented based on artificial intelligence (AI) technology. For example, the feature extraction of the to-be-processed text, the determination of the prediction uncertainty degree of the training sample, the training sample screening, and the identification of the logical relationship of the to-be-processed text can be implemented through a trained neural network model. AI is a theory, method, technology and application system for simulating, extending and expanding human intelligence by using a digital computer or a machine controlled by a digital computer, perceiving the environment, acquiring knowledge and using the knowledge to obtain the best results. With the research and progress of artificial intelligence technology, artificial intelligence technology has been widely researched and applied in many fields. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0063] Optionally, the model training and text processing related in the embodiments of the present application can be implemented based on cloud technology. For example, the uncertainty degree calculation in the training of the neural network model and the data calculation in the processing of the to-be-processed data can be implemented by cloud computing. Cloud technology refers to a kind of hosting technology that unifies a series of resources such as hardware, software and network in a wide area network or a local area network to realize data calculation, storage, processing and sharing. Cloud computing refers to a delivery and use mode of IT infrastructure, which means obtaining required resources in a scalable manner through a network. With the development of Internet, real-time data flow and diversified connected devices, and the promotion of search services, social networks, mobile commerce and open collaboration, cloud computing has developed rapidly. Unlike previous parallel distributed computing, the generation of cloud computing will revolutionize the entire Internet model and enterprise management model from the concept.

[0064] In order to make the purpose, technical solutions and advantages of the present application clearer, the embodiments of the present application will be described in further detail below with reference to the drawings.

[0065] Firstly, several terms related to the present application are introduced and explained:

[0066] 1) Dropout, a method to prevent overfitting. In each training, the model "drops" some model parameters with a probability p, and the "dropped" model parameters are not completely the same each time, so that the model is training a unique model in each training process.

[0067] 2) Uncertainty degree, which refers to the degree of disbelief of the neural network model for the prediction result. In related technologies, the relatively unreliable prediction results are usually filtered out according to the uncertainty degree.

[0068] 3) Existing logical relationship identification model:

[0069] Structured Joint Model model, which jointly optimizes event extraction and event temporal relation extraction, uses pre-trained language model BERT and BiLSTM to extract features, and then uses structured SVM to constrain the prediction results. However, this model only uses text information and ignores the statistical information of the data set, such as the distribution of each label.

[0070] End-to-end Inference model, which uses pre-trained language model BERT and LSTM to obtain the context representation of event pairs. In order to alleviate the problem of class imbalance in the data set, the model uses the Lagrange relaxation method to incorporate probabilistic domain knowledge into the model. This model only focuses on solving the problem of class imbalance in the data set, ignoring the distribution of labels.

[0071] HGRU model, which uses pre-trained language model BERT and LSTM to obtain the context representation of event pairs. In addition, this method also uses hyperbolic space to model the asymmetric relationship between labels, and uses some temporal common sense knowledge to assist the model prediction. This model cannot cope with the problem of insufficient training data, which is prone to overfitting.

[0072] The above methods are prone to ignore the problem of insufficient training data, cannot utilize a large amount of unlabeled data, and are prone to overfitting.

[0073] The model training method, text processing method, device, electronic equipment, computer readable storage medium and computer program product provided by the present application aim to solve the above technical problems of the prior art.

[0074] The technical solutions of the embodiments of the present application and the technical effects produced by the technical solutions of the present application will be described below through the description of several exemplary embodiments. It should be pointed out that the following embodiments can be mutually referenced, borrowed or combined. For the same terms, similar features and similar implementation steps in different embodiments, they will not be described repeatedly.

[0075] Please refer to Figure 1 , Figure 1 is a schematic diagram of an application environment provided by the embodiments of the present application, as shown in the figure, the application environment at least includes server 01 and terminal 02.

[0076] In the embodiments of the present application, the server 01 can be used to provide background services for the terminal 01, specifically, the training of the matter-relationship identification model can be performed, and the trained matter-relationship identification model is sent to the terminal 02. The server 01 can include a stand-alone physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms.

[0077] In the embodiments of the present application, the terminal 02 can be used to provide user-oriented text processing services, specifically, the identification of matter-relationship can be implemented based on the matter-relationship identification model trained by the server 01. The terminal 02 can include entity devices such as smart phones, desktop computers, tablet computers, notebook computers, smart speakers, digital assistants, augmented reality (AR) / virtual reality (VR) devices, and smart wearable devices. The entity device can also include software running in the entity device, such as an application program. The operating system running on the terminal 01 in the embodiments of the present application can include but is not limited to Android system, IOS system, Linux, Windows, etc.

[0078] In the embodiments of the present application, the terminal 01 and the server 02 described above can be directly or indirectly connected through wired or wireless communication, which is not limited in the present application.

[0079] In addition, it should be noted that in actual applications, the training of the matter-relationship identification model can also be implemented in the terminal 01. In the embodiments of the present application, the training of the matter-relationship identification model is preferably implemented in the server 01, so as to reduce the data processing pressure of the terminal and improve the device performance of the user-oriented terminal.

[0080] In a specific embodiment, the server 02 is a distributed system, which can be a blockchain system. When the distributed system 100 is a blockchain system, it can be formed by a plurality of nodes (any form of computing device in the access network, such as a server, a user terminal), and a peer-to-peer (P2P) network is formed between the nodes. The P2P protocol is an application layer protocol running on the transmission control protocol (TCP) protocol. In the distributed system, any machine such as a server or a terminal can join as a node, and the node includes a hardware layer, an intermediate layer, an operating system layer, and an application layer. Specifically, the functions of the nodes in the blockchain system include:

[0081] 1) Routing, a basic function of the node, used to support communication between nodes.

[0082] In addition to the routing function, the node can also have the following functions:

[0083] 2) Application, used to deploy in the blockchain, implement specific business according to actual business needs, record data related to the implementation function to form record data, carry a digital signature in the record data to represent the source of the task data, and send the record data to other nodes in the blockchain system. When the other nodes successfully verify the source and integrity of the record data, the record data is added to the temporary block.

[0084] The present application provides a model training method, as shown in Figure 2 The method comprises the following steps:

[0085] S101, obtaining a first sample set and a second sample set.

[0086] The first sample set of the present application comprises a plurality of first sample texts labeled with labels, and the second sample set comprises a plurality of second sample texts without labels. The domains of the sample texts in the first sample set and the second sample set are not limited to a single domain, and can cover as many domains as possible, for example, news, novels, academic papers, dialogue records, comment information, etc. The proportion of texts in different domains is not specifically limited.

[0087] The label of the first sample text represents the logical relationship between the event information contained in the first sample text. The number of event information contained in each first sample text can be two. The logical relationship is used to represent the logical relationship between the two event information. The specific type of the logical relationship is not specifically limited in the present application, for example, it can include:

[0088] Causal relationship, one event information causes another event information to occur. By obtaining the causal relationship, it can be applied to scenarios such as causal tracing and cause and effect, for example, a first sample text "In recent days, there is an avian flu in a certain place, which leads to the price rise of poultry in the place", which includes two event information "avian flu in a certain place" and "poultry price rise in the place", and "avian flu in a certain place" is the cause of "poultry price rise in the place", so the label of the first sample text is "causal relationship".

[0089] Conditional relationship, one event information leads to another event information under certain conditions. By obtaining the conditional relationship, it can be applied to scenarios such as timing determination, for example, a first sample text "If a certain foreign currency rises in price, then gold should fall", which includes two event information "a certain foreign currency rises in price" and "gold falls", and under the condition of triggering "a certain foreign currency rises in price", "gold falls" occurs, so the label of the first sample text is "conditional relationship".

[0090] Reverse relationship, one event information is opposite to another event information. By obtaining the reverse relationship, it can be applied to scenarios such as analyzing negative examples and preventing accidents, for example, a first sample text "Although the stock price rises slightly, the trading volume is not large", which includes two event information "stock price rises slightly" and "trading volume is not large", and the label of the first sample text is "reverse relationship".

[0091] Consent relationship, one event information occurs immediately after another event information. By obtaining the consent relationship, it can be applied to scenarios such as event evolution and future intention recognition, for example, a first sample text "If Zheng is destroyed and it is beneficial to the king", which includes two event information "Zheng is destroyed" and "it is beneficial to the king", and when "Zheng is destroyed" occurs, "it is beneficial to the king" occurs immediately, so the label of the first sample text is "consent relationship".

[0092] The specific form of the label corresponding to the embodiments of the present application is not limited, for example, it can be a number, a letter, a code, etc.

[0093] S102, training the initial reason relationship identification model according to the first sample set to obtain the first reason relationship identification model, and determining the uncertainty degree of the first reason relationship identification model predicted for each second sample text.

[0094] Specifically, the embodiment of the present application can train the initial causal relation identification model through a plurality of first sample texts labeled with labels in the first sample set, until the training end condition is met, and the neural network model meeting the training end condition is taken as the first causal relation identification model. During the training of the model, the input of the model is each first sample text, and the output is the causal relation prediction result between the event information contained in each first sample text, i.e., the result predicted by the model. The training loss is calculated by comparing the label (real result) corresponding to each first sample text with the prediction result, and the model parameters are adjusted based on the training loss.

[0095] The initial causal relation identification model is not limited in the embodiment of the present application, and can be selected according to actual needs. For example, it can be a combination of a pre-trained language model RoBERTa and a bidirectional long short-term memory network BiLSTM, and the pre-trained language model can also be replaced by other models, such as ALBERT.

[0096] After obtaining the first causal relation identification model, the uncertainty degree of each second sample text predicted by the first causal relation identification model can be obtained. The uncertainty degree is caused by too few training samples and incomplete training. The higher the uncertainty degree of the model prediction for a sample, the less confident the model prediction result for the sample is, and the lower the accuracy of the prediction result is. Conversely, the lower the uncertainty degree of the model prediction for a sample, the more confident the model prediction result for the sample is, and the higher the accuracy of the prediction result is.

[0097] S103, filtering a plurality of third sample texts from the second sample texts according to the uncertainty degree corresponding to each second sample text.

[0098] Since the uncertainty degree can reflect the accuracy of the model prediction for a sample, high-quality sample texts (i.e., third sample texts) meeting the retraining requirements can be selected based on the uncertainty degree. Specifically, some second sample texts with low uncertainty degree can be selected, and the prediction result of the second sample text with low uncertainty degree has high accuracy. Such samples are used as samples for retraining, which can not only quickly expand the number of samples, but also greatly save the time of manual labeling. Of course, some second sample texts with high uncertainty degree can also be selected, and the model is trained using the second sample texts with high uncertainty. Compared with the first causal relation identification model and the second causal relation identification model trained based on the second sample texts with low uncertainty, the model has more prominent robustness and is more suitable for complex text recognition.

[0099] S104, taking each first sample text with a label and each third sample text as a target sample text, iteratively training the first matter relationship identification model based on the plurality of target sample texts until a training stop condition is met, and obtaining a trained second matter relationship identification model.

[0100] In the process of training the first matter relationship identification model, the input of the model is the target sample text, i.e., the first sample text and the third sample text, and the output is the matter relationship prediction result between the event information contained in the target sample text, i.e., the result predicted by the model. The training loss is calculated by comparing the label (real result) corresponding to each target sample text with the prediction result, and the model parameters are adjusted based on the training loss.

[0101] It should be understood that for the third sample text, before step S104, the third sample text can also be input into the first matter relationship identification model to obtain the matter relationship information between the event information contained in the third sample text output by the first matter relationship identification model, and serve as the label of the third sample text.

[0102] The model training method of the embodiments of the present application obtains a first sample set and a second sample set, the first sample set includes a plurality of first sample texts labeled with labels, and the second sample set includes a plurality of second sample texts without labels. The first matter relationship identification model is trained based on the first sample set, the uncertainty degree of each second sample text predicted by the trained first matter relationship identification model is determined, and the third sample text with low uncertainty degree is selected from the second sample texts according to the uncertainty degree. This method can quickly expand the number of samples, alleviate the problem of insufficient labeled data, greatly save the time of manual labeling, and of course, it can also select some second sample texts with high uncertainty degree to train the model. Compared with the second matter relationship identification model obtained by retraining the first matter relationship identification model based on the first sample text and the third sample text with low uncertainty degree, the second matter relationship identification model has higher accuracy and robustness, and is more suitable for complex text recognition.

[0103] Based on the above embodiments, as an optional embodiment, the uncertainty degree of each second sample text predicted by the first matter relationship identification model is determined, comprising:

[0104] S201, identifying the matter relationship of each second sample text by the first matter relationship identification model to obtain the corresponding first identification result, and taking the first identification result as the label of the corresponding second sample text.

[0105] It should be understood that the first identification result is the logical relationship between the event information contained in each second sample text output by the first logical relationship identification model.

[0106] S202, in the case of activating dropout of the first logical relationship identification model, performing multiple logical relationship identifications on each second sample text by the first logical relationship identification model to obtain multiple second identification results of each second sample text.

[0107] See Figure 3 which exemplarily shows a schematic diagram of performing T times of feedforward operation in the case of activating dropout of the first logical relationship identification model of the embodiment of the present application, as shown in the figure, for the second sample text x u The parameters activated by the first logical relationship identification model are different each time the feedforward operation is performed, and the parameters of the first logical relationship identification model in the tth feedforward operation are The operation result of the tth feedforward operation is denoted as The method fuses the operation results of T times to obtain the final label result:

[0108]

[0109] Wherein, c' represents the identification result output each time, c represents the first identification result output by the first logical relationship identification model in the case of not activating dropout for x u argmax is a function of finding the parameter (set) of a function: when there is another function y = f(x), if there is a result x0 = argmax(f(x)), it means that when the function f(x) takes x = x0, the maximum value of the range of f(x) is obtained.

[0110] S203, determining the uncertainty degree of the prediction of the first logical relationship identification model for each second sample text according to the multiple second identification results of each second sample text.

[0111] When the model labels the unlabeled data, the more "confident" the model is, the higher the matching degree of the prediction result of the unlabeled data should be with the label. Therefore, for each second sample text, the matching degree of each second identification result with the label corresponding to the second sample text can be counted, and then according to all the matching degrees corresponding to the second sample text, the uncertainty degree of the prediction of the first logical relationship identification model for the second sample text is determined.

[0112] Specifically, the matching degree in the embodiments of the present application can be represented by the probability of the second recognition result output by the first causal relationship identification model when dropout is activated. In some embodiments, the degree of uncertainty of the prediction of the first causal relationship identification model for the second sample text is considered. The more dispersed the distribution of the second recognition result is, the more uncertain the prediction of the first causal relationship identification model for the second sample text is, that is, the more uncertain it is. Therefore, the degree of uncertainty can be represented by the degree of dispersion of the probability distribution. The embodiments of the present application do not make specific limitations on how to represent the degree of dispersion, for example, at least one of the expectation, variance, standard deviation, and mean square deviation can be used to represent the degree of dispersion.

[0113] When the dispersion degree of each second recognition result of the second sample text is represented by the expectation and variance (which is also the degree of uncertainty of the prediction of the first causal relationship identification model for the second sample text), it can be specifically represented by the following formula:

[0114]

[0115]

[0116] Wherein, Var(y) represents the variance of the matching degree of each second recognition result of the second sample text and the label (which can also be understood as the probability p of each second recognition result being the label); E(y) represents the expectation of the matching degree of each second recognition result of the second sample text and the label.

[0117] In the embodiments of the present application, the first causal relationship identification model is activated with dropout, and the first causal relationship identification model performs causal relationship identification on each second sample text multiple times to obtain multiple second recognition results of each second sample text. Since the parameters in the first causal relationship identification model are not the same when each second recognition result is obtained, the second recognition results obtained are also not the same. Therefore, the degree of uncertainty of the prediction of the first causal relationship identification model for each second sample text can be determined based on the distribution of the multiple second recognition results.

[0118] In order to better cope with the situation that the label of the third sample text is obtained by prediction rather than annotation according to the actual situation, and there is an error, the embodiments of the present application can consider the influence of sample label accuracy on the training target when retraining the first causal relationship identification model, and set a higher weight for the training target of the sample with a more concentrated recognition result distribution, so that the model pays more attention to the samples with more accurate recognition, and improves the prediction accuracy of the model.

[0119] In some embodiments, the embodiments of the present application further comprise: determining the dispersion degree of the multiple second recognition results of the third sample text in the multiple target samples.

[0120] Specifically, the embodiments of the present application can represent the dispersion degree with variance, covariance, standard deviation, etc.

[0121] Please refer to Figure 4 which exemplarily shows a flowchart of iterative training of the first cause-effect relationship identification model based on the plurality of target sample texts according to the embodiments of the present application. As shown in the figure, each first sample text and each third sample text in the plurality of target samples are input into the first cause-effect relationship identification model to obtain the predicted identification result of each first sample text and the predicted identification result of each third sample text.

[0122] The first target value is determined according to the predicted identification result of each first sample text in the plurality of target samples and the corresponding label;

[0123] The weight corresponding to each third sample text is determined according to the dispersion degree corresponding to each third sample text in the plurality of target samples. For each third sample text in the plurality of target samples, the initial second target value is determined according to the predicted identification result of the third sample text and the corresponding label, and the second target value is obtained by weighting the initial second target value with the weight corresponding to the third sample text.

[0124] The training target value of the first cause-effect relationship identification model is obtained according to the first target value and the second target value. If the training target value does not meet the condition, the parameters are adjusted, and the target sample texts and the adjusted parameters are used for continuous training.

[0125] It should be understood that the embodiments of the present application are equivalent to setting a weight (weight 1) of 1 for the first target value corresponding to each first sample text, and setting a weight (weight 2) determined based on the dispersion degree for the initial second target value corresponding to each third sample text. The weight is inversely proportional to the size of the dispersion degree, that is, if the dispersion degree of the third sample text is larger, the obtained weight is smaller. If the weight 2 is less than 1, the first target value will play a greater role in calculating the size of the training target value, which means that the model in the training will pay more attention to the first sample text. Conversely, if the weight 2 is greater than 1, it means that the model in the training will pay more attention to the second sample text. The present application can effectively deal with the noise problem in the back-label data by setting the weight for the second target value corresponding to the third sample text.

[0126] The weight set by the embodiments of the present application can be determined based on the dispersion degrees corresponding to all third sample texts, or the weight of each third sample text can be set based on the dispersion degree corresponding to the third sample text itself. Further, if the uniform weight is determined for all third sample texts, the dispersion degrees corresponding to multiple (which can be all) third sample texts are obtained first, then the mean of the multiple dispersion degrees is obtained, and then the weight is obtained based on the mean of the dispersion degrees. At this time, the obtained weight is the uniform weight applicable to all third sample texts.

[0127] In some embodiments, the first target value is used to describe the matching degree between the predicted recognition result of each first sample text and the corresponding label, and can be specifically represented based on the probability of the predicted recognition result of each first sample text being the corresponding label. The initial second target value is used to describe the matching degree between the predicted recognition result of each third sample text and the corresponding label, and can be specifically represented based on the probability of the predicted recognition result of each third sample text being the corresponding label.

[0128] The embodiments of the present application determine the first target value for the predicted recognition result and the corresponding label of each first sample text, determine the initial second target value for the predicted recognition result and the corresponding label of each third sample text, and weight each initial second target value based on the dispersion degree of the second recognition result of each third sample text, so that the model can pay more attention to samples with lower prediction uncertainty during training, thereby improving the accuracy of the model.

[0129] Based on the above embodiments, as an optional embodiment, the weight corresponding to each third sample text in the multiple target samples is determined according to the dispersion degree corresponding to the third sample text, including:

[0130] For each third sample text in the multiple target samples, the weight of the third sample text is determined according to the dispersion degree corresponding to the third sample text.

[0131] It should be noted that in order to enable the model to pay more attention to each third sample text in a more fine-grained and targeted manner, the embodiments of the present application determine the weight of each third sample text based on the dispersion degree of each second recognition result of the third sample text.

[0132] In some embodiments, the dispersion degree can be represented by variance, that is, the weight of the third sample text is determined by calculating the variance of the matching degree between each second recognition result of the third sample text and the preset label.

[0133] In some embodiments, the calculation formula of the training target value can be represented as:

[0134]

[0135]

[0136] Where, x l D represents the first sample set. l The first sample text in the middle; y l x represents l The tag; p(y) l |x l ,W) means that with model parameters W, the input x l The model outputs the recognition result as y l The probability, x u S represents the third sample text. u Var(y) represents the set of third sample texts selected from the second sample set, where Var(y) represents the input x when the first logical relationship recognition model activates dropout. u The variance of the probability that each second recognition result output during the T-th feedforward operation is the label y (where the model parameters during the t-th feedforward operation are...). The distribution of model parameters qθ(W*) and p(y|x) are satisfied when T feedforward operations are performed. u ,W) means that with model parameters W, the input x u The model outputs the probability that the recognition result is y. Iterative training stops when this training target value reaches its minimum.

[0137] In some embodiments, this application sets the weights of the initial second target value in different ways, trains a second reasoning relationship recognition model based on the weighted second target value, and statistically analyzes the performance of the second reasoning relationship model trained in each way, as shown in Table 1:

[0138]

[0139] Table 1 Comparison of the impact of different weighting methods on the F1 score

[0140] In Table 1, Mean refers to all third-sample texts having a weight of 1; Probability refers to using the degree of matching between the prediction result and the label (i.e., probability) as the training weight of the sample; and Uncertainty refers to using the inverse of the dispersion of the matching degree as the training weight of the sample. The sample weight allocation method proposed in this application significantly outperforms the method of setting the weight to 1, which also demonstrates the effectiveness of the proposed method.

[0141] According to the uncertainty degrees corresponding to the respective second sample texts, the plurality of third sample texts are screened from the respective second sample texts, including:

[0142] According to the size relationship of the uncertainty degrees corresponding to all the second sample texts, the second sample texts are sorted, and there are two sorting manners, one is to sort in the order from small to large of the uncertainty degrees, and the other is to sort in the order from large to small of the uncertainty degrees, the two sorting manners result in two sorting results, therefore, if it is desired to expand the number of training samples as soon as possible, a part of the second sample texts with lower uncertainty degrees can be selected as the third sample texts, and if it is desired that the second relationship identification model has more prominent robustness, a part of the second sample texts with higher uncertainty degrees can be selected as the third sample texts.

[0143] If the model always focuses on simple samples with lower uncertainty degrees, the prediction performance of the model will be improved, but the space for improvement is limited, and if the model always focuses on difficult samples with higher uncertainty degrees, the prediction performance of the model will be affected, in order to balance the benefits and prediction performance of the model, on the basis of the above embodiments, as an optional embodiment, according to the uncertainty degrees corresponding to the respective second sample texts, the plurality of third sample texts are screened from the respective second sample texts, including:

[0144] According to the size relationship of the uncertainty degrees corresponding to all the second sample texts, the probability of each second sample text being used as a third sample text is determined;

[0145] According to the probabilities corresponding to the respective second sample texts, a preset number of second sample texts are screened as third sample texts.

[0146] According to the size relationship of the uncertainty degrees corresponding to all the second sample texts, the probability of each second sample text being used as a third sample text is determined;

[0147] Since the various third sample texts obtained by the present application are not repeated, after determining a second sample text as a new third sample text each time, in order to avoid the problem of discarding and resampling when extracting a repeated third sample text, the present application screens the third sample text for non-replacement probability sampling. After screening a new third sample text from the second sample text set each time, the probability of each remaining third sample text as a third sample text can be recalculated based on the uncertainty degree corresponding to the remaining second sample texts in the second sample text set until the number of screened third sample texts meets the preset condition.

[0148] See Figure 5 which exemplarily shows a schematic diagram of screening a third sample text based on probability according to an embodiment of the present application. As shown in the figure, the second sample set includes five second sample texts, and two second sample texts need to be determined from the five second sample texts.

[0149] After determining the uncertainty degrees corresponding to the five sample texts, the probabilities corresponding to the five second sample texts are determined based on the size relationship of the uncertainty degrees corresponding to the five second sample texts. The present application can be set to the higher the uncertainty degree, the lower the corresponding probability. In the first screening of the third sample text, sampling is performed based on the probabilities corresponding to the five second sample texts to obtain a first third sample text. Then, the probabilities corresponding to the remaining four second sample texts are updated based on the size relationship of the uncertainty degrees corresponding to the remaining four second sample texts. Sampling is performed based on the updated probabilities corresponding to the four second sample texts to obtain a second third sample text.

[0150] The third sample texts screened by the present application based on probability will not be all low-uncertainty-degree third sample texts or all high-uncertainty-degree third sample texts as in the sorting method, and the benefits and prediction performance of the model can be balanced.

[0151] In some embodiments, the present application selects third sample texts from the second sample set in different ways respectively, and trains the second causal relationship identification model based on the selected third sample texts, and the performance of the second causal relationship model trained by each method is counted respectively. See Table 2:

[0152]

[0153] Table 2 Comparison table of influence of different sample selection methods on F1 index

[0154] Random in Table 2 refers to randomly selecting the third sample text; Probability refers to selecting the third sample text according to the predicted probability; Hard refers to selecting the difficult sample proposed in the application; Easy refers to selecting the simple sample proposed in the application, and MATRES and TB-Dense are two remote supervision relationship extraction data sets. 30% and 40% in the table refer to 30% or 40% of the first sample text in the data set. It can be seen that the performance of the three sample selection methods proposed in the embodiments of the application is better than that of the random selection method, which also illustrates the effectiveness of the method proposed in the embodiments of the application.

[0155] On the basis of the above embodiments, as an optional embodiment, the probability of determining that each second sample text is a third sample text comprises:

[0156] The ratio of the uncertainty degree corresponding to the second sample text to the sum of the uncertainty degrees corresponding to all second sample texts is taken as the probability of taking the second sample text as the third sample text.

[0157] Through the above method, the sum of the probabilities of all second sample texts being third sample texts is 100%, that is, a second sample text will be drawn as a third sample text in a sampling process. In some embodiments, the uncertainty degree can be determined according to the Bayesian Active Learning by Disagreement (BALD) index.

[0158] The calculation formula of the BALD index can be expressed as:

[0159]

[0160] Wherein, that is, in t times of feedforward operation (at this time, the model parameters are ), the input third sample text x u , the output second recognition result y u is the probability of the label c, D u represents the second sample set, and it should be understood that the label c is the recognition result output by the first relationship identification model without activating dropout for x u .

[0161] It should be understood that the application can represent the degree of uncertainty directly by the BALD index or by the value of (1-BALD index). When the degree of uncertainty is represented by the BALD index, the greater the value of BALD, the greater the corresponding degree of uncertainty. When the degree of uncertainty is represented by (1-BALD index), the greater the value of BALD, the smaller the corresponding degree of uncertainty.

[0162] Specifically, when the degree of uncertainty is represented by the BALD index, the calculation formula of the probability of the second sample text u as the third sample text can be represented as:

[0163]

[0164] When the degree of uncertainty is represented by (1-BALD index), the calculation formula of the probability of the second sample text u as the third sample text can be represented as:

[0165]

[0166] The embodiments of the application can select the third sample text for retraining the first causal relationship recognition model using one of them.

[0167] Please refer to Figure 6 which exemplarily shows a flowchart of a model training method according to another embodiment of the application. As shown in the figure, the method comprises the following steps:

[0168] Obtaining a first sample set and a second sample set, the first sample set comprising a plurality of first sample texts labeled with labels (labeled data), and the second sample set comprising a plurality of second sample texts without labels (unlabeled data);

[0169] Training an initial causal relationship recognition model by using the first sample set to obtain a first causal relationship recognition model;

[0170] Performing causal relationship recognition on each second sample text by using the first causal relationship recognition model to obtain a corresponding first recognition result, and taking the first recognition result as the label of the corresponding second sample text;

[0171] Determining the degree of uncertainty of the prediction of the first causal relationship recognition model for each second sample text, selecting a plurality of third sample texts from the second sample texts according to the degrees of uncertainty of the second sample texts, expanding the training sample size, taking each first sample text with a label and each third sample text as a target sample text, and iteratively training the first causal relationship recognition model based on the plurality of target sample texts until a training stop condition is met to obtain a trained second causal relationship recognition model.

[0172] The process of screening the third sample text and iteratively training the first cause-effect relationship identification model mainly includes three key processes:

[0173] (1) Uncertainty Estimation: using the method of activated dropout to estimate the uncertainty degree of the model

[0174] For the second sample text x u , the first cause-effect relationship identification model performs feedforward operation T times with activated dropout, and the parameters of the model at the t-th feedforward operation are The operation result is:

[0175]

[0176] The final prediction result y of the T times of operation results is:

[0177]

[0178] For each second sample text, the matching degree of the multiple second identification results of the second sample text and the label is determined, which can be represented by the probability p of the second identification result being the label. After obtaining the matching degree corresponding to each second identification result, the BALD index can be further used to determine the uncertainty degree of the prediction of the first cause-effect relationship identification model for the second sample text. As can be seen from the calculation formula of the BALD index in the foregoing embodiment, the BALD index is calculated according to the expectation of each second identification result.

[0179] (2) Sample Selection with Exploration: Two sample selection methods are designed, which are respectively based on sorting method and probability sampling method, to select simple samples and difficult samples for use;

[0180] Since the lower the uncertainty degree is, the stronger the confidence of the model prediction is, the uncertainty degree can be used as an index to screen simple samples or complex samples from the second sample set as third sample texts. The simple samples are samples with lower uncertainty degree, and the complex samples are samples with lower uncertainty degree.

[0181] The sorting method and the probability sampling method can be used to screen the third sample text. Specifically, when the sorting method is used, the uncertainty degrees of the second sample texts can be sorted, and a preset number of second sample texts can be selected from the sorting result as third sample texts in sequence.

[0182] When the probability sampling method is adopted, the ratio of the uncertainty degree corresponding to the second sample text to the sum of the uncertainty degrees corresponding to all the second sample texts can be taken as the probability of taking the second sample text as the third sample text. After determining the probability corresponding to each second sample text, sampling can be performed based on the probability.

[0183] (3) Uncertain-Aware Learning: The uncertainty predicted by the model is integrated into the objective function of training, so that the model pays more attention to samples with higher uncertainty and reduces the fitting of noisy data.

[0184] According to the dispersion degree corresponding to each third sample text, a weight corresponding to each third sample text is determined; the plurality of target samples (including the first sample text and the third sample text) are input into the first causal relationship identification model to obtain predicted identification results of the plurality of target samples respectively; a first target value is determined according to the predicted identification result of each first sample text and the corresponding label; for each third sample text, an initial second target value is determined according to the predicted identification result of the third sample text and the corresponding label, and the initial second target value is weighted by the weight corresponding to the third sample text to obtain a second target value; a training target value of the first causal relationship identification model is obtained according to the first target value and the second target value, and if the training target value does not meet the condition, the parameters are adjusted, and the training is continued based on the target sample text and the adjusted parameters.

[0185] Specifically, for the third sample text in the plurality of target samples, the dispersion degree of the plurality of second identification results of the third sample text is determined, which can be represented by variance;

[0186] Embodiments of the present application are inversely proportional to the size of the weight and the dispersion degree of the third sample text in the target sample, and the weight has two calculation methods:

[0187] (1) A uniform weight is set for all third sample texts: Specifically, the average of the dispersion degrees corresponding to all third sample texts can be calculated, and a weight is obtained based on the average dispersion degree.

[0188] (2) A personalized weight is set for each third sample text, that is, the average of the dispersion degrees corresponding to all third sample texts is not calculated, but the dispersion degree corresponding to the third sample text is directly used to obtain the corresponding weight. Specifically, the weight can be calculated as wherein Var(y) represents the variance.

[0189] Please refer to Figure 7 which exemplarily shows a flowchart of a text processing method according to an embodiment of the present application, as shown in the figure, which includes:

[0190] Obtaining text to be recognized.

[0191] Inputting the text to be recognized into a trained causal relation recognition model to obtain a causal relation between event information contained in the text to be recognized, the causal relation recognition model being trained according to the embodiments of the above model training methods.

[0192] It should be noted that the field of the text to be recognized is not specifically limited in the embodiments of the present application, for example, it can be news, novels, academic papers, dialogue records, comment information, and the like. The embodiments of the present application can further arrange a triple according to the output causal relation, the triple including two event information and the type of the causal relation. It should be understood that when the text to be recognized includes more than two event information, the causal relation between each two event information can be recognized by the causal relation recognition model.

[0193] The method provided by the embodiments of the present application can be applied to any scene in which the causal relation in the text needs to be recognized, for example, it can be an automatic question and answer scene, a hot news recognition scene, and the like. In order to better understand and illustrate the method and its effect provided by the embodiments of the present application, an optional implementation manner of the scheme provided by the present application is described below in combination with a specific application scene. The application scene is a game scene.

[0194] Figure 8 The structure schematic diagram of a text processing system applicable in the scene embodiments of the present application is shown, as shown in FIG. 1, the text processing system includes a user terminal, a sample text database, a model training server, a verification platform, a causal relation recognition server, and an intelligent chat server. Figure 8

[0195] ​The user terminal can be connected in communication with the intelligent chat server through a network, and the user terminal can be a user terminal of any consumer. The embodiments of the present application do not limit the type of chat application running on the user terminal, which can be a chat application that needs to be downloaded and installed by the user, a cloud chat application, or a chat application in a mini program. When the chat application runs on the user terminal, the user terminal sends chat information to the question and answer server through the network, the intelligent chat server receives the chat information, and generates the to-be-identified text according to the desensitized chat information and sends it to the fact-reason relationship identification server. Based on the trained fact-reason relationship identification model, the fact-reason relationship between the event information contained in the to-be-identified text is obtained, the instance relationship identification server returns the identified fact-reason relationship to the intelligent chat server, and the intelligent chat server returns the response information according to the fact-reason relationship. For example, the chat information is "I am happy because I know I got the first place", which contains event information including "sad" and "got the first place", and the fact-reason relationship between the two event information is causal relationship. The intelligent chat server determines that the user wants to emphasize why he is happy, and therefore generates relevant response information according to "got the first place", rather than "you have anything happy" and other response information related to "happy" - which will make the user feel off topic and reduce the user's enthusiasm for chatting. The intelligent chat server can also instruct the user to send feedback information when returning the response information. The feedback information is used to indicate the satisfaction of the accuracy of the response information. If the satisfaction is higher than a preset threshold, the intelligent chat server sends the desensitized chat information and the fact-reason relationship information contained in the chat information to the verification platform as a to-be-verified sample text. The sample auditing personnel logs in to the verification platform to manually judge whether the fact-reason relationship is accurate, and if it is accurate, a first sample text is constructed according to the chat information and the corresponding fact-reason relationship, and the first sample text is sent to the sample text database.

[0196] The sample text database stores a plurality of labeled samples (first sample texts) labeled with labels and a plurality of unlabeled samples (second sample texts) without labels. The embodiments of the present application can automatically capture text information on the Internet through a preset crawler program, and then send the text information to the sample text database. The sample text database stores the text information captured by the crawler program as a second sample text. When the number of updated samples in the text database meets a preset condition, the updated samples (including a plurality of first sample texts and a plurality of second sample texts) are sent to the model training server.

[0197] The model training server trains an initial rationale relation identification model according to a plurality of first sample texts, obtains a first rationale relation identification model, identifies rationale relations of each second sample text through the first rationale relation identification model, obtains a corresponding first identification result, and takes the first identification result as a label of the corresponding second sample text; in a case where the first rationale relation identification model is activated dropout, the first rationale relation identification model is used to identify rationale relations of each second sample text for multiple times, and a plurality of second identification results of each second sample text are obtained; the uncertainty degree of the first rationale relation identification model for predicting each second sample text is determined according to the plurality of second identification results of each second sample text, the probability of each second sample text being taken as a third sample text is determined according to the size relationship of the uncertainty degrees corresponding to all second sample texts, a preset number of second sample texts are selected as third sample texts according to the probabilities corresponding to the second sample texts, the dispersion degree of the plurality of second identification results of the third sample text in the plurality of target samples is determined, the weight corresponding to each third sample text is determined according to the dispersion degrees corresponding to the third sample texts in the plurality of target samples, wherein the weight is inversely proportional to the dispersion degree, the plurality of target samples are input into the first rationale relation identification model, and prediction identification results of the plurality of target samples are obtained, respectively, the first target value is determined according to the prediction identification results of each first sample text in the plurality of target samples and the corresponding label, the initial second target value is determined according to the prediction identification result of the third sample text in the plurality of target samples and the corresponding label, the second target value is obtained by weighting the initial second target value according to the weight corresponding to the third sample text, the training target value of the first rationale relation identification model is obtained according to the first target value and the second target value, the parameters are adjusted if the training target value does not meet the condition, the target sample text and the adjusted parameters are used for continuous training, until a trained second rationale relation identification model is obtained, so that the rationale relation identification server can determine the rationale relation in the to-be-identified text by calling the second rationale relation identification model.

[0198] To prove the effectiveness of the embodiments of the present application, the model training method (Uncertain Aware Self Training, UAST) provided in the present application is further compared with several existing model training methods, a rationale relation identification model is trained based on a public remote supervision relation extraction data set (including), and the effects of each rationale relation identification model are compared. The comparison results are shown in Table 3:

[0199]

[0200] Table 3: Comparison of overall effects

[0201] PLM+BiLSTM in Table 3 refers to using a pre-trained language model and a bidirectional long short-term memory network to extract the context representation of the event pair, and then using a softmax classifier for classification; Han et al. uses a pre-trained language model and a bidirectional long short-term memory network to extract the context representation of the event pair, and then uses a structured prediction network to ensure the consistency of the model prediction; Mean-Teacher: is a commonly used semi-supervised learning method, which makes the prediction results of different variants of the model for the same input data as consistent as possible; Self-Training: is also a commonly used semi-supervised learning method, which first trains the model using a small amount of labeled data, then uses the trained model to label unlabeled data, and then mixes the labeled data and the original labeled data to train the model. Repeat the above steps until the model converges. 30% and 40% in the table refer to 30% or 40% of the first sample texts in the data set. As can be seen from Table 3, the method provided by the embodiments of the present application is superior to the previous methods in all indicators (precision, recall, and comprehensive evaluation index F1) on the international public data set.

[0202] The embodiments of the present application provide a model training device, as shown in Figure 9 The model training device can include a sample set acquisition module 101, an uncertainty calculation module 102, a sample screening module 103, and a retraining module 104, wherein

[0203] The sample set acquisition module 101 is configured to acquire a first sample set and a second sample set, the first sample set includes a plurality of first sample texts labeled with labels, and the second sample set includes a plurality of second sample texts without labels, the label of the first sample text represents the relationship between the event information contained in the first sample text;

[0204] The uncertainty calculation module 102 is configured to train an initial relationship identification model according to the first sample set to obtain a first relationship identification model, and determine the uncertainty degree of the first relationship identification model for each second sample text;

[0205] The sample screening module 103 is configured to screen a plurality of third sample texts from each second sample text according to the uncertainty degree corresponding to each second sample text;

[0206] The retraining module 104 is configured to take each first sample text with a label and each third sample text as a target sample text, and iteratively train the first relationship identification model based on a plurality of target sample texts until a training stop condition is met, to obtain a trained second relationship identification model.

[0207] The apparatus in this application embodiment can execute the model training method provided in this application embodiment. The implementation principle is similar. The actions performed by each module in the apparatus of each embodiment of this application correspond to the steps in the model training method of each embodiment of this application. For detailed functional descriptions of each module of the apparatus, please refer to the descriptions in the corresponding methods shown above, which will not be repeated here.

[0208] Compared to existing technologies that acquire a first sample set and a second sample set (the first sample set includes multiple labeled first sample texts, and the second sample set includes multiple unlabeled second sample texts), this approach first trains a first logical relationship recognition model based on the first sample set. Then, it uses this trained model to determine the degree of uncertainty in the prediction for each second sample text. Finally, it uses this uncertainty level to select third sample texts with low uncertainty from the second sample texts. This method can quickly expand the number of samples, alleviate the problem of insufficient labeled data, and greatly save time spent on manual labeling. Alternatively, it can select some second sample texts with higher uncertainty and use them to train the model. Compared to the second logical relationship recognition model obtained by training the first logical relationship recognition model based on the first sample texts and third sample texts with low uncertainty, this approach has higher accuracy and robustness, making it more suitable for grammatically complex text recognition.

[0209] As an optional embodiment, the uncertainty calculation module is used to determine the degree of uncertainty in the first logical relationship recognition model's prediction for each second sample text when:

[0210] The first logic relationship recognition model is used to identify the logic relationship of each second sample text to obtain the corresponding first recognition result, and the first recognition result is used as the label of the corresponding second sample text.

[0211] With dropout activated in the first logical relationship recognition model, multiple logical relationship recognitions are performed on each second sample text using the first logical relationship recognition model to obtain multiple second recognition results for each second sample text.

[0212] Based on multiple second recognition results for each second sample text, determine the degree of uncertainty in the prediction of the first logical relationship recognition model for each second sample text.

[0213] As an optional embodiment, the model training device is also used to: determine the degree of dispersion of multiple second recognition results of the third sample text among multiple target samples;

[0214] The retraining module is used for iterative training of the first-order relation recognition model based on multiple target sample texts:

[0215] determine a weight corresponding to each third sample text according to a discrete degree corresponding to the third sample text in the plurality of target samples, wherein the weight is inversely proportional to the size of the discrete degree;

[0216] input the plurality of target samples into the first causal relation identification model to obtain predicted identification results of the plurality of target samples respectively;

[0217] determine a first target value according to the predicted identification results of each first sample text and the corresponding label in the plurality of target samples;

[0218] For each third sample text in the plurality of target samples, determine an initial second target value according to the predicted identification result of the third sample text and the corresponding label, and weight the initial second target value according to the weight corresponding to the third sample text to obtain a second target value.

[0219] obtain a training target value of the first causal relation identification model according to the first target value and the second target value, adjust the parameters if the training target value does not meet the condition, and continue training based on the target sample text and the adjusted parameters.

[0220] As an optional embodiment, when the retraining module determines the weight corresponding to each third sample text according to the discrete degree corresponding to the third sample text in the plurality of target samples, it is used for: determining the weight of each third sample text according to the discrete degree corresponding to the third sample text in the plurality of target samples.

[0221] As an optional embodiment, when the sample screening module screens the plurality of third sample texts from each second sample text according to the uncertainty degree corresponding to each second sample text, it is used for:

[0222] sort according to the size relationship of the uncertainty degrees corresponding to all second sample texts, and screen a preset number of second sample texts as third sample texts according to the sorting result.

[0223] As an optional embodiment, when the sample screening module screens the plurality of third sample texts from each second sample text according to the uncertainty degree corresponding to each second sample text, it is used for:

[0224] determine the probability of each second sample text as a third sample text according to the size relationship of the uncertainty degrees corresponding to all second sample texts;

[0225] screen a preset number of second sample texts as third sample texts according to the probability corresponding to each second sample text.

[0226] As an optional embodiment, when the sample screening module determines the probability of each second sample text as a third sample text, it is used for:

[0227] The ratio of the uncertainty degree corresponding to the second sample text to the sum of the uncertainty degrees corresponding to all the second sample texts is taken as the probability of taking the second sample text as the third sample text.

[0228] An embodiment of the present application provides a text processing apparatus, as shown in the figure, the model training apparatus can comprise a text acquisition module 201 and a matter-logic relationship analysis module 202, wherein, Figure 10

[0229] The text acquisition module 201 is configured to acquire the to-be-recognized text.

[0230] The matter-logic relationship analysis module 202 is configured to input the to-be-recognized text into the trained matter-logic relationship recognition model to obtain the matter-logic relationship between the event information contained in the to-be-recognized text.

[0231] The trained matter-logic relationship recognition model is obtained by using the model training apparatus.

[0232] An embodiment of the present application provides an electronic device, comprising a memory, a processor and a computer program stored in the memory, and the processor executes the computer program to realize the steps of the model training method and / or the text processing method. Compared with the related art, the number of samples can be quickly expanded, and the problem of insufficient labeled data can be solved, and the time of manual labeling can be greatly saved. Of course, some second sample texts with high uncertainty degree can also be screened out, and the model is trained by using the second sample texts with high uncertainty degree. Compared with the second matter-logic relationship recognition model obtained by retraining the first matter-logic relationship recognition model based on the first sample texts and the third sample texts with low uncertainty degree, the model has higher accuracy and robustness, and is more suitable for complex syntax text recognition.

[0233] In an optional embodiment, an electronic device is provided, as shown in the figure, Figure 11 Figure 11 ​​The electronic device 4000 shown includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 can also include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception, etc. It should be noted that the transceiver 4004 is not limited to one in actual application, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

[0234] The processor 4001 can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic device, a transistor logic device, a hardware component or any combination thereof. It can implement or execute various exemplary logical blocks, modules and circuits described in combination with the disclosure. The processor 4001 can also be a combination of computing functions, such as one or more microprocessor combinations, combinations of DSP and microprocessor, etc.

[0235] The bus 4002 can include a path for transmitting information between the above-mentioned components. The bus 4002 can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience, Figure 11 Only one thick line is used in the middle, but it does not mean that there is only one bus or one type of bus.

[0236] The memory 4003 can be a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, a RAM (Random Access Memory) or other type of dynamic storage device that can store information and instructions, an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, a magnetic disk storage or other magnetic storage devices, or any other medium capable of storing computer instructions and capable of being read by a computer, without limitation.

[0237] The memory 4003 is used to store a computer program for implementing the embodiments of the present application, and is controlled by the processor 4001 to execute. The processor 4001 is used to execute the computer program stored in the memory 4003 to realize the steps shown in the foregoing method embodiments.

[0238] The embodiments of the present application provide a computer readable storage medium, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to realize the steps and corresponding contents of the foregoing method embodiments. Compared with the prior art, by obtaining a first sample set and a second sample set, the first sample set includes a plurality of first sample texts labeled with a label, the second sample set includes a plurality of second sample texts without a label, a first matter relationship identification model is trained based on the first sample set, the uncertainty degree of each second sample text predicted by the trained first matter relationship identification model is determined, and the third sample text with a low uncertainty degree is screened from the second sample texts according to the uncertainty degree. The number of samples can be quickly expanded, the problem of insufficient labeled data can be solved, and the time of manual labeling can be greatly saved. Of course, the second sample texts with a high uncertainty degree can also be screened, and the model is trained by using the second sample texts with a high uncertainty degree. Compared with a second matter relationship identification model obtained by retraining the first matter relationship identification model based on the first sample texts and the third sample texts with a low uncertainty degree, the second matter relationship identification model has higher accuracy and robustness, and is more suitable for text recognition with complex syntax.

[0239] The embodiment of the present application also provides a computer program product comprising a computer program, which can realize the steps and corresponding contents of the foregoing method embodiment when executed by a processor. Compared with the prior art, by obtaining a first sample set and a second sample set, the first sample set comprising a plurality of first sample texts labeled with labels, the second sample set comprising a plurality of second sample texts without labels, a first matter relationship identification model is trained based on the first sample set, the uncertainty degree of each second sample text predicted by the trained first matter relationship identification model is determined, and the third sample text with low uncertainty degree is screened out from the second sample texts according to the uncertainty degree, which can quickly expand the number of samples, alleviate the problem of insufficient labeled data, greatly save the time of manual labeling, and of course, some second sample texts with high uncertainty degree can also be screened out, and the model is trained by using the second sample texts with high uncertainty, and compared with a second matter relationship identification model obtained by retraining the first matter relationship identification model based on the first sample texts and the third sample texts with low uncertainty degree, the model has higher accuracy and robustness, and is more suitable for complex text recognition.

[0240] The terms "first", "second", "third", "fourth", "1", "2", and the like (if any) in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described.

[0241] It should be understood that although the flowcharts of the embodiments of the present application indicate the respective operation steps by arrows, the implementation order of the steps is not limited to the order indicated by the arrows. Unless otherwise specified herein, the implementation steps in the flowcharts can be executed in other orders as required in some implementation scenarios of the embodiments of the present application. In addition, part or all of the steps in the flowcharts can include a plurality of sub-steps or a plurality of stages based on the actual implementation scenario. Part or all of the sub-steps or stages can be executed at the same time, and each of the sub-steps or stages can also be executed at different times. In the scenario where the execution times are different, the execution order of the sub-steps or stages can be flexibly configured according to requirements, and the embodiments of the present application do not limit this.

[0242] The above is only an optional implementation manner of some implementation scenarios of the present application, and it should be pointed out that, for ordinary skilled persons in the technical field, other similar implementation manners based on the technical idea of the present application without departing from the technical concept of the present application also belong to the protection scope of the embodiments of the present application.

Claims

1. A model training method, characterized in that, The method comprises the following steps: obtaining a first sample set and a second sample set, the first sample set comprising a plurality of first sample texts labeled with labels, and the second sample set comprising a plurality of second sample texts without labels, the labels of the first sample texts representing the logical relationship between the event information contained in the first sample texts; training an initial logical relationship identification model according to the first sample set to obtain a first logical relationship identification model, and determining the uncertainty degree of the prediction of the first logical relationship identification model for each second sample text; determining the probability of each second sample text as a third sample text according to the uncertainty degree of each second sample text in the second sample set, and randomly selecting a plurality of second sample texts as third sample texts from the second sample set according to the probabilities corresponding to the second sample texts, wherein after each second sample text is randomly selected, the selected second sample text is removed from the second sample set, and the probabilities corresponding to the remaining second sample texts in the second sample set are recalculated according to the uncertainty degrees corresponding to the remaining second sample texts; taking each first sample text with a label and each third sample text as a target sample text, and iteratively training the first logical relationship identification model based on a plurality of target sample texts until a training stop condition is met to obtain a trained second logical relationship identification model; wherein the iteratively training the first logical relationship identification model based on a plurality of target sample texts comprises: for each third sample text in the plurality of target sample texts, determining the dispersion degree of the second identification result of the third sample text; averaging the dispersion degrees corresponding to all third sample texts to obtain an average dispersion degree, and obtaining the same weight for all third sample texts based on the average dispersion degree; inputting each third sample text into the first logical relationship identification model to obtain a predicted identification result for each third sample text; for each third sample text, determining an initial second target value based on the predicted identification result of the third sample text and the corresponding label, and weighting the initial second target value based on the weight corresponding to the third sample text to obtain a second target value; the label of the third sample text is the logical relationship information between the event information contained in the third sample text output by the first logical relationship identification model; obtaining the training target value of the first logical relationship identification model according to the second target value of each third sample text.

2. The method of claim 1, wherein, The determination of the uncertainty degree of the prediction of the first logical relationship identification model for each second sample text comprises: performing logical relationship identification on each second sample text through the first logical relationship identification model to obtain a corresponding first identification result, and taking the first identification result as the label of the corresponding second sample text; in the case that the first logical relationship identification model is activated dropout, performing multiple logical relationship identifications on each second sample text through the first logical relationship identification model to obtain multiple second identification results of each second sample text; According to the plurality of second recognition results of each of the second sample texts, determine the uncertainty degree of the first cause-effect relationship recognition model predicted for each of the second sample texts.

3. The method of claim 2, wherein, The iterative training of the first cause-effect relationship recognition model based on the plurality of target sample texts comprises: According to the dispersion degree corresponding to each third sample text in the plurality of target samples, determine the weight corresponding to each third sample text; wherein the weight is inversely proportional to the size of the dispersion degree; Input each first sample text in the plurality of target samples into the first cause-effect relationship recognition model to obtain the predicted recognition result of each first sample text; According to the predicted recognition result of each first sample text and the corresponding label, determine a first target value; According to the first target value and the second target value, obtain a training target value of the first cause-effect relationship recognition model, if the training target value does not meet the condition, adjust the parameters, and continue training based on the target sample texts and the adjusted parameters.

4. The method of claim 3, wherein, The determination of the weight corresponding to each third sample text according to the dispersion degree corresponding to each third sample text in the plurality of target samples comprises: For each third sample text in the plurality of target samples, determine the weight of the third sample text according to the dispersion degree corresponding to the third sample text.

5. The method of claim 1, wherein, The screening of the plurality of third sample texts from each of the second sample texts according to the uncertainty degree corresponding to each of the second sample texts comprises: According to the size relationship of the uncertainty degrees corresponding to all second sample texts, sort and select a preset number of second sample texts as the third sample texts according to the sorting result.

6. The method of claim 1, wherein, The screening of the plurality of third sample texts from each of the second sample texts according to the uncertainty degree corresponding to each of the second sample texts comprises: According to the size relationship of the uncertainty degrees corresponding to all second sample texts, determine the probability of each of the second sample texts as a third sample text; According to the probability corresponding to each second sample text, select a preset number of second sample texts as the third sample texts.

7. The method of claim 6, wherein, The determination of the probability of each of the second sample texts as a third sample text comprises: The ratio of the uncertainty degree corresponding to the second sample text to the sum of the uncertainty degrees corresponding to all second sample texts is taken as the probability of the second sample text as the third sample text.

8. A text processing method characterized by, Comprise: Obtain the to be recognized text; Input the to be recognized text into the trained cause-effect relationship recognition model to obtain the cause-effect relationship between the event information contained in the to be recognized text; The trained cause-effect relationship recognition model is trained by the method of any one of claims 1-7.

9. A model training apparatus, comprising: Comprise: a sample set obtaining module, configured to obtain a first sample set and a second sample set, the first sample set comprising a plurality of first sample texts labeled with labels, and the second sample set comprising a plurality of second sample texts unlabeled with labels, the labels of the first sample texts representing a logical relationship between event information contained in the first sample texts; an uncertainty calculating module, configured to train an initial logical relationship identification model according to the first sample set to obtain a first logical relationship identification model, and determine a degree of uncertainty of the first logical relationship identification model for predicting each second sample text; a sample screening module, configured to determine a probability of each second sample text being used as a third sample text according to the degree of uncertainty of each second sample text in the second sample set, and randomly select a plurality of second sample texts from the second sample set as third sample texts according to the probabilities corresponding to the second sample texts, wherein, after each second sample text is randomly selected, the selected second sample text is removed from the second sample set, and the probabilities corresponding to the remaining second sample texts in the second sample set are recalculated according to the degrees of uncertainty of the remaining second sample texts; a retraining module, configured to use each first sample text and each third sample text as a target sample text, and iteratively train the first logical relationship identification model based on a plurality of target sample texts until a training stop condition is met, to obtain a trained second logical relationship identification model; wherein the retraining module iteratively trains the first logical relationship identification model based on a plurality of target sample texts, comprising: for a third sample text in the plurality of target sample texts, determining a degree of dispersion of a plurality of second identification results of the third sample text; averaging the degrees of dispersion corresponding to all third sample texts to obtain an average degree of dispersion, and obtaining a same weight for all third sample texts based on the average degree of dispersion; inputting each third sample text into the first logical relationship identification model to obtain a predicted identification result of each third sample text; for each third sample text, determining an initial second target value based on the predicted identification result of the third sample text and a corresponding label, weighting the initial second target value based on the weight corresponding to the third sample text to obtain a second target value, and the label of the third sample text being logical relationship information between event information contained in the third sample text output by the first logical relationship identification model; obtaining a training target value of the first logical relationship identification model according to the second target value of each third sample text.

10. A text processing apparatus characterized by comprising: comprising: a text obtaining module, configured to obtain a to-be-identified text; a logical relationship analysis module, configured to input the to-be-identified text into a trained logical relationship identification model to obtain a logical relationship between event information contained in the to-be-identified text; wherein the trained logical relationship identification model is trained by the model training device of claim 9.

11. An electronic device comprising a memory, a processor, and a computer program stored on the memory, wherein the computer program, when executed by the processor, is arranged to perform the method of any one of claims 1 to 10. The processor executes the computer program to implement the steps of the method of any one of claims 1-8.

12. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program, which when executed by a processor, implements the steps of the method according to any one of claims 1-8.

13. A computer program product comprising a computer program, characterized in that, The computer program, which when executed by a processor, implements the steps of the method according to any one of claims 1-8.

Images