Chart relation identification system

Abstract

The invention discloses a chapter relationship recognition system, which is based on an explicit chapter relationship data set, constructs an automatically labeled artificial implicit chapter relationship data set covering a plurality of fields, and extracts field shared features and field private features by taking the artificial implicit chapter relationship data set as training data. Meanwhile, adversarial training with a noise filtering mechanism is introduced, and noise interference introduced by pseudo labels can be effectively relieved; meanwhile, field generalization training of a hybrid expert mechanism is designed, the adaptability of the system to the unknown target field is enhanced, the recognition effect of the method is obviously better than that of an existing method when the method is applied to the unknown target field, and no manually-labeled implicit discourse relation data is needed during training, so that the recognition efficiency is greatly improved. The method can be applied to most languages and is wider in application range.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and particularly to a discourse relation recognition system. Background Art

[0002] With the continuous acceleration of the digitalization process, a large amount of text data continues to emerge in scenarios such as social media, news reports, and online reviews. Natural Language Processing (NLP) technology plays an increasingly crucial role in understanding and analyzing text information. Among them, discourse relation recognition, as the core task of understanding the structure and meaning of multi-sentence text, has received extensive attention. This task aims to identify the semantic relationships between adjacent or relevant text segments (such as sentences or clauses) in the text, such as causal, parallel, and contrast relationships. According to the presence or absence of explicit connectives, discourse relation recognition is usually divided into explicit discourse relation recognition (EDRR) and implicit discourse relation recognition (IDRR). The former has good model recognition results because the connectives provide direct clues; while the latter, due to the lack of obvious markers, requires the model to comprehensively reason from semantic, context, and even pragmatic levels, facing greater challenges. Although researchers have proposed various neural network models and pre-trained language models in recent years to improve the performance of IDRR, its recognition accuracy is still significantly lower than that of explicit discourse relation recognition, which limits its actual effect in downstream applications. Therefore, improving the performance of implicit discourse relation recognition remains a research hotspot and key problem in the current NLP field.

[0003] At present, implicit discourse relation recognition methods have evolved from traditional approaches relying on manual feature engineering to the stage of using deep learning to automatically learn distributed features. Early research mainly relied on manually constructed features at the syntactic, semantic, and lexical levels, combined with traditional machine learning algorithms such as support vector machines for relation classification. However, such methods have significant limitations in feature extraction and generalization capabilities. With the rapid development of deep learning technology, methods based on neural networks to automatically learn distributed features have gradually become mainstream. Recurrent neural networks (such as LSTM and GRU), convolutional neural networks (CNN), and attention mechanisms have been widely used to learn the semantic representations between arguments; graph neural networks (GNN) have also been introduced to construct complex argument relation graph structures, combined with attention mechanisms to enhance the model's ability to model structured information. In recent years, pre-trained language models (such as BERT, RoBERTa, etc.) have obtained powerful context modeling capabilities through pre-training on large-scale corpora and shown significant advantages after fine-tuning on discourse relation tasks. In addition, emerging technologies such as prompt learning and knowledge distillation have also begun to be used in the IDRR task, which helps to improve the recognition effect of the model.

[0004] However, the existing implicit discourse relation recognition methods still face two main problems: 1) Limited by the limited quantity and relatively single domain of manually annotated training corpora, the model performs poorly in cross-domain applications, especially when applied to unknown domains, the recognition effect drops significantly; 2) There is no manually annotated implicit discourse relation data in most languages, making it difficult for existing methods to be applied to these languages, greatly limiting their application scope. Summary of the Invention

[0005] Therefore, embodiments of the present invention propose a discourse relation recognition system to improve the recognition effect for unknown domains and expand the language application scope.

[0006] According to an embodiment of the present invention, a discourse relation recognition system includes an automatic annotation sub-module for discourse relation data, an implicit discourse relation recognition sub-module that fuses domain-shared and private features, an adversarial training sub-module with a noise filtering mechanism, and a domain generalization training sub-module based on a mixture of experts mechanism; In the automatic annotation sub-module for discourse relation data, explicit discourse relation instances in K domains are extracted based on heuristic rules to obtain K explicit discourse relation data sets. Any explicit discourse relation data set is divided into instances containing non-ambiguous connectives and instances containing ambiguous connectives. The instances containing non-ambiguous connectives and the instances containing ambiguous connectives are automatically annotated respectively, and then the automatic annotation results are merged to obtain an automatically annotated artificial implicit discourse relation data set covering K domains; In the implicit discourse relation recognition sub-module for sharing and private features in the fusion domain, based on the domain-shared feature extraction network, the domain-shared features of each instance in the artificial implicit discourse relation dataset are extracted, and based on the domain-private feature extraction network, the domain-private features of each instance in the artificial implicit discourse relation dataset are extracted. Then, based on the domain-shared features and domain-private features of each instance, the classification layer outputs the prediction results of implicit discourse relation recognition; In the adversarial training sub-module with a noise filtering mechanism, based on a domain discriminator shared by all experts, the predicted probability distribution of the training instances on K domains is predicted, and the weights of the training instances are dynamically calculated. Then, the cost function for the adversarial training stage with a noise filtering mechanism is constructed L 1, and the cost function L 1 is minimized until convergence; In the domain generalization training sub-module based on the mixture of experts mechanism, the domain experts and non-domain expert groups are defined, and then the domain experts and non-domain expert groups are aligned to construct the cost function L 21 , and alignment is performed among the non-domain expert groups to construct the cost function L 22 , and the cost functions L 21 and L 22 are minimized successively until convergence.

[0007] According to the discourse relation recognition system of the embodiments of the present invention, based on the explicit discourse relation dataset, an artificial implicit discourse relation dataset covering multiple domains and automatically labeled is constructed, and using this artificial implicit discourse relation dataset as training data, the domain-shared features and domain-private features are extracted, so as to output the prediction results of implicit discourse relation recognition. At the same time, the present invention introduces adversarial training with a noise filtering mechanism, which can effectively alleviate the noise interference introduced by pseudo-labels; at the same time, the present invention designs domain generalization training with a mixture of experts mechanism, which enhances the adaptability of the system to unknown target domains, making the recognition effect of the present invention significantly better than existing methods when applied to unknown target domains, and the present invention does not require any manually labeled implicit discourse relation data during training, can be applied to most languages, and has a wider application range. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The above and / or additional aspects and advantages of the embodiments of the present invention will become apparent and be readily understood from the description of the embodiments in conjunction with the following drawings, where: Figure 1 is a schematic structural diagram of a discourse relation recognition system according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0009] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.

[0010] Please refer to Figure 1 , an embodiment of the present invention provides a discourse relation recognition system, including an automatic annotation sub-module for discourse relation data, an implicit discourse relation recognition sub-module that integrates domain-shared and private features, an adversarial training sub-module with a noise filtering mechanism, and a domain generalization training sub-module based on a mixture of experts mechanism.

[0011] In the automatic annotation sub-module for discourse relation data, explicit discourse relation instances in K domains are extracted based on heuristic rules to obtain K explicit discourse relation data sets. Any explicit discourse relation data set is divided into instances containing non-ambiguous connectives and instances containing ambiguous connectives, and the instances containing non-ambiguous connectives and the instances containing ambiguous connectives are automatically annotated respectively, and then the automatic annotation results are merged to obtain an automatically annotated artificial implicit discourse relation data set covering K domains.

[0012] Among them, in most languages, it is relatively easy to obtain explicit discourse relation data, so a large amount of multi-domain data can be collected. On the one hand, by removing the connectives in the explicit discourse relation instances, artificial implicit discourse relation instances can be obtained; on the other hand, most connectives are not ambiguous. For example, "because" corresponds to "causal relationship", "but" corresponds to "transition relationship", etc.; only a small part of the connectives are ambiguous. For example, "therefore" can correspond to "causal relationship" or "temporal relationship".

[0013] Specifically, in the automatic annotation sub-module for discourse relation data, explicit discourse relation instances in K domains are first extracted based on heuristic rules to obtain K explicit discourse relation data sets , , , are the 1st, the k th, and the K th explicit discourse relation data sets respectively, , and the heuristic rules include an extraction template "Argument 1, Connective + Argument 2", a constraint rule "each argument must contain at least one verb", etc. According to whether the conjunctions in the examples are ambiguous, it is divided into an example set containing non-ambiguous conjunctions and an example set containing ambiguous conjunctions ; For any example in, first remove the conjunction. Since these conjunctions are not ambiguous, they can be directly mapped to the corresponding discourse relation category to obtain the first artificially labeled implicit discourse relation dataset ; For any example in, first remove the conjunction. Since these conjunctions are ambiguous, external verification is needed to determine which discourse relation to map to. Specifically, use two pre-trained models with different properties (e.g., RoBERTa and GPT) to predict the conjunctions suitable for connecting two arguments. If the two predicted conjunctions are different from the original conjunction but can both represent the same discourse relation, retain the example and automatically label it with the corresponding discourse relation; if the two predicted conjunctions are the same as the original conjunction, or the two predicted conjunctions are different from the original conjunction but cannot represent the same discourse relation, discard the example. Finally, obtain the second artificially labeled implicit discourse relation dataset ; Merge and the automatically labeled examples in to obtain the k automatically labeled artificial implicit discourse relation dataset for the th domain, and then obtain the automatically labeled artificial implicit discourse relation dataset covering K domains , which are the automatically labeled artificial implicit discourse relation datasets for the 1st and K th domains respectively. Among them, the th example in is , and the automatically labeled artificial implicit discourse relation of is

[0014] In the implicit discourse relation recognition sub-module that fuses domain-shared and private features, based on the domain-shared feature extraction network, the domain-shared features of each example in the artificial implicit discourse relation dataset are extracted, and based on the domain-private feature extraction network, the domain-private features of each example in the artificial implicit discourse relation dataset are extracted. Then, based on the domain-shared features and domain-private features of each example, the classification layer outputs the prediction result of implicit discourse relation recognition.

[0015] Among them, existing methods usually only apply domain-shared features to the unknown domain, ignoring the role of domain-private features. To better utilize domain-shared features and domain-private features, in the present invention, the domain-shared features and domain-private features are decoupled, and an implicit discourse relation recognition sub-module that fuses domain-shared and private features is constructed.

[0016] Specifically, in the implicit discourse relation recognition sub-module that fuses domain-shared and private features, first construct K implicit discourse relation recognition models, and denote them as K experts. Each expert consists of a domain-shared feature extraction network, a domain-private feature extraction network, and a classification layer. The domain-shared feature extraction networks of each expert adopt the same model parameters, while the domain-private feature extraction networks have their own independent model parameters.

[0017] Based on the domain-shared feature extraction network, extract the domain-shared features of each instance in the artificial implicit discourse relation dataset, and based on the domain-private feature extraction network, extract the domain-private features of each instance in the artificial implicit discourse relation dataset. The expressions are: ; ; Among them, is the th instance in the domain-shared feature extraction network shared by all experts, is k the domain-private feature extraction network of the th expert, is the domain-shared feature of is the domain-private feature of

[0018] It should be noted that in specific implementation, when instance is input, it usually needs to be specially processed according to the pre-trained language model adopted. For example, placeholders [CLS] and [SEP] are added at the beginning of the instance and between the two arguments respectively.

[0019] Based on the domain-shared features and domain-private features of each instance, the classification layer outputs the prediction result of implicit discourse relation recognition. The expression is: ; ; Among them, The predicted probability distribution over all discourse relation categories is the final semantic vector representation of is the normalization function, and is the parameter that the classification layer of the k th expert needs to learn, and

[0020] In the adversarial training sub-module with a noise filtering mechanism, based on a domain discriminator layer shared by all experts, the predicted probability distribution of training instances over K domains is obtained, and the weights of training instances are calculated dynamically, and then the cost function of the adversarial training phase with a noise filtering mechanism is constructed L 1, minimize the cost function L 1 until convergence.

[0021] Among them, when training multiple experts based on an automatically annotated multi-domain dataset for implicit discourse relation recognition in unknown domains, there are still problems of data noise, domain mismatch, and explicit / implicit data mismatch. To address the above problems, the adversarial training with a noise filtering mechanism proposed by the present invention is introduced below, and the main purposes are: 1) Filter out the noise instances existing in the automatically annotated data as much as possible, 2) Better learn the domain-shared features and their respective domain-private features of multiple domains to alleviate the domain mismatch problem.

[0022] To more effectively learn the domain-shared features across domains, the method of adversarial training is adopted. A domain discriminator layer shared by all experts is introduced to predict the domain to which the input instance belongs. If the discriminator layer is difficult to accurately identify the domain to which the instance belongs, it means that the extracted domain-shared features have strong domain irrelevance, indicating that the domain-shared features are learned more fully.

[0023] Specifically, in the adversarial training sub-module with a noise filtering mechanism, based on a domain discriminator layer shared by all experts, the predicted probability distribution of training instances over K domains is obtained, and the expression is: ; where is the predicted probability distribution over all K domains, and are the parameters that the domain discriminator layer needs to learn. The more uniform the probability value distribution in

[0024] To filter out as many noisy instances as possible existing in the automatically labeled data, the present invention preferentially selects samples with lower cross-entropy loss and higher adversarial training loss to participate in model training during the training process. A lower cross-entropy loss indicates that the automatic labeling result of this instance has a higher credibility; a higher adversarial training loss indicates that the domain features of this instance are not obvious, which is more conducive to learning domain-shared features that are irrelevant to the domain. In each round of training, the weights of the training instances are dynamically calculated, where a weight of 1 means that this instance is selected to participate in training, and a weight of 0 means that this instance is ignored. By selectively ignoring a certain proportion of samples, the purpose of filtering out noisy training instances can be achieved.

[0025] Specifically, during the process of dynamically calculating the weights of the training instances, the following formula is satisfied: ; where is the weight of , is 's automatically labeled artificial implicit discourse relation, is the cross-entropy cost function for classification, is the function for calculating entropy, and are set hyperparameters. The larger the entropy, the more difficult it is for the model to distinguish the domain of this instance, and the corresponding adversarial loss is also higher.

[0026] Based on the weights of the training instances calculated dynamically, train experts simultaneously on the automatically labeled artificial implicit discourse relation dataset K , and define the cost function L 1 of the adversarial training stage with a noise filtering mechanism as: ; where is the total number of training instances in , is the transpose operation of the vector. The closer the value of L 1 is to 0, the weaker the correlation between the domain-shared features and the domain-private features learned by the model, which helps to effectively decouple the two. The goal of the adversarial training with a noise filtering mechanism is to minimize L 1 until convergence.

[0027] In the domain generalization training sub-module based on the mixture of experts mechanism, define the domain expert group and the non-domain expert group, and then align the domain expert group and the non-domain expert group to construct the cost function L 21 , and align between the non-domain expert groups to construct the cost function L 22, minimize the cost function successively L 21 and L 22 , until convergence.

[0028] Among them, in most current studies, usually only the domain-shared features are applied to the unknown target domain, while the role of domain-private features is ignored. However, when the differences between different domains are large, simply relying on domain-shared features often fails to achieve ideal results. Therefore, in the domain generalization training stage based on the mixture of experts mechanism, the present invention further optimizes the domain-private feature extraction network, so that when integrating multiple experts to deal with the unknown target domain, the domain-private features can be more fully utilized to achieve complementary enhancement. This training stage includes two steps: The first step: alignment between the domain expert and the non-domain expert group, which is used to further strengthen the domain generalization ability; The second step: alignment between the non-domain expert groups, which is used to alleviate the explicit / implicit data mismatch problem.

[0029] In the first-step training, any domain is used as the current domain, and the th expert is the corresponding domain expert, and the remaining K -1 experts constitute the non-domain expert group. For the training instances in any dataset, the goal of this step of training is to make the prediction results of the corresponding domain expert as consistent as possible with the prediction results of the non-domain expert group. Therefore, the cost function L 21 is defined as: ; ; where represents the Kullback-Leibler distance calculation function, is the prediction result of the non-domain expert group, and is the prediction result of the th expert in the non-domain expert group.

[0030] The second-step training is used to alleviate the explicit / implicit data mismatch problem. First, collect the unlabeled implicit discourse relation instances of multiple domains (different from the K domains in the automatic annotation of discourse relations data), and construct the dataset ; Then, randomly divide all the non-domain experts in the non-domain expert group into two non-domain expert groups and , and align the prediction results of these two non-domain expert groups. Therefore, the cost function L 22 is defined as: ; ; ; Among them, is a dataset constructed from unannotated implicit discourse relation instances, is the total number of instances in is the th instance, is the prediction result of the first non-domain expert group for . is the prediction result of the second non-domain expert for . is the number of non-domain experts in the first non-domain expert group , is the number of non-domain experts in the second non-domain expert , is the prediction result of the th non-domain expert in the first non-domain expert group for . is the prediction result of the th non-domain expert in the second non-domain expert group for .

[0031] It should be noted that in the domain generalization training stage based on the mixture-of-experts mechanism, the parameters of the domain-shared feature extraction network are fixed, and only the parameters of the domain-private feature extraction networks of all expert models and the classification layers are optimized. In the domain generalization training based on the mixture-of-experts mechanism, the cost functions L 21 and L 22 are minimized successively until convergence, that is, the final trained K experts are obtained.

[0032] In specific applications, for the input instance in the unknown target domain, the combined K experts output the prediction result of implicit discourse relation recognition as : ; Among them, is the prediction result of the th expert for the input instance .

[0033] In summary, according to the discourse relation recognition system of this embodiment, based on the explicit discourse relation dataset, an automatically annotated artificial implicit discourse relation dataset covering multiple domains is constructed, and this artificial implicit discourse relation dataset is used as the training data to extract domain-shared features and domain-private features, so as to output the prediction result of implicit discourse relation recognition. At the same time, the present invention introduces adversarial training with a noise filtering mechanism, which can effectively alleviate the noise interference introduced by pseudo-labels. At the same time, the present invention designs domain generalization training with a mixture of experts mechanism, which enhances the adaptability of the system to unknown target domains, making the recognition effect of the present invention significantly better than existing methods when applied to unknown target domains. Moreover, the present invention does not require any manually annotated implicit discourse relation data during training, can be applied to most languages, and has a wider application scope.

[0034] In the description of this specification, the description referring to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0035] Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and purposes of the present invention. The scope of the present invention is defined by the claims and their equivalents.

Claims

1. A discourse relation recognition system, characterized in that, It includes an automatic annotation sub-module for discourse relation data, an implicit discourse relation recognition sub-module that integrates domain-shared and private features, an adversarial training sub-module with a noise filtering mechanism, and a domain generalization training sub-module based on a mixture of experts mechanism; In the automatic annotation sub-module of discourse relation data, explicit discourse relation instances in K domains are extracted based on heuristic rules to obtain K explicit discourse relation datasets. Any explicit discourse relation dataset is divided into instances containing non-ambiguous connectives and instances containing ambiguous connectives. Automatic annotation is performed on the instances containing non-ambiguous connectives and the instances containing ambiguous connectives respectively, and then the automatic annotation results are merged to obtain an automatically annotated artificial implicit discourse relation dataset covering K domains; In the implicit discourse relation recognition sub-module that integrates domain-shared and private features, the domain-shared features of each instance in the artificial implicit discourse relation dataset are extracted based on the domain-shared feature extraction network, and the domain-private features of each instance in the artificial implicit discourse relation dataset are extracted based on the domain-private feature extraction network. Then, based on the domain-shared and private features of each instance, the classification layer outputs the prediction results of implicit discourse relation recognition; In the adversarial training sub-module with a noise filtering mechanism, based on a domain discriminator layer shared by all experts, the predicted probability distribution of training instances on K the number of domains is predicted, and the weights of training instances are dynamically calculated. Then, the cost function of the adversarial training phase with a noise filtering mechanism is constructed L 1, and the cost function L 1 is minimized until convergence; In the domain generalization training sub-module based on the mixture of experts mechanism, domain experts and non-domain expert groups are defined, and then the domain experts and non-domain expert groups are aligned to construct a cost function L 21 , and alignment is performed among the non-domain expert groups to construct a cost function L 22 . The cost functions L 21 and L 22 are minimized successively until convergence 2. The discourse relationship recognition system according to claim 1, characterized in that, In the automatic annotation sub-module of discourse relation data, first extract explicit discourse relation instances in K domains to obtain K explicit discourse relation datasets , , , which are the 1st, k th, and K th explicit discourse relation datasets respectively, ; According to whether the conjunction in the example is ambiguous, divide it into an instance set containing non-ambiguous conjunctions , and an instance set containing ambiguous conjunctions ; For any instance in , remove the connectives first and then directly map to the corresponding discourse relation category to obtain the automatically annotated first artificial implicit discourse relation dataset ; For any instance in , first remove the connective, and then use two pre-trained models with different natures to predict the connectives suitable for connecting two arguments. If the two predicted connectives are different from the original connective but can both represent the same discourse relation, retain the instance and automatically label it with the corresponding discourse relation; if the two predicted connectives are the same as the original connective, or the two predicted connectives are different from the original connective but cannot represent the same discourse relation, discard the instance. Finally, obtain the second artificial implicit discourse relation dataset with automatic labeling ; Merge and the automatically annotated instances in, to obtain the k automatically annotated artificial implicit discourse relation dataset for the th domain, and further obtain the K automatically annotated artificial implicit discourse relation dataset covering domains, which are respectively the automatically annotated artificial implicit discourse relation datasets for the 1st and the K th domains.

3. The discourse relation recognition system according to claim 2, characterized in that In the sub-module for identifying implicit discourse relations of shared and private features in the fusion domain, first construct K implicit discourse relation recognition models, denoted as K experts. Then, based on the domain-shared feature extraction network, extract the domain-shared features of each instance in the artificial implicit discourse relation dataset, and based on the domain-private feature extraction network, extract the domain-private features of each instance in the artificial implicit discourse relation dataset. The expression is: ； ； Among them, is the th instance in the domain - shared feature extraction network shared by all experts, is k the domain - private feature extraction network of the th expert, is the domain - shared feature of is the domain - private feature of 4. The discourse relationship recognition system according to claim 3, wherein In the implicit discourse relation recognition sub-module that integrates domain-shared and private features, based on the domain-shared and private features of each instance, the classification layer outputs the prediction results of implicit discourse relation recognition, and the expression is: ； ； Among them, is the predicted probability distribution over all discourse relation categories, is the final semantic vector representation of is the normalization function, and are the parameters to be learned by the classification layer of the k -th expert, is the vector concatenation operation.

5. The discourse relation recognition system according to claim 4, wherein In the adversarial training sub-module with a noise filtering mechanism, based on a domain discriminator layer shared by all experts, the predicted probability distribution of the training instances on K domains is obtained, and the expression is: ； Among them, is the predicted probability distribution over all K domains, and are the parameters to be learned by the domain discriminator layer.

6. The discourse relationship recognition system according to claim 5, wherein In the adversarial training sub-module with a noise filtering mechanism, in the process of dynamically calculating the weights of training instances, the following formula is satisfied: ； Among them, is 's weight, is 's automatically labeled artificial implicit discourse relation, is the cross-entropy cost function for classification, is the function for calculating entropy, and are the set hyperparameters.

7. The discourse relationship recognition system according to claim 6, characterized in that, Cost function in the adversarial training phase with noise filtering mechanism L The expression of 1 is as follows: ； Among them, is the total number of training instances in the transpose operation of the vector.

8. The discourse relationship recognition system according to claim 7, wherein Cost function L 21 The expression of which is: ； ； Among them, represents the Kullback-Leibler distance calculation function, is the prediction result of the non-domain expert group, is the th expert's prediction result in the non-domain expert group.

9. The discourse relation recognition system according to claim 8, characterized in that, Cost function L 22 The expression is as follows: ； ； ； Among them, is a dataset constructed from unannotated implicit discourse relation instances, is the total number of instances in is the th instance in is the prediction result of the first non-domain expert group on ; is the prediction result of the second non-domain expert on ; is the number of non-domain experts in the first non-domain expert group ; is the number of non-domain experts in the second non-domain expert ; is the prediction result of the th non-domain expert in the first non-domain expert group on ; is the prediction result of the th non-domain expert in the second non-domain expert group on .

Images