Consistency evaluation method and system combining unified modeling and hybrid expert network

Abstract

This invention belongs to the field of text consistency evaluation and discloses a consistency evaluation method and system that combines unified modeling and hybrid expert networks. The method involves constructing a standard corpus; generating an initial vector sequence by manipulating the standard corpus samples; extracting features from the initial vector sequence and optimizing its positional information to obtain the hidden layer states; calculating the activation probabilities of each expert network; selecting the K experts with the highest probabilities; having each activated expert output its own feature vector; and then weighting and summing the outputs according to the weights assigned by the gating network to obtain a global feature vector. Consistency evaluation is performed based on the global feature vector, and the classification result is output. The model is then trained, and consistency evaluation is performed using the trained model. This invention can standardize and model data from different sources and in different formats, possessing strong generalization ability and versatility, and can adapt to various language processing tasks.

Description

Technical Field

[0001] This invention relates to the field of text consistency assessment technology, and in particular to a consistency assessment method and system that combines unified modeling and hybrid expert networks. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] From early template- or rule-based text generation systems to today's large language models (LLMs) based on massive amounts of data and parameters, the field of Natural Language Generation (NLG) has undergone significant development and progress. Current models can generate fluent and coherent text for various generation tasks. However, the generated text often contains factual consistency errors. Therefore, researchers aim to detect these illusions by predicting whether the generated text is factually consistent with the base text; this task is known as text consistency assessment.

[0004] Text consistency assessment not only effectively identifies and corrects factual errors in generated text, but also provides crucial assurance for the reliability and accuracy of model output. Traditional research typically employs manual evaluation to judge the accuracy and consistency of generated text, but this method is inefficient and highly subjective. With the development of natural language understanding tasks, researchers have begun to explore automated consistency assessment methods.

[0005] Traditional evaluation metrics, initially used in machine translation, have been widely applied to quality assessment in text generation tasks. These methods measure text quality by calculating the degree of lexical or phrase overlap between the generated and reference texts. For example, BLEU assesses text consistency by measuring the N-Gram overlap rate between the generated and reference texts. This method has limited ability to assess semantic consistency and is susceptible to differences in lexical representation. ROUGE is primarily used for summarization tasks, calculating recall and precision between the generated and reference texts. Similar to BLEU, this method focuses mainly on surface-level lexical matching and cannot identify semantic errors or factual deviations. BERTScore, an evaluation metric based on BERT representations, assesses generated text quality by calculating the word vector similarity between the generated and reference texts. This method captures semantic similarity and is more robust to lexical diversity, but it is only applicable to short text evaluation and depends on the quality of a specific pre-trained model. BLEURT is a combination of BLEU and BERT, predicting quality scores for generated text by fine-tuning the BERT model, supporting fine-grained evaluation. This method goes beyond surface matching, assessing the semantics and fluency of generated text, but its training relies on large-scale, high-quality labeled data, potentially limiting its generalization ability.

[0006] Natural Language Inference (NLI) and text consistency assessment tasks are naturally well-suited. NLI assesses the semantic consistency between generated and source texts by judging entailment, contradiction, and neutrality relationships between texts. In recent years, many NLI-based text consistency assessment methods have emerged. For example, the SummaC method focuses on assessing factual consistency in summarizing tasks, addressing the input granularity mismatch problem of previous NLI models when handling long documents. It divides the document into sentence units and calculates the score for each pair of sentences individually.

[0007] Question-Answering (QA) consistency assessment methods assume that if the generated text is consistent with the source text, it should be able to correctly answer questions designed based on the source text. This method assesses text consistency by extracting question-answer pairs and verifying their correctness. FEQA is a QA consistency assessment method that extracts key information points from the generated summary and verifies the correctness of this information in the original text. This method can effectively identify key factual errors in the generated text.

[0008] With the rise of pre-trained models such as BERT and DEBERTA, researchers have begun to leverage their powerful semantic modeling capabilities to improve consistency evaluation. Some researchers have used pre-trained models for feature representation and analyzed the performance of different pre-trained models on the ParaRel dataset for consistency tasks, finding that the models' consistency performance was poor, but their evaluation ability could be significantly improved through optimization. The WeCheck model employs a weakly supervised learning method, extracting consistency labels from the actual output of generative models to construct a lightweight consistency evaluation framework. This method uses the output of generative models as training samples, closely resembling real-world tasks, but the quality of its weak labels is not stable enough, affecting the model's generalization ability.

[0009] In recent years, some studies have attempted to achieve a unified evaluation of consistency issues in multi-task modeling. TRUE is a comprehensive fact-consistency evaluation benchmark and framework that compares methods based on NLI, QA, and other models, finding their performance complementary across different tasks. Therefore, TRUE combines state-of-the-art models of NLI and QA to obtain a more accurate consistency score. Its drawback lies in its high computational resource requirements for large-scale tasks.

[0010] In summary, research on text consistency assessment has gradually evolved from traditional assessment metrics to the exploration of unified modeling frameworks. However, the multi-task balancing ability, long text processing capabilities, and low-resource scenario support capabilities of models remain important research directions for the future. Compared to assessment methods relying on manual methods, using model-based methods for text consistency assessment can significantly improve assessment efficiency and save costs. At the same time, compared to statistical assessment methods, such as N-gram-based methods, model-based methods can more comprehensively and accurately understand language semantics, thus demonstrating more comprehensive and precise assessment capabilities. Current model methods have achieved considerable results on specific evaluation sets; however, they are often designed only for a particular task. For example, SummaC is only for consistency assessment in the summarization task and cannot comprehensively cover consistency issues in various natural language tasks; its generalization ability needs improvement. Summary of the Invention

[0011] To address the aforementioned issues, this invention proposes a consensus evaluation method and system that combines unified modeling with hybrid expert networks. It integrates the powerful language representation capabilities of DeBERTa-V3 with the high computational efficiency and flexibility of hybrid expert networks, enabling standardized processing and modeling of data from different sources and in different formats. It possesses strong generalization capabilities and versatility, and can adapt to various language processing tasks.

[0012] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a consensus evaluation method combining unified modeling and hybrid expert networks, comprising the following steps: Model samples from different task modes into a standard corpus with a unified pattern; Word embedding and positional encoding operations are performed on standard corpus samples to generate an initial vector sequence that integrates semantic and positional information. Feature extraction is performed on the initial vector sequence, and positional information is optimized using a relative position offset mechanism to obtain the hidden layer state. Based on the hidden layer state, the activation probability of each expert network is calculated using a gating network. The K expert networks with the highest activation probabilities are selected, and the activated expert networks process the hidden layer state respectively and output their respective feature vectors. Then, the outputs of the expert networks are weighted and summed according to the weights assigned by the gating network to obtain the integrated global feature vector. A consistency assessment is performed based on the integrated global feature vector, and the classification result of the consistency assessment is output. Define a loss function, train the consistency evaluation model to obtain a trained consistency evaluation model, and use the trained consistency evaluation model to perform consistency evaluation.

[0013] As an alternative implementation, samples from different task modes are modeled into a standard corpus with a unified pattern, specifically: The first mapping function extracts the "basic information" and "information to be evaluated" from the original task samples into "source text" and "target text" respectively. The second mapping function maps the labels of the original tasks into binary consistency labels. Through these two mapping functions, the original sample data of different tasks are standardized into a unified format of "source text-target text-consistency label".

[0014] As an alternative implementation, both the first mapping function and the second mapping function are structured transformation rules for different tasks, wherein the first mapping function is an information extraction rule and the second mapping function is a label mapping rule.

[0015] As an alternative implementation, the hidden layer state contains deep semantic features of "source text-target text" pairs.

[0016] As an alternative implementation method, different expert networks focus on different semantic patterns.

[0017] As an alternative implementation, Focal Loss can be used as the loss function. Focal Loss introduces a loss function into the loss function. Adjustment factors can solve the class imbalance problem.

[0018] Secondly, the present invention provides a consensus evaluation system that combines unified modeling and hybrid expert networks, comprising: The standard corpus construction module is configured to model samples from different task modes into a standard corpus with a unified pattern. The data processing module is configured to: perform word embedding and position encoding operations on standard corpus samples to generate an initial vector sequence that integrates semantic and positional information; extract features from the initial vector sequence and optimize the positional information using a relative position offset mechanism to obtain the hidden layer state; calculate the activation probability of each expert network using a gating network based on the hidden layer state; select the K expert networks with the highest activation probabilities; have the activated expert networks process the hidden layer state respectively and output their respective feature vectors; and then perform a weighted summation of the outputs of the expert networks according to the weights assigned by the gating network to obtain the integrated global feature vector. The consistency assessment module is configured to perform consistency assessment based on the integrated global feature vector and output the classification result of the consistency assessment. The model training module is configured to: define a loss function, train the consistency evaluation model, obtain the trained consistency evaluation model, and use the trained consistency evaluation model to perform consistency evaluation.

[0019] Thirdly, the present invention provides an electronic device including a memory and a processor, and computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the method described in the first aspect.

[0020] Fourthly, the present invention provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the method described in the first aspect.

[0021] Fifthly, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect.

[0022] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention proposes a consensus assessment method combining unified modeling and hybrid expert networks. It leverages the powerful language representation capabilities of DeBERTa-V3 and the high computational efficiency and flexibility of hybrid expert networks to construct the UDeMoE model. This model can standardize and model data from different sources and formats, overcoming the model incompatibility issues caused by diverse data sources in previous consensus assessments. It provides a general framework for subsequent model construction. Not only does it perform excellently on consensus assessment tasks, but it also demonstrates good application results on hallucination detection datasets. This cross-task application capability fully demonstrates the model's strong generalization ability and versatility, enabling it to adapt to different language processing tasks.

[0023] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0024] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0025] Figure 1 This is the method for constructing the unified mode corpus of the present invention; Figure 2 This is a framework diagram of the consistency evaluation method combining unified modeling and hybrid expert networks provided in Embodiment 1 of the present invention. Detailed Implementation

[0026] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0027] It should be noted that the following detailed description is exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0028] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments of the invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. Furthermore, it should be understood that the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but includes other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0030] Example 1 like Figure 2 As shown, this embodiment provides a consensus evaluation method that combines unified modeling and hybrid expert networks, including the following steps: Model samples from different task modes into a standard corpus with a unified pattern; Word embedding and positional encoding operations are performed on standard corpus samples to generate an initial vector sequence that integrates semantic and positional information. Feature extraction is performed on the initial vector sequence, and positional information is optimized using a relative position offset mechanism to obtain the hidden layer state. Based on the hidden layer state, the activation probability of each expert network is calculated using a gating network. The K expert networks with the highest activation probabilities are selected, and the activated expert networks process the hidden layer state respectively and output their respective feature vectors. Then, the outputs of the expert networks are weighted and summed according to the weights assigned by the gating network to obtain the integrated global feature vector. A consistency assessment is performed based on the integrated global feature vector, and the classification result of the consistency assessment is output. Define a loss function, train the consistency evaluation model to obtain a trained consistency evaluation model, and use the trained consistency evaluation model to perform consistency evaluation.

[0031] The present invention will now be described in detail.

[0032] In natural language generation tasks, generated text often contains factual consistency errors. Existing text consistency evaluation methods lack generalization ability and have poor multi-task compatibility. To address this, this paper proposes a consistency evaluation model, UDeMoE, which combines unified modeling and hybrid expert networks (MoE), to construct a general framework for handling text consistency evaluation problems in multi-task scenarios. Specifically, by using a unified modeling paradigm, seven datasets from five types of tasks, including natural language inference and question answering systems, are transformed into "source text-target text" pairs, constructing a standardized corpus containing 3.252 million samples to solve the data format incompatibility problem. The model uses DeBERTa-V3 as a base to capture the deep semantics of the text, followed by a MoE network to achieve multi-expert dynamic routing to optimize processing efficiency, and Focal Loss is used to address the data imbalance problem.

[0033] (1) Construct a standard corpus with a unified model To address the consistency evaluation challenges across multiple natural language processing tasks using a unified evaluation model, this model needs to possess a general evaluation method. Summarizing the commonalities in the definition of consistency evaluation, it can be found that the evaluation process involves two texts: the source text and the target text, and it is necessary to evaluate the consistency of the target text compared to the source text. The idea behind this invention's definition of consistency evaluation is: calculate the overlap between the target text and the source text; if all information in the target text is contained within the source text and does not contradict it, then the target text is considered consistent with the source text. This invention models this unified consistency evaluation paradigm as a function, as shown in equation (1). The original samples are text information from various tasks, denoted as . The original labels corresponding to these samples are ,Will Through function Convert to source and target text, pass Convert to consistent labels: (1) Among them, the source text list is This indicates that the target text list is... express, This indicates a list of evaluation results.

[0034] This unified evaluation paradigm accepts various language forms for different tasks, such as text summarization, open-domain dialogue, and machine translation. The paradigm converts samples from different tasks into source and target texts, and combines the original labels with textual information to convert them into consistent labels, thus achieving a unified consistency evaluation. The evaluation result is a binary discrete value, either "consistent" or "inconsistent".

[0035] Through function The "basic information" and "information to be evaluated" in the original task data are extracted as "source text" and "target text," respectively; this is done using a function. Map the original task labels (such as "implication / contradiction" for NLI, "correct / incorrect" for QA, etc.) to binary consistency labels ("consistent" or "inconsistent").

[0036] function and It is not an abstract mathematical function, but a structured transformation rule for different tasks: It is an "information extraction rule": based on the characteristics of the task, locate the "factual evidence text (source text)" and the "text to be evaluated (target text)" from the raw data; It is a “label mapping rule”: converting task-specific labels into “consistent / inconsistent” binary labels to uniformly evaluate the target.

[0037] Through these two mapping functions, the raw data of different tasks (NLI, QA, summarization, etc.) are standardized into a unified format of "source text-target text-consistency label", which provides a foundation for training a consistency evaluation model across tasks.

[0038] function and Specific steps (with task example): 1. Function Extract source and target text.

[0039] Its function is to separate the "base text (source text)" and the "text to be evaluated (target text)" from the raw task data. The specific operation varies from task to task. Natural Language Inference (NLI): The original data contains "premise" and "hypothesis". F1 uses "premise" as source text (factual basis) and "hypothesis" as target text (reasoning conclusion to be evaluated).

[0040] Example: Premise "Conceptually, cream skimming has two..." → Source text; Assumption "Product and geography are what..." → Target text Figure 1 Example). Question Answering System (QA): The raw data contains “context (article)”, “question” and “answer”. F1 uses “context (article)” as the source text (factual basis) and “question” and “answer” as the target text (whether the answer to be evaluated fits the context).

[0041] Example: Article "The rain had continued for a week..." → Source text; Answer "Protecting cows from being drowned" → Target text ( Figure 1 Example).

[0042] Facts prove: The raw data contains "evidence" and "statement". F1 uses "evidence" as the source text (factual basis) and "statement" as the target text (whether the statement to be evaluated conforms to the evidence).

[0043] Example: Evidence “Nikolaj Coster-Waldau worked with ..." → Source text; Statement “The Fox Broadcasting Company often ..." → Target text Figure 1 Example).

[0044] Rewrite recognition: The original data contains "two sentences (sentence 1 / sentence 2)". f1 uses "sentence 1" as the source text (baseline text) and "sentence 2" as the target text (whether the sentence to be evaluated is semantically equivalent to sentence 1).

[0045] Example: Sentence 1 "Im Oktober 1560 traf er sich in Paris ..." → source text; Sentence 2 "Im Oktober 1560 traf er sich heimlich ..." → target text ( Figure 1 Example).

[0046] Text summary: The raw data contains "document" and "summary". F1 uses "document" as the source text (factual basis) and "summary" as the target text (whether the summary to be evaluated is faithful to the document).

[0047] Example: Document "If you're a photographer, keep all ..." → Source text; Summary "Keep related supplies in the same ..." → Target text Figure 1 Example).

[0048] 2. Functions Mapping consistency label.

[0049] Its function is to convert the original task labels (mostly task-specific labels) into "consistent / inconsistent" binary labels. The core logic is: if the target text and the source text are consistent in fact and contain no redundant or irrelevant information, then it is "consistent"; otherwise, it is "inconsistent". The specific mapping rules are as follows: Natural Language Inference (NLI): The original tags were "entailment", "contradiction", and "neutral".

[0050] Mapping rules: "Contradiction" → "Inconsistency"; "Implication" is "consistent" if it means "the target text does not exceed the factual scope of the source text," and "inconsistent" if it means "the target text adds information outside the source text." Figure 1 In the example, the "implication" label is mapped to "inconsistency," presumably because it assumes information not mentioned in the source text; "neutral" → "inconsistency" (because the target text has no clear factual connection with the source text).

[0051] Question Answering System (QA): The original label is either "Correct Answer" or "Incorrect Answer" (depending on whether the context supports it).

[0052] Mapping rules: "Correct answer (consistent with context)" → "Consistent"; "Incorrect answer (contradictory or irrelevant to context)" → "Inconsistent" Figure 1 In the example, the correct answer is mapped to "consistent".

[0053] Facts prove: The original labels were "Supported", "Refuted", and "Not EnoughInfo".

[0054] Mapping rules: "Support" → "Consistent"; "Refute" → "Inconsistent"; "Insufficient information" → "Inconsistent" (due to inability to confirm factual consistency) Figure 1 In the example, "Refuted" is mapped to "inconsistent".

[0055] Rewrite recognition: The original label is either "1 (semantic equivalence)" or "0 (semantic inequivalence)".

[0056] Mapping rules: "1 (correct rewrite, semantically equivalent)" → "consistent"; "0 (incorrect rewrite, semantically inequivalent)" → "inconsistent" Figure 1 In the example, label 0 is mapped to "inconsistency".

[0057] Text summary: The original tags implicitly indicate whether the abstract is faithful to the document (based on human annotation or high-quality abstract standards).

[0058] Mapping rules: "The summary is faithful to the document (no factual errors)" → "Consistent"; "The summary contains errors or irrelevant information" → "Inconsistent". Figure 1 In the example, high-quality summaries are mapped to "consistent".

[0059] (2) Construction of UDeMoE model The unified modeling consistency evaluation model UDeMoE is constructed based on the DeBERTa-V3 model and a hybrid expert network. DeBERTa-V3, as a pre-trained model, contains rich knowledge, and its powerful understanding capabilities enable it to deeply analyze text semantics, accurately grasping both grammatical structure and underlying logic. The MoE network, on the other hand, has excellent processing capabilities for multi-feature, multi-domain datasets, flexibly selecting appropriate processing methods based on data characteristics, greatly improving processing efficiency. This study concatenates the MoE network after the DeBERTa-V3 model; the two work together to enable the evaluation model to both deeply understand and efficiently process complex and diverse texts, thus accurately completing the consistency evaluation task.

[0060] Model framework such as Figure 2 As shown, the input data, after word embedding and positional encoding, is used to extract features through the multi-head attention and feedforward neural network of the DeBERTa-V3 model. Relative positional offsets are used to capture sequence relationships, generating hidden layer states. Subsequently, the states are input into a Top K gating network and distributed to experts according to their activation probabilities. The outputs of each expert are integrated using a weighted average strategy, processed by a classification layer, and then the prediction result is output.

[0061] DeBERTa model: The DeBERTa model is an advanced pre-trained language model that uses the Transformer architecture as its core building block and stacks multiple layers of Transformer structures. Each layer can further encode and decode the input, progressively extracting higher-level semantic features and contextual information.

[0062] The main innovation of the DeBERTa model lies in its decoupled attention mechanism and enhanced mask decoder. Thanks to this, the model is able to capture semantic and positional information in the text more precisely.

[0063] DeBERTa-V3 is an improved version of the DeBERTa series, significantly enhancing its performance in natural language processing tasks through optimizations in model architecture and training methods. Compared to previous versions, DeBERTa-V3 primarily introduces an improved masked language modeling objective and an improved decoupled attention mechanism.

[0064] The improved masked language modeling goal used in DeBERTa-V3 introduces the replacement prediction technique proposed by ELECTRA into model training. Unlike traditional masked language modeling goals, ELECTRA's replacement prediction goal requires the model to distinguish between real words and replacement words, enabling the model to not only generate semantically rich representations but also enhance its understanding of the context through replacement prediction.

[0065] Thanks to the aforementioned improvements, DeBERTa-V3 performs exceptionally well in NLP tasks, particularly in sentence classification, text matching, and question answering. Its superior semantic understanding and syntactic capture capabilities make it suitable for various tasks requiring complex semantic parsing and fine-grained information differentiation, such as text consistency assessment, semantic similarity calculation, and contextual understanding.

[0066] Hybrid expert network: Mixture-of-Experts (MoE) networks are efficient neural network architectures designed to handle complex tasks collaboratively by multiple expert models. They have seen widespread application in recent years with the development of large-scale models, particularly in natural language processing and computer vision. The core idea of ​​MoE networks is to decompose complex tasks into multiple sub-tasks, each handled by an "expert" model. The outputs of these expert models are weighted and combined through a "gating network" to generate the final model output. By activating only a small subset of expert nodes, MoE can achieve high performance on large-scale tasks while reducing computational resource consumption.

[0067] Hybrid expert networks typically consist of three main parts: 1) Experts: The core of the model, each expert is responsible for processing a specific aspect or subtask of the input data. Experts can be simple linear models or complex neural networks; 2) Gating network: Assigns weights to each expert, determining the degree of contribution of each expert to the final output. A gating network is usually a simple neural network whose input is the original input data (or its feature representation), and whose output is a probability distribution representing the importance of each expert; 3) Combination module: Weights and sums the outputs of each expert according to the weights assigned by the gating network to generate the final model output. When the original input data is a vector... When the output of a hybrid expert network is given, it can be expressed as: (2) in, y This is the final output. N It is the number of experts. It is the first i The weight of each expert, It is the first i A network of experts on the input x The processing results. The mathematical description of a gating network is: (3) in, It is the weight matrix of the gated network. This is a noise term used to introduce a certain degree of randomness. Top-K represents the selection of the K experts with the highest probability.

[0068] In this study, the structure of each expert network in MoE is the same as the feedforward neural network structure in Transformer, consisting of two fully connected layers and an activation function, as expressed in the formula: (4) in, It is the weight matrix of the first fully connected layer. It is the bias vector of the first fully connected layer. It is the weight matrix of the second fully connected layer. It is the bias vector of the second fully connected layer, and ReLU is the activation function used to introduce nonlinearity.

[0069] This invention adds a MoE network after the last hidden layer of the DeBERTa-V3 model. Compared to using MoE alone, the UDeMoE architecture can leverage the pre-trained knowledge of DeBERTa-V3 to provide rich semantic information, capturing deep-level language structure, contextual connections, and fine-grained semantic features when processing text. This powerful language understanding capability is indispensable for consistency evaluation tasks. Furthermore, compared to fine-tuning the DeBERTa-V3 model alone, the UDeMoE architecture can utilize the MoE network to further refine the model's processing capabilities, allowing the model to select the most suitable expert based on the different needs of the task. This enables MoE not only to be optimized for different tasks but also to more effectively mine complex patterns and features in the data, especially since consistency evaluation tasks may involve specific patterns or data distributions.

[0070] The specific data processing flow is as follows: The data processing flow of the UDeMoE model starts with standardized inputs, gradually processes data through multiple layers of models, and finally outputs a consistency evaluation result. The input, output, and connection logic of each stage are as follows: The first step is input preprocessing, which converts the "source text-target text" pairs into a format that the model can recognize: the input is the "source text-target text" pairs in the unified corpus. After word embedding (converting words into vectors) and position encoding (marking the position of word sequences), an initial vector sequence that integrates semantic and positional information is generated, which serves as the input to the DeBERTa-V3 model.

[0071] Next is the feature extraction of DeBERTa-V3: After inputting the above initial vector sequence, the model uses a multi-head attention mechanism and a feedforward neural network to deeply encode the text semantics, and combines the "relative position offset" mechanism to optimize the modeling of position information. Finally, it outputs the hidden layer state of the last Transformer, which contains the deep semantic features of the "source text-target text" pair and is the core data connecting the subsequent MoE network.

[0072] The next step is the processing of the Hybrid Expert Network (MoE), which is the key connection between DeBERTa and downstream classification: the input is the hidden layer state output by DeBERTa, which is first processed by the Top K Gated Network to calculate the activation probability of each expert, and the K experts with the highest probabilities are selected; the activated expert networks (each performs targeted processing on the hidden layer state (different experts focus on different semantic patterns)) output their respective feature vectors; then the expert outputs are weighted and summed according to the weights assigned by the gated network to obtain the integrated global feature vector.

[0073] Finally, the classification output is as follows: the input global feature vector integrated by MoE is mapped to a binary probability of "consistent" or "inconsistent" through a linear classification layer, and the final output is the classification result of consistency assessment (0 for "inconsistent", 1 for "consistent").

[0074] Throughout the process, the hidden layer state of DeBERTa acts as a semantic bridge, passing deep features to MoE. MoE then achieves adaptive processing of complex text patterns through dynamic expert selection and weighted integration, ultimately outputting accurate evaluation results through the classification layer.

[0075] (3) Loss function design The training corpus collected in this invention has a significant imbalance in binary classification labels. Of the total 2.47 million training samples, only 423,000 are labeled "consistent".

[0076] Focal Loss is a loss function designed to solve the class imbalance problem. Its calculation formula is as follows: (5) in, This represents the model's predicted probability of the true class. For balancing parameters, This is a control factor (usually set to 2) used to control the attention given to difficult-to-classify samples. When At this time, the loss of easily classified samples will be reduced, causing the model to favor minority class samples.

[0077] Compared to the weighted cross-entropy loss method, which requires manual setting of sample weights, Focal Loss introduces a different approach into its loss function. The adjustment factor automatically reduces the loss for easily classified samples while increasing the loss for difficult-to-classify samples, thus effectively improving the model's focus on the minority class. Furthermore, traditional weighted cross-entropy or oversampling methods often lead to the model overfitting to noisy samples in the minority class, whereas the Focal Loss adjustment factor... It can suppress the loss value of easily classified samples, allowing the model to focus on truly difficult samples, reducing sensitivity to noise, and thus improving the model's generalization performance.

[0078] Experiments show that UDeMoE achieves AUC-ROC scores exceeding 0.9 on six test sets, including SummEval and QAGS, significantly outperforming methods such as BLEURT and TRUE. Ablation experiments and leave-one-variable analysis validate the effectiveness of each component and the improvement in generalization ability through multi-task training. Furthermore, it outperforms baseline methods on the hallucination detection dataset WBGH. This research provides a general solution for cross-task consistency evaluation, contributing to improved reliability of natural language generation systems.

[0079] This invention proposes a consistency evaluation model for UDeMoE. To train a model that can effectively judge the consistency of various natural language text formats, UDeMoE transforms multiple datasets with different format characteristics into a unified consistency evaluation corpus. Simultaneously, using the DeBERTa-V3 model as the base model, a hybrid expert network with multiple expert networks is connected after its last hidden layer, aiming to allow different experts to learn and process text information with different features. To address the problem of imbalanced datasets, Focal Loss is used instead of the cross-entropy loss function. Ablation experiments demonstrate the effectiveness of the DeBERTa-V3 model, the hybrid expert network, and Focal Loss. Leave-one-variable analysis on multiple natural language processing tasks confirms its contribution to the UDeMoE model. Performance comparisons with existing models highlight the advancement of the UDeMoE model. Furthermore, this invention also verifies the crucial role of the consistency evaluation model in hallucination detection.

[0080] Example 2 This embodiment provides a consistency evaluation system that combines unified modeling and hybrid expert networks, including: The standard corpus construction module is configured to model samples from different task modes into a standard corpus with a unified pattern. The data processing module is configured to: perform word embedding and position encoding operations on standard corpus samples to generate an initial vector sequence that integrates semantic and positional information; extract features from the initial vector sequence and optimize the positional information using a relative position offset mechanism to obtain the hidden layer state; calculate the activation probability of each expert network using a gating network based on the hidden layer state; select the K expert networks with the highest activation probabilities; have the activated expert networks process the hidden layer state respectively and output their respective feature vectors; and then perform a weighted summation of the outputs of the expert networks according to the weights assigned by the gating network to obtain the integrated global feature vector. The consistency assessment module is configured to perform consistency assessment based on the integrated global feature vector and output the classification result of the consistency assessment. The model training module is configured to: define a loss function, train the consistency evaluation model, obtain the trained consistency evaluation model, and use the trained consistency evaluation model to perform consistency evaluation.

[0081] It should be noted that the above modules correspond to the steps in Embodiment 1, and the examples and application scenarios implemented by the above modules and their corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the above modules can be executed in a computer system as part of the system.

[0082] In further embodiments, the following is also provided: An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the method described in Embodiment 1. For brevity, further details are omitted here.

[0083] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0084] A computer-readable storage medium for storing computer instructions that, when executed by a processor, perform the method of Embodiment 1.

[0085] The method in Example 1 can be directly executed by a hardware processor, or it can be executed by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.

[0086] A computer program product includes a computer program that, when executed by a processor, implements the method in Embodiment 1.

[0087] The present invention also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions included in program modules, which execute in a device on a target real or virtual processor to perform the processes / methods described above. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc., that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules can be combined or divided among program modules as needed. The machine-executable instructions for the program modules can execute within a local or distributed device. In a distributed device, the program modules can reside in both local and remote storage media.

[0088] The computer program code used to implement the methods of the present invention may be written in one or more programming languages. This computer program code may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the computer or other programmable data processing device, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a computer, partially on a computer, as a stand-alone software package, partially on a computer and partially on a remote computer, or entirely on a remote computer or server.

[0089] In the context of this invention, computer program code or related data may be carried by any suitable carrier to enable a device, apparatus, or processor to perform the various processes and operations described above. Examples of carriers include signals, computer-readable media, and the like. Examples of signals may include electrical, optical, radio, sound, or other forms of propagation signals, such as carrier waves, infrared signals, etc.

[0090] Those skilled in the art will recognize that the units and algorithm steps described in conjunction with the embodiments herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0091] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method of consistency evaluation that combines unified modeling and hybrid expert networks, characterized in that, Includes the following steps: Model samples from different task modes into a standard corpus with a unified pattern; Word embedding and positional encoding operations are performed on standard corpus samples to generate an initial vector sequence that integrates semantic and positional information. Feature extraction is performed on the initial vector sequence, and positional information is optimized using a relative position offset mechanism to obtain the hidden layer state. Based on the hidden layer state, the activation probability of each expert network is calculated using a gating network. The K expert networks with the highest activation probabilities are selected, and the activated expert networks process the hidden layer state respectively and output their respective feature vectors. Then, the outputs of the expert networks are weighted and summed according to the weights assigned by the gating network to obtain the integrated global feature vector. A consistency assessment is performed based on the integrated global feature vector, and the classification result of the consistency assessment is output. Define a loss function, train the consistency evaluation model to obtain a trained consistency evaluation model, and use the trained consistency evaluation model to perform consistency evaluation.

2. The method for consistency evaluation of combining unified modeling with hybrid expert network according to claim 1, wherein, Samples from different task modes are modeled into a standard corpus with a unified pattern, specifically as follows: The first mapping function extracts the "basic information" and "information to be evaluated" from the original task samples into "source text" and "target text" respectively. The second mapping function maps the labels of the original tasks into binary consistency labels. Through these two mapping functions, the original sample data of different tasks are standardized into a unified format of "source text-target text-consistency label".

3. The method for consistency evaluation of combining unified modeling with hybrid expert network according to claim 2, wherein, Both the first and second mapping functions are structured transformation rules for different tasks. The first mapping function is an information extraction rule, and the second mapping function is a label mapping rule.

4. The method for consistency evaluation of combining unified modeling with hybrid expert network according to claim 1, wherein, The hidden layer state contains deep semantic features of the "source text - target text" pair.

5. The method for consistency evaluation of combining unified modeling with hybrid expert networks as claimed in claim 1, wherein, Different expert networks focus on different semantic patterns.

6. The consistency evaluation method combining unified modeling and hybrid expert networks as described in claim 1, characterized in that, Focal Loss is used as the loss function, Focal Loss introduces a modulating factor, which can solve the class imbalance problem.

7. A conformance evaluation system that combines unified modeling and hybrid expert networks, characterized by, include: The standard corpus construction module is configured to model samples from different task modes into a standard corpus with a unified pattern. The data processing module is configured to: perform word embedding and position encoding operations on standard corpus samples to generate an initial vector sequence that integrates semantic and positional information; extract features from the initial vector sequence and optimize the positional information using a relative position offset mechanism to obtain the hidden layer state; calculate the activation probability of each expert network using a gating network based on the hidden layer state; select the K expert networks with the highest activation probabilities; have the activated expert networks process the hidden layer state respectively and output their respective feature vectors; and then perform a weighted summation of the outputs of the expert networks according to the weights assigned by the gating network to obtain the integrated global feature vector. The consistency assessment module is configured to perform consistency assessment based on the integrated global feature vector and output the classification result of the consistency assessment. The model training module is configured to: define a loss function, train the consistency evaluation model, obtain the trained consistency evaluation model, and use the trained consistency evaluation model to perform consistency evaluation.

8. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, perform the method described in any one of claims 1-6.

10. A computer program product, characterized in that, Includes a computer program, which, when executed by a processor, implements the method described in any one of claims 1-6.

Images