An AI simultaneous interpretation quality evaluation method based on evaluation of a large model

Abstract

This invention proposes an AI simultaneous interpreting quality assessment method based on a large-scale evaluation model, addressing the shortcomings of existing AI simultaneous interpreting quality assessment methods, such as low quantitative scoring accuracy, poor interpretability, and insufficient scenario adaptability. Its core idea is to use a single large language model as the core of the evaluation, employing an end-to-end architecture to unify the entire process of input processing, quality analysis, and score generation for simultaneous interpreting evaluation. Through an innovative Token As Score scoring mechanism, integer scores from 0 to 100 are mapped to the token space of an LLM (Local Language Model). An analysis-driven strategy is used to first output multi-dimensional interpretable text and then output accurate scores. A two-stage training strategy achieves an organic unity between general evaluation capabilities and vertical domain adaptability.

Description

Technical Field

[0001] This invention belongs to the field of simultaneous interpretation. Background Technology

[0002] With the acceleration of globalization and the increasing frequency of international exchanges, simultaneous interpretation, as a core technology for real-time cross-language communication, plays an irreplaceable role in scenarios such as international conferences, diplomatic negotiations, business negotiations, and academic exchanges. In recent years, the rapid development of artificial intelligence technology has driven breakthroughs in AI simultaneous interpretation technology. End-to-end simultaneous interpretation models based on deep learning can now achieve translation quality close to that of human interpreters in multiple language pairs.

[0003] However, AI simultaneous interpretation technology faces a key bottleneck in practical deployment and application: the lack of accurate, interpretable, and scenario-adaptive quality assessment methods. Currently, simultaneous interpretation quality assessment mainly relies on the following methods: (1) traditional automatic assessment indicators (such as BLEU), which are based on n-gram overlap or edit distance calculations and cannot effectively measure semantic accuracy and translation fluency in simultaneous interpretation scenarios; (2) manual assessment methods, which rely on language experts to subjectively score translation quality, resulting in high costs, low efficiency, and poor consistency; (3) rule-based assessment systems, which score based on predefined error types and weights, resulting in poor scalability and difficulty in covering complex translation error types.

[0004] The aforementioned evaluation methods generally suffer from three core flaws: low quantitative scoring accuracy (traditional indicators cannot provide continuous scores that are highly correlated with human expert judgments), poor interpretability (only outputting numerical scores without explaining the scoring criteria), and insufficient scenario adaptability (difficult to quickly adapt to the evaluation needs of different language pairs and vertical fields). Therefore, there is an urgent need for an end-to-end quality evaluation method for AI simultaneous interpreting that can simultaneously achieve accurate quantitative scoring and expert-level interpretable analysis.

[0005] The following 8 relevant patents were obtained through a full-network patent search: 1.2.1 CN119312818A - Simultaneous Interpretation Quality Assessment Methods and Supporting Devices, Equipment and Storage Media Applicant: iFlytek Co., Ltd. Application date: 2024-12-17 Authorization Date: 2025-06-17 Technical Solution Summary: This patent discloses a method for evaluating the quality of simultaneous interpretation. The simultaneous interpretation text of the target speech is segmented into several sub-texts, and the word-writing data of the sub-speech is obtained (including several texts from the first word to the gradual correction until the final translation into the sub-text). The simultaneous interpretation quality score of the sub-speech is obtained based on the word-writing data analysis, and the target score of the simultaneous interpretation quality of the target speech is obtained by fusion.

[0006] Disadvantages compared to this invention: Quality scoring based on text-based translation (intermediate translation process text) is a process data-driven evaluation method that requires obtaining intermediate data within the simultaneous interpretation system and is highly intrusive to the evaluation object. It does not adopt a large model architecture, lacks end-to-end evaluation capabilities, and the evaluation process requires multiple modules to be connected in series. The scoring method is a fusion of scores at the sub-text level, and it does not achieve continuous and accurate quantitative scoring from 0 to 100 points. It lacks interpretable analytical capabilities, only outputting scores without expert commentary; The Token As Score scoring mechanism is not involved, and the scoring accuracy is limited.

[0007] 1.2.2 CN110148413A - Speech evaluation method and related apparatus Applicant: iFlytek Co., Ltd. Application date: 2019-05-21 Authorization Date: 2021-10-08 Technical Summary: This patent discloses a speech evaluation method. It acquires a first speech as the evaluation standard and a second speech to be evaluated under a first evaluation mode. These are then processed to obtain a first text and a second text, respectively. A corresponding text detection strategy is obtained, and the two texts are processed according to the strategy to obtain the detection result. The method supports paraphrasing and interpretation testing modes. Alignment information is determined by calculating a mutual translation confidence matrix, thereby evaluating translation quality.

[0008] Disadvantages compared to this invention: The text comparison and evaluation based on ASR transcription from speech to text is not an end-to-end large model architecture, and the evaluation link is lengthy. The evaluation method, which uses mutual translation confidence matrix and alignment path, is relatively traditional and has limited model capacity and expressive power. It only supports two modes: paraphrase test and interpretation test, which limits its adaptability to different scenarios. The scoring method is based on classification levels (missed translation rate, multiple translation rate, etc.), and is not a continuous, precise quantitative method. The analysis output is uninterpretable and cannot generate expert-level comments.

[0009] 1.2.3 CN113674764A - Interpreting Evaluation Method, System and Equipment Based on Bidirectional Recurrent Neural Network Applicant: Guangdong University of Foreign Studies Application date: 2021-08-20 Authorization Date: Unauthorized (Rejected) Technical Summary: This patent discloses an interpretation evaluation method based on a bidirectional recurrent neural network. The method transforms the acoustic features of the interpretation into vectors, outputs feature vectors, obtains a unified probability distribution, and uses the probability distribution to obtain the quality vector and context word vector of the interpretation result. The quality vector and context feature sentence vector are then input into the bidirectional recurrent neural network to obtain the overall feature vector of the interpretation result, which is used to calculate the interpretation quality score.

[0010] Disadvantages compared to this invention: It adopts a bidirectional recurrent neural network (BiRNN) architecture, which is not a large model architecture, and has limited model capacity and expressive power; The evaluation is based on the fusion of acoustic and textual features, focusing on oral interpretation rather than AI simultaneous interpretation. The scoring method is based on quality score calculation using feature vectors, rather than the Token As Score precise quantization mechanism; Lacks interpretable analysis; only outputs scores without expert commentary. Limited scenario adaptability, and no vertical domain fine-tuning strategy involved.

[0011] 1.2.4 CN120375864A - A Multimodal Interpreting Training Evaluation Method and Evaluation Device Applicant: Shanghai Lingji Information Technology Co., Ltd. Application date: 2025-06-27 Authorization Date: 2025-09-16 Technical Summary: This patent discloses a multimodal interpreting training evaluation method. It determines the semantic correctness and emotional tendency of the translation by performing standard text matching and sentiment score calculation on the text data, extracts multiple features from the audio data to determine the translation fluency, and evaluates the trainee's translation behavior based on the semantic score, sentiment score, and fluency score.

[0012] Disadvantages compared to this invention: The focus is on interpreting training evaluation (human training assessment), not AI simultaneous interpreting quality assessment. It employs multimodal (text + audio) evaluation and is not an end-to-end large model architecture; The evaluation dimensions are semantic score, sentiment score, and fluency score, on a three-point scale, rather than a continuous and precise score from 0 to 100. Rule / feature combination evaluation based on NLP and speech processing technologies, non-large model generative evaluation; The output lacks interpretability and the ability to generate expert-level comments.

[0013] 1.2.5 CN117910482B - An Automatic Evaluation Method for Machine Translations Based on Deep Difference Features Applicant: Jiangxi Normal University Application date: 2024-03-19 Authorization Date: 2024-05-28 Technical Summary: This patent discloses an automatic evaluation method for machine translations based on deep difference features. It extracts vectors representing the differences between the machine translation and the human-translated reference, as well as vectors representing the semantic differences between the machine translation and the source language sentences. These two difference features are then interacted to extract deep difference features, which are input into a feedforward neural network to predict the quality score of the machine translation. A RoBERTa pre-trained language model is used to extract word-level semantic representation vectors.

[0014] Disadvantages compared to this invention: Based on differential feature extraction of the pre-trained language model RoBERTa, an end-to-end evaluation architecture for non-large models; It focuses on machine translation evaluation rather than simultaneous interpretation evaluation, and does not take into account the temporal and streaming characteristics of simultaneous interpretation; The scoring method uses a feedforward neural network to output scores, rather than a Token As Score precise quantization mechanism; The lack of interpretable analysis prevents the generation of expert-level comments. Without a two-stage training strategy, its scenario adaptability and generalization ability are limited.

[0015] 1.2.6 CN116542260B - A method and system for assessing the quality of translated text based on a large natural language model Applicant: Civil Aviation Flight University of China Application date: 2023-07-05 Authorization Date: 2023-09-26 Technical Summary: This patent discloses a method for evaluating the quality of translated text based on a large natural language model. It constructs a dataset from the civil aviation field, selects the GPT-4 model as the base model, defines objective evaluation indicators, and trains the GPT-4 model using Finetune to obtain a transfer model. This transfer model is then applied to evaluate the quality of translated English texts from civil aviation perspectives. It can assess the accuracy, fluency, professionalism, and security of the translation results from a professional standpoint.

[0016] Disadvantages compared to this invention: Translation quality assessment is based on GPT-4 fine-tuning, using a large model but not an end-to-end evaluation architecture; It is limited to the single field of civil aviation English and has limited ability to generalize to various scenarios. The evaluation metrics are a weighted combination of traditional metrics such as accuracy and recall, and non-Token As Score is continuously and accurately quantified. It lacks a two-stage training strategy (pre-training + fine-tuning) and only uses a single Finetune method; It has limited interpretability and outputs evaluation index scores rather than expert-level analytical text.

[0017] 1.2.7 CN111931519B - Translation evaluation method, apparatus, storage medium and processor Applicant: Alibaba Group Holding Limited Application date: 2019-04-28 Authorization Date: 2023-11-17 Technical Summary: This patent discloses a translation evaluation method that involves acquiring vocabulary belonging to a specified expression type (translation error type) in a target translation paragraph, determining the tag to which the vocabulary belongs (type in the target domain), determining the weight of the vocabulary in the target domain based on the specified expression type and tag, and thus determining the evaluation value of the target translation paragraph. A weighted scoring method is used by constructing an error type-domain tag matrix.

[0018] Disadvantages compared to this invention: Rule-weighted scoring based on translation error type and domain label, non-large model architecture; It focuses on evaluating the quality of translations in specific fields, not simultaneous interpreting. The scoring method is a weighted average of error type matrices, with non-continuous, precise quantitative scoring. Completely lacking in large-scale models and interpretable analytical capabilities; There is no scene generalization or vertical domain adaptation mechanism.

[0019] 1.2.8 WO2023 / 011125A1 - A simultaneous interpretation method, apparatus, device and storage medium Applicant: iFlytek Co., Ltd. Application date: 2022-07-13 Authorization Date: 2023-02-09 Technical Summary: This international patent discloses a simultaneous interpretation method that processes the current input data unit and the currently obtained output data unit, predicts whether data output will occur at the current input position based on the processing result, determines the output data unit and outputs it when data output is predicted, with the aim of jointly optimizing translation quality and translation delay.

[0020] Disadvantages compared to this invention: This patent is for a simultaneous interpreting method (generating end) and a non-simultaneous interpreting quality assessment method (evaluation end), which belong to different technical directions; Focus on the joint optimization of translation quality and latency, rather than quality assessment scores; There is no large-scale evaluation model architecture and no scoring mechanism; Uninterpretable analysis and quantitative scoring capabilities.

[0021] 1.3 Summary of the Comprehensive Disadvantages of Existing Technologies Based on the above analysis of existing technologies, the current field of AI simultaneous interpretation quality evaluation faces the following three core problems: (1) Low accuracy of quantitative scoring: Existing evaluation methods mostly use classification scoring (such as 5-point system, grade system) or indirect scoring based on traditional indicators (BLEU, TER), which cannot provide a continuous and accurate quantitative score of 0-100 points that is highly correlated with human expert judgment. Although iFlytek's CN202411858505.1 patent has achieved simultaneous interpretation quality scoring, its method of sub-text segment level score fusion has a relatively coarse scoring granularity, and its correlation with human expert judgment still needs to be improved.

[0022] (2) Poor interpretability: Existing evaluation methods only output numerical scores or grades and cannot generate interpretable evaluation analysis text. The evaluation results lack expert comments on multiple dimensions such as translation accuracy, fluency, and terminology consistency, making it difficult for users and developers to understand the basis of the evaluation results and the direction for improvement.

[0023] (3) Insufficient scenario adaptability: Existing evaluation methods are mostly designed for specific language pairs or specific fields, lacking a universal end-to-end evaluation architecture and flexible fine-tuning strategies, making it difficult to quickly adapt to the simultaneous interpretation evaluation needs of different language pairs and different professional fields. Patent CN116542260B only covers the single field of civil aviation English, and patent CN113674764A is based on the BiRNN architecture with limited expressive power, neither of which can achieve efficient scenario generalization. Summary of the Invention

[0024] This invention addresses three core problems in existing AI simultaneous interpretation quality evaluation methods: low quantitative scoring accuracy, poor interpretability, and insufficient scenario adaptability. The following technical issues are proposed for resolution: Question 1: How to achieve continuous and accurate quantitative scoring of AI simultaneous interpretation results from 0 to 100? Existing evaluation methods suffer from two limitations: first, classification-based discrete scoring (such as a 1-5 scale) has coarse granularity, failing to distinguish subtle quality differences; second, indirect scoring based on traditional automated metrics (such as BLEU) has low correlation with human expert judgments, especially in simultaneous interpreting scenarios where the correlation between metric values ​​and actual translation quality is further weakened due to the diversity of translation strategies. This invention aims to address the technical challenge of reconciling the output capabilities of large models with the need for precise quantitative scoring.

[0025] Question 2: How can we provide expert-level interpretable analysis while quantifying the scoring? Existing evaluation methods typically present a trade-off between scoring and interpretability: traditional automated metrics only output scores without interpretability, while manual evaluation is interpretable but costly and inconsistent. This invention aims to address the technical challenge of simultaneously achieving accurate quantitative scoring and expert-level interpretable analysis output within a single end-to-end model, ensuring that evaluation results reflect both the precise level of translation quality and provide multi-dimensional professional evaluation criteria.

[0026] Question 3: How to achieve generalization of evaluation capabilities for multiple scenarios and rapid adaptation to vertical fields? AI simultaneous interpreting has diverse applications, involving different language pairs (Chinese-English, Chinese-Japanese, Chinese-German, etc.) and different professional fields (medical, legal, financial, technological, etc.). Existing evaluation methods lack a unified evaluation architecture and flexible adaptation mechanisms, requiring a redesign of the evaluation scheme for each scenario change. This invention aims to solve the technical problem of how to achieve scenario generalization of evaluation capabilities and rapid adaptation to vertical fields through a unified model architecture and training strategy.

[0027] • Core Invention Point 1: Token As Score Scoring Mechanism (Analysis-Driven Continuous Quantitative Scoring) ◦ Technical Problem Solved: Existing simultaneous interpretation evaluation methods have low quantitative scoring accuracy and cannot achieve continuous and accurate scoring from 0 to 100 points. ◦ Technical Approach: Mapping integer scores from 0 to 100 to the token space of a large language model, and using the model to predict the probability distribution of scoring tokens to achieve continuous and accurate scoring; employing an analysis-driven strategy, first generating multi-dimensional quality analysis text, and then generating scoring tokens based on the analysis conclusions, ensuring consistency between the scoring and analysis logic. Specifically, the Token As Score scoring mechanism transforms the scoring problem into a token prediction problem. Given standardized input pairs... (Including source and target languages) and the analysis text already generated by the model The probability distribution prediction of the rating token is shown in equation (1):

[0028] in, To analyze text The hidden state vector at the end position. The rating projection matrix is ​​defined as follows (|V| is the size of the rating token vocabulary, in this invention |V|=101). As the bias vector, softmax normalizes logits into a probability distribution.

[0029] To obtain a continuous and accurate score, the probability of each candidate score token is weighted and expected to obtain the final continuous score from 0 to 100, as shown in the following formula (2):

[0030] in, To ensure continuous and accurate scoring, For candidate discrete scores, This predicts the probability of the token with this score for the model. This expectation mechanism increases the scoring granularity from the traditional 5-point system (granularity 20) to continuous values, while ensuring the consistency between the scoring and analysis logic by using the analyzed text A as a condition.

[0031] Technical Achievements: Achieves continuous and accurate quantitative scoring from 0 to 100 points, with a Pearson correlation of over 0.85 with human expert scores, while providing expert-level interpretable analysis. Core Invention Point 2: Two-stage training strategy (general pre-training + domain-specific fine-tuning for simultaneous interpretation evaluation). ◦ Technical problems to be solved: Existing evaluation models have poor cross-scenario generalization ability and insufficient adaptability to vertical domains. ◦ Technical means adopted: The first stage is to pre-train on large-scale multi-domain simultaneous interpretation evaluation data to obtain general evaluation capabilities, and the second stage is to fine-tune and adapt the data for specific scenarios / domains.

[0032] Technical results achieved: Cross-scenario evaluation relevance improved by more than 20%, and vertical domain adaptability significantly enhanced.

[0033] Core Invention Point 3: End-to-End LLM Evaluation Large Model Architecture ◦ Technical problem to be solved: Information loss and error accumulation in existing multi-module serial evaluation architectures ◦ Technical approach adopted: A single large language model is used to unify the entire process of input processing, quality analysis and score generation. The analysis text and score are output collaboratively through a joint decoder in a single inference. The end-to-end architecture outputs the analysis text A and score S collaboratively through a joint decoder in a single inference. Its joint probability decomposition follows the analysis-driven principle, as shown in the following equation (3):

[0034] in, To analyze the probability of text generation, This represents the predicted probability of a rating given the analyzed text. This decomposition guarantees the rating... Always based on analysis This serves as a condition to achieve cognitive consistency in "analyzing first and then scoring," eliminating information loss and error accumulation in the interconnection of multiple modules.

[0035] The text analysis The tokens are generated using an autoregressive method, as shown in equation (4) below:

[0036] Where T is the length of the analyzed text sequence. For the t-th analysis token, This is the generated prefix sequence. The analysis text and score are output collaboratively in the same forward inference, avoiding efficiency losses and inconsistencies caused by multiple inferences.

[0037] Technical effects achieved: Eliminate information loss and error accumulation between modules, and improve the Inter-Module Conformity (ICC) from 0.60-0.70 to over 0.85.

[0038] The innovative idea of ​​this invention stems from a systematic analysis of existing simultaneous interpretation evaluation technologies. Existing technologies, represented by CN117910482B (Automatic Evaluation Method for Machine Translation Based on Deep Difference Features), employ a multi-module serial architecture of "pre-trained model → feature extraction → feedforward network," which suffers from three core contradictions: coarse scoring granularity, lack of interpretability, and fragmented architecture. TRIZ analysis identifies the following key contradictions: ① Physical contradiction – the same model needs to simultaneously output continuous numerical values ​​(scoring) and discrete text (analysis), which is addressed using a time separation strategy (first generating the analysis text, then generating scoring tokens based on the analysis); ② Technical contradiction – improving scoring accuracy requires increasing model complexity but also increases inference latency. The TokenAs Score mechanism transforms the scoring problem into a token prediction problem, leveraging the powerful language understanding capabilities of LLM to directly output scores without the need for an additional regression module; ③ System evolution direction – evolving from a multi-module serial architecture to an end-to-end unified architecture, integrating the evaluation function into a single LLM. The end-to-end evaluation model architecture of this invention is finally formed. Through the Token AsScore scoring mechanism and the two-stage training strategy, it simultaneously achieves three major goals: accurate quantitative scoring, expert-level interpretable analysis, and scenario generalization adaptation.

[0039] This invention proposes an AI-based simultaneous interpreting quality assessment method based on a large evaluation model. Its core idea is to use a single large language model as the core of the assessment, employing an end-to-end architecture to uniformly model the entire process of input processing, quality analysis, and score generation for simultaneous interpreting evaluation. Through an innovative Token As Score scoring mechanism, integer scores from 0 to 100 are mapped to the token space of an LLM (Local Language Model). An analysis-driven strategy is used to first output multi-dimensional interpretable text and then output accurate scores. A two-stage training strategy achieves an organic unity between general evaluation capabilities and vertical domain adaptation.

[0040] The quality assessment model for AI simultaneous interpreting of this invention adopts an end-to-end architecture, consisting of five core components: an input interface (4), an assessment model (1), a Token As Score module (2), a two-stage training module (3), and an output decoder (5). Among them, the input interface (4) is responsible for receiving and standardizing the simultaneous interpreting source language and target language input; the assessment model (1) serves as the core inference engine; the Token As Score module (2) implements analysis-driven continuous quantitative scoring; the two-stage training module (3) is responsible for the model's training strategy; and the output decoder (5) jointly outputs the analysis text and the scoring results.

[0041] Detailed description of each component: The quality assessment model of this invention consists of the following five core components: First, the assessment model (1), which adopts a large language model based on the Transformer decoder architecture, with a parameter count of 7B to 70B, is responsible for the unified modeling and assessment process, and has language understanding and generation capabilities. Second, the Token As Score module (2), which includes an analysis text generator (20) and a score token generator (21), as well as a score mapping layer (22) and a probability distribution layer (23), with the embedding dimension consistent with the base model, to achieve analysis-driven 0-100 continuous quantitative scoring. Third, the two-stage training module (3), which includes a pre-training stage (30) and a fine-tuning stage (31), which respectively use general assessment data (32) and domain-adapted data (33), wherein the pre-training data is no less than 1 million records and the fine-tuning data is no less than 10,000 records, to achieve general assessment capabilities and vertical domain adaptation. Fourth, the input interface (4) includes a unified input format (40) and standardization processing (41), which, together with the input encoder (12), supports multilingual pairs and is responsible for receiving and standardizing the source and target language inputs in simultaneous interpretation. Fifth, the output decoder (5) includes a joint decoder (50), which outputs analysis output (51) and scoring output (52), and adopts an autoregressive decoding method to jointly output analysis text and scoring results.

[0042] The positional relationship of each component: The input interface (4) is located at the front end of the model and receives external inputs and passes them to the evaluation model (1); the Token AsScore module (2) is embedded in the output path of the evaluation model (1); the two-stage training module (3) is independent of the inference process and acts on the model training stage; the output decoder (5) is located at the back end of the model and outputs the final evaluation results.

[0043] The connections between the components are as follows: First, the data inflow connection uses tensor transfer to connect the input interface (4) to the evaluation model (1), realizing the transfer of standardized input encoding to the large model. Second, the score generation connection uses autoregressive decoding to connect the evaluation model (1) to the Token As Score module (2), triggering the generation of score tokens after the large model generates analysis text. Third, the training control connection uses parameter update to connect the two-stage training module (3) to the evaluation model (1), updating model parameters during the training stage. Fourth, the result output connection uses decoding output to connect the evaluation model (1) to the output decoder (5), jointly decoding and outputting analysis text and scores.

[0044] The working principle of this invention is as follows: After receiving the source language input and target language (translation result) input from the AI ​​simultaneous interpretation, the input is uniformly formatted and standardized through the input interface, and the encoded input is sent to the evaluation model. The evaluation model first performs quality analysis on each dimension through the analysis text generator to generate interpretable analysis text. Then, based on the analysis conclusion, the scoring token generator maps the 0-100 scores to the token space through the score mapping layer, and outputs the final scoring token using the probability distribution layer. Finally, the evaluation result is output through the joint decoder, including multi-dimensional analysis text and continuous accurate scores from 0 to 100.

[0045] Workflow / Operation Steps: Step 1: Receive simultaneous interpretation source language and target language input (S1) • Execution subject: Input interface (4) • Specific actions: Receive the source language text (10) and target language translation text (11) output by the AI ​​simultaneous interpretation system, and combine them into a standardized input pair format through a unified input format (40). • Mechanism of action: Unifies heterogeneous multilingual inputs into a standard format that the model can process, ensuring the standardization of subsequent processing. Step 2: Input Encoding and Feature Extraction (S2) • Execution subject: Input encoder (12) • Specific actions: Tokenization and positional encoding are performed on the input pairs after standardization (41) to convert the text into a vector representation that the model can process. • Mechanism of action: Encodes textual information into high-dimensional semantic vectors, capturing the semantic correspondence between the source language and the target language. Step 3: Multi-dimensional quality analysis text generation (S3) • Execution entity: Text analysis generator (20) • Specific actions: Based on the input encoding, a detailed quality analysis text is generated through autoregression from five dimensions: accuracy, fluency, completeness, terminology consistency, and pragmatic appropriateness. • Mechanism of Action: Utilizing the text generation capabilities of a large language model, it systematically analyzes simultaneous interpretation quality from the perspective of a professional translator, generating traceable and interpretable analytical text. Step 4: Analysis-driven rating token generation (S4) • Execution entity: scoring token generator (21) • Specific actions: Based on the analysis text generated in step 3, the integers from 0 to 100 are mapped to the token space of the model through the score mapping layer (22), and the token corresponding to the score is predicted using the probability distribution layer (23). • Mechanism of Action: The scoring problem is transformed into a token prediction problem. The model's understanding of the analyzed text drives accurate scoring, ensuring consistency between the scoring and the analysis logic. Step 5: Output the evaluation results (S5) • Execution entity: Joint decoder (50) • Specific actions: Combine the analysis text (51) generated in step 3 and the scoring results (52) generated in step 4 to output a complete evaluation report. • Mechanism of action: A single inference process completes the collaborative output of the analyzed text and the score, avoiding the efficiency loss and inconsistency caused by multiple inferences. Training process: Step 6: Pre-training with large-scale, multi-domain evaluation data (S6) • Execution subject: Pre-training phase (30) • Specific actions: Pre-training on large-scale, multi-domain simultaneous interpretation evaluation data (32), covering multiple scenarios such as international conferences, business negotiations, and academic reports, and using an autoregressive language model to train the target. • Mechanism of action: Enables the model to master general simultaneous interpretation evaluation knowledge and scoring models, establishing benchmark evaluation capabilities. Step 7: Fine-tuning of vertical domain evaluation data (S7) • Implementation entity: Fine-tuning stage (31) • Specific actions: Supervised fine-tuning of simultaneous interpretation evaluation data (33) for specific vertical fields (such as medical, legal, and technical fields), using evaluation data annotated by domain experts. • Mechanism of action: Adapting general evaluation capabilities to the evaluation needs of specific fields, thereby improving the accuracy of evaluations in vertical fields. Interactions between the components: The interactions between the components are as follows: First, the input-driven role: through data transmission, the input interface (4) provides standardized input to the evaluation model (1). Second, the analysis-driven scoring role: through autoregressive contextualization, the analysis text generator (20) drives the scoring token generator (21), ensuring that the scoring and analysis logic are consistent. Third, the general capability transfer role: through parameter inheritance, the pre-training stage (30) transfers general evaluation capabilities to the fine-tuning stage (31), realizing capability transfer to domain adaptation. Fourth, the joint output role: through decoding collaboration, the analysis output (51) and the scoring output (52) jointly output the evaluation results, completing dual outputs in one inference.

[0046] The present invention achieves the following six technical effects: (1) It realizes continuous and accurate quantitative scoring from 0 to 100 points. Existing technologies can only provide discrete scores or binary judgments from 1 to 5 points, improving the scoring granularity by 20 times. (2) The Pearson correlation coefficient with human expert judgment is greater than or equal to 0.85, while the correlation coefficient of existing technologies is only 0.55-0.65, improving by more than 30%. (3) It realizes expert-level multi-dimensional interpretable analysis. Existing technologies have no interpretability, achieving a qualitative leap from nothing to something. (4) The cross-scenario evaluation correlation is improved by more than 20%. Existing technologies have poor cross-scenario generalization ability, while two-stage training significantly improves generalization. (5) The evaluation consistency (ICC) is greater than or equal to 0.85, while the ICC of existing technologies is 0.60-0.70, improving by more than 20%. (6) The end-to-end one-time reasoning completes the analysis and scoring. Existing technologies require multiple modules to be connected and multiple reasonings, improving efficiency and ensuring information integrity.

[0047] The above-mentioned technical effects can be quantitatively measured using the following evaluation indicators. The consistency with human expert judgment is measured using the Pearson correlation coefficient, as shown in equation (5):

[0048] in, For the first Model score for each sample Rate it according to the corresponding human experts. , These are the averages of the two, respectively. This invention... ≥0.85, significantly better than BLEU ( =0.48), COMET ( Existing metrics such as (=0.63) indicate that the Token As Score mechanism achieves accurate quantitative scoring that is highly correlated with human experts.

[0049] Consistency was measured using the intragroup correlation coefficient (ICC), as shown in equation (6) below:

[0050] in, For the mean square of the raters, For the mean square error, The number of raters is denoted by 0.85. This invention achieves an ICC score of ≥0.85, representing an improvement of over 20% compared to existing technologies. This indicates that the end-to-end architecture effectively eliminates information loss and error accumulation between modules, ensuring high consistency in the evaluation results.

[0051] The reasons for the above technical effects are as follows: (1) The 0-100 continuous accurate score is due to the Token As Score mechanism, which transforms the scoring problem into a token prediction problem and uses the existing language understanding ability of LLM to directly output a fine score; (2) The high correlation with human experts is due to the analysis-driven strategy. The model first generates professional analysis and then scores based on it, simulating the cognitive process of human experts "analyzing first and then scoring"; (3) The interpretability comes from the systematic analysis of the five quality dimensions by the analysis text generator. Each score has a traceable analytical basis; (4) The cross-scene generalization comes from the two-stage training strategy. The general evaluation knowledge established in the pre-training stage can be transferred to different scenarios; (5) The evaluation consistency comes from the end-to-end architecture, which eliminates information loss and error accumulation between modules; (6) The improved inference efficiency comes from the joint decoder completing dual outputs in one inference. Attached Figure Description

[0052] Figure 1 This is a flowchart of the overall architecture, type A, showing the overall architecture of the evaluation model and the data flow relationships between each module.

[0053] Figure 2 This is a flowchart of the Token As Score scoring mechanism, type A, illustrating the internal working mechanism of the Token As Score module.

[0054] Figure 3 This is a flowchart of a two-stage training strategy, type A, showing the two-stage training process of pre-training and fine-tuning.

[0055] Figure 4 A flowchart for analysis-scoring joint generation, type A, is provided to illustrate the complete workflow of analysis-driven scoring.

[0056] In the diagram, 1-Evaluation model, 2-Token As Score module, 3-Two-stage training module, 4-Input interface, 5-Output decoder, 10-Source language input, 11-Target language input, 12-Input encoder, 20-Analysis text generator, 21-Score token generator, 22-Score mapping layer, 23-Probability distribution layer, 30-Pre-training stage, 31-Fine-tuning stage, 32-General evaluation data, 33-Domain-adapted data, 40-Unified input format, 41-Standardization processing, 50-Joint decoder, 51-Analysis output, 52-Score output, S1-Receive simultaneous interpretation source and target language inputs, S2-Input encoding and feature extraction, S3-Multi-dimensional quality analysis text generation, S4-Analysis-driven score token generation, S5-Output evaluation results, S6-Large-scale multi-domain evaluation data pre-training, S7-Vertical domain evaluation data fine-tuning. Detailed Implementation

[0057] This embodiment provides a large-scale model for quality assessment of AI simultaneous interpreting, and its overall architecture is as follows: Figure 1 As shown. The model uses a large language model based on the Transformer decoder architecture as the evaluation model (1), with a parameter scale of 13B. The model's input interface (4) receives source language input (10) and target language input (11), and combines the two into a standard prompt template format of "[Source Language]: {Source Text}\n[Target Language]: {Translated Text}\n[Evaluation]:" through a unified input format (40). After standardization (41), the input encoder (12) performs tokenization and position encoding.

[0058] The evaluation model (1) embeds a Token As Score module (2), which includes an analysis text generator (20) and a rating token generator (21). The analysis text generator (20) generates quality analysis text from five dimensions: accuracy, fluency, completeness, terminology consistency, and pragmatic appropriateness. The rating token generator (21) includes a score mapping layer (22) and a probability distribution layer (23). The score mapping layer (22) maps integer scores from 0 to 100 to specific token IDs in the model vocabulary (e.g., mapping score 0 to tokenid0, score 1 to tokenid1, ..., score 100 to tokenid100). The probability distribution layer (23) predicts the probability distribution of the rating tokens based on the generated analysis text context and takes the token with the highest probability as the score output.

[0059] The joint decoder (50) in the output decoder (5) combines the analysis output (51) and the scoring output (52) into a complete evaluation report, completing dual outputs in one inference.

[0060] In another implementation, the parameter size of the evaluation large model (1) can be replaced with 7B or 70B, the choice of which depends on the trade-off between computing resources and accuracy requirements.

[0061] In another implementation, the mapping method of the score mapping layer (22) can be replaced by dividing the score range of 0-100 into several intervals (such as 0-20, 21-40, 41-60, 61-80, 81-100), mapping each interval to a token, and calculating the final score by weighted expectation to reduce the number of token categories that the model needs to predict.

[0062] Work process: See Figure 4 The working process of this embodiment is as follows: Step S1: The input interface (4) receives the source language text (such as the original English text "The speaker emphasized the importance of sustainable development") and the target language translation text (such as the Chinese translation "The speaker emphasized the importance of sustainable development") output by the AI ​​simultaneous interpretation system. The input interface (4) combines the source language text (such as the original English text "The speaker emphasized the importance of sustainable development") into a standard prompt template through a unified input format (40). After standardization processing (41), the input encoder (12) encodes the template into a vector representation.

[0063] Steps S2-S3: Evaluation of the large model (1) First, the text generator (20) generates multi-dimensional quality analysis text through autoregression. For example: "Accuracy: The translation accurately conveys the core meaning of the original text. 'emphasized' is translated as 'emphasis' appropriately, and 'sustainable development' is translated as 'sustainable development' accurately; Fluency: The translation is natural and fluent, in line with Chinese expression habits; Completeness: All key information in the original text is fully conveyed without any omissions; Terminology consistency: Professional terms are translated accurately and consistently; Pragmatic appropriateness: The tone of the translation matches the original text and is suitable for formal speech scenarios." Step S4: Based on the above analysis text, the rating token generator (21) maps the 0-100 rating to the token space through the score mapping layer (22), and the probability distribution layer (23) predicts the rating token, for example, the output token corresponds to the score 92.

[0064] Step S5: The joint decoder (50) combines the analysis text (51) and the score 92 (52) to output a complete evaluation report.

[0065] Technical effects: This embodiment achieves a continuous and accurate quantitative score from 0 to 100 through the Token As Score scoring mechanism. The scoring and analysis logic are self-consistent and highly correlated with human expert judgment.

[0066] Control experiment / experimental data: To verify the effectiveness of the method of this invention, the following control experiments were set up: Control group 1 was evaluated using the BLEU index, with a scoring granularity of sentence level 0-1, a Pearson correlation coefficient of 0.48, an ICC of 0.52, and no interpretability. Control group 2 was evaluated using the COMET index, with a scoring granularity of sentence level 0-1, a Pearson correlation coefficient of 0.63, an ICC of 0.65, and no interpretability. Control group 3 was evaluated using the CN117910482B method, with continuous scores, a Pearson correlation coefficient of 0.62, an ICC of 0.68, and no interpretability. Control group 4 was evaluated using the CN116933806A method, with binary scoring, a Pearson correlation coefficient of N / A, an ICC of 0.55, and no interpretability. The experimental group was evaluated using the method of this invention, with continuous scores from 0-100, a Pearson correlation coefficient of 0.87, an ICC of 0.86, and providing five-dimensional interpretable analysis.

[0067] Experimental data analysis: The method of this invention improves the Pearson correlation coefficient by 38.1% compared with the optimal control group (COMET) and the ICC coefficient by 26.5% compared with the optimal control group (CN117910482B method). It is also the only method that provides multi-dimensional interpretable analysis.

[0068] 2. Example 2 Difference from Example 1: This example, based on Example 1, implements a two-stage training strategy for simultaneous interpretation evaluation in the medical field.

[0069] Specific structure: The specific implementation of the two-stage training module (3) is as follows: Pre-training phase (30): Pre-training was performed on a dataset containing 1.2 million simultaneous interpretation evaluation data points from multiple domains (32). The data covered scenarios such as international conferences, business negotiations, academic reports, and press conferences. Each data point included source language text, target language text, expert scores (0-100 points), and expert analysis text. The training objective was an autoregressive language model with a learning rate of 2e-5, a batch size of 128, and a training duration of 3 epochs.

[0070] Fine-tuning phase (31): Fine-tuning was performed on a dataset containing 15,000 medical simultaneous interpretation evaluation data (33), covering sub-scenarios such as medical lectures, clinical consultations, and medical device instructions, and annotated by professional translators with medical translation backgrounds. The training adopted the LoRA fine-tuning strategy, with rank=16, alpha=32, learning rate of 1e-5, batch size of 64, and training for 5 epochs.

[0071] In another implementation, full parameter fine-tuning can be used instead of LoRA fine-tuning in the fine-tuning stage (31). Full parameter fine-tuning is more effective when there is sufficient data (≥50,000 records).

[0072] In another implementation, the domain-adapted data (33) can be replaced with data from vertical domains such as law, technology, and finance to adapt to different application scenarios.

[0073] Work process: The model first learns general evaluation capabilities in the pre-training phase (30), and then adapts them in the fine-tuning phase (31) for medical terminology (such as "myocardial infarction"), specific sentence structures (such as the standardized expression in drug instructions), and evaluation criteria (such as higher weighting for the accuracy of medical terminology). During inference, the model can identify medical characteristics and provide more accurate evaluations.

[0074] Technical effects: To verify the effectiveness of the two-stage training strategy, the following controlled experiment was conducted: The control group used only general pre-training, with a Pearson correlation coefficient of 0.82 for general evaluation and 0.71 for domain evaluation, resulting in a terminology accuracy of 78%. The experimental group used general pre-training plus medical fine-tuning, with a Pearson correlation coefficient of 0.81 for general evaluation and 0.89 for domain evaluation, resulting in a terminology accuracy of 94%.

[0075] Experimental data analysis: The two-stage training strategy, while maintaining general assessment capabilities, improved the relevance of assessments in the medical field by 25.4% and the accuracy of terminology by 16 percentage points.

Claims

1. A method for evaluating the quality of AI simultaneous interpreting based on a large evaluation model, characterized by: Using a single large language model as the core of the evaluation, an end-to-end architecture is adopted to unify the entire process of simultaneous interpretation evaluation, including input processing, quality analysis and score generation. Through an innovative Token As Score scoring mechanism, integer scores from 0 to 100 are mapped to the token space of LLM. An analysis-driven strategy is used to first output multi-dimensional interpretable analysis text and then output accurate scores. A two-stage training strategy is used to achieve general evaluation capabilities and vertical domain adaptation. The Token As Score scoring mechanism maps integer scores from 0 to 100 to the token space of a large language model, using the model to predict the probability distribution of score tokens to achieve continuous and accurate scoring. It employs an analysis-driven strategy, first generating multi-dimensional quality analysis text, and then generating score tokens based on the analysis conclusions, ensuring consistency between the scoring and analysis logic. Specifically, the Token As Score scoring mechanism transforms the scoring problem into a token prediction problem; given standardized input pairs... and the analysis text generated by the model X includes the source language and the target language. The probability distribution prediction of the rating token is shown in the following formula (1): （1）； in, To analyze text The hidden state vector at the end position. Let |V| be the rating projection matrix, and |V| be the size of the rating token vocabulary. As a bias vector, softmax normalizes logits into a probability distribution; The weighted expected value of the probabilities of each candidate scoring token is calculated to obtain the final continuous score of 0-100, as shown in the following formula (2): （2）； in, To ensure continuous and accurate scoring, For candidate discrete scores, Predict the probability of this score token for the model.

2. The method according to claim 1, characterized in that, The end-to-end LLM evaluation big model architecture is as follows: a single big language model is used to unify the entire process of input processing, quality analysis and score generation, and the analysis text and score are output collaboratively through a joint decoder in one inference. The end-to-end architecture outputs the analysis text A and score S collaboratively through a joint decoder in one inference. Its joint probability decomposition follows the analysis-driven principle, as shown in the following equation (3): （3）； in, To analyze the probability of text generation, The predicted probability of a score given the analyzed text. The text analysis The tokens are generated using an autoregressive method, as shown in equation (4) below: （4）； Where T is the length of the analyzed text sequence. For the t-th analysis token, This is the already generated prefix sequence.

3. A large-scale quality assessment model for AI simultaneous interpreting, employing an end-to-end architecture, characterized by: It consists of five core components: input interface (4), evaluation big model (1), Token As Score module (2), two-stage training module (3), and output decoder (5). Among them, input interface (4) is responsible for receiving and standardizing the simultaneous interpretation source language and target language input; evaluation big model (1) serves as the core inference engine; Token As Score module (2) realizes analysis-driven continuous quantitative scoring; two-stage training module (3) is responsible for the model training strategy; and output decoder (5) jointly outputs the analysis text and scoring results.

4. The large-scale quality assessment model for AI simultaneous interpreting according to claim 3, characterized in that: It consists of the following five core components: First, the evaluation big model (1), which adopts a big language model based on the Transformer decoder architecture with a parameter count of 7B to 70B, and is responsible for the unified modeling and evaluation process; Second, the Token As Score module (2), which includes an analysis text generator (20) and a score token generator (21), as well as a score mapping layer (22) and a probability distribution layer (23), with the embedding dimension consistent with the base model, to achieve analysis-driven 0-100 continuous quantitative scoring; Third, the two-stage training module (3), which includes a pre-training stage (30) and a fine-tuning stage (31), which use general evaluation data (32) and domain-adapted data (33) respectively; Fourth, the input interface (4), which includes a unified input format (40) and standardization processing (41), which, together with the input encoder (12), is responsible for receiving and standardizing the simultaneous interpretation source language and target language input; Fifth, the output decoder (5), which includes a joint decoder (50), which outputs analysis output (51) and score output (52), and jointly outputs analysis text and score results.

5. The large-scale quality assessment model for AI simultaneous interpreting according to claim 3, characterized in that, The positional relationship of each component: The input interface (4) is located at the front end of the model and receives external inputs and passes them to the evaluation model (1); the Token As Score module (2) is embedded in the output path of the evaluation model (1); the two-stage training module (3) is independent of the inference process and acts on the model training stage; the output decoder (5) is located at the back end of the model and outputs the final evaluation results; The connections between the components are as follows: First, the data inflow connection uses tensor transfer to connect the input interface (4) to the evaluation model (1) to transfer the standardized input encoding to the model; Second, the score generation connection uses autoregressive decoding to connect the evaluation model (1) to the Token As Score module (2), which generates the analysis text and triggers the generation of the score token; Third, the training control connection uses parameter update to connect the two-stage training module (3) to the evaluation model (1), which updates the model parameters during the training stage; Fourth, the result output connection uses decoding output to connect the evaluation model (1) to the output decoder (5), which jointly decodes and outputs the analysis text and score.

6. The method according to claim 1, characterized in that, Includes the following steps: Step 1: Receive simultaneous interpretation source language and target language input (S1); • The input interface (4) receives the source language text (10) and target language translation text (11) output by the AI ​​simultaneous interpretation system, and combines the two into a standardized input pair format through a unified input format (40); unifying the heterogeneous multilingual input pairs into a standard format that the model can process; Step 2: Input encoding and feature extraction (S2); The input encoder (12) performs tokenization and position encoding on the input pairs after normalization (41) to convert the text into a vector representation that the model can process; • Encode text information into high-dimensional semantic vectors to capture the semantic correspondence between the source language and the target language; Step 3: Multi-dimensional quality analysis text generation (S3); The analysis text generator (20) generates quality analysis text based on input encoding and through autoregression. Step 4: Analysis-driven rating token generation (S4); The rating token generator (21) maps integers from 0 to 100 to the token space of the model through the score mapping layer (22) based on the analysis text generated in step 3, and uses the probability distribution layer (23) to predict the token corresponding to the rating. Step 5: Output the evaluation results (S5); The joint decoder (50) combines the analysis text (51) generated in step 3 and the scoring results (52) generated in step 4 to output a complete evaluation report; Step 6: Pre-training with large-scale, multi-domain evaluation data (S6); Pre-training was performed on large-scale, multi-domain simultaneous interpretation evaluation data (32), and the target was trained using an autoregressive language model.

Images