Artificial intelligence language ability evaluation method based on parallel corpus and multi-dimensional information

Abstract

The application discloses an artificial intelligence language ability evaluation method based on parallel corpus and multi-dimensional information. The method comprises the following steps: reading a natural language text to be processed; determining an extraction dimension; extracting multi-dimensional text information by using a first working large language model; generating a generated text with a length equivalent to that of the natural language text by using a subject large language model; determining the consistency score of the natural language text and the generated text in each extraction dimension by using a second working large language model; determining the comprehensive consistency score of the natural language text, and further determining the comprehensive language ability score of the subject large language model. The application innovatively evaluates the language ability of the large language model based on the consistency between the quality of the generated text and the natural language text, reduces the cost of data set construction, has strong interpretability, can accurately locate the shortcomings of the model in a specific dimension, avoids the overfitting of the model on high-frequency language styles in the training data, can adapt to the characteristics of different text types, and provides more accurate evaluation.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an artificial intelligence language ability assessment method based on parallel corpora and multidimensional information. Background Technology

[0002] Currently, the main automatic evaluation metrics for the output quality of large language models include the following: (i) Evaluation metrics based on N-gram overlap (1) BLEU index BLEU (Bilingual Evaluation Understudy) was proposed by Papineni et al. in 2002. [1] BLEU is the most widely used automatic evaluation metric in the field of machine translation. BLEU measures translation quality by calculating the N-gram precision between the generated text and the reference text.

[0003] (2) ROUGE index ROUGE (Recall-Oriented Understudy for Gisting Evaluation) was proposed by Lin in 2004. [2] It is mainly used for evaluation of text summarization tasks.

[0004] (3) METEOR indicator METEOR (Metric for Evaluation of Translation with Explicit Ordering) was proposed by Banerjee and Lavie in 2005. [3] It comprehensively considers precision, recall, and word order information, and supports synonym matching and stemming matching.

[0005] (ii) Evaluation metrics based on semantic representation (1) BERTScore index BERTScore was proposed by Zhang et al. (2020). [4] We use pre-trained language models (such as BERT) to calculate the word embedding vectors in the generated text and the reference text, and measure the degree of semantic overlap by calculating the cosine similarity between the word vectors.

[0006] (2) Other semantic evaluation indicators In recent years, researchers have proposed BARTScore. [5] GPTScore [6] InstructScore [7]Evaluation methods based on large language models attempt to improve evaluation results by leveraging the semantic understanding capabilities of pre-trained models.

[0007] (III) Performance Evaluation Indicators (1) It can only output text similarity or generate an overall score of text quality.

[0008] (2) The results of most indicators are uninterpretable, and only some indicators have limited interpretability, which is highly limited by the parameters that the algorithm itself can provide.

[0009] (3) One series of works uses the language representation ability of pre-trained language models to calculate the similarity score between the generated text and the reference text in the semantic space, represented by BERTScore; another series of works uses the generation ability of pre-trained models to obtain the evaluation score through generation probability or generation result, represented by BARTScore.

[0010] Among them, the patent titled "An Evaluation Method, System, Device, and Storage Medium for a Large Language Model" (CN120336464A) proposes an evaluation method, system, device, and storage medium for a large language model. The method includes: constructing a sample dataset, which includes input data and its corresponding reference answers; inputting the input data into the model being evaluated to generate output results; designing a multi-dimensional evaluation index system according to task requirements and assigning dynamic weights to each evaluation index, with each index accompanied by a scoring standard description, including at least two of contextual relevance, terminology consistency, language fluency, and expression accuracy; combining the input data, reference answers, output results, and scoring standard descriptions into standardized input instructions; calling the evaluation model to perform multi-dimensional scoring on the standardized input instructions to generate evaluation results; and analyzing the evaluation results according to a preset threshold to generate structured feedback suggestions containing improvement directions. This invention can effectively support the rapid optimization and iteration of large language models. However, this technical solution has the following technical drawbacks: 1. High dataset requirements: This solution has high data requirements, requiring reference answers to compare the generated results with the reference answers. This model belongs to labeled datasets, which greatly increases the cost of building datasets and the requirements for datasets. Because the sample dataset is a limited set that requires fine manual annotation, the testing system faces two risks: (1) Test set contamination: The limited test set data may be inadvertently included in the training material, resulting in distorted evaluation results and failure to reflect the true generalization ability of the model. (2) Test set saturation: As the large model develops and iterates, the performance reaches its limit on the limited content that the test set can reflect (such as model capacity saturation, profit saturation). 2. Rigid evaluation system: The dynamic evaluation system is used to evaluate based on a limited set of indicators. The evaluation indicators are fixed closed categories, including context relevance, terminology consistency, language fluency and expression accuracy. The dynamic evaluation system only adjusts the weight of the indicators and cannot increase the number of indicator categories. Because the evaluation indicators are limited, the testing system faces two risks: (1) Ignoring individuality: Because the same indicators are used to evaluate different categories of texts, the indicators can only select the greatest common divisor of all types of texts, that is, the common dimension of all types of texts. This approach can only discard the characteristics and individuality of the text, which are often the key factors that determine the quality of the generated text. (2) Test set saturation: As large models iterate and develop, performance reaches its limit on limited test metrics. 3. Limited task types: Due to the difficulty of constructing labeled datasets, this approach can only select limited specific task types such as machine translation and text summarization for evaluation. 4. Supervised evaluation: The input data, reference answer, output results, and scoring criteria description are combined into standardized input instructions. The evaluation model is then called to perform multi-dimensional scoring on the standardized input instructions to generate evaluation results. The evaluation model relies on the reference answer and cannot achieve unsupervised evaluation.5. Templated Optimization Suggestions: Analyzes the evaluation results based on preset thresholds and generates structured feedback suggestions containing directions for improvement. These feedback suggestions are templated responses based on the set thresholds, rather than customized responses tailored to specific generated content.

[0011] The patent (CN117634468A) entitled "A General Text Quality Assessment Method Based on a Large Language Model" includes the following: 1. Constructing a general text quality assessment model using a large language model; 2. Constructing training data: the input of the training data consists of prompt words and evaluation inputs, and the output is the evaluation result. The prompt words include instructions, scoring rules, and output formats. Constructing the training data includes constructing training data with reference answers and constructing training data without reference answers; 3. Training the general text quality assessment model using the training data; 4. Using the trained text quality assessment model to perform quality assessment on general texts; 5. It can simultaneously obtain high-quality training data with and without reference answers, improving data quality and performance under the setting of no reference answers. The proposed technical solution has the following technical defects: 1. Difficulty in achieving both universality and specialization: Different text types (such as recipes and poems) vary greatly, and the evaluation standards differ. To make the text quality evaluation model applicable to all types of text, it is necessary to perform overtraining for each type of text to combat model forgetting and achieve ability transfer, or to use new data for each type of text and conduct targeted training to achieve specialization, training multiple models for different text types; regardless of the method used, it will incur high costs. 2. Reliance on finely labeled training data: The training data is also labeled data, which is limited in itself, costly to construct, and can only be used for supervised training. 3. Reliance on reference answer alignment: The solution still requires reference answers to guide the generation of evaluation results in the absence of reference answers during the training phase, essentially still relying on the guidance of reference answers. It cannot achieve unsupervised training and evaluation without reference. 4. Lack of transparency in the evaluation process and mechanism: The text quality evaluation results rely entirely on the knowledge and abilities internalized by the model, which is essentially a series of weight matrices. The evaluation process and mechanism are untraceable, unanalyzable, and lack reference. 5. The evaluation only focuses on the quality of the text and does not involve the evaluation of the language capabilities of the large model. Moreover, the evaluation lacks a reference, such as the original text used to obtain the generated text, which makes the evaluation lack a foundation and reduces the credibility of the evaluation results.

[0012] In summary, existing large language model language ability assessment techniques have the following main shortcomings: 1. High Data Costs Due to Reliance on Labeled Datasets: Existing assessment solutions generally employ question-answering frameworks, requiring the pre-construction of test datasets containing standard answers, and then comparing the generated results with the reference answers. This model belongs to the supervised learning paradigm, which has high data requirements, significantly increasing the cost of building datasets. This contradicts the development trend in the field of machine learning, which is shifting from supervised training to unsupervised training.

[0013] 2. Limited Task Types and Insufficient Generality: Due to the difficulty of constructing labeled datasets, existing evaluation schemes can either only evaluate specific task types such as machine translation, text summarization, and question answering, or require training targeted models on domain-specific data. However, comparing AI-generated parallel text with natural language text is a more widely applicable and general task, and existing schemes fail to provide such a universal evaluation framework.

[0014] 3. Lack of interpretability in evaluation metrics: Existing automated evaluation metrics such as BLEU and ROUGE only output a final score, failing to provide multi-dimensional, fine-grained evaluation information and accurately pinpointing specific error types. This "black box" evaluation result is difficult to provide targeted guidance for model optimization and also fails to meet users' needs for interpretable evaluation results.

[0015] 4. Lack of attention to distinctive Chinese language registers: Existing assessment benchmarks focus primarily on conventional registers and general tasks, while lacking targeted assessment capabilities for distinctive Chinese language registers such as those used in traditional opera, documentaries, and legal regulations. However, the performance of large language models in handling these distinctive registers precisely reflects the boundaries of their true language proficiency.

[0016] 5. Fixed evaluation dimensions and insufficient flexibility: Existing solutions typically pre-set evaluation dimensions within the system, lacking the ability to flexibly adapt to different textual styles (such as narrative and expository styles) by setting different evaluation dimension standards. Different types of texts have different core characteristics, and using a uniform evaluation dimension makes it difficult to accurately assess the quality of text generation for various types of texts. The references involved include: [1]PAPINENI K, ROUKOS S, WARD T, et al. Bleu: a Method for AutomaticEvaluation of Machine Translation[C / OL] / / ISABELLE P, CHARNIAK E, LIN D.Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. Philadelphia, Pennsylvania, USA:Association for Computational Linguistics,2002:311-318[2026-03-26].https: / / aclanthology.org / P02-1040 / . DOI:10.3115 / 1073083.1073135. [2]LINCY.ROUGE:A Package for Automatic Evaluation of Summaries [C / OL] / / Text Summarization Branches Out. Barcelona, Spain:Association for Computational Linguistics, 2004: 74-81[2026-03-26]. https: / / aclanthology.org / W04-1013 / . [3]BANERJEE S, LAVIE A. METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments[C / OL] / / GOLDSTEIN J, LAVIE A, LIN C Y, et al. Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translationand / or Summarization. Ann Arbor, Michigan: Association for Computational Linguistics, 2005: 65-72[2026-03-26]. https: / / aclanthology.org / W05-0909 / . [4]ZHANG T, KISHORE V, WU F, et al. BERTScore: Evaluating TextGeneration with BERT[A / OL]. arXiv, 2020[2026-03-26]. http: / / arxiv.org / abs / 1904.09675. DOI:10.48550 / arXiv.1904.09675. [5]YUAN W, NEUBIG G, LIU P. BARTScore: Evaluating Generated Text as Text Generation[A / OL]. arXiv, 2021[2026-03-26]. http: / / arxiv.org / abs / 2106.11520. DOI:10.48550 / arXiv.2106.11520. [6]FU J, NG SK, JIANG Z, et al. GPTScore: Evaluate as You Desire[A / OL]. arXiv, 2023[2026-03-26]. http: / / arxiv.org / abs / 2302.04166. DOI:10.48550 / arXiv.2302.04166. [7]XU W, WANG D, PAN L, et al. INSTRUCTSCORE: Explainable Text Generation Evaluation with Finegrained Feedback[A / OL]. arXiv, 2023[2026-03-26]. http: / / arxiv.org / abs / 2305.14282. DOI:10.48550 / arXiv.2305.14282. Summary of the Invention In order to solve the problems existing in the prior art, the present invention provides the following technical solution.

[0017] This invention provides an artificial intelligence language ability assessment method based on parallel corpora and multidimensional information, including: Read the natural language text to be processed from a natural language corpus with Chinese-specific language styles; Determine the extraction dimensions of the natural language text; Based on the extraction dimensions, multi-dimensional text information of the natural language text is extracted using a first working large language model; the first working large language model is a fine-tuning model for text information extraction tasks. Based on the multidimensional text information, the subject's large language model is used to generate a generated text of similar length to the natural language text. Using the second working large language model, the consistency scores between the natural language text and the generated text on each extraction dimension are determined; the second working large language model is a fine-tuning model for the consistency assessment task. Based on the consistency scores of each extraction dimension, the overall consistency score of the natural language text is determined. Based on the comprehensive consistency score of multiple natural language texts, the overall language ability score of the subjects' large language model is determined.

[0018] Preferably, the evaluation method further includes: evaluating the consistency between the natural language text and the generated text across each extraction dimension based on consistency scores across each extraction dimension.

[0019] Preferably, the evaluation method further includes: evaluating the quality of the generated text based on the comprehensive consistency score.

[0020] Preferably, the text information extraction dimension of the natural language text is determined by the following method: based on the style category identifier of the natural language text, the text information extraction dimension corresponding to the style category identifier is retrieved from the text information extraction dimension standard library.

[0021] Preferably, the multidimensional text information of the natural language text is extracted using the first working large language model using the following method: Construct text information extraction prompts; The first working large language model is invoked, the input text information is used to extract prompt words, and the output multi-dimensional text information is obtained; The output multidimensional text information is subjected to structured parsing.

[0022] Preferably, the following method is used to generate text of similar length to the natural language text using the subject's large language model: Construct text to generate prompts; The subject's large language model is invoked, the input text is used to generate prompt words, and the output generated text is obtained; Record the generated text and related information.

[0023] Preferably, the consistency score between the natural language text and the generated text on each extraction dimension is determined using the second working large language model as follows: For each extracted dimension, construct consistency assessment prompts; Call the second working large language model, input the consistency assessment prompt words, and obtain the output consistency score; Iterate through all extracted dimensions to obtain a set of consistency scores for each dimension.

[0024] Preferably, the overall consistency score is determined using the following formula: ; in, The number of dimensions to extract from text information; For the first Weight coefficients for each dimension; For the first Consistency scores across all dimensions.

[0025] Preferably, the comprehensive language proficiency score is determined using the following formula: ; in, The number of natural language texts; For the first A comprehensive consistency score for a natural language text.

[0026] Preferably, the comprehensive language proficiency score is determined using the following formula: in, The number of major genres, For the first Weight of each major genre category For belonging to the first The average of the overall consistency scores of multiple natural language texts across a major genre.

[0027] Compared with the prior art, the beneficial effects of the present invention include: (1) Innovative assessment approach This invention presents an innovative approach to evaluating the language capabilities of large language models based on the quality of generated text (its consistency with natural language text). Compared to existing technologies that focus on comprehension assessment, this invention prioritizes the quality of generated text, objectively evaluating the language capabilities of large language models by assessing the multi-dimensional consistency between AI-generated text and the original natural language text.

[0028] (2) Evaluation paradigm for unlabeled datasets This invention employs an innovative task design that eliminates the need for reference answers in the data, directly comparing the generated text with the original. This evaluation paradigm using unlabeled datasets significantly reduces dataset construction costs and improves the versatility and scalability of the evaluation method. This difference is analogous to the paradigm shift in machine learning from supervised to unsupervised training, and is revolutionary.

[0029] (3) Multidimensional interpretable evaluation results This invention evaluates consistency across multiple dimensions, including content summary, paragraph structure, emotional tone, and language style, by setting dimensional standards for text information extraction. It provides a score and evaluation rationale for each dimension. Compared to existing "black box" evaluations that only output a total score, this invention offers highly interpretable results, accurately pinpointing the model's shortcomings in specific dimensions.

[0030] (4) Pay attention to the distinctive style of Chinese language This invention evaluates language based on a natural language corpus featuring distinctive Chinese registers, focusing on registers such as those found in traditional Chinese opera, documentaries, laws and regulations, and classical literature, rather than the conventional registers frequently handled by large language models. This design allows for a comprehensive and effective measurement of the boundaries of a large language model's true language capabilities, preventing the model's "overfitting" performance on high-frequency registers in the training data from masking its true capabilities.

[0031] (5) Flexible dimensional configuration mechanism The text information extraction dimension standards of this invention can be set in multiple ways to suit different major language categories. The dimensions for extracting text information can differ between major categories, while the dimensions for extracting text information within a major category remain consistent. This flexible dimension configuration mechanism allows the evaluation method to adapt to the characteristics of different text types, providing more accurate assessments.

[0032] (6) Robust method design The technical approach of this invention is ingeniously designed, with text information extraction dimensional standards running throughout the entire process of information extraction, text generation, and consistency assessment. Because all three stages are based on the same dimensional standards, the relative comparability of test results from different subjects' large language models is ensured. Even if there are some deviations in the design of the dimensional standards, it does not affect the horizontal comparison of test results between different models. Attached Figure Description

[0033] Figure 1 This is a flowchart illustrating the artificial intelligence language ability assessment method based on parallel corpora and multidimensional information described in this invention. Figure 2 This is a schematic diagram illustrating the working principle of the text information extraction module described in this invention. Figure 3 This is a schematic diagram illustrating the working principle of the AI ​​text generation module described in this invention; Figure 4 This is a schematic diagram illustrating the working principle of the consistency evaluation module described in this invention. Detailed Implementation

[0034] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0035] The method provided by this invention can be implemented in a terminal environment that may include one or more of the following components: a processor, a memory, and a display screen. The memory stores at least one instruction, which is loaded and executed by the processor to implement the method described in the following embodiments.

[0036] A processor may include one or more processing cores. The processor uses various interfaces and lines to connect various parts of the terminal, and performs various functions and processes data by running or executing instructions, programs, code sets or instruction sets stored in memory, and by calling data stored in memory.

[0037] Memory can include random access memory (RAM) or read-only memory (ROM). Memory can be used to store instructions, programs, code, code sets, or instructions.

[0038] The display screen is used to show the user interface of each application.

[0039] In addition, those skilled in the art will understand that the structure of the terminal described above does not constitute a limitation on the terminal. The terminal may include more or fewer components, or combine certain components, or have different component arrangements. For example, the terminal may also include radio frequency circuits, input units, sensors, audio circuits, power supplies, and other components, which will not be described in detail here.

[0040] To address the shortcomings of existing language proficiency assessment schemes using large language models, such as reliance on labeled datasets, limited task types, lack of interpretability of evaluation results, and lack of attention to Chinese-specific registers, this invention provides an AI-based parallel corpus of Chinese-specific registers and multi-dimensional text information for assessing AI language proficiency. Through innovative task design, this invention comprehensively, systematically, objectively, and scientifically evaluates AI language proficiency based on the quality of generated text (its consistency with the corresponding natural language text). This invention does not require data to include reference answers; the generated text is directly compared with the original text across multiple dimensions. It belongs to the assessment paradigm of unlabeled datasets, significantly reducing dataset construction costs while providing multi-dimensional and interpretable evaluation results.

[0041] like Figure 1As shown, this embodiment of the invention provides an artificial intelligence language ability assessment method based on parallel corpora and multidimensional information, including the following steps: S101, read natural language text from a natural language corpus of Chinese characteristic styles. This corpus can include various types of Chinese characteristic texts such as opera, documentaries, and laws and regulations. The natural language text can be taken from multiple types of Chinese characteristic texts. As an example, natural language text can have the following attribute information: a unique text identifier (Text_ID); text content (Text_Content); a genre category identifier (Genre_Category): such as narrative, descriptive, and argumentative styles; a subtype identifier (Genre_Subtype): such as opera, documentaries, laws and regulations, and classical literature; and text metadata: including source, time, and author. The method of this invention focuses on Chinese characteristic styles and evaluates based on a natural language corpus of Chinese characteristic styles, enabling a comprehensive and effective measurement of the true language capability boundary of a large language model when processing characteristic styles.

[0042] S102, Determine the extraction dimensions of natural language text. Since natural language texts have different stylistic categories, it is necessary to first determine the extraction dimensions of the natural language text when extracting text information.

[0043] S103, Based on the extraction dimensions, the first working large language model is used to extract multi-dimensional text information from the natural language text. The first working large language model is a fine-tuned model for the text information extraction task. It can be understood as a model based on a general large language model, finely tuned for a specific text information extraction task, used to perform the text information extraction task.

[0044] S104, based on multidimensional text information, uses the subject's large language model to generate text of comparable length to natural language text. Here, the subject's large language model can be understood as the large language model released by various vendors and made available to the public.

[0045] S105, using the second working large language model, determines the consistency scores between the natural language text and the generated text across each extraction dimension; wherein, the second working large language model is a fine-tuned model for the consistency assessment task. The second working large language model can be understood as a model based on a general large language model, finely tuned for a specific consistency assessment task, used to perform the consistency assessment task.

[0046] S106. Based on the consistency scores of each extraction dimension, determine the comprehensive consistency score of the natural language text. S107. Based on the comprehensive consistency score of multiple natural language texts, determine the comprehensive language ability score of the subject's large language model.

[0047] This invention extracts multi-dimensional information from natural language text, instructs the subject's large language model to generate text based on this information, and then compares the generated text with the original natural language text in terms of multi-dimensional consistency. This method can use evaluation datasets that do not require manual annotation, belonging to the category of unlabeled datasets. Therefore, using the method of this invention can significantly reduce the cost of dataset construction.

[0048] The AI-generated parallel text and natural language text comparison proposed in this invention is a widely applicable general task, suitable for various text styles and not limited by specific task types. Therefore, the method of this invention has universality. This invention follows the basic logic of "generating text based on text and then evaluating text consistency." The first stage takes text information as input and outputs text, making it a general task of constructing text based on information. It is mainly completed by the large model being evaluated, applicable to all types of text data, and does not require the establishment of evaluation models corresponding to different data categories. In the second stage, only a general model for evaluating text parallelism needs to be established. The logic of this invention can support truly unsupervised evaluation and training.

[0049] The method provided by this invention can be implemented in a corresponding evaluation system. This evaluation system may include the following layers: (1) Data layer, used to store and manage the basic data required for the assessment, including: Chinese characteristic style natural language corpus: storing natural language texts of various Chinese characteristic styles; text information extraction dimension standard library: storing text information extraction dimension standards for different styles; subject large language model configuration information library: storing API interface configuration, calling parameters and other information of the subject large language model.

[0050] (2) Processing layer, the core computing module of the system, is used to execute the main process of the evaluation, including: text information extraction, which extracts multi-dimensional text information from natural language text by calling the first working large language model; AI text generation, which generates AI text based on multi-dimensional text information by calling the subject's large language model; and consistency evaluation, which evaluates the multi-dimensional consistency between the original text and the AI-generated text by calling the second working large language model.

[0051] (3) Model layer, which provides the large language model required for the assessment, including: a first working large language model, which is a fine-tuned model specifically for text information extraction tasks; a second working large language model, which is a fine-tuned model specifically for consistency assessment tasks; a subject large language model, and one or more target large language models to be tested.

[0052] (4) Output layer, used to generate and display assessment results, including: single text score, comprehensive consistency score and sub-item score of each text; model comprehensive score, comprehensive language ability score of the subject's large language model; Chinese characteristic style AI parallel corpus, AI text corpus generated during the assessment process; assessment report, which includes detailed evaluation results and analysis.

[0053] In one embodiment of the present invention, the evaluation method may further include: evaluating the consistency between natural language text and generated text in each extraction dimension based on the consistency score of each extraction dimension.

[0054] In one embodiment of the present invention, the evaluation method may further include: evaluating the quality of the generated text based on the comprehensive consistency score.

[0055] In one embodiment of the present invention, the text information extraction dimension of the natural language text can be determined by the following method: based on the style category identifier of the natural language text, the text information extraction dimension corresponding to the style category identifier is retrieved from the text information extraction dimension standard library.

[0056] This process can be completed in the text information extraction module. The text information extraction module is responsible for extracting multi-dimensional text information from natural language text and is one of the core components of the technical solution of this invention. This module is implemented based on the first working large language model and achieves accurate extraction of multi-dimensional features of the text through carefully designed prompt word engineering. Its working principle is as follows: Figure 2 As shown in the diagram, the extraction dimensions are defined in detail within this module. Corresponding text information extraction dimensions are defined for different genre categories. These text information extraction dimensions can be understood as the set of text information dimensions to be extracted for a specific genre category. In embodiments of this invention, multiple sets of text information extraction dimensions can be set, applicable to different genre categories. Different genre categories correspond to different dimension standards, while all texts within the same genre category use the same dimension standard. This achieves flexible dimension adaptation and evaluation for different text types. As an example, the text information extraction dimensions can be shown in Table 1, the extraction dimensions for narrative genres can be shown in Table 2, and the extraction dimensions for descriptive genres can be shown in Table 3.

[0057] Table 1 Table 2 Table 3 This invention, by setting dimensions for text information extraction, can evaluate the consistency between the generated text and the original natural language text across multiple dimensions, including content summary, paragraph structure, emotional tone, language style, and length, ultimately deriving a comprehensive score. This multi-dimensional evaluation method possesses strong interpretability and can accurately pinpoint the model's deficiencies in specific dimensions. Furthermore, in this invention, the text information extraction dimensions are integrated throughout the entire process of information extraction, text generation, and consistency evaluation, ensuring the relative comparability of test results from different large language models.

[0058] In one embodiment of the present invention, the multidimensional text information of the natural language text can be extracted using the first working large language model as follows: Construct text information extraction prompts; The first working large language model is invoked, the input text information is used to extract prompt words, and the output multi-dimensional text information is obtained; The output multidimensional text information is subjected to structured parsing.

[0059] Specifically, the following steps can be taken to implement this: Step 1: Construct text information extraction prompts (Prompts). The prompts may include: task instructions, which explicitly instruct the model to perform the text information extraction task; dimension definitions, which list the dimensions to be extracted and their definitions; output format, which specifies the structured output format of the extraction results; and text to be processed, which describes the content of the natural language text.

[0060] Step two: Call the API interface of the first working large language model, input the constructed prompt words, and obtain the multi-dimensional text information output by the model.

[0061] Step 3: Perform structured parsing on the multi-dimensional text information output by the model to obtain multi-dimensional text information, the data structure of which is as follows: Multidimensional text information A = { "Text_ID": A unique identifier for the text. "Dimension_Standard_ID": Identifier of the dimension standard used. "Extracted_Dimensions": { "Dimension 1 Name": "Dimension 1 Extracted Content" "Dimension 2 Name": "Dimension 2 Extracted Content" ... "Dimension N Name": "Content Extracted from Dimension N" }, "Extraction_Timestamp": Extracts the timestamp. "Model_Version": Version information for the first working large language model. } .

[0062] In one embodiment of the present invention, the following method is used to generate text of similar length to the natural language text using the subject's large language model: Construct text to generate prompts; The subject's large language model is invoked, the input text is used to generate prompt words, and the output generated text is obtained; Record the generated text and related information.

[0063] This process can be completed in the AI ​​text generation module. This module is responsible for calling the subject's large language model and generating AI text of similar length to the original text based on the extracted multi-dimensional text information. This module is the core component for evaluating the language ability of the subject's large language model. Its working principle can be described as follows: Figure 3 As shown.

[0064] Specifically, the following steps can be taken to implement this: Step one involves constructing a text generation prompt, which includes: task instructions to explicitly direct the subject's large language model to generate text based on given multi-dimensional text information; dimensional information, including the extracted content from each dimension of the multi-dimensional text information; and generation requirements, including constraints such as target word count and formatting requirements. The following principles can be followed when generating the prompt: (1) Information integrity: Ensure that the information of each dimension of the multidimensional text information is completely transmitted to the subject model.

[0065] (2) Clarity of instructions: Clearly state the goals and requirements of the generated task.

[0066] (3) Reasonable constraints: Set reasonable generation constraints, such as word count range, format requirements, etc.

[0067] As an example, the template for generating prompt words can be: You are a professional text creator. Please create a text that meets the requirements based on the following multi-dimensional text information.

[0068] # Content Summary {content_summary} # Paragraph Structure {paragraph_structure} # Emotional Tone {emotional_tone} # Language Style {language_style} # Generation Requirements - Target word count: Approximately {target_word_count} words - Please ensure that the generated text is consistent with the above description in terms of content, structure, sentiment, and style. - Please directly output the generated text content; no further explanation is needed. Please begin creating.

[0069] Step two: Call the API interface of the subject's large language model, input the constructed prompt words, and obtain the AI ​​text generated by the model.

[0070] Step 3: Record the AI-generated text and its metadata. The data structure is as follows: AI-generated text = { "Generated_Text_ID": Generates a unique identifier for the text. "Source_Text_ID": The corresponding natural language text identifier. "Model_ID": The subject's large language model identifier. "Generated_Content": The generated text content. "Generation_Timestamp": Generates a timestamp. "Model_Parameters": Call parameter information } .

[0071] In one embodiment of the present invention, the consistency score between the natural language text and the generated text on each extraction dimension can be determined using the second working large language model as follows: For each extracted dimension, construct consistency assessment prompts; Call the second working large language model, input the consistency assessment prompt words, and obtain the output consistency score; Iterate through all extracted dimensions to obtain a set of consistency scores for each dimension.

[0072] This process can be completed in the consistency evaluation module. This module is responsible for evaluating the degree of consistency between natural language text and AI-generated text across various dimensions, and is a key component for achieving interpretability evaluation in this invention. This module is implemented based on the second working large language model. The working principle of the consistency evaluation module is as follows: Figure 4 As shown.

[0073] In this embodiment of the invention, a second working large language model can be used to evaluate the consistency between natural language text and generated text in each dimension based on the text information extraction dimension standard, and obtain a consistency score for each dimension.

[0074] Specifically, the multi-dimensional consistency assessment process can be implemented using the following steps: Step 1: For each assessment dimension, construct a consistency assessment prompt, including: task instruction, which explicitly instructs the model to assess the consistency of two texts on a specific dimension; dimension definition, which defines the current assessment dimension and the assessment criteria; scoring rules, which quantifies the consistency score (e.g., a 1-10 scale); original text, which is the content of natural language text read from a Chinese-specific natural language corpus; and generated text, which is the content of text generated using the subject's large language model.

[0075] As an example, a consistency assessment prompt template could be: You are a professional text evaluation expert. Please evaluate the consistency of the following two texts along the {dimension_name} dimension.

[0076] # Evaluation Dimension Definition {dimension_definition} # Original text {original_text} # Generate text {generated_text} # Scoring Criteria - 9-10 points: Highly consistent, almost no difference - 7-8 points: Basically the same, with minor differences. - 5-6 points: Partially consistent, with significant differences. - 3-4 points: Low consistency, significant differences - 1-2 points: Almost inconsistent # Please output in the following format Rating: [Integers from 1 to 10] Evaluation rationale: [Explain the scoring criteria in detail, pointing out any consistency and inconsistencies] Step two: Call the API interface of the second working large language model, input the prompt words to be constructed, and obtain the consistency score and evaluation reasons output by the model.

[0077] Step 3: Iterate through all evaluation dimensions to obtain a set of consistency scores for each dimension: Dimensional consistency score = { "Dimension 1 Name": {"score": score value, "reason": "reason for evaluation"}, “Dimension 2 Name”: {"score": score value, "reason": “reason for evaluation”}, ... “Dimension N Name”: {"score": score value, "reason": "reason for evaluation"} } .

[0078] In one embodiment of the present invention, the consistency assessment of each dimension adopts a unified scoring standard, as shown in Table 4.

[0079] Table 4 In one embodiment of the present invention, the overall consistency score can be determined using the following formula: ; in, The number of dimensions to extract from text information; For the first Weight coefficients for each dimension; For the first Consistency scores across all dimensions.

[0080] Based on the consistency scores across various dimensions, a comprehensive consistency score can be calculated using a weighted average or other aggregation methods. The weighting coefficients can be configured according to actual needs; by default, equal weights are used. .

[0081] The overall consistency score serves as the main indicator for evaluating the quality of the generated text. The higher the consistency, the higher the quality of the generated text, reflecting the language ability performance of the subject's large language model on that text.

[0082] In one embodiment of the present invention, the generated text and related scoring information can be stored in a parallel corpus of Chinese characteristic language styles AI.

[0083] The data structure of the parallel corpus can be as follows: AI Parallel Corpus Records = { "Record_ID": Records a unique identifier. "Source_Text": { "Text_ID": Natural Language Text Identifier "Content": Natural language text content, "Genre_Category": Style category, "Genre_Subtype": Genre Subtype }, "Generated_Text": { "Model_ID": The subject's large language model identifier. "Content": AI-generated text content "Generation_Timestamp": Generates a timestamp }, "Evaluation_Result": { "Dimension_Scores": Consistency scores across all dimensions. "Overall_Score": Overall consistency score. "Evaluation_Timestamp": Evaluation timestamp } } .

[0084] By traversing all texts in the corpus and all subject large language models, a complete parallel AI corpus of Chinese-specific language styles was constructed. In this corpus, each natural language text corresponds to multiple AI-generated texts generated by different subject large language models.

[0085] In one embodiment of the present invention, the comprehensive language proficiency score can be determined using the following formula: ; in, The number of natural language texts; For the first A comprehensive consistency score for a natural language text.

[0086] In another embodiment of the present invention, the comprehensive language proficiency score can be determined using the following formula: in, The number of major genres, For the first The weight of each major style category For belonging to the first The average of the overall consistency scores of multiple natural language texts across a major genre.

[0087] In one embodiment of the present invention, the following centralized evaluation mode can be adopted.

[0088] First, a comprehensive evaluation model is adopted, which systematically traverses all texts in the natural language corpus with Chinese characteristic style to conduct a comprehensive evaluation of the subject's large language model.

[0089] The execution process may include the following steps: Step 1. Traverse all natural language text in the corpus; Step 2. Perform the complete evaluation process for each text; Step 3. Construct a complete parallel AI corpus of Chinese-specific language styles; Step 4. Calculate the overall language proficiency score of the participants in the large language model.

[0090] It can be applied to the following scenarios: pre-release evaluation of large language models; comprehensive capability analysis in academic research; and construction of AI parallel corpus resources.

[0091] Second, the rapid evaluation mode: the system extracts a representative portion of text from the corpus as a test dataset for rapid evaluation.

[0092] The execution process may include the following steps: Step 1. Select a test dataset from the corpus according to the preset sampling strategy; Step 2. Perform the evaluation process on the text in the test dataset; Step 3. Quickly estimate the overall language proficiency score based on the scores from the test dataset.

[0093] Several sampling strategies can be employed, including: stratified sampling, which stratifies the language by major and minor categories to ensure representativeness; random sampling, which involves random sampling within each stratum; and a configurable sampling ratio.

[0094] It can be applied to the following scenarios: rapid iterative testing of large language models; preliminary comparison and screening of multiple models; and evaluation under resource constraints.

[0095] Third, the single-text evaluation mode allows users to specify a single natural language text for evaluation and obtain a detailed assessment report.

[0096] The execution process may include the following steps: Step 1. The user enters or selects a single natural language text; Step 2. Perform the complete assessment process; Step 3. Generate a detailed single-text evaluation report; It can be applied to the following scenarios: in-depth analysis of specific texts; debugging and verifying system functions; and evaluation requirements for user-defined texts.

[0097] The artificial intelligence language ability assessment method based on parallel corpora and multidimensional information provided by this invention can achieve the following beneficial effects: (1) Reduce assessment costs By employing an unlabeled dataset evaluation paradigm, this invention significantly reduces the manual cost of constructing evaluation datasets. Users only need to prepare natural language text, without needing to annotate reference answers for each text.

[0098] (2) Improve assessment efficiency This invention provides multiple modes such as comprehensive evaluation, rapid evaluation, and single-text evaluation. Users can choose the appropriate evaluation strategy according to their actual needs and achieve a balance between evaluation accuracy and efficiency.

[0099] (3) Enhance the interpretability of results The multi-dimensional consistency evaluation results provide targeted guidance for model optimization. Developers can clearly understand which dimensions the model performs well and which dimensions it is lacking, thus enabling them to make targeted improvements.

[0100] (4) Generates by-product value The parallel corpus of Chinese-specific language styles constructed during the evaluation process records texts generated by various large language models and is an important material for studying the characteristics of AI-generated language. This corpus has broad application prospects in applied stylistics research and AI-generated text identification.

[0101] (5) Supports dynamic updates The corpus and evaluation system of this invention support continuous expansion and updates. As new corpora are included and new large language models are released, the system can continuously accumulate data and dynamically update the evaluation results.

[0102] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if these modifications and modifications of the invention fall within the scope of the claims and their equivalents, the invention is also intended to include these modifications and modifications.

Claims

1. An artificial intelligence language ability assessment method based on parallel corpora and multidimensional information, characterized in that, include: Read natural language text from a natural language corpus with Chinese-specific language styles; Based on the style category identifier of the natural language text, the extraction dimension corresponding to the style category identifier is retrieved from the extraction dimension standard library; based on the extraction dimension, the multi-dimensional text information of the natural language text is extracted using the first working large language model; the first working large language model is a fine-tuning model for the text information extraction task. Based on the multidimensional text information, the subject's large language model is used to generate a generated text of similar length to the natural language text. Using the second working large language model, the consistency scores between the natural language text and the generated text on each extraction dimension are determined; the second working large language model is a fine-tuning model for the consistency assessment task. Based on the consistency scores of each extraction dimension, the overall consistency score of the natural language text is determined. Based on the comprehensive consistency score of multiple natural language texts, the comprehensive language ability score of the subjects' large language model is determined; The overall language proficiency score is determined using the following formula: in, The number of major genres, For the first The weight of each major style category For belonging to the first The average of the overall consistency scores of multiple natural language texts across a single genre category; Alternatively, the following formula can be used to determine the overall language proficiency score: ； in, The number of natural language texts, For the first A comprehensive consistency score for a natural language text.

2. The artificial intelligence language ability assessment method based on parallel corpora and multidimensional information as described in claim 1, characterized in that, The evaluation method also includes: evaluating the consistency between natural language text and generated text across each extraction dimension based on consistency scores.

3. The artificial intelligence language ability assessment method based on parallel corpora and multidimensional information as described in claim 1, characterized in that, The evaluation method further includes: evaluating the quality of the generated text based on the comprehensive consistency score.

4. The artificial intelligence language ability assessment method based on parallel corpora and multidimensional information as described in claim 1, characterized in that, The following method is used to extract multidimensional textual information from the natural language text using the first working large language model: Construct text information extraction prompts; The first working large language model is invoked, the input text information is used to extract prompt words, and the output multi-dimensional text information is obtained; The output multidimensional text information is subjected to structured parsing.

5. The artificial intelligence language ability assessment method based on parallel corpora and multidimensional information as described in claim 1, characterized in that, The following method was used to generate text of similar length to the natural language text using the subject's large language model: Construct text to generate prompts; The subject's large language model is invoked, the input text is used to generate prompt words, and the output generated text is obtained; Record the generated text and related information.

6. The artificial intelligence language ability assessment method based on parallel corpora and multidimensional information as described in claim 1, characterized in that, The consistency scores between the natural language text and the generated text across each extraction dimension are determined using the second working large language model as follows: For each extracted dimension, construct consistency assessment prompts; Call the second working large language model, input the consistency assessment prompt words, and obtain the output consistency score; Iterate through all extracted dimensions to obtain a set of consistency scores for each dimension.

7. The artificial intelligence language ability assessment method based on parallel corpora and multidimensional information as described in claim 1, characterized in that, The overall consistency score is determined using the following formula: ； in, The number of dimensions to extract; For the first Weight coefficients for each dimension; For the first Consistency scores across all dimensions.

Images