An AI-based method and system for automatically generating a large model evaluation question bank

Abstract

The present invention relates to an AI-based large-scale model evaluation question bank automatic generation method and system, which dynamically acquires knowledge and establishes a knowledge base, generates questions from different dimensions and performs enhanced processing on the questions, and after completion of verification, performs difficulty calibration and grade calibration on the questions, establishes a question bank, and updates the knowledge base or question bank by conditional triggering; the system uses an input unit to input or update knowledge, and uses a question bank generation unit to adopt a method to dynamically acquire knowledge, establish a knowledge base, generate questions, perform difficulty calibration and grade calibration, and then generate a question bank, and configure an interactive unit with an interactive port to realize the output of different questions in the question bank. The present invention automatically generates a model evaluation question bank through dynamic knowledge extraction, multi-dimensional question generation, intelligent difficulty calibration and adversarial verification, with high generation efficiency, high question quality and strong diversity, and is particularly suitable for artificial intelligence evaluation scenarios.

Description

Technical Field

[0001] The present invention relates to the technical field of electronic digital data processing, and in particular to an AI-based method and system for automatically generating a large model evaluation question bank. Background Art

[0002] Machine learning, especially deep learning, is transforming numerous industries with its powerful predictive capabilities and wide-ranging application scenarios. With the rapid development of artificial intelligence (AI), large-scale machine learning models, such as GPT and BERT, are increasingly being used across various fields. Given the high reliance on data and computing resources and the inherent limitations of model interpretability, model evaluation is essential. Model evaluation is a crucial step in machine learning, enabling performance assessment of trained models and optimizing the final model for deployment.

[0003] To evaluate the performance of these models, it is usually necessary to construct a large number of evaluation questions. However, the traditional method of manually constructing question banks is inefficient and difficult to cover diverse test scenarios.

[0004] Therefore, there is an urgent need for an automated method to generate high-quality and diverse assessment question banks. Summary of the Invention

[0005] The present invention solves the problems existing in the prior art and provides an AI-based method and system for automatically generating a large-scale model evaluation question bank, thereby solving the problems of low efficiency and low question quality in manually generating evaluation question banks in the prior art.

[0006] The technical solution adopted by the present invention is a method for automatically generating a large model evaluation question bank based on AI, which includes the following steps:

[0007] S1 dynamically acquires knowledge and builds a knowledge base;

[0008] S2 generates questions from different dimensions based on the knowledge base and enhances the questions;

[0009] After S3 completes the verification, the difficulty level and grading of the questions are calibrated and a question bank is established;

[0010] S4 If the conditions for updating the knowledge base or question bank are met, repeat S1 or S2.

[0011] Preferably, in S1, a self-supervised pre-trained language model is used to extract feature vectors of the input multi-source data, and the concepts and relationships therein are obtained based on the feature vectors, a concept relationship network is constructed, and a knowledge graph is established as a knowledge base; in the knowledge graph, an importance score is assigned to any concept match.

[0012] Preferably, the concepts and relationships in the knowledge graph are matched with timestamps.

[0013] Preferably, S2 comprises the following steps:

[0014] S2.1 Screen the topic dimensions and generate input sequences based on the topic dimensions;

[0015] S2.2 uses position encoding to fix the position features of the input sequence, which is then fed into the hybrid attention module to dynamically calculate the importance weights of different positions in the input sequence and generate new questions;

[0016] S2.3 Enhance the questions.

[0017] Preferably, in S2.2, the hybrid attention module includes parallel multi-head attention units, self-attention units and cross-attention units, and an input layer is provided in front of the multi-head attention units, self-attention units and cross-attention units, and an importance review layer and an output layer are provided after the multi-head attention units, self-attention units and cross-attention units.

[0018] Preferably, in S2.3, the enhancement processing includes adding interference items, inserting multimodal elements, and generating adversarial variants.

[0019] Preferably, S3 includes the following steps:

[0020] S3.1 Preprocess the questions generated in S2;

[0021] S3.2 Use the pre-adjusted T5 model to evaluate the performance of the pre-processed questions to determine whether they meet the requirements of the questions;

[0022] S3.3 Establish a question bank after difficulty calibration and grading calibration of the questions.

[0023] Preferably, in S3.3, the difficulty calibration is associated with the computational complexity, the capacity of the hybrid attention module, and the difficulty score of the data feature, and the difficulty score of the data feature is associated with the sequence length and feature dimension of the question.

[0024] Preferably, an adversarial generative model is established and trained, and the difficulty of the simulated questions generated by the generator in the trained adversarial generative model is calibrated and reviewed together with the real questions.

[0025] An AI-based automatic generation system for a large-scale model assessment question bank, the system comprising:

[0026] an input unit, used to input or update knowledge into the system;

[0027] A question bank generation unit, which uses the AI-based large-model evaluation question bank automatic generation method to dynamically acquire knowledge, establish a knowledge base, generate questions, and perform difficulty calibration and grade calibration to generate a question bank;

[0028] An interactive unit is configured with an interactive port for outputting different questions in the question bank.

[0029] The present invention relates to an AI-based large-model evaluation question bank automatic generation method and system, which dynamically acquires knowledge and establishes a knowledge base, generates questions from different dimensions and performs enhancement processing on the questions, and after completion of verification, performs difficulty calibration and grade calibration on the questions to establish a question bank, and updates the knowledge base or question bank by conditional triggering; the system uses an input unit to input or update knowledge, and uses a question bank generation unit to adopt a method to dynamically acquire knowledge, establish a knowledge base, generate questions, perform difficulty calibration and grade calibration, and then generate a question bank, and an interactive unit configured with an interactive port realizes the output of different questions in the question bank.

[0030] The beneficial effect of the present invention is that it automatically generates a model evaluation question bank through dynamic knowledge extraction, multi-dimensional question generation, intelligent difficulty calibration and adversarial verification, with high generation efficiency, high question quality and strong diversity, and is particularly suitable for artificial intelligence evaluation scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a flow chart of the method of the present invention;

[0032] Figure 2 It is a schematic block diagram of the system structure of the present invention;

[0033] Figure 3 Schematic diagram of the hybrid attention module in the present invention;

[0034] Figure 4 This is a flow chart of establishing a question bank once in the present invention. DETAILED DESCRIPTION

[0035] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0036] The present invention relates to an AI-based method for automatically generating a large model evaluation question bank, the method comprising the following steps:

[0037] S1 dynamically acquires knowledge and builds a knowledge base;

[0038] S2 generates questions from different dimensions based on the knowledge base and enhances the questions;

[0039] After S3 completes the verification, the difficulty level and grading of the questions are calibrated and a question bank is established;

[0040] S4 If the conditions for updating the knowledge base or question bank are met, repeat S1 or S2.

[0041] The following is a detailed implementation description based on the steps.

[0042] S1 dynamically acquires knowledge and builds a knowledge base;

[0043] In S1, a self-supervised pre-trained language model is used to extract feature vectors of the input multi-source data, and the concepts and relationships therein are obtained based on the feature vectors. A concept relationship network is constructed, and a knowledge graph is established as a knowledge base. In the knowledge graph, an importance score is matched for any concept.

[0044] In the implementation of the present invention, multi-source data comes from, including but not limited to, textbooks, papers, technical documents, forum discussions, etc., which are regarded as knowledge metadata. The self-supervised pre-trained language model adopted is RoBERTa-large, which is used to extract concepts and their relationships. When constructing the concept relationship network, ConceptNet is considered to make full use of its technical characteristics of connecting various concepts to each other and assigning weights to these relationships to represent semantic knowledge.

[0045] In order to better implement importance scoring, the concept of timestamp is introduced in the present invention, that is, the timestamp of the concept and relationship matching in the knowledge graph is used to evaluate the importance of the concept based on the time decay factor, and the time decay factor γ is defined as γ=△T -a , where △T is the duration of knowledge appearance and a is an adjustable parameter.

[0046] S2 generates questions from different dimensions based on the knowledge base and enhances the questions;

[0047] Specifically, S2 covers the generation and enhancement of basic questions, including the following steps:

[0048] S2.1 Screen the topic dimensions and generate input sequences based on the topic dimensions;

[0049] Generally speaking, the existing question bank can be used as the dimension of the seed question bank. The dimensions of the questions include content security, model security, data security, model robustness, application and infrastructure security, etc. The dimension of the refused-answer question bank is also set to generate the seed question bank at the same time.

[0050] Here, "content security" is used as the dimension of the input sequence. The input sequence (such as "content security") is encoded into a vector (such as word embedding) to form a matrix X, whose dimension is sequence length × model dimension d model .

[0051] Since the attention mechanism itself does not contain sequence order information, it is necessary to inject position features into the input through position encoding.

[0052] S2.2 uses positional encoding to fix the positional features of the input sequence. A sine function or a learnable vector is used to generate the positional encoding matrix P. This matrix is ​​added to the input to obtain X = X + P. This matrix is ​​then fed into the hybrid attention module. Through the core component of the attention mechanism in the Transformer, the importance weights of different positions in the input sequence are dynamically calculated to focus on key information and generate new questions based on this information.

[0053] In S2.2, the hybrid attention module includes parallel multi-head attention units, self-attention units and cross-attention units, and an input layer is provided in front of the multi-head attention units, self-attention units and cross-attention units, and an importance review layer and an output layer are provided after the multi-head attention units, self-attention units and cross-attention units.

[0054] S2.2.1 Generate three key vectors of X through linear transformation:

[0055] Query (Q): the element currently being focused on;

[0056] Key (Key, K): the element being compared;

[0057] Value (V): The element that actually provides information;

[0058] Calculation formula: Q=XW Q , K=XW K , V=XW V , where W Q 、W K 、W V is a learnable weight matrix;

[0059] S2.2.2 Calculate attention weights:

[0060] Similarity calculation, measure the similarity between query Q and key K by dot product, Attention Scores = QK T , T is transposed, the larger the value of Attention Scores is, the stronger the correlation is;

[0061] Scaling, divided by , get Scaled Scores = , to prevent the dot product from being too large and causing gradient instability, where d k is the dimension of the key vector;

[0062] Softmax normalization converts the scores into probability distributions with weights summed to 1, giving ;

[0063] S2.2.3. Add the attention weights to the value vector V and obtain the final attention output, Output = Attention Weights⋅V.

[0064] S2.2.4 The obtained attention weights and corresponding output values ​​are then fed into the input layer of the hybrid attention module, where they are processed by parallel multi-head attention units, self-attention units, and cross-attention units, respectively.

[0065] S2.2.4.1 uses multi-head attention to capture diverse dependencies in different subspaces, allowing the hybrid attention module to "analyze the same passage from multiple perspectives," focusing on information from different dimensions simultaneously and then combining the conclusions of each group to produce the final result.

[0066] First, split Q, K, V into h "heads" head1,...,head h , used to represent each group of independent subspaces and calculate attention in parallel;

[0067] Attention is calculated independently, with each head focusing on different aspects of information (such as keywords, semantics, etc.), and finally the results are concatenated and fused through a linear layer to meet the requirements.

[0068] MultiHead(Q, K,V)=Concat(head1,...,head h )W o

[0069] Among them, head i =Attention (QW i Q ,KW i K ,VW i V ), W o is the weight matrix;

[0070] S2.2.4.2 uses self-attention to process the same information internally. In self-attention, Q, K, and V all come from the same input sequence (such as within the encoder) to capture the internal relationships within the sequence. In this process, the relationship between each word in the sentence and other words is analyzed to capture the "relationship";

[0071] S2.2.4.3 uses cross-attention. In cross-attention, Q comes from the decoder, and K and V come from the encoder output. For example, in machine translation, the decoder pays attention to the encoder information, allowing the two pieces of information to obtain key information across the sequence, similar to associating different modalities or languages.

[0072] S2.2.5 For the output of the parallel multi-head attention units, self-attention units, and cross-attention units, the importance review layer dynamically calculates the importance weights of different positions in the input sequence, selects the content with the highest importance weight, and outputs the calculation results (generates questions) in the output layer to achieve focus on key information; the dynamic calculation here is related to the aforementioned importance score for any concept match; of course, manual fine-tuning is also possible.

[0073] Through the hybrid attention mechanism, Transformer achieves efficient modeling of complex dependencies, dynamically adjusts the focus according to the input, and fully utilizes dynamic weights; realizes long-range dependencies, directly models the relationship between elements of any distance, and solves the gradient vanishing problem of RNN; without sequential processing, parallel computing is achieved, improving training efficiency.

[0074] S2.3 Enhance the questions.

[0075] In S2.3, the enhancement processing includes adding interference terms, inserting multimodal elements, and generating adversarial variants.

[0076] In the present invention, the enhanced processing includes but is not limited to adding interference items for multi-round question and answer, inserting multimodal elements including but not limited to charts and code snippets, and generating adversarial variants, especially what kind of phenomenon will occur in the model when the number of heads of multi-head attention approaches infinity.

[0077] After S3 completes the verification, the difficulty level and grading of the questions are calibrated and a question bank is established;

[0078] S3 includes the following steps:

[0079] S3.1 Preprocess the questions generated in S2, including:

[0080] Clean the text to remove HTML tags, special characters, extra spaces and other noise;

[0081] Unify text formats, such as converting all text to lowercase;

[0082] Use a tokenizer to tokenize the text, such as the T5Tokenizer pre-trained with the T5 model.

[0083] S3.2 Use the pre-adjusted T5 model to evaluate the performance of the pre-processed questions to determine whether they meet the requirements of the questions;

[0084] The T5 (Text-to-Text Transfer Transformer) model here is used to detect question ambiguity and verify factual accuracy based on the knowledge graph. After loading, the prepared test dataset is organized into a format suitable for model input, generally as input data-label pairs, where the labels are 0 or 1, corresponding to non-questions or questions, respectively. Fine-tuning is performed using the Trainer class in the transformers library. The fine-tuned model is evaluated using the test dataset to check the T5 model's accuracy, recall, F1 value, and other indicators. Finally, the fine-tuned model is used to predict whether a new question is a question (the output label is 1).

[0085] Questions that do not meet the requirements will be eliminated this time.

[0086] S3.3 Establish a question bank after difficulty calibration and grading calibration of the questions.

[0087] In S3.3, the difficulty calibration is associated with the computational complexity C, the capacity M of the hybrid attention module, and the difficulty score D of the data feature. The difficulty score of the data feature is associated with the sequence length and feature dimension of the question.

[0088] Regarding the difficulty calibration of computational complexity, when processing the question text sequence, the time complexity of dot product attention is O(n 2 d), where n is the length of the question text sequence and d is the feature dimension. For some more complex questions, the additive attention mechanism is needed. Due to the addition of additional linear transformation and activation function calculation, the time complexity will increase accordingly. Let the additional complexity of linear transformation and activation function calculation in additive attention be O(n 2 d 2 ), then the total time complexity C of additive attention is approximately O(n 2 d+ n 2 d 2 ); The higher the complexity, the more difficult the problem is in terms of calculation;

[0089] Furthermore, model capacity reflects the model's ability to learn complex patterns. This also applies to models that handle question difficulty calibration. In the hybrid attention module, capacity is evaluated by counting the number of its parameters. When questions involve more complex semantics and logical relationships, a larger capacity model is required to process them. In other words, more parameters mean a larger model capacity M, which may also correspond to higher question difficulty.

[0090] Furthermore, we need to consider the characteristics of the question text data to assess difficulty. When the sequence length of the question text is longer, it means that more information fragments need to be processed; and when the feature dimension is higher, it indicates that the semantics, grammar and other features contained in the question are more complex. Based on this, we define a difficulty index that comprehensively considers sequence length and feature dimension, such as difficulty score D = sequence length × feature dimension.

[0091] In summary, the computational complexity C, the capacity M of the hybrid attention module, and the difficulty score D of the data features are weighted and integrated. Through a large amount of experimental data and machine learning methods, the weights w1, w2, and w3 of each dimension are determined, thereby obtaining the final question difficulty calibration value S, S=w1C+w2M+w3D.

[0092] In practical applications, the present invention also establishes and trains an adversarial generative model, and performs difficulty calibration and review on the simulated questions generated by the generator in the trained adversarial generative model together with the real questions, thus completing adversarial verification.

[0093] The adversarial generative model includes a generator and a discriminator;

[0094] In the context of question difficulty calibration, the generator's task is to generate simulated question feature data that is as close as possible to the distribution of real question data. For example, this can involve generating question text vectors with different sequence lengths and feature dimension combinations, or simulating data with different attention mechanism computational complexities and model capacities.

[0095] The discriminator is responsible for determining whether the input data is real question features or simulated data generated by the generator. The discriminator also calibrates the difficulty of the questions, distinguishing between real questions and questions generated by the generator based on the difficulty calibration results of computational complexity, model capacity and data characteristics; and checks the stability and accuracy of the calibration results.

[0096] The adversarial generative model herein is easily understood by those skilled in the art, and those skilled in the art can set it up as needed.

[0097] During the implementation process, observe the fluctuation of the difficulty calibration value before and after adding simulated data. If the fluctuation is small, it means that adversarial verification helps improve the stability and generalization ability of the difficulty calibration algorithm.

[0098] S4 If the conditions for updating the knowledge base or question bank are met, repeat S1 or S2.

[0099] In the present invention, the knowledge base is automatically updated by real-time monitoring of data sources such as academic papers and technical blogs, and an incremental learning algorithm is used to keep the knowledge fresh.

[0100] When no update is required, the question bank is continuously output.

[0101] The present invention also relates to an AI-based automatic generation system for a large-scale model assessment question bank, the system comprising:

[0102] an input unit, used to input or update knowledge into the system;

[0103] A question bank generation unit, which uses the AI-based large-model evaluation question bank automatic generation method to dynamically acquire knowledge, establish a knowledge base, generate questions, and perform difficulty calibration and grade calibration to generate a question bank;

[0104] An interactive unit is configured with an interactive port for outputting different questions in the question bank.

[0105] In the present invention, the input unit refers to a module that automatically inputs and updates the knowledge base by real-time monitoring of data sources such as academic papers and technical blogs. It can be implemented by data crawling or actively entered by technical personnel through the interactive terminal; the interactive unit is generally an application layer, which can provide functions including but not limited to display and downloading.

[0106] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0107] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0108] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0109] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0110] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0111] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A method for automatically generating a large model evaluation question bank based on AI, characterized by: The method comprises the following steps: S1 dynamically acquires knowledge and establishes a knowledge graph as a knowledge base; in the knowledge graph, any concept matching importance score is scored; the concepts and relationships in the knowledge graph match timestamps, which are used to evaluate the importance of concepts based on the time decay factor, and define the time decay factor γ. γ =△T -a , where △T is the duration of knowledge appearance and a is an adjustable parameter; S2 generates questions from different dimensions based on the knowledge base and performs enhanced processing on the questions, including the following steps: S2.1 Screen the topic dimensions and generate input sequences based on the topic dimensions; S2.2 uses position encoding to fix the position features of the input sequence, and then inputs it into a hybrid attention module to dynamically calculate the importance weights of different positions in the input sequence to generate new questions; the hybrid attention module includes parallel multi-head attention units, self-attention units, and cross-attention units, and an input layer is provided before the multi-head attention units, self-attention units, and cross-attention units. An importance review layer and an output layer are provided after the multi-head attention units, self-attention units, and cross-attention units. The importance review layer dynamically calculates the importance weights of different positions in the input sequence, selects the content with the highest importance weight, and outputs the calculation results through the output layer; S2.3 Enhance the questions; enhancements include adding distractors for multiple rounds of question answering, inserting multimodal elements, and generating adversarial variants; multimodal elements include diagrams and code snippets; After S3 completes the verification, the difficulty of the questions is calibrated and graded to establish a question bank; the difficulty calibration is related to the computational complexity, the capacity of the hybrid attention module, and the difficulty score of the data features; the computational complexity is related to the sequence length and feature dimension of the question text; the difficulty score of the data features is related to the sequence length and feature dimension of the question; If S4 meets the conditions for updating the knowledge base or updating the question bank, S1 or S2 is repeated.

2. The method for automatically generating a large model assessment question bank based on AI according to claim 1, characterized in that: In S1, a self-supervised pre-trained language model is used to extract the feature vectors of the input multi-source data, and the concepts and relationships therein are obtained based on the feature vectors. A concept relationship network is constructed, and a knowledge graph is established as a knowledge base.

3. The method for automatically generating a large model assessment question bank based on AI according to claim 1, characterized in that: S3 includes the following steps: S3.1 pre-processes the questions generated in S2; S3.2 Use the pre-adjusted T5 model to evaluate the performance of the pre-processed questions to determine whether they meet the requirements of the questions; S3.3 Establish a question bank after calibrating the difficulty and grading of the questions.

4. The method for automatically generating a large model assessment question bank based on AI according to claim 1, characterized in that: A generative adversarial model is established and trained, and the difficulty of the simulated questions generated by the generator in the trained generative adversarial model is calibrated and reviewed together with the real questions.

5. An AI-based automatic generation system for a large-scale model assessment question bank, characterized by: The system comprises: an input unit, used to input or update knowledge into the system; A question bank generation unit, which uses the AI-based large-model evaluation question bank automatic generation method according to any one of claims 1 to 4, and is used to dynamically acquire knowledge, establish a knowledge base, generate questions, perform difficulty calibration and grade calibration, and then generate a question bank; An interactive unit is configured with an interactive port for outputting different questions in the question bank.

Images