AI literacy conversational evaluation data construction method based on multi-agent collaboration

Abstract

This invention provides a method for constructing AI literacy conversational assessment data based on multi-agent collaboration, relating to the field of artificial intelligence technology. The method includes: constructing at least one virtual student agent with preset characteristic attributes; deploying multiple expert agents in parallel, each expert agent analyzing and evaluating the answers generated by the virtual student agent in the dialogue from different dimensions and generating corresponding evaluation feedback; deploying a chair agent, which, based on the virtual student agent's answers and the evaluation feedback generated by the expert agents, and in conjunction with a preset interaction strategy, generates and initiates a new round of interaction content to the virtual student agent; and continuously repeating the above steps to generate a conversational assessment dataset including multiple rounds of interaction. This invention effectively solves the problems of traditional AI literacy conversational assessment data synthesis methods, such as the inability to simultaneously achieve assessment depth and scale, and insufficient data annotation completeness.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method for constructing AI literacy conversational assessment data based on multi-agent collaboration. Background Technology

[0002] Artificial intelligence (AI) has profoundly impacted people's production methods and lifestyles, and will play an increasingly important role in future society. AI literacy refers to an individual's comprehensive ability to understand AI, use it correctly, and make responsible judgments about its impact. It is a core competency needed for individuals to adapt to an intelligent society, and its scientific assessment is key to promoting literacy education and improvement. AI literacy generally includes three core components: first, understanding AI knowledge, i.e., knowing relevant AI knowledge, what AI can do, and mastering the principles of AI technology; second, mastering AI skills, i.e., being able to select and use effective AI tools according to specific production or life scenarios; and third, possessing an AI attitude, i.e., being able to identify risks, maintain critical thinking, and adhere to ethical norms in its use.

[0003] Conversational assessment, through open-ended questions and in-depth probing, can capture an individual's thought processes and values, making it an effective method for in-depth qualitative assessment of AI literacy. However, this method heavily relies on human expert resources, greatly limiting its scalability; when faced with large-scale assessment needs, it cannot balance depth and efficiency, leading to an inherent contradiction between assessment depth and scale.

[0004] Large Language Models (LLMs) have demonstrated tremendous potential in dialogue interaction, dynamic probing, and real-time feedback, making them a promising new tool for learning diagnosis and competency assessment. This alleviates the aforementioned contradictions and provides a new technological foundation for conversational assessment of AI competence. However, general-purpose LLMs are typically trained on open corpora. Because they are not trained on real-world conversational assessment data and lack scientific conversational assessment strategies, they not only fail to provide differentiated feedback based on learners' levels, attitudes, or depths of understanding, but also fail to infer core dimensions of AI competence such as knowledge comprehension, practical skills, and ethical judgment from learners' responses.

[0005] In existing technologies, the construction of conversational assessment data for such vertical domains mainly falls into two categories: The first category is manual collection methods based on real-world scenarios. This method records real interviews or teaching dialogues between experts and learners, and forms a dataset after desensitization, transcription, and manual annotation. While the data quality is high, it suffers from extremely high acquisition costs, low collection efficiency, and involves sensitive personal information and privacy issues, severely limiting the data scale and failing to meet the demand for massive amounts of high-quality data for large-scale model training. The second category is simple synthesis methods based on general-purpose large language models. This method directly generates simulated question-and-answer dialogues by inputting simple prompt words into general-purpose large models such as ChatGPT and DeepSeek. However, the data generated by this simple synthesis method suffers from two fundamental technical problems: (1) Lack of multi-dimensional process labeling: Existing methods can usually only generate a surface-level "question-answer" text flow, while completely lacking the crucial internal logical data in professional assessment; for example: which dimensions of AI knowledge, skills, and attitudes do students' answers specifically reflect? What are the evaluation criteria and thought processes of experts? Why choose to ask follow-up questions instead of providing direct guidance at a specific point? This black-box generated data lacks structured evaluation labels, making it difficult for models trained on it to learn professional evaluation logic and decision mapping.

[0006] (2) The dialogue logic is flat and lacks educational depth: The dialogues generated by simple role-playing often remain at the level of information exchange and cannot simulate the complex teaching intervention strategies dynamically adopted by human experts in assessment; for example, they cannot guide and correct students' misconceptions. Therefore, the generated datasets are seriously lacking in the authenticity, educational value and guidance of the interaction.

[0007] In summary, the current technological field lacks a method to automatically and efficiently generate AI literacy conversational assessment data that combines high educational authenticity, complex interactive logic, and rich multi-dimensional fine-grained assessment labels. This directly hinders the development of large-scale models in vertical fields specifically for AI literacy assessment, making it difficult to achieve intelligent assessments that maintain professional depth while possessing the potential for large-scale application. Summary of the Invention

[0008] To address the aforementioned problems, this invention provides a method for constructing AI literacy conversational assessment data based on multi-agent collaboration. This method is used to automatically and efficiently generate AI literacy conversational assessment data that is highly educational, features complex interactive logic, and is rich in multi-dimensional fine-grained assessment labels. This solves the technical problems of traditional methods, such as the inability to balance assessment depth and scale, and insufficient data labeling completeness.

[0009] This invention provides a method for constructing AI literacy conversational assessment data based on multi-agent collaboration, the method comprising: S100, Construct at least one virtual student intelligent agent with preset characteristic attributes; S200, multiple expert agents are deployed in parallel. Each expert agent is used to analyze and evaluate the answers generated by the virtual student agent in the dialogue from different dimensions and generate corresponding evaluation feedback. S300, Deploy the chairman agent. The chairman agent generates and initiates a new round of interactive content to the virtual student agent based on the answers of the virtual student agent and the evaluation feedback generated by the expert agent, combined with a preset interaction strategy. S400, repeat steps S200 and S300 to generate a conversational assessment dataset including multi-turn interactions; wherein, the conversational assessment dataset includes a conversational text sequence, student feature attribute labels corresponding to the virtual student agent, evaluation feedback labels generated by the expert agent, and decision reasoning labels for the chairman agent to make teaching decisions.

[0010] Furthermore, the preset feature attributes include one or more of the following: grade level, interest direction, AI knowledge level, AI ethical inclination, AI tools used, and courses taken.

[0011] Furthermore, multiple expert agents include at least: Knowledge expert agents are configured to assess the depth of understanding of AI concepts reflected in the responses; Skills expert agents are configured to assess the ability to use AI tools to solve real-world problems as demonstrated in responses; The attitude expert agent is configured to evaluate the value judgments reflected in the responses regarding AI ethics, fairness, privacy protection, and social impact.

[0012] Furthermore, the evaluation feedback generated by the expert agent is structured data, including at least qualitative judgments and corresponding analytical reasons.

[0013] Furthermore, the method also includes: accessing an external knowledge base for the chairman agent; wherein the external knowledge base includes at least one or more of the following: an AI literacy assessment framework, an AI ethics case set, and an AI application scenario set.

[0014] Furthermore, the external knowledge base is provided to the chairman agent in a retrieval-enhanced manner.

[0015] Furthermore, the preset interaction strategies include exploratory questioning strategies, in-depth probing strategies, and guided teaching strategies; The exploratory questioning strategy is used to broadly scan students' knowledge base; the in-depth questioning strategy is used to guide critical thinking in response to vague or open-ended answers; and the instructional guidance strategy is used to correct and guide students in response to cognitive biases or ethical deviations identified in the assessment feedback.

[0016] Furthermore, the chairperson agent performs a comprehensive analysis based on the historical dialogue context, the current response of the virtual student agent, and the evaluation feedback generated by all expert agents, dynamically selects an interaction strategy, and generates a new round of interaction content based on the selected interaction strategy.

[0017] Furthermore, the method also includes: integrating the complete dialogue data generated by repeatedly executing steps S200 and S300, as well as the automatically generated labels in each step, to construct a structured conversational assessment dataset; and using the structured conversational assessment dataset to perform supervised fine-tuning of the target large language model to obtain a domain model for AI literacy conversational assessment.

[0018] Furthermore, the virtual student agent, the expert agent, and the chairman agent are all based on a large language model, and their roles are realized by configuring corresponding prompt words for them.

[0019] In summary, this invention provides a method for constructing AI literacy conversational assessment data based on multi-agent collaboration. Compared with existing technologies, the technical solution conceived in this invention can achieve the following beneficial effects: (1) This invention proposes a multi-agent collaborative closed-loop data synthesis architecture consisting of a chairman agent for overall control, multi-dimensional expert agents for collaborative evaluation, and virtual student agents for simulated interaction. This architecture simulates a real, professional teaching and assessment interaction process, enabling fully automatic and batch generation of deep structured dialogue data with "student profile attribute labels, multi-dimensional fine-grained assessment feedback labels, and decision-making reasoning labels for teaching decision-making logic." Compared to manual data collection methods relying on human experts, this method boasts extremely high synthesis efficiency, overcoming limitations in cost, privacy, and scale. Compared to simple question-and-answer pair synthesis methods, the data generated by this method embeds expert assessment thinking and teaching intervention strategies, exhibiting significant educational depth and professionalism, thus providing a crucial data foundation for training high-performance vertical domain assessment models. In summary, this invention achieves efficient and automated synthesis of high-quality AI literacy assessment data, fundamentally resolving the technical contradiction between assessment depth and assessment scale.

[0020] (2) This invention generates structured data rich in multidimensional procedural annotations, effectively solving the problem of existing synthetic data being black-boxed and difficult to use for training professional assessment models. During the data synthesis process, multiple expert agents work in parallel, automatically generating assessment feedback labels with qualitative judgments and analytical reasons for each round of student responses from the three dimensions of knowledge, skills, and attitudes. At the same time, the dynamic strategy selection process of the chairman agent also automatically generates decision-making inference labels for teaching interventions. These automatically generated, high-quality, labeled structured data provide clear supervision signals for subsequent fine-tuning of the large language model, enabling the model to learn the complete mapping logic from student responses to expert analysis to the optimal guidance strategy, thereby significantly improving the performance of the fine-tuned model in terms of assessment accuracy, relevance, and heuristics.

[0021] (3) The chairman agent of this invention adopts dynamic strategy selection. This is not a static knowledge-based question-and-answer process, but rather allows the chairman agent to proactively trigger guidance and teaching strategies to guide and correct students when expert evaluations point out cognitive or ethical misconceptions. This makes the final synthesized dialogue data not only an assessment of students' literacy but also a dynamic learning process. This ensures that the generated data can simulate the most valuable interactions in real educational scenarios, enabling the model trained based on this data to master advanced abilities to promote learning in assessments, thereby enhancing the educational value and interactive authenticity of the generated data. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of the method steps for constructing AI literacy conversational assessment data based on multi-agent collaboration provided by the present invention; Figure 2 This is a schematic diagram illustrating the principle of a method for constructing AI literacy conversational assessment data based on multi-agent collaboration, as provided by the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings and embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0025] It should be noted that, in the description of the embodiments of the present invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a method, step, or apparatus that includes a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to the method, step, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the method, step, or apparatus that includes said element.

[0026] To automate and efficiently generate AI-powered conversational assessment data for literacy that combines high educational authenticity, complex interactive logic, and rich multi-dimensional, fine-grained evaluation feedback labels, and to address the technical challenges of traditional methods such as the trade-off between assessment depth and scale, and insufficient data labeling completeness, this invention provides a method for constructing AI-powered conversational assessment data based on multi-agent collaboration. This method does not rely on real human interviews but instead uses multiple agents to simulate a complete and in-depth educational assessment dialogue, automatically generating high-quality dialogue data with rich labels during the process. This method significantly improves data synthesis efficiency while ensuring the educational depth and professionalism of the data. It resolves the contradiction between the high cost and small scale of manual data collection methods and the low quality and flat logic of simple synthesis methods, overcoming the bottleneck of traditional methods in balancing assessment depth, data scale, and labeling completeness.

[0027] like Figure 1 and Figure 2 As shown, the chair agent leads the dialogue process and interacts with virtual student agents with different characteristics. Simultaneously, multiple expert agents evaluate the virtual student agents' AI knowledge, AI tool application skills, and AI attitudes during the dialogue. Based on expert feedback and the students' actual responses, the chair agent autonomously decides which questioning strategy to use in the next round or provides guidance.

[0028] Specifically, the method of the present invention includes: S100, construct at least one virtual student intelligent agent with preset characteristic attributes.

[0029] The purpose of building at least one virtual student agent is to create diverse, configurable student roles and ensure that the data built can cover students from different backgrounds and levels.

[0030] It should be noted that the preset feature attributes are used to define the background, abilities, and tendencies of the virtual student agent. This virtual student agent is based on a large language model and implemented through the configuration of specific prompt words; that is, preset feature attributes are created for each virtual student agent, and specific prompt words are set to drive the large language model to perform role-playing based on these feature attributes, enabling the virtual student agent to generate logically consistent and personalized responses in dialogues based on its feature attributes.

[0031] As an example, the preset feature attributes may include one or more of the following: grade level, interest direction, AI knowledge level, AI ethical orientation, AI tools used, and courses taken.

[0032] To ensure that the constructed data covers a wide range of students with different characteristics, this invention defines a set of virtual student agents as follows: , This represents the number of virtual students. Simultaneously, for each virtual student intelligent agent... Preset feature attributes are defined. Different virtual student agents can have their feature attributes set by the user or generated randomly. For example, a virtual student agent can be configured with the following feature attributes: {Grade: "Second-year undergraduate", Interest: "Natural Language Processing", AI Knowledge Level: "Mastering basic principles", AI Ethical Orientation: "Neutral", AI Tools Used: ["ChatGPT"]}. Multiple different virtual student agents can be obtained in this way. These configured feature attributes serve as student feature attribute labels for the virtual student agent's profile and are stored in the dataset.

[0033] In addition, to broaden the scope of the data, information on actual students can be added.

[0034] Specifically, virtual student intelligent agents A set of preset feature attributes is randomly selected as the virtual student intelligent agent. The features and attributes are defined, such as [a first-year undergraduate student, interested in machine learning, but with only basic AI knowledge and an open-minded view of AI ethics...]. Based on these features and attributes, prompts are designed to drive an intelligent agent in a large language model like Qwen2.5-7B-Instruct, according to the virtual student's characteristics and attributes. Role-playing based on the characteristics and attributes of virtual students enables them to become intelligent agents. In subsequent dialogues, the system can generate responses that match the character's background and cognitive level based on the established persona. , Represents virtual student intelligent agent At any moment The content of the speech.

[0035] The S200 deploys multiple expert agents in parallel. Each expert agent is used to analyze and evaluate the answers generated by the virtual student agent in the dialogue from different dimensions and generate corresponding evaluation feedback.

[0036] To achieve fine-grained process evaluation, this invention deploys multiple domain-specific expert agents in parallel, each responsible for evaluating and labeling the virtual student agent's responses in real time from different dimensions. This allows the evaluation feedback from multiple expert agents to the virtual student agent's responses to be used as multi-dimensional, fine-grained evaluation feedback labels and stored in the dataset.

[0037] As one example, the multiple expert agents include at least a knowledge expert agent ( Skilled expert intelligent agents ( ) and attitude expert intelligent agents ( ).

[0038] The knowledge expert agent is configured to assess the depth of understanding of AI concepts reflected in the responses; these AI concepts include AI definitions, AI reasoning, and AI algorithm principles. The prompts require it to focus on the accuracy and depth of its assessment of the AI ​​definitions, technical principles, and algorithmic logic involved in the responses.

[0039] The expert agent is configured to assess the ability to use AI tools to solve real-world problems as demonstrated in the responses; this includes practical operational skills with AI tools and computational thinking. The prompts require it to focus on assessing the ability and computational thinking demonstrated in the responses to utilize AI tools to solve specific tasks such as information retrieval, code generation, and copywriting.

[0040] The attitude expert agent is configured to evaluate the value judgments reflected in responses regarding AI ethics, fairness, privacy protection, and social impact. The prompts require it to focus on the understanding and judgment tendencies regarding AI ethics, privacy, fairness, and social responsibility reflected in the responses.

[0041] It should be noted that each expert agent is also built based on a large language model, which can be the same or different large language models. Specifically, by setting prompt words for each expert agent, its role and evaluation focus are clearly defined, and it is instructed to generate evaluation feedback for each round of answers from the virtual student agent.

[0042] As an example, the evaluation feedback generated by the expert agent is structured data, including at least qualitative judgments and corresponding analytical reasons. The qualitative judgments include positive, neutral, and negative.

[0043] For example, at time In this round of dialogue, the expert agents received responses from the virtual student agents. and output structured analytical feedback respectively. , and For example, a knowledge expert agent might output: {Dimension: "Knowledge", Qualitative judgment: "Understanding is basically correct but lacks depth", Reasoning: "The student can describe the basic concepts of neural networks, but failed to explain the specific process of backpropagation..."}. Finally, the analysis feedback from all expert agents is integrated to form structured data. This serves as a multi-dimensional, fine-grained evaluation feedback label for this round of dialogue.

[0044] ; It should be noted that the integration involves directly splicing together the analytical feedback outputs from multiple expert agents. This includes qualitative judgments and corresponding analytical reasons, serving as multidimensional, fine-grained evaluation feedback labels for subsequent data training.

[0045] S300 deploys a chairman agent. Based on the answers from the virtual student agent and the evaluation feedback generated by the expert agent, the chairman agent generates and initiates a new round of interactive content to the virtual student agent, combined with a preset interaction strategy.

[0046] The chair agent, acting as the control center of the dialogue process, guides the flow of conversation and implements instructional interventions. It receives the current responses from the virtual student agent, historical dialogue context, and evaluation feedback generated by all expert agents, dynamically assessing the student's current AI literacy in real time. Based on this, the chair agent, combining various pre-set interaction strategies, dynamically decides the next action, generating corresponding interactive content, such as new questions or guiding statements, and sends it to the virtual student agent. The chair agent's decision-making process for determining the next dialogue strategy serves as the decision-making reasoning label for dialogue strategies and instructional interventions.

[0047] As one embodiment, the method further includes: accessing an external knowledge base for the chairman agent; wherein the external knowledge base includes at least one or more of the following: an AI literacy assessment framework, an AI ethics case set, and an AI application scenario set. Further, the external knowledge base is provided to the chairman agent in a retrieval-enhanced generation manner.

[0048] Accessing external knowledge bases for the chairman agent, i.e., mounting external knowledge bases, includes one or more of the following: publicly available AI literacy assessment frameworks (K), manually compiled AI ethics case studies (C), and AI application scenario sets (S). This knowledge is generated in a retrieval-enhanced manner for the chairman agent to refer to during decision-making.

[0049] The AI ​​literacy assessment framework adopts the AI ​​literacy framework proposed in publicly available academic papers; the AI ​​ethics case set is manually annotated and includes classic controversial cases on algorithmic bias, data privacy breaches, and attribution of responsibility; the AI ​​application scenario set is manually annotated and includes typical and specific student use scenarios such as generative writing, programming assistance, and image generation.

[0050] In each round of interaction, the chair agent receives input including: the historical dialogue context, the current response from the virtual student agents, and the evaluation feedback generated by all expert agents. The chair agent then performs a comprehensive analysis based on this input.

[0051] As an example, the chairperson agent bases its actions on the historical dialogue context and the current response of the virtual student agent. and the evaluation feedback generated by all expert agents. A comprehensive analysis is conducted to dynamically select an interaction strategy, and a new round of interactive content is generated based on the selected strategy.

[0052] As an example, the preset interaction strategies include exploratory questioning, in-depth probing, and guided instruction strategies. That is, based on comprehensive analysis, the chairperson agent autonomously decides on one of these interaction strategies as the next action. For example, if the attitude expert's feedback indicates that the student's current answer shows a tendency to "ignore data privacy risks," and the skills expert's feedback shows that the student has strong tool usage skills, the chairperson agent may determine that it is appropriate to trigger the guided instruction strategy.

[0053] Among them, the exploratory questioning strategy is used to broaden students' knowledge; the in-depth probing strategy is used to guide critical thinking for vague answers or answers with room for further exploration; and the instructional strategy is used to correct and guide students to address cognitive biases or ethical deviations pointed out in the assessment feedback.

[0054] Specifically, when it is necessary to broaden the scope of students' knowledge, exploratory questioning strategies are triggered, such as: "What AI tools do you usually use?" or "What other types of machine learning algorithms do you know?", to broaden the topic.

[0055] When expert agents provide feedback indicating that a student's answer is vague or has room for further exploration, a deep questioning strategy is triggered to guide critical thinking. For example, questions such as "How do you view the impact of using ChatGPT to assist with assignments on your independent thinking ability?" and "You just mentioned that AI can improve efficiency, so in what ways will it reduce work efficiency or bring new problems?" are used to stimulate deep thinking.

[0056] When expert agents point out cognitive misconceptions or ethical deviations in students, guidance strategies are triggered, embodying the "Assessment as Learning" concept. For example, statements such as "Please note that responsible use of AI should balance efficiency and privacy protection" or "You mentioned efficiency improvements when using AI tools, which is good. However, data privacy issues need to be considered, for example..." are used to correct misconceptions and impart correct concepts.

[0057] Finally, the chair agent generates specific new questions or instructions based on the selected interaction strategy and relevant information from an external knowledge base. This will initiate the next round of dialogue.

[0058] It should be noted that learning through assessment emphasizes that assessment itself is part of learning. Assessment is not only about how much students have mastered, but also about helping students reflect on and adjust their learning process in a timely manner, thereby promoting continuous learning. For example, if misconceptions are found in students during the assessment process, they should be guided and corrected.

[0059] This invention integrates advanced educational concepts into data synthesis in an engineered manner. Through the dynamic decision-making of the chairman agent and the triggering of strategies such as exploratory questioning, in-depth questioning, and guidance of teaching, the concept of learning through assessment is solidified in the data generation process. This makes the generated conversational assessment dataset not only an evaluation record, but also reflects the learning process of guidance, error correction, and reflection, significantly improving the educational value and interactive authenticity of the data, and laying the foundation for cultivating a more educationally intelligent AI assessment system.

[0060] S400, repeat steps S200 and S300 to generate a conversational assessment dataset including multi-turn interactions; wherein, the conversational assessment dataset includes a conversational text sequence, student feature attribute labels corresponding to the virtual student agent, evaluation feedback labels generated by the expert agent, and decision reasoning labels for the chairman agent to make teaching decisions.

[0061] In other words, the dialogue is repeated until a preset number of rounds (such as 10 rounds) or other termination conditions are met (such as the chairman agent judging that the evaluation has covered the core dimensions), thereby simulating a complete conversational evaluation dataset that includes multiple rounds of interaction.

[0062] Specifically, a structured, complete conversational assessment dataset is obtained through synthesis. This includes the dialogue text sequence, student feature attribute labels corresponding to the virtual student agent, and implicit inference data generated at each step, namely the multi-dimensional evaluation feedback labels and decision inference labels corresponding to each round of interaction.

[0063] For example, a complete structured conversational assessment data for: { “dialog_id”: “001” “student_profile”: {“Grade”: “Second year of undergraduate studies”, …}, “dialog_sequence”: [ {"turn": 1, "chair_utterance": “student_utterance”: “expert_feedback”: }, {"turn": 2, "chair_utterance": “student_utterance”: ,, “expert_feedback”: ,, “chair_strategy”: “In-depth questioning”}, ... ] } By running the above process in batches, a large-scale, high-quality AI literacy conversational assessment dataset with multi-dimensional fine-grained labels can be synthesized. This dataset can be directly used to supervise and fine-tune a dedicated large language model to obtain a high-performance AI literacy automatic assessment model.

[0064] As an example, the method further includes: integrating the complete dialogue data generated by repeatedly executing steps S200 and S300, as well as the automatically generated labels in each step, to construct a structured conversational assessment dataset; and using the structured conversational assessment dataset to perform supervised fine-tuning of the target large language model to obtain a domain model for AI literacy conversational assessment.

[0065] In addition, to enhance the accuracy of the model, some real assessment data can be appropriately added to the AI ​​literacy conversational assessment dataset.

[0066] During the dialogue generation process, multiple labels are automatically generated in parallel, including student feature attribute labels for student profiles, multi-dimensional fine-grained evaluation feedback labels, and decision reasoning labels for teaching decision-making logic. This generates structured data rich in multi-dimensional process annotations. This high-quality structured data provides key supervision signals for the subsequent training of professional AI literacy assessment models, enabling the target large language model to learn the inherent evaluation logic and effectively solving the problem of existing synthetic data being black-boxed and difficult to use for model training.

[0067] As an example, the virtual student agent, expert agent, and chairman agent are all based on a large language model, and their roles are implemented by configuring corresponding prompt words.

[0068] It should be noted that the virtual student agent, expert agent, and chairman agent can be driven by large language models of different architectures or scales. For example, a model with a larger parameter scale can be used to drive the chairman agent to increase the complexity of its decision-making and guidance. The content and organization of the external knowledge base can be adjusted and expanded according to the specific AI literacy assessment framework. The interaction strategy library can also be further expanded, for example, by adding more complex strategies such as case analysis and scenario simulation.

[0069] To verify the higher quality of the data constructed in this invention, the model can learn professional language strategies and structured interaction norms in the evaluation dialogue during the model training process. This invention first uses this method to construct conversational assessment data, then uses this data to perform supervised fine-tuning of the open-source model Qwen-2.5-7B-Instruct, and finally verifies the performance of the supervised fine-tuned model in conversational AI literacy assessment.

[0070] All agents used in this experiment were driven by the Qwen-2.5-7B-Instruct model, and prompt words were used to assign roles to the models. The experiment used 10,000 data entries, and the average synthesis time for each data entry was only 5.35 seconds. This synthesis efficiency is difficult to achieve by manual methods.

[0071] First, using 10,000 data points constructed according to this invention, the open-source model Qwen-2.5-7B-Instruct was subjected to supervised fine-tuning to obtain the fine-tuned model, which is the model of this invention. Simultaneously, six large language models were selected for comparison: DeepSeek-V3, Hunyuan-A13B-Instruct, Qwen3-235B-A22B-Instruct, Qwen2.5-7B-Instruct, Qwen2.5-14B-Instruct, and Qwen2.5-32B-Instruct. A total of seven models were used as interviewers and interviewees to compare existing large language models with the model of this invention.

[0072] Next, the interviewees, i.e., the virtual student agents, were designed. This invention designed a total of 4×6×6×4=576 virtual students, which are the permutations and combinations of the feature attributes in Table 1.

[0073] Table 1 Preset Feature Attribute Parameters of Virtual Student Intelligent Agent

[0074] Next, a conversational assessment was conducted to obtain conversational assessment data. The model had a conversation with each student, resulting in a total of 576 conversation records.

[0075] Then, an evaluation of the test results was conducted. This invention designed four indicators in the experiment: accuracy, relevance, heuristics, and professionalism.

[0076] Accuracy is determined by comparing the assessment results of the "interviewers" with the pre-defined characteristics of the virtual students; the higher the similarity, the more accurate the test.

[0077] Relevance measures whether the interviewer's questions match the student's characteristics, background, and interests; this metric affects the overall conversation experience.

[0078] The effectiveness of the assessment measures whether the "interviewer" can effectively guide students to think and reflect deeply during the dialogue process, which reflects whether the model implements the concept of "assessment as learning".

[0079] Professionalism measures whether the "interviewer's" assessment reflects the structured and professional standards of AI literacy assessment, that is, whether the assessment dimensions are comprehensive.

[0080] The evaluation method involves using Qwen2.5-72B-Instruct as a judge to evaluate the four indicators and assign an evaluation score between 0 and 10.

[0081] Table 2. Experimental comparison results of multiple models

[0082] Finally, the experimental results, as shown in Table 2, demonstrate that after fine-tuning Qwen2.5-7B-Instruct with the data constructed using this invention, the model achieved significant improvements in accuracy, relevance, inspiration, and professionalism in the conversational AI literacy assessment. Particularly in the inspiration indicator, the fine-tuned model scored 2.74 points higher than the untuned model. Furthermore, despite having only 7B parameters, the fine-tuned model outperformed the 671B DeepSeek-V3 model in all indicators. This indicates that through learning, the model can master the expert thought processes embedded in the synthetic data obtained in this invention, thereby mastering how to analyze student responses from multiple perspectives and establishing a precise mapping logic from student feedback to optimal teaching intervention strategies, thus significantly improving the accuracy and interaction quality of the assessment.

[0083] In summary, this invention, through a closed-loop architecture of multi-agent collaboration, can fully automatically and in batches simulate in-depth teaching and assessment interactions. While ensuring the depth and professionalism of the data in education, it significantly improves data construction efficiency, resolving the contradiction between the high cost and small scale of manual data collection methods and the low quality and flat logic of simple synthesis methods. This achieves efficient and automated generation of high-quality assessment data. Furthermore, unlike traditional static question-and-answer generation, the chairman agent in this architecture can dynamically decide on the next round of interaction strategies based on real-time, fine-grained evaluation feedback from expert agents on student agent responses, combined with an external professional knowledge base. This not only simulates the complex processes of assessment, feedback, and guidance in real-world educational scenarios, realizing the data-level implementation of the concept of learning through assessment, but also automatically generates highly structured labeled data, including expert evaluation thought chains and teaching intervention logic, during the construction process. This effectively solves the technical challenges of the scarcity of high-quality assessment data in vertical fields and the difficulty in balancing assessment depth and scale.

[0084] It should be noted that, for the sake of simplicity, the foregoing embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0085] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0086] In the several embodiments provided in this application, it should be understood that the disclosed methods or systems can be implemented in other ways. For example, the embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0087] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0088] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0089] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0090] Those skilled in the art will understand that all or part of the circuits in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0091] The foregoing description is merely an exemplary embodiment of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Those skilled in the art will readily conceive of embodiments of this disclosure upon considering the specification and practicing the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described herein. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

[0092] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0093] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for constructing AI literacy conversational assessment data based on multi-agent collaboration, characterized in that, The method includes: S100, Construct at least one virtual student intelligent agent with preset characteristic attributes; S200, multiple expert agents are deployed in parallel. Each expert agent is used to analyze and evaluate the answers generated by the virtual student agent in the dialogue from different dimensions and generate corresponding evaluation feedback. S300, Deploy the chairman agent. The chairman agent generates and initiates a new round of interactive content to the virtual student agent based on the answers of the virtual student agent and the evaluation feedback generated by the expert agent, combined with a preset interaction strategy. S400, repeat steps S200 and S300 to generate a conversational assessment dataset including multi-turn interactions; wherein, the conversational assessment dataset includes a conversational text sequence, student feature attribute labels corresponding to the virtual student agent, evaluation feedback labels generated by the expert agent, and decision reasoning labels for the chairman agent to make teaching decisions.

2. The method for constructing AI literacy conversational assessment data based on multi-agent collaboration as described in claim 1, characterized in that, The preset feature attributes include one or more of the following: grade level, interest direction, AI knowledge level, AI ethical orientation, AI tools used, and courses taken.

3. The method for constructing AI literacy conversational assessment data based on multi-agent collaboration according to claim 1, characterized in that, Multiple expert agents include at least: Knowledge expert agents are configured to assess the depth of understanding of AI concepts reflected in the responses; Skills expert agents are configured to assess the ability to use AI tools to solve real-world problems as demonstrated in responses; The attitude expert agent is configured to evaluate the value judgments reflected in the responses regarding AI ethics, fairness, privacy protection, and social impact.

4. The method for constructing AI literacy conversational assessment data based on multi-agent collaboration according to claim 3, characterized in that, The evaluation feedback generated by the expert agent is structured data, including at least qualitative judgments and corresponding analytical reasons.

5. The method for constructing AI literacy conversational assessment data based on multi-agent collaboration according to claim 1, characterized in that, The method further includes: accessing an external knowledge base for the chairman agent; wherein the external knowledge base includes at least one or more of the following: an AI literacy assessment framework, an AI ethics case set, and an AI application scenario set.

6. The method for constructing AI literacy conversational assessment data based on multi-agent collaboration according to claim 5, characterized in that, The external knowledge base is provided to the chairman agent in a retrieval-enhanced manner.

7. The method for constructing AI literacy conversational assessment data based on multi-agent collaboration according to claim 1, characterized in that, The preset interaction strategies include exploratory questioning strategies, in-depth probing strategies, and guided teaching strategies; The exploratory questioning strategy is used to broadly scan students' knowledge base; the in-depth questioning strategy is used to guide critical thinking in response to vague or open-ended answers; and the instructional guidance strategy is used to correct and guide students in response to cognitive biases or ethical deviations identified in the assessment feedback.

8. The method for constructing AI literacy conversational assessment data based on multi-agent collaboration according to claim 7, characterized in that, The chairperson agent performs a comprehensive analysis based on the historical dialogue context, the current response of the virtual student agent, and the evaluation feedback generated by all expert agents, dynamically selects an interaction strategy, and generates a new round of interaction content based on the selected interaction strategy.

9. The method for constructing AI literacy conversational assessment data based on multi-agent collaboration according to claim 1, characterized in that, The method further includes: integrating the complete dialogue data generated by repeatedly executing steps S200 and S300, as well as the automatically generated labels in each step, to construct a structured conversational assessment dataset; and using the structured conversational assessment dataset to perform supervised fine-tuning of the target large language model to obtain a domain model for AI literacy conversational assessment.

10. The method for constructing AI literacy conversational assessment data based on multi-agent collaboration according to claim 1, characterized in that, The virtual student agent, the expert agent, and the chairman agent are all based on a large language model, and their roles are realized by configuring corresponding prompt words.

Images