Automatic homework evaluation method and system based on multi-agent parallel voting

Abstract

The invention provides an automatic homework evaluation method and system based on multi-agent parallel voting, and the method comprises the steps: receiving a to-be-corrected target homework answer, and distributing a converted task to a parallel scoring agent cluster for processing; performing parallel scoring operation on the target homework answers, and respectively outputting respective correction scores and corresponding confidence indexes; collecting a correction result from each agent, and calculating to obtain a final correction score of the target job according to a preset dynamic weight distribution mechanism and an abnormal value elimination mechanism; and returning the calculated final correction score and an interpretable report including the scoring details of each scoring model to the terminal. According to the method, multiple scoring models are adopted for parallel scoring, and the voting aggregation technology is combined, so that the scoring variance of a single model is effectively reduced, and the overall accuracy is improved; and the model weight is dynamically adjusted according to the historical accuracy and the real-time confidence, so that the adaptability and robustness to different question types and different answers can be improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and educational informatization, specifically to an automatic evaluation method for assignments based on parallel voting by multiple agents, applicable to online education platforms and intelligent teaching systems. Background Technology

[0002] With the rapid development of internet and artificial intelligence technologies, online education platforms and intelligent education systems are becoming increasingly popular. Teachers face a large volume of student assignments during the teaching process, and manual grading is not only inefficient but also easily influenced by subjective factors, making it difficult to guarantee the accuracy and consistency of the grading results.

[0003] Traditional automated scoring systems primarily rely on single models or rule-based scoring engines. These systems perform well for structured, fixed-format objective questions (such as multiple-choice and fill-in-the-blank questions). However, their performance is less satisfactory for open-ended questions (such as subjective, essay, and proof questions). Using a single model for scoring is susceptible to biases in training data and limitations in model structure, leading to significant variations in scoring results. When rule-based engines encounter situations with diverse answer expressions and complex logic, they often fail to make accurate decisions due to insufficient rule coverage.

[0004] Most existing automatic scoring systems are based on a single model, making it impossible to deploy multiple models in parallel or aggregate votes. A single model is easily affected by factors such as training data bias and model structure limitations, leading to significant differences in calibration results. For complex question types such as subjective questions, essay questions, and proof questions, the accuracy and robustness are difficult to meet the requirements. Summary of the Invention

[0005] The purpose of this invention is to provide an automatic job evaluation method and system based on multi-agent parallel voting, which significantly reduces evaluation errors and improves evaluation accuracy. Specifically, this invention employs multiple independent scoring models for parallel scoring, combined with voting aggregation technology, to effectively reduce the scoring variance of individual models and improve overall accuracy. It also introduces a vertical question type model and a real-time confidence assessment mechanism, dynamically adjusting model weights based on historical accuracy and real-time confidence, thereby improving adaptability and robustness to different question types and answers.

[0006] The present invention achieves the above objectives through the following technical solutions: An automatic job evaluation method based on multi-agent parallel voting includes the following steps: The system receives the answers to the target assignments to be graded, converts the answers into a unified task format using an assignment distributor, and then distributes the converted tasks to a cluster of parallel scoring agents for processing. By utilizing multiple heterogeneous or homogeneous scoring models in a parallel scoring agent cluster, parallel scoring operations are performed on the answers to the target assignment, and each model outputs its own graded score and corresponding confidence index. The central aggregator collects the grading results from each agent, calculates the final grading score of the target job based on the preset dynamic weight allocation mechanism and outlier removal mechanism, and optimizes and adjusts the voting weights. The final graded score, along with an interpretable report including details of each grading model, is returned to the client or teacher via the results feedback mechanism. Simultaneously, online fine-tuning is performed based on teacher feedback on the grading results.

[0007] According to the present invention, an automatic job evaluation method based on multi-agent parallel voting is provided. In the job dispatcher, the original text is first evaluated. Preprocessing is performed, then a globally unique identifier (task_id) and metadata are generated for each answer, and the processing results are integrated into a message structure {task_id, The data is pushed to the message queue in the format of , metadata, and distributed to the parallel scoring agent cluster for processing.

[0008] The present invention provides an automatic job evaluation method based on multi-agent parallel voting, which evaluates the original text. x During preprocessing, character filtering is performed first, using a preset set of legal characters to remove characters from the original text. x The text is then processed to identify all illegal characters not included in the set; next, text standardization is performed, converting English words to lowercase and Chinese words from traditional to simplified characters; then, grammar validation and correction are performed, marking grammatical errors in the text and attempting to automatically correct them based on a pre-defined grammar rule library. When generating a globally unique identifier task_id, the system combines the current system's precise timestamp with a random string generated based on a specific algorithm. The timestamp and the random string are then combined according to a preset concatenation rule to ensure the uniqueness of task_id throughout the entire lifecycle of the system and across different job batches. When generating metadata, question information is retrieved by matching from a pre-built and real-time updated question database. Question type determination is based on preset question type feature rules and is made through multi-feature data analysis. The multi-feature data includes at least the original text. xThe format, content structure, etc., are evaluated based on multi-dimensional data calculated by a preset difficulty calculation model. The multi-dimensional data includes at least the average score rate and answering time of the question in the historical homework data. The student ID is obtained by connecting to the school's student information management system through a secure interface and based on the student's identity information when submitting the homework.

[0009] According to the present invention, an automatic job evaluation method based on multi-agent parallel voting is provided, wherein the parallel scoring agent cluster consists of... It consists of agents containing heterogeneous or homogeneous scoring models. Each agent is independently deployed on isolated containers or different GPU nodes to perform inference tasks in parallel. When each agent receives a preprocessed answer task assigned by the job distributor, Answer The input is fed into its dedicated preprocessing pipeline, which processes the answer according to preset rules. After feature extraction and format adjustment, the raw scores are generated. ,and ∈[0,1]; Each agent calculates a confidence index based on the output distribution entropy. , ∈[0,1], expressed by the following formula:

[0010] in, , K For the preset number of categories or rating-related parameters, p i ( v ) represents an intelligent agent i The answer belongs to the first v Probability estimation of class or corresponding rating situation.

[0011] According to the automatic job evaluation method based on multi-agent parallel voting provided by the present invention, the operation process of the central aggregator module includes the following steps and rules: The central aggregator module receives data from... M Fractional vectors of agents With confidence vector Afterwards, among them, s i For the first i The raw scores given by each agent. c i For the first i The confidence index calculated by each agent. i =1,...,M ; First, calculate the fractional vector. the median of Absolute deviation from the median ; Set anomaly detection threshold For each agent's score s i If satisfied The scoring result of the agent is then determined to be an outlier, and the corresponding model is removed from subsequent calculations. After removing outlier models, the central aggregator summarizes the scores of the remaining models and combines them with the confidence index corresponding to each model. c i The final score of the target task is calculated and generated using a weighted average or other preset aggregation algorithm.

[0012] According to the automatic job evaluation method based on multi-agent parallel voting provided by the present invention, for the remaining models that have not been eliminated in the central aggregator, the weight calculation, final score aggregation, and weight optimization process includes: For each remaining model, read the historical error variance. And combined with the current confidence index of the model c i Calculate the initial weights, expressed as:

[0013] in, It is a very small positive number to avoid the case where the denominator is zero; The initial weights are normalized to obtain the final weights, which are expressed as follows:

[0014] Based on the calculated final weights of each remaining model and the corresponding raw scores s i The system's overall score is calculated using the weighted average formula and is expressed as follows:

[0015] After obtaining the system's overall score Then, map it to the desired interval; After aggregation, if there is manual scoring by teachers... Then the exponentially weighted mean squared error is updated immediately, expressed as:

[0016] by Used to adjust and optimize the weights of each model.

[0017] According to the automatic job evaluation method based on multi-agent parallel voting provided by the present invention, the result feedback module will finally calculate the overall system score. And the scoring details for each model, including the score for each model. Confidence index and weight The grading results are presented in a JSON structure via a standardized API, and the results are displayed as a visual report on the teacher's interface. When teachers give a comprehensive score to the system Make corrections and submit a revised version. When the system includes the job sample and correction label in the online fine-tuning pipeline, it periodically triggers incremental training tasks according to preset cycles or trigger conditions, and uses the newly included sample data to train the scoring model, thereby achieving continuous closed-loop optimization and adaptive adjustment of the system.

[0018] According to the present invention, an automatic job evaluation method based on multi-agent parallel voting is provided, which schedules and configures three scoring model agents—Qwen, Doubao, and DeepSeek—on a Linux server, including: API Client Initialization: In a Linux server environment, using Python 3.8 as the programming language, and with the help of the Requests library and the OpenAI Python SDK, the corresponding API clients for the three scoring model agents, Qwen, Doubao, and DeepSeek, were initialized. During the initialization process, a unique access key was configured for each model agent, and the corresponding request parameters were set according to the characteristics and requirements of each model to ensure that requests could be sent to each model correctly and responses could be received. Asynchronous call management: All API calls to these three scoring model agents are managed using asynchronous coroutines (asyncio). Call strategy settings: Set the timeout for API calls to K, meaning that if no response is received from the model agent within K days from the start of the request, the request is considered to have timed out; set the number of retries to k, so that the system will automatically retry when the request fails due to network jitter or service unavailability.

[0019] According to the automatic evaluation method for jobs based on parallel voting by multiple agents provided by the present invention, on a Linux server, Docker container technology is used to create an independent running environment for each agent. First, the Docker engine is installed, and the corresponding Docker image is customized according to the operating system version, library files and running environment requirements on which each scoring model agent depends. The code, configuration files, and related data of each scoring model agent are packaged into a corresponding Docker container, and the container is started using the Docker command, so that each scoring model agent runs in an independent container. To enable the job dispatcher to communicate with the scoring model agent deployed in containers, the network mode of the Docker containers is configured to adopt the bridged network mode, and each container is assigned an independent IP address and port, so that the job dispatcher can send the job tasks to be graded to the corresponding container through the specified IP address and port. Based on the computational requirements of the scoring model agents and the resource availability of the GPU nodes, N scoring model agents are rationally allocated to different GPU nodes; the code and related data of the scoring model agents are deployed to the corresponding GPU nodes; on each GPU node, a task scheduling tool is used to manage the inference tasks of the scoring model agents, and the task scheduling tool arranges the execution order of tasks according to the resource usage of the GPU nodes and the priority of the tasks. The scoring model agent deployed on a container or GPU node receives the job tasks to be graded from the job distributor through a pre-configured network interface; after receiving the task, the agent parses the task data and extracts the key information that needs to be processed. Based on the parsed job information, the agent calls the pre-trained model to perform inference calculations. After completing the inference calculations, the agent encapsulates the obtained score results and related confidence indicators, and returns them to the central aggregator through the network interface according to the pre-agreed communication protocol and data format.

[0020] An automatic job evaluation system based on multi-agent parallel voting includes: The assignment distributor is used to receive the answers to the target assignments to be graded and push the received answers to each agent in the parallel scoring agent cluster according to preset rules and strategies, providing data input for the parallel scoring operation. Parallel scoring agent cluster: consists of N heterogeneous or homogeneous scoring models. Each scoring model acts as an independent agent, used to independently analyze and process the received target homework answers, and output the corresponding grading score and a confidence index reflecting the reliability of the scoring. Central aggregator: Used to collect the grading scores and confidence information fed back by each agent in the parallel scoring agent cluster. According to the pre-set dynamic weight allocation mechanism and outlier removal mechanism, the central aggregator performs comprehensive processing on the collected information and calculates the final grading score of the target job. At the same time, the central aggregator also calculates the historical feasibility based on historical data and uses the historical feasibility to optimize and adjust the voting weights of each agent. Results Feedback Unit: Returns the final graded scores and generated interpretable reports calculated by the central aggregator to the client or teacher; in addition, the results feedback unit also feeds back the teacher-annotated data to the system for online fine-tuning of the grading model.

[0021] Therefore, compared with the prior art, the automatic job evaluation method and system based on multi-agent parallel voting proposed in this invention has the following beneficial effects: 1. This invention employs a multi-agent parallel voting method. Even if individual models fail or produce abnormal scores, other normally functioning models can still provide reasonable scores. The voting aggregation mechanism effectively suppresses large biases and variances caused by the randomness of individual models. This invention maintains stable performance and provides reliable scoring results even in various complex situations, greatly enhancing the system's robustness.

[0022] 2. This invention proposes a real-time weighting mechanism based on question type history and confidence level. It automatically adjusts the voting weights of each model according to its past performance on other question types and its current confidence interval. For a specific question type, if a model has demonstrated high accuracy in scoring that question type in the past and has high confidence level in the current scoring, its weight in the final vote will increase accordingly. Conversely, if a model has low accuracy in scoring that question type in the past, or its current confidence level is low, its weight will decrease. This allows the system to better adapt to the characteristics and requirements of different question types, improving its adaptability and scoring accuracy for different question types and answers.

[0023] 3. When a teacher corrects the system's score and submits a revised version, the system will include the sample and the corrected label in the online fine-tuning pipeline, periodically triggering incremental training tasks. Through this method, the system can continuously learn and accumulate new knowledge, optimizing the model's parameters and performance to better meet the needs of actual teaching. Over time and with the continuous accumulation of data, the system can automatically adapt to the characteristics of different teaching scenarios and student groups, achieving continuous closed-loop optimization and self-adaptation, further improving the accuracy and reliability of the scoring.

[0024] 4. The interpretable reports and detailed scoring provided by this invention help teachers better understand students' learning progress and existing problems, offering strong support for teaching decisions. For students, accurate scoring and detailed feedback help them understand their learning achievements and shortcomings in a timely manner, enabling them to make targeted improvements and thereby enhance the overall teaching quality.

[0025] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0026] Figure 1 This is a flowchart of an embodiment of an automatic job evaluation method based on multi-agent parallel voting according to the present invention.

[0027] Figure 2 This is a schematic diagram of an embodiment of an automatic job evaluation system based on multi-agent parallel voting according to the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0029] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0030] An Example of an Automatic Job Evaluation Method Based on Multi-Agent Parallel Voting See Figure 1 This embodiment provides an automatic job evaluation method based on multi-agent parallel voting, including: The system receives the answers to the target assignments to be graded, converts the answers into a unified task format using an assignment distributor, and then distributes the converted tasks to a cluster of parallel scoring agents for processing. By utilizing multiple heterogeneous or homogeneous scoring models in a parallel scoring agent cluster, parallel scoring operations are performed on the answers to the target assignment, and each model outputs its own graded score and corresponding confidence index. The central aggregator collects the grading results from each agent, calculates the final grading score of the target job based on the preset dynamic weight allocation mechanism and outlier removal mechanism, and optimizes and adjusts the voting weights. The final graded score, along with an interpretable report including details of each grading model, is returned to the client or teacher via the results feedback mechanism. Simultaneously, online fine-tuning is performed based on teacher feedback on the grading results.

[0031] In the job dispatcher, the original text is first processed. Preprocessing is performed, then a globally unique identifier `task_id` and metadata (such as question, question type, difficulty, student ID, etc.) are generated for each answer, and the processing results are integrated into a message structure `{task_id, ...}`. The data is pushed to the message queue in the format of , metadata, and distributed to the parallel scoring agent cluster for processing.

[0032] Specifically, for the original text x During preprocessing, character filtering is performed first, using a preset set of legal characters to remove characters from the original text. x The text is then processed to identify all illegal characters not included in the set. Next, text standardization is performed, converting English words to lowercase and Chinese words from traditional to simplified characters. Then, grammar validation and correction are performed, marking grammatical errors in the text and attempting to automatically correct them based on a pre-defined grammar rule library.

[0033] When generating the globally unique identifier task_id, the system combines the current system's precise timestamp, accurate to the nanosecond level, with a random string generated based on a specific algorithm. The timestamp and the random string are combined according to a preset concatenation rule to ensure the uniqueness of task_id throughout the entire lifecycle of the system and across different job batches.

[0034] When generating metadata, question information is precisely matched and obtained from a pre-built and real-time updated question database. Question type is determined based on preset question type feature rules by analyzing the format, content structure, and other features of the original text x. Difficulty assessment is based on multi-dimensional data such as the average score rate and answering time of the question in historical homework data, and is calculated by a preset difficulty calculation model. Student ID is accurately obtained by connecting to the school's student information management system through a secure interface, based on the identity information of the student who submitted the homework.

[0035] In this embodiment, the parallel scoring agent cluster consists of It consists of agents containing heterogeneous or homogeneous scoring models. Each agent is independently deployed on isolated containers or different GPU nodes to perform inference tasks in parallel. When each agent receives a preprocessed answer task assigned by the job distributor, Answer The input is fed into its dedicated preprocessing pipeline, which processes the answer according to preset rules. After feature extraction and format adjustment, the raw scores are generated. ,and ∈[0,1]; Each agent calculates a confidence index based on the output distribution entropy. , ∈[0,1], expressed by the following formula:

[0036] in, , K For the preset number of categories or rating-related parameters, p i ( v ) represents an intelligent agent i The answer belongs to the first v Probability estimation of class or corresponding rating situation.

[0037] The entire system is deployed using the Ollama framework, which supports batch inference, enabling the processing of multiple jobs at once and effectively improving system throughput. It also supports low-latency real-time mode, meeting the needs of job grading scenarios with high real-time requirements. Each agent calculates the raw scores... and confidence index Pack the task identifier (task_id) into a data packet {task_id, , }, and send it to the central aggregator.

[0038] In this embodiment, the operation of the central aggregator module includes the following steps and rules: The central aggregator module receives data from... M The score vector of each model With confidence vector Afterwards, among them, s i For the first i The raw scores given by each agent. c i For the first i The confidence index calculated by each agent.i =1,..., M ; First, calculate the fractional vector. the median of Absolute deviation from the median ; Set anomaly detection threshold For each agent's score s i If satisfied The scoring result of the agent is then determined to be an outlier, and the corresponding model is removed from subsequent calculations. After removing outlier models, the central aggregator summarizes the scores of the remaining models and combines them with the confidence index corresponding to each model. c i The final score of the target task is calculated and generated using a weighted average or other preset aggregation algorithm.

[0039] For the remaining models that were not eliminated in the central aggregator, the weight calculation, final score aggregation, and weight tuning process includes: For each remaining model, read the historical error variance. And combined with the current confidence index of the model c i Calculate the initial weights, expressed as:

[0040] in, It is a very small positive number to avoid the case where the denominator is zero; The initial weights are normalized to obtain the final weights, which are expressed as follows:

[0041] Based on the calculated final weights of each remaining model and the corresponding raw scores s i The system's overall score is calculated using the weighted average formula and is expressed as follows:

[0042] After obtaining the system's overall score Then, map it to the desired range (e.g., 0-100 points). After aggregation, if there is manual scoring by teachers... Then the exponentially weighted mean squared error is updated immediately, expressed as:

[0043] by Used to adjust and optimize the weights of each model.

[0044] In this embodiment, the result feedback module will output the final calculated system comprehensive score. And the scoring details for each model, including the score for each model. Confidence index and weight The results are returned to the client or teacher in JSON format via a standardized API. { "task_id":"...","final_score":87.5, "breakdown":[{"model":"m1","score":0.89,"conf":0.92,"weight":0.35},…], "excluded":["m3"] } Meanwhile, the grading results are presented in the form of a visual report on the teacher interface. The visual report includes the system's overall score, charts showing the scoring of each model, and information on abnormal models, so that teachers can intuitively understand the details of the homework grading.

[0045] When teachers give a comprehensive score to the system Make corrections and submit a revised version. When the system includes the job sample and correction label in the online fine-tuning pipeline, it periodically triggers incremental training tasks according to preset cycles or trigger conditions. The scoring model is trained using the newly included sample data, realizing continuous closed-loop optimization and adaptive adjustment of the model, and continuously improving the accuracy and reliability of job grading.

[0046] In this embodiment, the scheduling and configuration of three scoring model agents—Qwen, Doubao, and DeepSeek—are performed on a Linux server, including: API Client Initialization: In a Linux server environment, using Python 3.8 as the programming language, and with the help of the Requests library and the OpenAI Python SDK (v0.27.0), the corresponding API clients for the three scoring model agents, Qwen, Doubao, and DeepSeek, were initialized respectively. During the initialization process, a unique access key was configured for each model agent, and the corresponding request parameters were set according to the characteristics and requirements of each model to ensure that requests could be sent to each model correctly and responses could be received. Asynchronous call management: All API calls to these three scoring model agents are managed using asynchronous coroutines (asyncio). Asyncio enables concurrent request processing, improving system resource utilization and overall response efficiency, allowing the system to interact with multiple model agents simultaneously and reducing waiting time. Call strategy settings: Set the timeout for API calls to K, meaning that if no response is received from the model agent within K days from the start of the request, the request is considered to have timed out; set the number of retries to k, so that the system will automatically retry when network jitter or service unavailability causes the request to fail, thereby increasing the probability of successful requests while ensuring stable system response and guaranteeing the smooth progress of the job grading process.

[0047] On a Linux server, Docker container technology is used to create an independent runtime environment for each agent. First, the Docker engine is installed, and then the corresponding Docker images are customized according to the operating system version, library files and runtime environment requirements that each scoring model agent depends on. For example, for scoring models that depend on specific Python versions and deep learning frameworks, the corresponding Python version and the required deep learning libraries (such as TensorFlow, PyTorch, etc.) are pre-installed in the Docker image.

[0048] The code, configuration files, and related data of each scoring model agent are packaged into a corresponding Docker container. The containers are then started using the Docker command, ensuring that each scoring model agent runs independently within a container, isolated from each other, thus avoiding operational issues caused by resource contention or environmental conflicts. For example, the container is started using the `docker run` command, specifying resource limits (such as the number of CPU cores and memory size) to ensure stable operation of each agent.

[0049] To enable the job dispatcher to communicate with the scoring model agent deployed in containers, the Docker container network mode is configured to use bridged networking. Each container is assigned an independent IP address and port, allowing the job dispatcher to send jobs to be graded to the corresponding container via the specified IP address and port. Simultaneously, a corresponding network listening service is configured inside the container to receive and process requests from the job dispatcher.

[0050] Ensure that the appropriate GPU driver and CUDA (Compute Unified Device Architecture) toolkit are installed on the Linux server to support accelerated computation of the scoring model agent on the GPU. Based on the server's GPU model and the requirements of the scoring model, install the corresponding versions of the GPU driver and CUDA toolkit, and configure them correctly so that the operating system can recognize and utilize GPU resources.

[0051] Based on the computational requirements of the scoring model agents and the resource availability of the GPU nodes, the N scoring model agents are rationally allocated to different GPU nodes. Agents with high computational demands and high GPU resource requirements are assigned to higher-performance GPU nodes; agents with relatively low computational demands can be assigned to lower-performance GPU nodes to optimize resource utilization. The code and related data of the scoring model agents are deployed to the corresponding GPU nodes and configured as necessary to enable them to correctly utilize GPU resources for inference computation.

[0052] On each GPU node, a task scheduling tool (such as Slurm) is used to manage the inference tasks of the scoring model agents. The task scheduler rationally arranges the execution order of tasks based on the resource usage of the GPU node and the priority of the tasks, ensuring that multiple scoring model agents can execute inference tasks in parallel on the GPU node, thus improving overall computational efficiency. For example, when multiple scoring model agents need to use GPU resources simultaneously, the task scheduler will allocate appropriate GPU computing resources to each agent according to a preset strategy, enabling them to perform inference computations in parallel.

[0053] The scoring model agent, deployed on containers or GPU nodes, receives assignments to be graded from the assignment distributor via a pre-configured network interface. The assignments are encapsulated in a specific data format (such as JSON) and contain assignment identification information, question content, student answers, and other relevant data. Upon receiving the assignment, the agent parses the data, extracts key information that needs processing, and prepares for subsequent inference calculations.

[0054] Based on the parsed assignment information, the scoring model agent invokes its pre-trained internal model for inference computation. For heterogeneous models, different models may employ different algorithm architectures and computational methods, but all aim to provide corresponding scoring results based on the input student answers. During the inference computation process, the agent fully utilizes the computing resources provided by containers or GPU nodes to accelerate the model's computation process and improve inference efficiency. After the scoring model agent completes its inference calculations, it encapsulates the obtained scoring results and related confidence indicators, and returns them to the central aggregator via a network interface according to a pre-agreed communication protocol and data format. The returned result data is also organized in formats such as JSON, containing key information such as job identifier, scoring score, and confidence level, so that the central aggregator can perform subsequent processing and aggregation.

[0055] An embodiment of an automatic job evaluation system based on multi-agent parallel voting like Figure 2 As shown, this embodiment provides an automatic job evaluation system based on multi-agent parallel voting, comprising: The assignment distributor is used to receive the answers to the target assignments to be graded and push the received answers to each agent in the parallel scoring agent cluster according to preset rules and strategies, providing data input for the parallel scoring operation. Parallel scoring agent cluster: consists of N heterogeneous or homogeneous scoring models. Each scoring model acts as an independent agent, used to independently analyze and process the received target homework answers, and output the corresponding grading score and a confidence index reflecting the reliability of the scoring. The central aggregator is used to collect the grading scores and confidence information from each agent in the parallel scoring agent cluster. Based on a pre-set dynamic weight allocation mechanism and outlier removal mechanism, it comprehensively processes the collected information and calculates the final grading score of the target job. At the same time, the central aggregator also calculates historical feasibility based on historical data and uses this historical feasibility to optimize and adjust the voting weights of each agent in order to continuously improve the scoring performance of the system. Results Feedback Unit: This unit returns the final graded scores and generated interpretable reports calculated by the central aggregator to the client or teacher, enabling users to clearly understand the grading status of assignments. Furthermore, the Results Feedback Unit also feeds back teacher-annotated data to the system for online fine-tuning of the grading model, further improving the model's adaptability to different types of assignments and grading accuracy.

[0056] In practical applications, this embodiment schedules three scoring model agents—Qwen, Doubao, and DeepSeek—on a Linux server by calling the API interfaces provided by OpenAI. The system uses Python 3.8, the Requests library, and the OpenAI Python SDK (v0.27.0) to initialize the API client for each model, configuring the corresponding keys and request parameters. All API calls are managed using asynchronous coroutines (asyncio), with a 2-second timeout and a 2-retry strategy to ensure stable response even during network jitter or service unavailability.

[0057] Student answer text After being uploaded to the teaching platform, the assignment distributor sends grading requests to the three OpenAI API endpoints—Qwen, Doubao, and DeepSeek—asynchronously and concurrently. An example request body is shown below: {"model":"qwen","input": x, "parameters": {...}} Each model returns a score pair Among them, confidence level Calculated using output distribution entropy:

[0058] The call sets a timeout threshold of 2 seconds and retryes once on failure. After obtaining the three scores, it immediately proceeds to aggregation.

[0059] The central aggregation script reads the historical variance of each model from a local file. And scored on three routes. Perform median-MAD outlier removal:

[0060] Eliminate satisfied For the model that retains the model, calculate the dynamic weights:

[0061] And obtain a comprehensive score. .

[0062] In this embodiment, the system uses a REST API to deliver the final score. and each model The details are returned to the teaching platform. After the tens of thousands of samples have been reviewed by the teachers, they are automatically merged into the fine-tuning dataset and online incremental training is triggered in batches on the Bailian platform to fine-tune the agents and continuously improve the system's adaptive capabilities.

[0063] In summary, the present invention proposes the following innovative points: 1. Introducing multi-model voting technology—borrowing the bagging concept from ensemble learning, multiple heterogeneous or homogeneous scoring models with independent parameters are deployed to score the same answer independently and in parallel. By voting on or averaging the outputs of each model, the scoring variance of a single model can be significantly reduced, improving the accuracy and robustness of the overall scoring and avoiding extreme scoring errors caused by the failure of individual models.

[0064] 2. A dynamic weighting and trust mechanism is designed—based on multi-model voting technology, more specialized models for vertical questions (such as text-based question answering models and mathematical evaluation models) are introduced and given higher base weights. Simultaneously, the confidence level of each model for the current answer is calculated in real time, and the final weight is dynamically adjusted based on its past accuracy. Models with high reliability and good performance play a crucial role in scoring, while models with lower reliability are automatically weakened. This mechanism further optimizes the model's contribution to different question types and answers, resulting in higher score stability and interpretability.

[0065] This invention effectively solves the problems of single-model failure, large score differences, and lack of interpretability in existing automatic scoring systems through the above-mentioned multi-model voting and dynamic weighting design, and greatly improves the scoring accuracy of complex question types.

[0066] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0067] The above embodiments are merely preferred embodiments of the present invention and should not be construed as limiting the scope of protection of the present invention. Any non-substantial changes and substitutions made by those skilled in the art based on the present invention shall fall within the scope of protection claimed by the present invention.

Claims

1. An automatic job evaluation method based on multi-agent parallel voting, characterized in that, Includes the following steps: The system receives the answers to the target assignments to be graded, converts the answers into a unified task format using an assignment distributor, and then distributes the converted tasks to a cluster of parallel scoring agents for processing. By utilizing multiple heterogeneous or homogeneous scoring models in a parallel scoring agent cluster, parallel scoring operations are performed on the answers to the target assignment, and each model outputs its own graded score and corresponding confidence index. The central aggregator collects the grading results from each agent, calculates the final grading score of the target job based on the preset dynamic weight allocation mechanism and outlier removal mechanism, and optimizes and adjusts the voting weights. The final graded score, along with an interpretable report including details of each grading model, is returned to the client or teacher via the results feedback mechanism. Simultaneously, online fine-tuning is performed based on teacher feedback on the grading results.

2. The method according to claim 1, characterized in that: In the job dispatcher, the original text is first processed. Preprocessing is performed, then a globally unique identifier (task_id) and metadata are generated for each answer, and the processing results are integrated into a message structure {task_id, The data is pushed to the message queue in the format of , metadata, and distributed to the parallel scoring agent cluster for processing.

3. The method according to claim 2, characterized in that: For the original text x During preprocessing, character filtering is performed first, using a preset set of legal characters to remove characters from the original text. x The text is then processed to identify all illegal characters not included in the set; next, text standardization is performed, converting English words to lowercase and Chinese words from traditional to simplified characters; then, grammar validation and correction are performed, marking grammatical errors in the text and attempting to automatically correct them based on a pre-defined grammar rule library. When generating a globally unique identifier task_id, the system combines the current system's precise timestamp with a random string generated based on a specific algorithm. The timestamp and the random string are then combined according to a preset concatenation rule to ensure the uniqueness of task_id throughout the entire lifecycle of the system and across different job batches. When generating metadata, question information is retrieved by matching from a pre-built and real-time updated question database. Question type determination is based on preset question type feature rules and is made through multi-feature data analysis. The multi-feature data includes at least the original text. x The format, content structure, etc., are evaluated based on multi-dimensional data calculated by a preset difficulty calculation model. The multi-dimensional data includes at least the average score rate and answering time of the question in the historical homework data. The student ID is obtained by connecting to the school's student information management system through a secure interface and based on the student's identity information when submitting the homework.

4. The method according to claim 1, characterized in that: Parallel scoring agent clusters are composed of It consists of agents containing heterogeneous or homogeneous scoring models. Each agent is independently deployed on isolated containers or different GPU nodes to perform inference tasks in parallel. When each agent receives a preprocessed answer task assigned by the job distributor, Answer The input is fed into its dedicated preprocessing pipeline, which processes the answer according to preset rules. After feature extraction and format adjustment, the raw scores are generated. ,and ∈[0,1]; Each agent calculates a confidence index based on the output distribution entropy. , ∈[0,1], expressed by the following formula: in, , K For the preset number of categories or rating-related parameters, p i ( v ) represents an intelligent agent i The answer belongs to the first v Probability estimation of class or corresponding rating situation.

5. The method according to claim 4, characterized in that, The operation of the central aggregator module includes the following steps and rules: The central aggregator module receives data from... M Fractional vectors of agents With confidence vector Afterwards, among them, s i For the first i The raw scores given by each agent. c i For the first i The confidence index calculated by each agent. i =1,..., M ; First, calculate the fractional vector. the median of Absolute deviation from the median ; Set anomaly detection threshold For each agent's score s i If satisfied The scoring result of the agent is then determined to be an outlier, and the corresponding model is removed from subsequent calculations. After removing outlier models, the central aggregator summarizes the scores of the remaining models and combines them with the confidence index corresponding to each model. c i The final score of the target task is calculated and generated using a weighted average or other preset aggregation algorithm.

6. The method according to claim 5, characterized in that: For the remaining models that were not eliminated in the central aggregator, the weight calculation, final score aggregation, and weight tuning process includes: For each remaining model, read the historical error variance. And combined with the current confidence index of the model c i Calculate the initial weights, expressed as: in, It is a very small positive number to avoid the case where the denominator is zero; The initial weights are normalized to obtain the final weights, which are expressed as follows: Based on the calculated final weights of each remaining model and the corresponding raw scores s i The system's overall score is calculated using the weighted average formula and is expressed as follows: After obtaining the system's overall score Then, map it to the desired interval; After aggregation, if there is manual scoring by teachers... Then the exponentially weighted mean squared error is updated immediately, expressed as: by Used to adjust and optimize the weights of each model.

7. The method according to claim 6, characterized in that: The result feedback module will ultimately calculate the system's overall score. And the scoring details for each model, including the score for each model. Confidence index and weight The grading results are presented in a JSON structure via a standardized API, and the results are displayed as a visual report on the teacher's interface. When teachers give a comprehensive score to the system Make corrections and submit a revised version. When the system includes the job sample and correction label in the online fine-tuning pipeline, it periodically triggers incremental training tasks according to preset cycles or trigger conditions, and uses the newly included sample data to train the scoring model, thereby achieving continuous closed-loop optimization and adaptive adjustment of the system.

8. The method according to any one of claims 1 to 7, characterized in that, The scheduling and configuration of three scoring model agents—Qwen, Doubao, and DeepSeek—are performed on a Linux server, including: API Client Initialization: In a Linux server environment, using Python 3.8 as the programming language, and with the help of the Requests library and the OpenAI Python SDK, the corresponding API clients for the three scoring model agents, Qwen, Doubao, and DeepSeek, were initialized. During the initialization process, a unique access key was configured for each model agent, and the corresponding request parameters were set according to the characteristics and requirements of each model to ensure that requests could be sent to each model correctly and responses could be received. Asynchronous call management: All API calls to these three scoring model agents are managed using asynchronous coroutines (asyncio). Call strategy settings: Set the timeout for API calls to K, meaning that if no response is received from the model agent within K days from the start of the request, the request is considered to have timed out; set the number of retries to k, so that the system will automatically retry when the request fails due to network jitter or service unavailability.

9. The method according to any one of claims 1 to 7, characterized in that: On a Linux server, Docker container technology is used to create an independent runtime environment for each agent. First, the Docker engine is installed, and then the corresponding Docker images are customized according to the operating system version, library files and runtime environment requirements that each scoring model agent depends on. The code, configuration files, and related data of each scoring model agent are packaged into a corresponding Docker container, and the container is started using the Docker command, so that each scoring model agent runs in an independent container. To enable the job dispatcher to communicate with the scoring model agent deployed in containers, the network mode of the Docker containers is configured to adopt the bridged network mode, and each container is assigned an independent IP address and port, so that the job dispatcher can send the job tasks to be graded to the corresponding container through the specified IP address and port. Based on the computational requirements of the scoring model agents and the resource availability of the GPU nodes, N scoring model agents are rationally allocated to different GPU nodes; the code and related data of the scoring model agents are deployed to the corresponding GPU nodes; on each GPU node, a task scheduling tool is used to manage the inference tasks of the scoring model agents, and the task scheduling tool arranges the execution order of tasks according to the resource usage of the GPU nodes and the priority of the tasks. The scoring model agent deployed on a container or GPU node receives the job tasks to be graded from the job distributor through a pre-configured network interface; after receiving the task, the agent parses the task data and extracts the key information that needs to be processed. Based on the parsed job information, the agent calls the pre-trained model to perform inference calculations. After completing the inference calculations, the agent encapsulates the obtained score results and related confidence indicators, and returns them to the central aggregator through the network interface according to the pre-agreed communication protocol and data format.

10. An automatic job evaluation system based on multi-agent parallel voting, characterized in that, include: The assignment distributor is used to receive the answers to the target assignments to be graded and push the received answers to each agent in the parallel scoring agent cluster according to preset rules and strategies, providing data input for the parallel scoring operation. Parallel scoring agent cluster: consists of N heterogeneous or homogeneous scoring models. Each scoring model acts as an independent agent, used to independently analyze and process the received target homework answers, and output the corresponding grading score and a confidence index reflecting the reliability of the scoring. Central aggregator: Used to collect the grading scores and confidence information fed back by each agent in the parallel scoring agent cluster. According to the pre-set dynamic weight allocation mechanism and outlier removal mechanism, the central aggregator performs comprehensive processing on the collected information and calculates the final grading score of the target job. At the same time, the central aggregator also calculates the historical feasibility based on historical data and uses the historical feasibility to optimize and adjust the voting weights of each agent. Results Feedback Unit: Returns the final graded scores calculated by the central aggregator and the generated interpretable report to the client or teacher. Furthermore, the results feedback device will also feed back the teacher-annotated data to the system for online fine-tuning of the scoring model.

Images