Algorithm exercise knowledge point automatic labeling method and system based on multi-agent cooperation

Abstract

The invention provides an algorithm exercise knowledge point automatic labeling method and system based on multi-agent cooperation, and belongs to the field of artificial intelligence and educational information, and the method comprises the steps: S1, inputting a to-be-labeled algorithm exercise into a plurality of labeling expert agents, and obtaining a preliminary knowledge point labeling set outputted by each labeling expert agent; s2, jointly inputting the preliminary knowledge point labeling set and a predefined standard knowledge point set into an alignment expert agent to obtain an alignment knowledge point labeling set; s3, inputting the aligned knowledge point labeling set into a review expert agent to obtain a knowledge point labeling result; and S4, taking the knowledge point labeling result and the to-be-labeled algorithm exercises as input, and repeatedly executing the steps S1 to S3 until the maximum number of iterations is reached, so as to obtain a final knowledge point labeling result. According to the method, comprehensive and accurate marking of the knowledge points is realized through accurate distinguishing of the knowledge points and deep analysis of complex problem solving logic.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of artificial intelligence and education information, and particularly relates to an algorithm exercise knowledge point automatic labeling method and system based on multi-agent cooperation. BACKGROUND

[0002] Online programming practice platforms accumulate a large amount of algorithm exercises. The knowledge points examined by these exercises are the core metadata for building personalized learning paths and recommendation systems. However, current platforms generally face the problem of missing, incomplete, or inconsistent exercise knowledge point labeling.

[0003] Existing automatic labeling methods are mainly based on the content of the exercise question text. There are significant defects: first, it is difficult to identify knowledge points that are implied in the problem-solving logic but not explicitly stated in the question; second, it has poor understanding of non-traditional natural language content such as formulas or symbols; finally, in the fine-grained, multi-label scenario, it lacks the ability to distinguish between highly similar knowledge points.

[0004] Large language models provide a new way for automatic labeling, but single models have problems such as unstable output, redundant labels, and poor standardization in complex tasks, making it difficult to directly meet the high requirements of label consistency for downstream applications. Therefore, there is an urgent need for an automatic algorithm exercise knowledge point labeling method based on multi-agent cooperation to obtain more accurate and comprehensive knowledge point labeling results. SUMMARY

[0005] To solve the above technical problems, the application provides an automatic algorithm exercise knowledge point labeling method based on multi-agent cooperation, comprising the following steps:

[0006] Step S1: input the algorithm exercise to be labeled into a labeling expert agent group composed of multiple labeling expert agents, and obtain a preliminary knowledge point labeling set output by each labeling expert agent based on the analysis of the problem-solving process of the algorithm exercise;

[0007] Step S2: input the preliminary knowledge point labeling set and the pre-defined standard knowledge point set into the alignment expert agent, perform semantic matching and granularity standardization processing by the alignment expert agent, and obtain an aligned knowledge point labeling set;

[0008] Step S3: input the aligned knowledge point labeling set into the review expert agent, and perform screening and integration by the review expert agent to obtain a knowledge point labeling result;

[0009] Step S4: input the knowledge point labeling result and the algorithm exercise to be labeled as input, repeat steps S1-S3 until the maximum number of iterations is reached, and obtain the final knowledge point labeling result.

[0010] The application provides a kind of algorithm exercise knowledge point automatic labeling method based on multi-agent cooperation has following beneficial effects:

[0011] 1, deep analysis and high recall rate: the application can mine the implicit knowledge points not directly embodied in the question text by simulating the problem solving process of the labeling expert agent, improving the comprehensiveness and recall rate of labeling.

[0012] 2, labeling specification and high accuracy: the application ensures the consistency of the output result and the predefined standard set by aligning the semantic mapping and granularity of the expert agent, effectively eliminates the redundancy and ambiguity of expression, improves the accuracy and usability of labeling.

[0013] 3, collaborative decision making and strong robustness: the application provides multiple perspectives through multiple expert agent initial labeling, then integrates the decision of review expert, and combines dropout mechanism, reduces the sensitivity to single model output noise, enhances the stability and robustness of the overall method.

[0014] 4, professional division of labor and high performance: the application fine-tunes the agents of different roles, and each agent is more professional in its own task, so that the overall labeling performance is better than that of general large model or single model, especially in fine-grained knowledge point distinction. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 It is a kind of algorithm exercise knowledge point automatic labeling method based on multi-agent cooperation of the application;

[0016] Figure 2 It is a kind of algorithm exercise knowledge point automatic labeling method based on multi-agent cooperation of the application;

[0017] Figure 3 It is a kind of algorithm exercise knowledge point automatic labeling method based on multi-agent cooperation of the application;

[0018] Figure 4 It is a kind of algorithm exercise knowledge point automatic labeling method based on multi-agent cooperation of the application;

[0019] Figure 5 It is a kind of algorithm exercise knowledge point automatic labeling method based on multi-agent cooperation of the application;

[0020] Figure 6 It is a kind of algorithm exercise knowledge point automatic labeling method based on multi-agent cooperation of the application; DETAILED DESCRIPTION

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0022] Example 1

[0023] like Figure 1 As shown in the figure, an automatic annotation method for algorithm exercises based on multi-agent cooperation provided by an embodiment of the present invention includes the following steps:

[0024] Step S1: Input the algorithm problem to be labeled into a group of labeling expert agents consisting of multiple labeling expert agents, and obtain the preliminary set of knowledge point labels output by each labeling expert agent after analyzing the problem-solving process of the algorithm problem;

[0025] Step S2: Input the preliminary knowledge point annotation set and the predefined standard knowledge point set into the alignment expert agent, which will then perform semantic matching and granular normalization to obtain the aligned knowledge point annotation set.

[0026] Step S3: Input the aligned knowledge point annotation set into the review expert agent, which will then filter and integrate the data to obtain the knowledge point annotation results.

[0027] Step S4: Take the knowledge point annotation results and the algorithm exercises to be annotated as input, and repeat steps S1 to S3 until the maximum number of iterations is reached to obtain the final knowledge point annotation results.

[0028] In one embodiment, step S1 above: inputting the algorithm problem to be labeled into a group of labeling expert agents composed of multiple labeling expert agents, and obtaining the preliminary knowledge point labeling set output by each labeling expert agent after analyzing the problem-solving process of the algorithm problem, specifically including:

[0029] Step S11: Obtain an algorithm exercise training dataset with labels of real knowledge points;

[0030] This invention relates to algorithm problem annotation based on the Codeforces dataset. 9,775 programming problems were collected from the Codeforces platform via its official API, containing 36 predefined knowledge point tags, as shown in Table 1. This invention uses the set of all knowledge point tags defined by the platform as the predefined knowledge point set for Embodiment 1.

[0031] Table 1. Predefined knowledge point tags for Codeforces datasets

[0032] The data set is divided into a training set, a validation set and a test set in a ratio of 7:2:1.

[0033] Step S12: clustering the pre-defined standard knowledge point set to form N knowledge point classifications;

[0034] In the embodiment of the application, the definition text of each knowledge point of the standard knowledge point set is modeled by using the BERT model, the generated embedding is pooled, and is converted into a feature vector of each knowledge point label. Then, according to the number N=3 of the labeling expert agent, the K-means algorithm is used to cluster the knowledge points to form three knowledge point classifications.

[0035] Step S13: dividing the algorithm exercises in the algorithm exercise training data set into corresponding knowledge point classifications according to their real knowledge point labels;

[0036] All exercises are classified into corresponding classifications according to their knowledge point labels. If an exercise can be classified into several classifications at the same time, the exercise is classified into each corresponding classification.

[0037] Step S14: using the algorithm exercises classified into different knowledge point classifications to respectively fine-tune different labeling expert agents to obtain a group of fine-tuned labeling expert agents;

[0038] The preset role prompt words for the labeling expert agent are designed by the application, as shown in Table 2:

[0039] Table 2 Labeling expert agent prompt words

[0040] The fine-tuning of the labeling expert agent adopts a generative loss function:

[0041] ;

[0042] wherein, is an input text sequence, denotes the Sth token in the input sequence; is a target output knowledge point sequence, denotes the Tth token in the target sequence; denotes the probability of generating the correct token at position under the condition that the given input sequence and the sequence already generated before;

[0043] Step S15: Input the algorithm exercises to be labeled into the fine-tuned group of labeling expert agents, so that each labeling expert agent focuses on labeling the algorithm exercises of its corresponding category, and constructs a preliminary set of knowledge point labels.

[0044] This invention constructs a preliminary set of knowledge point annotations by having each annotation expert agent construct the annotation set based on their understanding of the questions and simulated problem-solving processes. By leveraging the differences in perspectives among different experts and the randomness of the large model, diverse preliminary knowledge point annotation results are generated to comprehensively cover the knowledge points involved in the exercises.

[0045] In one embodiment, step S2 above, which involves inputting the preliminary knowledge point annotation set and the predefined standard knowledge point set into the alignment expert agent, and having the alignment expert agent perform semantic matching and granular normalization processing to obtain the aligned knowledge point annotation set, specifically includes:

[0046] Step S21: For each standard knowledge point in the standard knowledge point set, generate multiple representation variants, including synonym substitution and granular extension types;

[0047] In this embodiment of the invention, the GPT-3.5-Turbo API is used to generate several variants for each knowledge point in a predefined set of knowledge points, including synonym substitution, coarse / fine granular extension, and other types.

[0048] Step S22: Use randomly combined variant expressions as input and corresponding standard knowledge points as target labels to construct an alignment fine-tuning training dataset;

[0049] Step S23: Using the alignment fine-tuning training dataset, fine-tune the alignment expert agent using semantic mapping and granular normalization fine-tuning strategies so that it learns to map knowledge points with different expressions to standard knowledge points, and obtain the fine-tuned alignment expert agent.

[0050] This invention has designed preset role prompts specifically for the alignment expert agent, as shown in Table 3:

[0051] Table 3 Alignment Expert Agent Prompts

[0052] The fine-tuning of the alignment expert agent uses the cross-entropy loss function of multi-label classification:

[0053] ;

[0054] Where C is the size of the predefined set of standard knowledge points (i.e., the total number of predefined knowledge point labels); This is the unnormalized logits vector output by the model; For multi-hot encoding of the real label vector, if the first... if the knowledge point exists in the real label, then 0 otherwise; is a Sigmoid activation function, and the logit value corresponding to each knowledge point is independently mapped into a probability value;

[0055] Step S24: input the preliminary knowledge point annotation set and the standard knowledge point set into the alignment expert agent for alignment processing to obtain an aligned knowledge point annotation set.

[0056] The alignment expert agent of the present application is responsible for semantic matching of the knowledge points in the preliminary knowledge point annotation result with the standard knowledge point set, replacing the knowledge points with synonymous different expressions or existing coarse / fine-grained extension relationship with the standard names in the standard set, and eliminating the items that cannot be matched, and outputting the standardized aligned knowledge point annotation set.

[0057] In one embodiment, the step S3 described above: input the aligned knowledge point annotation set into the review expert agent, and the review expert agent is used for screening and integration to obtain a knowledge point annotation result, which specifically includes:

[0058] Step S31: for the real knowledge point label of the algorithm exercise in the algorithm exercise training data set, randomly add irrelevant knowledge points to construct a to-be-screened knowledge point set containing noise;

[0059] Step S32: taking the to-be-screened knowledge point set and the corresponding algorithm exercise content as input, and taking the real knowledge point label as the target, a review fine-tuning training data set is constructed;

[0060] Step S33: fine-tune the review expert agent using the review fine-tuning training data set, so that it learns the ability to screen the correct result from multiple candidate knowledge points, thereby obtaining a fine-tuned review expert agent;

[0061] The present application designs preset role prompt words for the review expert agent, as shown in Table 4:

[0062] Table 4: Review expert agent prompt words

[0063] The fine-tuning of the review expert agent of the present application adopts the same cross-entropy loss function as the alignment expert agent.

[0064] Step S34: after dropout processing of the aligned knowledge point annotation set, input the fine-tuned review expert agent to obtain a knowledge point annotation result;

[0065] The present application carries out dropout processing on the aligned knowledge point annotation set, randomly discards a part of the annotation results, and then inputs the remaining results into the fine-tuned review expert agent. The review expert agent filters out the correct knowledge points from multiple candidate results based on domain knowledge and pre-set review rules (such as excluding irrelevant items, retaining correct items, and merging complementary descriptions), and integrates to form the final accurate and complete annotation results.

[0066] The annotation expert agent, the alignment expert agent and the review expert agent of the present application are based on the same base large language model, but realize role division and ability strengthening through targeted professional fine-tuning strategy.

[0067] The three agents of the present application are based on the GLM-4-32B-Base-0414 base large language model, and are fine-tuned through Low-Rank Adaptation (LoRA) technology. The specific parameter settings are shown in Table 5:

[0068] Table 5 Fine-tuning parameter settings of agent

[0069] In one embodiment, the above step S4: taking the knowledge point annotation result and the algorithm exercise to be annotated as input, repeatedly executing steps S1-S3 until the maximum iteration number is reached, and obtaining the final knowledge point annotation result.

[0070] In order to comprehensively evaluate the performance of the method on the knowledge point annotation task, the present application uses four indexes of accuracy (Precision), recall (Recall), micro average F1 value (F1-micro) and macro average F1 value (F1-macro) for evaluation, which are defined as follows:

[0071] ;

[0072] ;

[0073] Among them, represents a set of real positive samples, represents a set of predicted positive samples.

[0074] The micro average F1 value is calculated by globally counting true positives (TP), false positives (FP) and false negatives (FN):

[0075] ;

[0076] And the macro average F1 value is the arithmetic average of the F1-score of each class:

[0077] ;

[0078] In the above two formulas, C is the size of the predefined knowledge point set, represents a certain specific knowledge point, and the F1-score is the harmonic mean of precision and recall.

[0079] In the classification task, accuracy, recall, and F1 value are generally the core indicators for evaluating model performance. Accuracy reflects the accuracy of the model in predicting positive samples, and recall measures the coverage ability of the model for true positive samples. On this basis, micro-average F1 value and macro-average F1 value are more in line with the characteristics of multi-label tasks. Micro-average F1 value is calculated by globally counting TP, FP, and FN, which is more suitable for scenarios where samples are evenly distributed in each class. Macro-average F1 value takes the arithmetic mean of F1 values for each class, which can effectively capture the processing ability of the model for minority classes in the case of class imbalance. In the knowledge point labeling task, micro-average F1 can evaluate the overall labeling accuracy, macro-average F1 can detect the coverage ability of the model for unpopular knowledge points, and F1-score can reflect the comprehensive quality of single knowledge point labeling.

[0080] Observation Figure 4 The performance experiment results of the knowledge point labeling method under different numbers of labeling experts show that when the number of labeling experts N increases from 1 to 3, the F1-micro index increases from 0.49 to 0.576, which indicates that the proposed multi-agent collaborative labeling method enhances the coverage ability of different knowledge through complementary labeling results generated by different labeling experts. When N exceeds 3, the performance of the method improves little, and the complementary gain brought by multi-expert labeling has almost saturated.

[0081] Observation Figure 5 The performance experiment results of the knowledge point labeling method under different numbers of iterations. In the first round, multiple labeling expert agents generate preliminary knowledge point labeling sets from different perspectives, using their professional division of labor and randomness to cover as many explicit and implicit knowledge points as possible that may be involved in the exercises. In the second iteration, the knowledge point labeling results of the first round and the original exercises are input into the agent collaboration process again, and each labeling expert will refer to the labeling results of the previous round to make supplementary or adjustments according to the requirement in the prompt word that "use the labeling results of the previous round as a reference." This achieves accurate selection and consistency enhancement of knowledge points, mainly solving the calibration and optimization problem of labeling results. Through continuous optimization and iteration of the multi-agent collaboration framework, different labeling experts can understand the labeling results of other experts, thereby achieving the effect of collecting ideas from all walks of life. When the number of iterations num_epoch increases to 2, the overall performance of the method has almost reached the best, and considering the dramatic increase in time and resource cost with each additional iteration, the overall cost performance is also declining.

[0082] The two-round iteration mechanism achieves full coverage in the first round and precise calibration in the second round, fully leveraging the closed-loop advantages of "division of labor-feedback-optimization" in multi-agent collaboration.

[0083] As shown in Table 6, a comparative experiment was designed to compare the performance of the method with six different baseline methods (GLM-4-air, GPT-4o, GPT-4.1, etc.) in accuracy, recall, micro-average F1 value, and macro-average F1 value on the Codeforces dataset.

[0084] Table 6 Performance comparison of knowledge point labeling methods on Codeforces dataset

[0085] On the Codeforces dataset, the accuracy (0.6800), recall (0.4995), micro-average F1 value (0.5760), and macro-average F1 value (0.5758) of the method were significantly better than those of other baseline models, including large language models (GLM-4-air, GPT-4o, GPT-4.1) and traditional deep learning models (TextCNN with Attention, AC-BiLSTM, PBAM), indicating that it has stronger comprehensive performance in fine-grained knowledge point recognition.

[0086] The method solves the key problems of traditional automated labeling methods in handling implicit knowledge points, distinguishing fine-grained similar knowledge points, and having low accuracy, poor consistency, and difficulty in direct use for downstream tasks. The specific innovations can be divided into the following three levels:

[0087] A collaboration system composed of three types of role agents, "labeling experts", "alignment experts", and "review experts", was constructed, and a multi-round iteration and self-optimization closed-loop workflow was designed. This simulates the high-quality collaboration process of human expert teams "initial labeling->standard alignment->cross-auditing->feedback correction", rather than a single generation of a single model.

[0088] For different role agents, a differentiated fine-tuning strategy was designed, rather than just using prompt words to distinguish roles:

[0089] For labeling expert agents, different experts focus on different categories of exercises. The categories of exercises are determined by the number N of labeling experts using the K-means algorithm. All exercises are classified into corresponding categories according to their knowledge point labels. Labeling experts use the exercise data corresponding to a knowledge point category and the true knowledge point labels to train and become "domain experts", improving their ability to distinguish complex and fine-grained knowledge points.

[0090] For the alignment expert agent, the model is trained to accurately map non-standard expressions to the predefined knowledge point system by generating synonyms and fine-grained variants of knowledge points, ensuring consistency in the output.

[0091] For the review expert agent, the agent is trained to filter out the correct item from multiple candidate results that may contain errors by adding noise to the true label, achieving denoising and fusion of the results.

[0092] Before the labeled results are passed to the review expert, a "dropout" mechanism is introduced to randomly discard part of the preliminary results of the labeling experts.

[0093] Example 1 For the three agents after fine-tuning, as shown in Figure 2 the following operations are performed:

[0094] 1. Input the algorithm problem to be labeled "Given an integer n, how many binary search trees with n nodes and node values from 1 to n that are mutually different" into the labeling expert agent group, including three labeling expert agents:

[0095] Labeling expert agent 1 analyzes that the problem needs to calculate the number of binary search trees of different structures through dynamic programming, identifies that the core is a combination counting problem, and outputs: ["dp", "combinatorics", "math"].

[0096] Labeling expert agent 2 analyzes the counting problem of tree data structure from the structural characteristics of binary search tree, and outputs: ["trees", "data structures", "divide and conquer"].

[0097] Labeling expert agent 3 analyzes that the problem needs to decompose subproblems and combine solutions from the algorithm design perspective of problem solving, and outputs: ["dp", "trees", "math"].

[0098] Construct a preliminary knowledge point labeling set from the outputs of the three labeling expert agents.

[0099] 2. Input the preliminary knowledge point labeling set and the predefined standard knowledge point set into the alignment expert agent, and the reasoning process is as follows:

[0100] All preliminary knowledge point labeling results are within the predefined knowledge point set, and no synonym replacement is needed.

[0101] Check the consistency of knowledge point granularity and confirm that all labels meet the predefined standards.

[0102] Merge duplicate labels: "dp", "trees", "math" appear twice, merge them into one.

[0103] Output aligned knowledge point annotation set: [“dp”,“combinatorics”,“math”,“trees”,“datastructures”,“divide and conquer”].

[0104] Then, the aligned knowledge point annotation set is subjected to dropout processing, and the annotation results of the annotation expert agent 2 are randomly discarded, and the remaining knowledge points are the aligned knowledge point annotation set obtained after alignment by the annotation expert agent 1 and the annotation expert agent 3: [“dp”,“combinatorics”,“math”,“trees”].

[0105] 3、Input the aligned knowledge point annotation set into the review expert agent. The multi-label classification reasoning process of the review expert agent is as shown in Figure 3 , which includes:

[0106] The input text is converted into token ID, position encoding, and attention mask by Tokenizer;

[0107] The hidden state is outputted through the multi-layer Transformer architecture of the large language model;

[0108] The hidden state is pooled and mapped to the classification space through a linear layer;

[0109] The logits are mapped to the interval [0, 1] using the Sigmoid function, and the class higher than the threshold value 0.5 is selected.

[0110] The review expert agent filters and integrates based on domain knowledge:

[0111] Confirm that “dp” is the core solution to the problem (calculate the number of binary search trees corresponding to different n values through dynamic programming); “combinatorics” accurately describes the essence of the problem of combination counting; “math” covers the mathematical modeling characteristics of the problem; confirm that “trees” accurately reflects the data structure background of binary search trees.

[0112] It is believed that “data structures” is related, but has been covered by the more specific “trees”; it is believed that “divide and conquer” is a solution approach, but the essence has been included in “dp”.

[0113] Therefore, the output knowledge point annotation result is: [“dp”,“combinatorics”,“math”,“trees”].

[0114] 4. The first round of knowledge point annotation results ["dp", "combinatorics", "math", "trees"] and the content of the exercises to be annotated were used as input. Annotation expert agents, alignment expert agents, and review expert agents respectively performed a second round of annotation, alignment, and review. In the second round, each annotation expert referenced existing annotation results, resulting in more consistent and accurate outputs. The final results stabilized on the same set of knowledge points, verifying the accuracy and stability of the annotation.

[0115] Example 2: This embodiment of the invention uses algorithm exercises from the Luogu OJ platform as an example to illustrate the implementation process of the invention in fine-grained knowledge point annotation tasks.

[0116] The Luogu dataset contains 1,731 questions with knowledge point tags, encompassing 149 different fine-grained knowledge point tags. As shown in Table 7, the Luogu OJ platform categorizes these knowledge point tags into 10 pre-defined knowledge modules.

[0117] Table 7. Examples of knowledge modules and knowledge points in the Luogu dataset.

[0118] In response to the characteristics of the Luogu dataset—a large number of knowledge points with fine granularity—the following adjustments were made during implementation:

[0119] Set the number of annotation experts N=5, and use the K-means algorithm to cluster the knowledge points of different modules into 5 categories;

[0120] The number of iteration rounds is set to 3 to accommodate more complex annotation tasks;

[0121] In the alignment expert fine-tuning, special emphasis was placed on training to differentiate similar knowledge points, such as distinguishing between easily confused knowledge points like "digital DP" and "interval DP", and "strongly connected components" and "connected components".

[0122] The performance of various knowledge point annotation methods on the Luogu dataset is shown in Table 8. Our proposed method significantly outperforms all baseline methods in four metrics: accuracy (0.4554), recall (0.4615), micro-average F1 score (0.4584), and macro-average F1 score (0.2716). Experimental results indicate that traditional deep learning models perform poorly overall on this task, particularly in accuracy and macro-average F1 score, demonstrating their inability to effectively handle the challenges of fine-grained, multi-label knowledge point annotation in this task. In contrast, while the performance of the base-based large language model is relatively better than traditional methods, it still lags significantly behind our proposed method.

[0123] Table 8 Performance Comparison of Knowledge Point Annotation Methods on Luogu Dataset

[0124] The method can effectively process the labeling task of fine-grained knowledge points. Through accurate differentiation of multiple similar knowledge points and deep analysis of complex problem solving logic, comprehensive and accurate labeling of algorithm exercise knowledge points is realized.

[0125] Embodiment two

[0126] As shown in Figure 6 The embodiment of the present application provides an algorithm exercise knowledge point automatic labeling system based on multi-agent cooperation, which comprises the following modules:

[0127] The labeling expert agent module 41 is used for inputting the algorithm exercise to be labeled into a labeling expert agent group composed of multiple labeling expert agents, and acquiring a preliminary knowledge point labeling set output by each labeling expert agent based on the problem solving process analysis of the algorithm exercise;

[0128] The alignment expert agent module 42 is used for inputting the preliminary knowledge point labeling set and the pre-defined standard knowledge point set into an alignment expert agent, performing semantic matching and granularity standardization processing by the alignment expert agent, and acquiring an aligned knowledge point labeling set;

[0129] The review expert agent module 43 is used for inputting the aligned knowledge point labeling set into a review expert agent, performing screening and integration by the review expert agent, and obtaining a knowledge point labeling result.

[0130] An algorithm exercise knowledge point automatic labeling device based on multi-agent cooperation comprises one or more electronic devices, wherein the one or more electronic devices are used to implement an algorithm exercise knowledge point automatic labeling method based on multi-agent cooperation.

[0131] An electronic device comprises one or more processors, a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement an algorithm exercise knowledge point automatic labeling method based on multi-agent cooperation.

[0132] A computer readable storage medium has executable instructions stored thereon, which are executed by a processor to make the processor implement an algorithm exercise knowledge point automatic labeling method based on multi-agent cooperation.

[0133] A non-transitory computer readable storage medium has a computer program stored thereon, which is executed by a processor to implement an algorithm exercise knowledge point automatic labeling method based on multi-agent cooperation.

[0134] The foregoing is considered as illustrative only of the principles of the application. Numerous modifications and changes will readily occur to those skilled in the art, and the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Therefore, the application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for automatically labeling knowledge points in algorithmic exercises based on multi-agent collaboration, characterized in that, include: Step S1: Input the algorithm problem to be labeled into a group of labeling expert agents consisting of multiple labeling expert agents, and obtain the preliminary knowledge point labeling set output by each labeling expert agent after analyzing the problem-solving process of the algorithm problem; Step S2: Input the preliminary knowledge point annotation set and the predefined standard knowledge point set into the alignment expert agent, and the alignment expert agent performs semantic matching and granular normalization processing to obtain the aligned knowledge point annotation set; Step S3: Input the aligned knowledge point annotation set into the review expert agent, which will then filter and integrate the data to obtain the knowledge point annotation results; Step S4: Take the knowledge point annotation results and the algorithm exercises to be annotated as input, and repeat steps S1 to S3 until the maximum number of iterations is reached to obtain the final knowledge point annotation results.

2. The method for automatically labeling algorithmic exercise knowledge points based on multi-agent collaboration as described in claim 1, characterized in that, Step S1: Input the algorithm problem to be labeled into a group of labeling expert agents consisting of multiple labeling expert agents, and obtain the preliminary knowledge point labeling set output by each labeling expert agent after analyzing the problem-solving process of the algorithm problem, specifically including: Step S11: Obtain an algorithm exercise training dataset with labels of real knowledge points; Step S12: Cluster the predefined set of standard knowledge points to form N knowledge point categories; Step S13: Divide the algorithm exercises in the algorithm exercise training dataset into the corresponding knowledge point categories according to their real knowledge point labels; Step S14: Using algorithm exercises categorized into different knowledge point classes, fine-tune different annotation expert agents to obtain the fine-tuned annotation expert agent groups; Step S15: Input the algorithm exercises to be labeled into the fine-tuned group of labeling expert agents, so that each labeling expert agent focuses on labeling the algorithm exercises of its corresponding category, and constructs a preliminary set of knowledge point labels.

3. The automatic annotation method for algorithm exercises based on multi-agent collaboration according to claim 2, characterized in that, Step S2: The preliminary knowledge point annotation set and the predefined standard knowledge point set are input into the alignment expert agent, which performs semantic matching and granular normalization processing to obtain the aligned knowledge point annotation set, specifically including: Step S21: For each standard knowledge point in the standard knowledge point set, generate multiple representation variants, including synonym substitution and granular extension types; Step S22: Using the randomly combined variants of the expression as input and the corresponding standard knowledge points as target labels, construct an alignment fine-tuning training dataset; Step S23: Using the alignment fine-tuning training dataset, fine-tune the alignment expert agent using semantic mapping and granular normalization fine-tuning strategies so that it learns to map knowledge points with different expressions to standard knowledge points, and obtain the fine-tuned alignment expert agent. Step S24: Input the preliminary knowledge point annotation set and the standard knowledge point set into the alignment expert agent for alignment processing to obtain the aligned knowledge point annotation set.

4. The automatic annotation method for algorithm exercises based on multi-agent collaboration according to claim 3, characterized in that, Step S3: The aligned knowledge point annotation set is input into the review expert agent, which then filters and integrates the annotations to obtain the knowledge point annotation results, specifically including: Step S31: For the real knowledge point labels of the algorithm exercises in the algorithm exercise training dataset, randomly add irrelevant knowledge points to construct a set of knowledge points to be screened that contains noise; Step S32: Using the set of knowledge points to be screened and the corresponding algorithm exercises as input, and the real knowledge point labels as targets, construct a review and fine-tuning training dataset; Step S33: Use the review fine-tuning training dataset to fine-tune the review expert agent so that it learns the ability to select the correct result from multiple candidate knowledge points, thereby obtaining the fine-tuned review expert agent; Step S34: After performing dropout processing on the aligned knowledge point annotation set, input the fine-tuned review expert agent to obtain the knowledge point annotation results.

5. An automatic annotation system for algorithmic exercises and knowledge points based on multi-agent collaboration, characterized in that, Includes the following modules: The annotation expert agent module is used to input the algorithm problem to be annotated into an annotation expert agent group consisting of multiple annotation expert agents, and obtain the preliminary knowledge point annotation set output by each annotation expert agent after analyzing the problem-solving process of the algorithm problem; The alignment expert agent module is used to input the preliminary knowledge point annotation set and the predefined standard knowledge point set into the alignment expert agent, which then performs semantic matching and granular normalization processing to obtain the alignment knowledge point annotation set. The review expert agent module is used to input the aligned knowledge point annotation set into the review expert agent, which then filters and integrates the data to obtain the knowledge point annotation results.

6. An automatic annotation device for algorithmic exercises and knowledge points based on multi-agent collaboration, characterized in that, It includes one or more electronic devices, wherein the one or more electronic devices are used to implement the method of any one of claims 1 to 4.

7. An electronic device, characterized in that, include: One or more processors; A memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method of any one of claims 1 to 4.

8. A computer-readable storage medium, characterized in that, It stores executable instructions that, when executed by a processor, cause the processor to implement the method described in any one of claims 1 to 4.

9. A non-transitory computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 4.

Images