Multi-role student adaptive grouping method based on reinforcement learning and linear programming

Abstract

The invention discloses a multi-role student adaptive grouping method based on reinforcement learning and linear programming, and relates to the field of teaching education. The method comprises the steps of performing fuzzy clustering on students through multi-dimensional features of the students, filtering grouping of a large-scale student group by utilizing reinforcement learning to reduce a problem scale, rating filtered rules by utilizing linear programming, feeding back to a deep network for learning, and finally outputting an effective grouping scheme. The beneficial effects of the invention are that the method considers that the students may have multiple roles, carries out the fuzzy clustering of the students through the multi-dimensional features of the students, carries out the filtering of the grouping of a large-scale student group through the reinforcement learning to greatly reduce the problem scale, carries out the rating of the filtered rules through the linear programming, and achieves the recognition of the students. And finally, a grouping scheme which is effective for the current student group is output, so that a good grouping result can be obtained in a relatively short time.

Description

Technical Field

[0001] The present invention relates to the field of teaching and education, and in particular to a multi-role student adaptive grouping method based on reinforcement learning and linear programming. Background Art

[0002] Advances in technology and innovations in pedagogy are reshaping the learning experience in today’s education sector. Collaborative learning, a key component of educational strategies that promote deep student engagement and communication, is gaining new momentum with these emerging trends. This learning strategy helps students develop critical thinking and deep learning skills by encouraging deep communication and knowledge sharing among peers, thereby promoting long-term knowledge retention and learning effectiveness. In addition, studies have shown that collaborative learning plays a vital role in developing students’ social skills and teamwork abilities.

[0003] The formation of learning groups is crucial to the success of collaborative learning, and grouping strategies directly affect learning outcomes. According to the analysis in, the method of grouping students greatly affects the learning experience and its results. Inappropriate grouping can lead to negative learning experiences and lower learning outcomes. Therefore, it is crucial to develop a scientific grouping strategy.

[0004] With the development of the Internet, large-scale online courses have become more and more mature, and more and more students are using online courses to acquire knowledge. In large-scale online courses, it is unrealistic to group many students relying solely on limited teacher resources. In recent years, more and more research has focused on using computer-supported collaborative learning to assist teachers in grouping students, using methods such as genetic algorithms and linear programming. As the number of students increases, the combinatorial complexity of grouping also rises sharply, resulting in an explosive growth in computing time, making it difficult to effectively apply these algorithms in actual teaching scenarios.

[0005] Most of the existing studies on student grouping classify students into a fixed category without considering the diversity and transformability of student roles in reality. This paper considers that students can belong to multiple roles at the same time and solves the large-scale student grouping problem based on reinforcement learning and linear programming. Summary of the invention

[0006] The purpose of the present invention is to provide a multi-role student adaptive grouping method based on reinforcement learning and linear programming in response to the shortcomings and defects in the prior art. In this method, taking into account that students may have multiple roles, students are fuzzy clustered according to their multidimensional characteristics. Secondly, reinforcement learning is used to filter the groups of large-scale student groups to greatly reduce the scale of the problem. Linear programming is then used to rate the filtered rules and feed them back to the deep network for learning. Finally, a grouping scheme that is effective for the current student group is output, which can obtain good grouping results in a relatively short time.

[0007] To achieve the above-mentioned purpose, the present invention adopts the following technical scheme: a multi-role student adaptive grouping method based on reinforcement learning and linear programming, which includes the following specific methods: fuzzy clustering of students through their multidimensional characteristics, and then using reinforcement learning to filter the groups of large-scale student groups to reduce the scale of the problem, and then using linear programming to rate the filtered rules, feed back to the deep network for learning, and finally output an effective grouping scheme.

[0008] Furthermore, the method is as follows: firstly, the silhouette coefficient is used to analyze the multidimensional characteristics of the students to determine the optimal number of clusters k, and then the fuzzy C-means clustering algorithm is used to cluster the students according to their multidimensional characteristics. After the students are clustered, the present invention determines the classification of the students according to the membership degree. Each student j has a membership degree u to cluster i. ij , usually, students are assigned to the group with the highest membership: Cluster(i) = argmax i u ij .

[0009] Furthermore, the difference between the maximum membership and the second largest membership is further analyzed, wherein, if the difference is less than 0.05, the present invention considers that the student may belong to two clusters at the same time, that is, it is possible to adaptively switch between the two clusters.

[0010] A multi-role student adaptive grouping method based on reinforcement learning and linear programming is characterized in that it also includes a dual-depth Q network with a dynamic reward function and 200 different grouping rules for five-person group allocation methods.

[0011] Furthermore, the dual-depth Q network with a dynamic reward function includes a state space and an action space: the state space is composed of a 200-dimensional vector, where each dimension corresponds to one of the 200 rules, and the initial state is an all-zero vector, indicating that no rule is selected. When a rule is executed by an action, the corresponding zero element in the state matrix will be assigned a value of 1, indicating that the rule has been selected; the action space is the index of the candidate rules, including 200 candidate rules, and therefore, there are also 200 corresponding actions. Each time an action is executed, the selected rule will be added to the selected rule subset, and the rule cannot be selected again in this training.

[0012] Furthermore, the deep network uses a sparse reward mechanism, and the quality of the solution is evaluated through linear programming only after the deep network completes the complete rule combination.

[0013] Furthermore, since the rule subsets selected each time are different, linear programming is not always successful, so a failure penalty is added. This method hopes that the deep network will find a better solution during the training process, rather than just avoiding the failure penalty of linear programming. Therefore, a negative reward is set for the case where the linear programming is successful but the solution quality is poor; if the linear programming is successful, a dynamic reward function is used to encourage the deep network to get as close to the optimal solution as possible during the exploration process, where s is used. i To represent the success score of linear programming in the i generation before the current iteration, the final scores of nearly 50 successful groupings of the deep network are recorded and the average is taken to obtain the average line of grouping success: After obtaining 20 selected rules in a single round, the following rewards are set: Among them, score is the score obtained by the successful linear programming in the current iteration, and evenline is the average score of the last 50 successful groupings. As the deep network learns and improves, the probability of successfully forming a group will increase. However, the quality of these groups may still vary greatly. In the reward function, only groups that meet high standards can receive positive rewards, while groups that are below the threshold after successful grouping will receive negative rewards.

[0014] Furthermore, in the process of constructing and optimizing the grouping strategy, the deep network is first initialized. Each network contains two hidden layers, each hidden layer has 128 neurons, the input layer receives the current state, and the output layer corresponds to the predicted Q value of each possible action. The action is selected using an exploration-exploitation strategy. After obtaining a complete subset of rules, the grouping configuration is evaluated and scored. The score is based on a combination of multiple indicators, including student participation, teamwork efficiency, and overall satisfaction. After obtaining a reward, the current state, the selected action, the new state after execution, and the corresponding reward are stored in the experience replay buffer. The deep network extracts samples from the experience replay buffer at each step, and uses these samples to update the parameters of the deep network. The network parameters are gradually optimized by calculating the difference between the predicted Q value and the actual reward.

[0015] Furthermore, linear programming is used to group the current population so that they can be assigned to different groups. After obtaining the rule set through deep reinforcement learning, a rule matrix Rule of size R×S and a score array Score of size R are obtained, where Rule ij represents the number of students belonging to cluster j in rule i, Score i represents the score of rule i, R is the number of rules, and S is the number of clusters. The total number of people in each cluster is obtained through fuzzy clustering, and array C is used to store this information. jrepresents the number of people in cluster j. For some students with multi-role characteristics, a matrix is ​​used to represent their transferability, T jk represents the maximum number of people in group j that can be transferred to group k.

[0016] Furthermore, after obtaining the above information, a linear programming model was established to solve the number of times each rule is used x i and the actual group transfer matrix y jk : The objective function is to maximize the total score of the groups, where constraint 1 ensures that the number of people in a group in the rule used is equal to the total number of people in the original group plus the number of people transferred from other groups to this group, minus the number of people transferred from this group to other groups. Constraint 2 ensures that the number of times the rule is used is at least 0. Constraint 3 ensures that the number of people transferred to each cluster is at least 0 and cannot exceed the maximum number of possible transfers.

[0017] After adopting the above technical scheme, the beneficial effects of the present invention are as follows: in this method, it is taken into account that students may have multiple roles, and students are fuzzy clustered according to their multidimensional characteristics. Secondly, reinforcement learning is used to filter the groups of large-scale student groups to greatly reduce the scale of the problem, and then linear programming is used to rate the filtered rules, which are fed back to the deep network for learning. Finally, a grouping scheme that is effective for the current student group is output, and good grouping results can be obtained in a shorter time. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0019] Figure 1 It is a schematic diagram of the algorithm flow of the present invention. DETAILED DESCRIPTION

[0020] See also Figure 1As shown, the technical solution adopted by this specific implementation method is: it includes the following specific methods: fuzzy clustering students through their multidimensional characteristics, then using reinforcement learning to filter the grouping of large-scale student groups to reduce the scale of the problem, and then using linear programming to rate the filtered rules, and feeding them back to the deep network for learning, and finally outputting an effective grouping scheme, which is specifically: first using the silhouette coefficient to analyze the multidimensional characteristics of the students to determine the optimal number of clusters k, and then using the fuzzy C-means clustering algorithm to cluster the students according to their multidimensional characteristics. After completing the clustering of the students, the present invention determines the classification of the students according to the membership degree, and each student j has a membership degree u to cluster i. ij , students are usually assigned to the group with the highest affiliation:

[0021] Cluster(i)=argmax i u ij

[0022] The difference between the maximum membership and the second largest membership is further analyzed. If the difference is less than 0.05, the present invention considers that the student may belong to two clusters at the same time, that is, the student can switch adaptively between the two clusters.

[0023] A multi-role student adaptive grouping method based on reinforcement learning and linear programming is characterized in that it also includes a dual-depth Q network with a dynamic reward function and 200 different grouping rules for five-person group allocation methods. A dual-depth Q network with a dynamic reward function includes a state space and an action space:

[0024] The state space consists of 200-dimensional vectors, where each dimension corresponds to one of the 200 rules. The initial state is an all-zero vector, indicating that no rule is selected. When a rule is executed by an action, the corresponding zero element in the state matrix will be assigned a value of 1, indicating that the rule has been selected.

[0025] The action space is the index of the candidate rules, including 200 candidate rules, and therefore, there are 200 corresponding actions. Each time an action is executed, the selected rule will be added to the selected rule subset, and the rule cannot be selected again in this training. There are 20 steps in a round. Each step is executed, that is, a new rule is added to the selected rule subset. When there are 20 rules in the rule subset, the rule is output to the linear programming and the current round ends after feedback.

[0026] Furthermore, the deep network uses a sparse reward mechanism, and the quality of the solution is evaluated through linear programming only after the deep network completes the complete rule combination.

[0027] Furthermore, since the rule subsets selected each time are different, linear programming does not always succeed, so a failure penalty is added. This method hopes that the deep network will find a better solution during the training process, rather than just avoiding the failure penalty of linear programming. Therefore, a negative reward is also set for the case where the linear programming succeeds but the solution quality is poor;

[0028] If the linear programming is successful, a dynamic reward function is used to motivate the deep network to get as close to the optimal solution as possible during the exploration process, where s i To represent the success score of linear programming in the i generation before the current iteration, the final scores of nearly 50 successful groupings of the deep network are recorded and the average is taken to obtain the average line of grouping success:

[0029]

[0030] After obtaining 20 selected rules in a single round, the following rewards are set:

[0031]

[0032] Among them, score is the score obtained by the successful linear planning in the current iteration, and evenline is the average score of the last 50 successful groupings;

[0033] As the deep network learns and improves, the probability of successfully forming groups will also increase. However, the quality of these groups may still vary greatly. In the reward function described above, only groups that meet high standards can receive positive rewards, while groups that are below the threshold after successful grouping will receive negative rewards.

[0034] Furthermore, in the process of constructing and optimizing the grouping strategy, the deep network is first initialized. Each network contains two hidden layers, each hidden layer has 128 neurons, the input layer receives the current state, and the output layer corresponds to the predicted Q value of each possible action. The exploration-exploitation strategy is used to select actions; the exploration-exploitation strategy is a balanced method of exploration and utilization, that is, randomly selecting actions according to the probability of the exploration rate, or selecting the action with the highest current Q value. Whether greedy actions or random actions are selected, repeated selection of actions is prohibited. The value of the exploration rate will decrease as the training process progresses, thereby utilizing more learned information.

[0035] After obtaining the complete subset of rules, the grouping configuration will be evaluated and scored based on a combination of multiple indicators, including student participation, teamwork efficiency and overall satisfaction. After receiving the reward, the current state, the selected action, the new state after execution and the corresponding reward will be stored in the experience replay buffer. The deep network will extract samples from the experience replay buffer at each step, and use these samples to update the parameters of the deep network. By calculating the difference between the predicted Q value and the actual reward, the network parameters are gradually optimized.

[0036] More specifically, linear programming is used to group the current population so that they can be assigned to different groups. After obtaining the rule set through deep reinforcement learning, a rule matrix Rule of size R×S and a score array Score of size R are obtained, where Rule ij represents the number of students belonging to cluster j in rule i, Score i represents the score of rule i, R is the number of rules, and S is the number of clusters;

[0037] The total number of people in each cluster is obtained through fuzzy clustering, and array C is used to store this information. j represents the number of people in cluster j. For some students with multi-role characteristics, a matrix is ​​used to represent their transferability, T jk represents the maximum number of people in group j that can be transferred to group k.

[0038] After obtaining the above information, a linear programming model was established to solve the number of times each rule is used x i and the actual group transfer matrix y jk :

[0039]

[0040] The objective function is to maximize the total score of the groups, where constraint 1 ensures that the number of people in a group in the rule used is equal to the total number of people in the original group plus the number of people transferred from other groups to this group, minus the number of people transferred from this group to other groups. Constraint 2 ensures that the number of times the rule is used is at least 0. Constraint 3 ensures that the number of people transferred to each cluster is at least 0 and cannot exceed the maximum number of possible transfers.

[0041] The above description is only used to illustrate the technical solution of the present invention rather than to limit it. Other modifications or equivalent substitutions made to the technical solution of the present invention by ordinary technicians in this field should be included in the scope of the claims of the present invention as long as they do not depart from the spirit and scope of the technical solution of the present invention.

Claims

1. A multi-role student adaptive grouping method based on reinforcement learning and linear programming, characterized by: It includes the following specific methods: fuzzy clustering of students through their multi-dimensional features, then using reinforcement learning to filter the groups of large-scale student groups to reduce the scale of the problem, and then using linear programming to rate the filtered rules, feeding them back into the deep network for learning, and finally outputting an effective grouping plan.

2. The multi-role student adaptive grouping method based on reinforcement learning and linear programming according to claim 1 is characterized by: Specifically, the method is as follows: firstly, the silhouette coefficient is used to analyze the multidimensional characteristics of students to determine the optimal number of clusters k, and then the fuzzy C-means clustering algorithm is used to cluster students according to their multidimensional characteristics. After the students are clustered, the method determines the classification of students according to the membership degree. Each student j has a membership degree u to cluster i. ij , usually, students are assigned to the group with the highest membership: Cluster(i) = argmax i u ij .

3. The multi-role student adaptive grouping method based on reinforcement learning and linear programming according to claim 2 is characterized by: The difference between the maximum membership and the second largest membership is further analyzed. If the difference is less than 0.05, the present invention considers that the student may belong to two clusters at the same time, that is, the student can switch adaptively between the two clusters.

4. A multi-role student adaptive grouping method based on reinforcement learning and linear programming, characterized in that: It also includes a dual deep Q-network with a dynamic reward function and grouping rules for 200 different five-person team assignment methods.

5. The multi-role student adaptive grouping method based on reinforcement learning and linear programming according to claim 4 is characterized by: The dual-depth Q network with a dynamic reward function includes a state space and an action space: The state space consists of 200-dimensional vectors, where each dimension corresponds to one of the 200 rules. The initial state is an all-zero vector, indicating that no rule is selected. When a rule is executed by an action, the corresponding zero element in the state matrix will be assigned a value of 1, indicating that the rule has been selected. The action space is an index of candidate rules, including 200 candidate rules, and therefore, 200 corresponding actions. Each time an action is executed, the selected rule will be added to the selected rule subset, and the rule cannot be selected again in this training.

6. The multi-role student adaptive grouping method based on reinforcement learning and linear programming according to claim 4 is characterized by: The deep network uses a sparse reward mechanism, and the quality of the solution is evaluated through linear programming only after the deep network has completed a complete rule combination.

7. The multi-role student adaptive grouping method based on reinforcement learning and linear programming according to claim 4 is characterized by: Since the rule subsets selected each time are different, linear programming does not always succeed, so a failure penalty is added. This method hopes that the deep network will find a better solution during the training process, rather than just avoiding the failure penalty of linear programming. Therefore, a negative reward is also set for the case where the linear programming succeeds but the solution quality is poor; If the linear programming is successful, a dynamic reward function is used to motivate the deep network to get as close to the optimal solution as possible during the exploration process, where s i To represent the success score of linear programming in the i generation before the current iteration, the final scores of nearly 50 successful groupings of the deep network are recorded and the average is taken to obtain the average line of grouping success: After obtaining 20 selected rules in a single round, the following rewards are set: Among them, score is the score obtained by the successful linear planning in the current iteration, and evenline is the average score of the last 50 successful groupings; As the deep network learns and improves, the probability of successfully forming groups will also increase. However, the quality of these groups may still vary greatly. In the reward function described above, only groups that meet high standards can receive positive rewards, while groups that are below the threshold after successful grouping will receive negative rewards.

8. The multi-role student adaptive grouping method based on reinforcement learning and linear programming according to claim 1 is characterized by: In the process of constructing and optimizing the grouping strategy, the deep network is first initialized. Each network contains two hidden layers, each hidden layer has 128 neurons, the input layer receives the current state, and the output layer corresponds to the predicted Q value of each possible action. The exploration-exploitation strategy is used to select the action; After obtaining the complete subset of rules, the grouping configuration will be evaluated and scored based on a combination of multiple indicators, including student participation, teamwork efficiency and overall satisfaction. After receiving the reward, the current state, the selected action, the new state after execution and the corresponding reward will be stored in the experience replay buffer. The deep network will extract samples from the experience replay buffer at each step, and use these samples to update the parameters of the deep network. By calculating the difference between the predicted Q value and the actual reward, the network parameters are gradually optimized.

9. The multi-role student adaptive grouping method based on reinforcement learning and linear programming according to claim 1 is characterized by: Linear programming is used to group the current population so that they can be assigned to different groups. After obtaining the rule set through deep reinforcement learning, a rule matrix Rule of size R×S and a score array Score of size R are obtained, where Rule ij represents the number of students belonging to cluster j in rule i, Score i represents the score of rule i, R is the number of rules, and S is the number of clusters; The total number of people in each cluster is obtained through fuzzy clustering, and array C is used to store this information. j represents the number of people in cluster j. For some students with multi-role characteristics, a matrix is ​​used to represent their transferability, T jk represents the maximum number of people in group j that can be transferred to group k.

10. The multi-role student adaptive grouping method based on reinforcement learning and linear programming according to claims 8-9, characterized in that: After obtaining the above information, a linear programming model was established to solve the number of times each rule is used x i and the actual group transfer matrix y jk : The objective function is to maximize the total score of the groups, where constraint 1 ensures that the number of people in a group in the rule used is equal to the total number of people in the original group plus the number of people transferred from other groups to this group, minus the number of people transferred from this group to other groups. Constraint 2 ensures that the number of times the rule is used is at least 0. Constraint 3 ensures that the number of people transferred to each cluster is at least 0 and cannot exceed the maximum number of possible transfers.

Images