Multi-role student adaptive grouping method based on reinforcement learning and linear programming

Abstract

This invention relates to an adaptive grouping method for multi-role students based on reinforcement learning and linear programming, applicable to the field of teaching and education. It employs fuzzy clustering of students based on their multidimensional features, followed by reinforcement learning to filter the grouping of large student populations, reducing the problem size. Linear programming is then used to rate the filtered rules, which are fed back into a deep network for further learning. Finally, an effective grouping scheme is output. The advantages of this invention are: considering that students may have multiple roles, this method performs fuzzy clustering based on their multidimensional features, then uses reinforcement learning to filter the grouping of large student populations, significantly reducing the problem size. Linear programming is then used to rate the filtered rules, which are fed back into a deep network for further learning, ultimately outputting an effective grouping scheme for the current student population. This approach can achieve good grouping results in a relatively short time.

Description

Technical Field

[0001] This invention relates to the field of teaching and education, specifically to a multi-role student adaptive grouping method based on reinforcement learning and linear programming. Background Technology

[0002] Technological advancements and innovative pedagogical approaches are reshaping the learning experience in education today. Collaborative learning, a key component of educational strategies that promote deep student engagement and communication, is showing renewed vitality alongside these emerging trends. This learning strategy helps students develop critical thinking and deep learning skills by encouraging in-depth communication and knowledge sharing among peers, thereby promoting long-term knowledge retention and learning effectiveness. Furthermore, research indicates that collaborative learning plays a crucial role in cultivating students' social skills and teamwork abilities.

[0003] The formation of learning groups is crucial for the success of collaborative learning, and grouping strategies directly impact learning outcomes. According to the analysis, the method of student grouping significantly affects the learning experience and its results. Inappropriate grouping can lead to negative learning experiences and lower learning outcomes. Therefore, developing scientific grouping strategies is essential.

[0004] With the development of the internet, large-scale online courses (MLAs) have become increasingly mature, and more and more students are using online courses to acquire knowledge. However, in large-scale online courses, relying solely on limited teacher resources to group numerous students is impractical. In recent years, more and more research has focused on using computer-supported collaborative learning to assist teachers in grouping students, employing methods such as genetic algorithms and linear programming. As the number of students increases, the complexity of group combinations also rises sharply, leading to an explosive increase in computation time, making it difficult to effectively apply these algorithms in real-world teaching scenarios.

[0005] Most existing student grouping studies fix students into a single category without considering the diversity and variability of student roles in reality. This paper considers that students can simultaneously belong to multiple roles and solves large-scale student grouping problems based on reinforcement learning and linear programming. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings and defects of existing technologies by providing a multi-role student adaptive grouping method based on reinforcement learning and linear programming. This method considers that students may have multiple roles and performs fuzzy clustering of students based on their multidimensional features. Next, reinforcement learning is used to filter the grouping of large-scale student groups to significantly reduce the problem size. Then, linear programming is used to rate the filtered rules and feed them back into a deep network for learning. Finally, an effective grouping scheme for the current student group is output, which can obtain good grouping results in a short time.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: a multi-role student adaptive grouping method based on reinforcement learning and linear programming, which includes the following specific methods: performing fuzzy clustering of students through multi-dimensional features of students, then using reinforcement learning to filter the grouping of large-scale student groups to reduce the problem size, then using linear programming to rate the filtered rules, feeding them back into a deep network for learning, and finally outputting an effective grouping scheme.

[0008] Furthermore, specifically: first, the silhouette coefficient is used to analyze the multidimensional features of students to determine the optimal number of clusters k; then, the fuzzy C-means clustering algorithm is used to cluster students according to their multidimensional features. After the student clustering is completed, this invention determines the student classification based on membership degree, where each student j has a membership degree u to cluster i. ij Typically, students are grouped into the group with the highest membership degree: Cluster(i) = argmax i u ij .

[0009] Furthermore, the difference between the largest membership degree and the second largest membership degree is further analyzed. If the difference is less than 0.05, the present invention considers that the student may belong to two clusters at the same time, that is, the student can adaptively switch between the two clusters.

[0010] The multi-role student adaptive grouping method based on reinforcement learning and linear programming is characterized by including a dual-deep Q-network with a dynamic reward function and grouping rules for 200 different five-person group assignment methods.

[0011] Furthermore, the aforementioned dual-depth Q-network with a dynamic reward function includes a state space and an action space: the state space consists of a 200-dimensional vector, where each dimension corresponds to one of 200 rules. The initial state is a vector of all zeros, indicating that no rule has been selected. When a rule is executed by an action, the corresponding zero element in the state matrix is ​​assigned a value of 1, indicating that the rule has been selected. The action space is an index of candidate rules, including 200 candidate rules, and therefore 200 corresponding actions. Each time an action is executed, the selected rule is added to the selected rule subset, and this rule cannot be selected again in this training.

[0012] Furthermore, the deep network uses a sparse reward mechanism, and the quality of the solution is evaluated by linear programming only after the deep network has completed a full combination of rules.

[0013] Furthermore, since the subset of rules chosen each time is different, linear programming does not always succeed. Therefore, a failure penalty is added. Because this method aims for the deep network to find better solutions during training, rather than simply avoiding the failure penalty of linear programming, a negative reward is also set for cases where linear programming succeeds but the solution quality is poor. If linear programming succeeds, a dynamic reward function is used to incentivize the deep network to approach the optimal solution as closely as possible during the exploration process, where s... i Let represent the score of successful linear programming within the i-th generation before the current iteration. The final scores of the deep network's last 50 successful groupings are recorded, and their average is taken to obtain the average line of successful grouping: The following reward is set after obtaining 20 selected rules in a single round: Here, score is the score obtained from the successful linear programming in the current iteration, and evenline is the average score of the last 50 successful groupings. As the deep network learns and improves, the probability of successfully forming groups will also increase. However, the quality of these groups may still vary greatly. In the reward function, only groups that reach high standards can receive positive rewards, while groups that are below the threshold after successful grouping will receive negative rewards.

[0014] Furthermore, in the process of constructing and optimizing the grouping strategy, a deep network is first initialized. Each network contains two hidden layers, each with 128 neurons. The input layer receives the current state, and the output layer corresponds to the predicted Q-value for each possible action. An exploration-exploitation strategy is used to select actions. After obtaining a complete subset of rules, the grouping configuration is evaluated and scored. The scoring is based on a combination of multiple indicators, including student participation, teamwork efficiency, and overall satisfaction. After receiving a reward, the current state, the selected action, the new state after execution, and the corresponding reward are stored in an experience replay buffer. At each step, the deep network extracts samples from the experience replay buffer and uses these samples to update the parameters of the deep network. By calculating the difference between the predicted Q-value and the actual reward, the network parameters are gradually optimized.

[0015] Furthermore, linear programming is used to group the current population so that it can be assigned to different groups. After obtaining the rule set through deep reinforcement learning, a rule matrix of size R×S and a score array of size R are obtained, where Rule... ij Score represents the number of students in cluster j belonging to rule i. i R represents the score of rule i, R is the number of rules, and S is the number of clusters. Fuzzy clustering is used to obtain the total number of people in each cluster, and this information is stored in array C. jThe number of people in cluster j is represented by a matrix T. For students with multiple roles, their transferability is represented by a matrix T. jk This represents the maximum number of people in group j who can be transferred to group k.

[0016] Furthermore, after obtaining the above information, a linear programming model was established to solve for the number of times each rule is used, x. i And the actual group transition matrix y jk : The objective function is to maximize the total score of the groups. Constraint 1 ensures that the number of people in a group under the rule used is equal to the total number of people in the original group plus the number of people transferred from other groups to this group, minus the number of people transferred from this group to other groups. Constraint 2 ensures that the rule is used at least 0 times. Constraint 3 ensures that the number of people transferred to each cluster is at least 0 and cannot exceed the maximum number of people that can be transferred.

[0017] After adopting the above technical solution, the beneficial effects of the present invention are as follows: This method takes into account that students may have multiple roles, performs fuzzy clustering of students through multidimensional features of students, then uses reinforcement learning to filter the grouping of large-scale student groups to greatly reduce the problem size, then uses linear programming to rate the filtered rules, feeds them into a deep network for learning, and finally outputs a grouping scheme that is effective for the current student group, which can obtain good grouping results in a short time. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the algorithm flow of the present invention. Detailed Implementation

[0020] See Figure 1As shown, the technical solution adopted in this specific implementation is as follows: it includes the following specific methods: fuzzy clustering of students based on their multidimensional features; secondly, filtering of large-scale student groups using reinforcement learning to reduce the problem size; thirdly, rating the filtered rules using linear programming and feeding them back into a deep network for learning; and finally, outputting an effective grouping scheme. Specifically, it involves: firstly analyzing the multidimensional features of students using silhouette coefficients to determine the optimal number of clusters k; then using the fuzzy C-means clustering algorithm to cluster students based on their multidimensional features; after completing the student clustering, the invention determines the student classification based on membership degree, with each student j having a membership degree u to cluster i. ij Typically, students are grouped into the group with the highest membership level:

[0021] Cluster(i) = argmax i u ij

[0022] Further analysis of the difference between the largest and second largest membership degree reveals that if the difference is less than 0.05, the present invention considers that a student may belong to two clusters simultaneously, meaning that the student can adaptively switch between the two clusters.

[0023] A multi-role student adaptive grouping method based on reinforcement learning and linear programming is characterized by including a dual-deep Q-network with a dynamic reward function and grouping rules for 200 different five-person group assignment methods. The dual-deep Q-network with a dynamic reward function includes a state space and an action space.

[0024] The state space consists of a 200-dimensional vector, where each dimension corresponds to one of the 200 rules. The initial state is a vector of all zeros, indicating that no rule has been selected. When a rule is executed by an action, the corresponding zero element in the state matrix will be assigned a value of 1, indicating that the rule has been selected.

[0025] The action space is an index of candidate rules, containing 200 candidate rules and therefore 200 corresponding actions. Each time an action is executed, the selected rule is added to the selected rule subset, and this rule cannot be selected again in the current training. There are 20 steps in an epoch. Each step adds a new rule to the selected rule subset. When there are 20 rules in the rule subset, the rule is output to the linear programming algorithm, and the current epoch ends after receiving feedback.

[0026] Furthermore, the deep network uses a sparse reward mechanism, and the quality of the solution is evaluated by linear programming only after the deep network has completed a full combination of rules.

[0027] Furthermore, since the subset of rules chosen each time is different, linear programming will not always succeed. Therefore, a failure penalty is added. Since this method hopes that the deep network will find better solutions during training, rather than just avoiding the failure penalty of linear programming, a negative reward is also set for the case where linear programming is successful but the solution quality is poor.

[0028] If the linear programming is successful, a dynamic reward function is used to incentivize the deep network to approach the optimal solution as closely as possible during the exploration process, where s is used. i Let represent the score of successful linear programming within the i-th generation before the current iteration. The final scores of the deep network's last 50 successful groupings are recorded, and their average is taken to obtain the average line of successful grouping:

[0029]

[0030] The following reward is set after obtaining 20 selected rules in a single round:

[0031]

[0032] Where score is the score obtained by the successful linear programming in the current iteration, and evenline is the average score of the last 50 successful groupings;

[0033] As deep networks learn and improve, the probability of successfully forming groups will increase. However, the quality of these groups may still vary greatly. In the reward function, only groups that meet high standards can receive positive rewards, while groups that are below the threshold after successful formation will receive negative rewards.

[0034] Furthermore, in the process of constructing and optimizing the grouping strategy, a deep network is first initialized. Each network contains two hidden layers, each with 128 neurons. The input layer receives the current state, and the output layer corresponds to the predicted Q-value for each possible action. An explore-exploitation strategy is used to select an action. The explore-exploitation strategy is a balance between exploration and exploitation; that is, an action is randomly selected according to the probability of the exploration rate, or the action with the highest current Q-value is selected. Regardless of whether a greedy action or a random action is selected, repeated selection of actions is prohibited. The exploration rate decreases as the training process progresses, thereby utilizing more learned information.

[0035] After obtaining the complete subset of rules, the group configuration is evaluated and scored. The scoring is based on a combination of multiple indicators, including student participation, teamwork efficiency, and overall satisfaction. After receiving a reward, the current state, the selected action, the new state after execution, and the corresponding reward are stored in the experience replay buffer. At each step, the deep network extracts samples from the experience replay buffer and uses these samples to update the parameters of the deep network. By calculating the difference between the predicted Q value and the actual reward, the network parameters are gradually optimized.

[0036] More specifically, linear programming is used to group the current population so that it can be assigned to different groups. After obtaining the rule set through deep reinforcement learning, a rule matrix of size R×S and a score array of size R are obtained, where Rule... ij Score represents the number of students in cluster j belonging to rule i. i R represents the score of rule i, R is the number of rules, and S is the number of clusters.

[0037] The total number of people in each cluster is obtained through fuzzy clustering, and this information is stored in an array C. j The number of people in cluster j is represented by a matrix T. For students with multiple roles, their transferability is represented by a matrix T. jk This represents the maximum number of people in group j who can be transferred to group k.

[0038] After obtaining the above information, a linear programming model was established to solve for the number of times each rule is used, x. i And the actual group transition matrix y jk :

[0039]

[0040] The objective function is to maximize the total score of the groups. Constraint 1 ensures that the number of people in a group under the rule used is equal to the total number of people in the original group plus the number of people transferred from other groups to this group, minus the number of people transferred from this group to other groups. Constraint 2 ensures that the rule is used at least 0 times. Constraint 3 ensures that the number of people transferred to each cluster is at least 0 and cannot exceed the maximum number of people that can be transferred.

[0041] The above description is only used to illustrate the technical solution of the present invention and is not intended to limit it. Any other modifications or equivalent substitutions made by those skilled in the art to the technical solution of the present invention, as long as they do not depart from the spirit and scope of the technical solution of the present invention, should be covered within the scope of the claims of the present invention.

Claims

1. A multi-role student adaptive grouping method based on reinforcement learning and linear programming, characterized in that: It comprises the following specific methods: fuzzy clustering of students through their multi-dimensional characteristics, filtering of the grouping of a large number of students by using reinforcement learning to reduce the problem size, rating of the filtered rules by using linear programming, feedback to the deep network for learning, and finally output of an effective grouping scheme; and the specific methods are as follows: First, the profile coefficient is used to analyze the multi-dimensional characteristics of students to determine the optimal cluster number Then, the fuzzy C-means clustering algorithm is used to cluster students according to the multi-dimensional characteristics of students. After clustering the students, the classification of students is determined according to the membership degree, and each student has a membership degree Under normal circumstances, students are classified into the group with the highest membership degree: ; Further analysis of the difference between the maximum membership and the second maximum membership, wherein if the difference is less than 0.05, the application considers that the student may belong to two clusters at the same time, i.e., adaptive switching between the two clusters is possible; The deep network uses a sparse reward mechanism, and only after the deep network completes the complete rule combination, the quality of the solution is evaluated by linear programming; Linear programming is used to group the current population into different groups, and after obtaining the rule set through deep reinforcement learning, a set of rules of size [value missing] is obtained. The rule matrix and a size of Array of scores ,in, Representative Rules Middle school students belong to The number of clusters, Representative Rules The score, It is the number of rules. It is the number of clusters; The total number of people in each cluster is obtained by fuzzy clustering, using an array to store this information, denotes the number of people in a cluster For some students with multi-role characteristics, a matrix is used to represent their transferability, denotes the maximum number of people in a group that can be transferred to a group ; After obtaining the above information, a linear programming model is established to solve the number of times each rule is used and the actual group transition matrix : The objective function is to maximize the total score of the groups, where constraint 1 ensures that the number of people in a group using a rule is equal to the total number of people in the original group plus the number of people transferred into that group from other groups minus the number of people transferred from that group to other groups, constraint 2 ensures that the number of times the rule is used is at least 0, and constraint 3 ensures that the number of people transferred to each cluster is at least 0 and cannot exceed the maximum number of people that can be transferred.

2. The method of claim 1, wherein: It also comprises a double deep Q network with a dynamic reward function and 200 different five-person team allocation methods.

3. The method of claim 2, wherein: The double deep Q network with a dynamic reward function comprises a state space and an action space: The state space is composed of a 200-dimensional vector, wherein each dimension corresponds to one of the 200 rules, the initial state is a zero vector, indicating that no rule has been selected, and when a rule is executed by an action, the corresponding zero element in the state matrix is assigned a value of 1, indicating that the rule has been selected; The action space is the index of the candidate rules, including 200 candidate rules, so there are also 200 corresponding actions, and each time an action is executed, the selected rule is added to the selected rule subset, and the rule cannot be selected again in this training.

4. The method of claim 2, wherein: Since the selected rule subset is different each time, linear programming may not always be successful, so a failure penalty is added, because the method expects the deep network to find a better solution during training, not just avoid the failure penalty of linear programming, so a negative reward is also set for the case where linear programming is successful but the solution quality is poor; If the linear programming is successful, a dynamic reward function is used to encourage the deep network to get as close to the optimal solution as possible during the exploration process, where represents the fraction of successful linear programming within the iteration before the current iteration, the final scores of the last 50 successful groups of the deep network are recorded, and the average value is taken to obtain the average line of the success of the group. As the deep network learns and improves, the probability of successfully assembling a group also increases, however, the quality of these groups can still vary greatly, and in the described reward function, only groups that meet a high standard receive a positive reward, while groups that fall below a threshold after a successful assembly receive a negative reward.

5. The method of claim 1, wherein: In the process of constructing and optimizing the grouping strategy, the deep network is first initialized, each network contains two hidden layers, each hidden layer has 128 neurons, the input layer receives the current state, and the output layer corresponds to the predicted Q value of each possible action, and an exploration-exploitation strategy is used to select the action; After obtaining the complete rule subset, the grouping configuration is evaluated and scored, the score is based on the combination of multiple indicators, including student participation, team cooperation efficiency and overall satisfaction, after obtaining the reward, the current state, the selected action, the new state after execution and the corresponding reward are stored in the experience replay buffer, the deep network extracts samples from the experience replay buffer at each step, and the parameters of the deep network are updated using these samples, and the network parameters are gradually optimized by calculating the difference between the predicted Q value and the actual reward.

Images