Reinforcement learning grouping methods, devices, and media based on typical user group characteristics

Abstract

This invention discloses a reinforcement learning grouping method, apparatus, and medium based on typical user group characteristics. The steps are as follows: 1) Extracting original features from user profiles in a large-scale online learning system; 2) Constructing a user network graph using group relationships, and using a graph convolutional neural network to perform dimensionality-upgrading mapping on individual user features to form group features; 3) Clustering the group features using a learnable clustering algorithm to form multiple user group types; 4) Using a Q-learning reinforcement learning algorithm, learning a grouping model in the typical feature space of the group to finally obtain the optimal combination pattern of typical features. The method of this invention is applicable to student user grouping in large-scale online learning scenarios such as discussions, practices, and collaborations. It offers high grouping efficiency, good accuracy, and strong interpretability. This invention has significant practical application value for user grouping teaching practices in large-scale online learning platforms.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence, and in particular relates to a reinforcement learning grouping method, device and medium based on typical characteristics of a user group. Background Art

[0002] Solving the problem of collaborative decision-making in the multi-level natural interaction mechanism of teacher-machine-student is the key to realizing human-machine enhanced intelligent organization. In the traditional one-to-many large-scale online group learning scenario, neither teachers nor machines can cope with the needs of grouping massive user groups alone, resulting in the difficulty of adapting the group learning experience. How to integrate the search, trial and error of machine intelligence and human feedback capabilities to establish an organizational mechanism in the many-to-many scenarios of "teacher-machine" and "student" is the key to solving the difficulty of adapting the group learning experience. Unlike traditional clustering problems, group grouping cannot simply group students of similar categories into one group. Under natural conditions, the complementary and diverse natures of the members in the group often form better learning effects. Therefore, how to obtain the optimization of the grouping method from the perspective of user feedback and realize a system that is truly suitable for user experience is a key issue in intelligent group grouping.

[0003] In response to this problem, a variety of intelligent group grouping methods have been developed at home and abroad. Shanghai University of Finance and Economics proposed a discrete-time heterogeneous multi-agent system, mainly studying the optimal grouping consistency control problem. The system discussed not only includes agents with completely unknown first-order and second-order dynamics, but also includes the intensity of cooperation-competition interaction. The University of Science and Technology of China proposed a two-stage reinforcement learning clustering algorithm. This method first performs state space learning of the continuous state space adaptively discretized using the K-means clustering algorithm; then, it finds the optimal strategy for strategy learning using the alternative suitable trace Sarsa learning algorithm. In recent years, hierarchical reinforcement learning has made significant progress in overcoming the curse of dimensionality. Typical results include Option, HAM and MAXQ methods, among which Option and MAXQ are currently more widely used. The Option method is convenient for automatically dividing subtasks (especially partition or segment subtasks), and the granularity of subtasks is easy to control. However, when using prior knowledge to divide subtasks, the task division results are not clear enough, and the internal strategy of subtasks is difficult to determine; the MAXQ method has strong online learning ability, but weak automatic stratification ability, and the stratification granularity is not fine enough, making it difficult to further decompose some subtasks that are still large in scale. Summary of the invention

[0004] The purpose of the present invention is to overcome the existing deficiencies and provide a reinforcement learning grouping method, device and medium based on typical characteristics of user groups.

[0005] To achieve the purpose of the present invention, the technical solution provided is as follows:

[0006] In a first aspect, the present invention provides a reinforcement learning grouping method based on typical characteristics of a user group, the steps of which are as follows:

[0007] S1: Extract and filter the target user’s profile information on the online learning platform, and combine the personal information profile of each target user and the interaction behavior profile between the user and the platform as the user profile feature;

[0008] S2: Build a user relationship graph based on the user portrait features of all users. Each node in the graph corresponds to a target user. Then use a graph convolutional neural network to extract and aggregate features of the user relationship graph to obtain the group features corresponding to each node in the graph.

[0009] S3: Cluster the group characteristics of all nodes in the user relationship graph to form multiple types of typical user groups;

[0010] S4: Based on the group characteristics obtained in S2 and the typical user groups obtained by clustering in S3, the reinforcement learning model is used to predict the typical types of the group, so that the users in each typical user group are gradually divided into different groups, and finally the optimal grouping of all target users on the platform is obtained.

[0011] As a preferred embodiment of the first aspect above, the specific implementation method of step S1 is as follows:

[0012] S11: extracting the target user's portrait information from the database of the online learning platform, wherein the portrait information includes the personal information portrait and the interaction behavior portrait between the user and the platform;

[0013] S12: For each user i’s portrait information, retain the features related to the grouping task, and digitize the retained portrait information dimensions to the real number space, and then concatenate the portrait information of all dimensions together to obtain the D-dimensional user portrait feature f i ; User portrait features f of all N target users on the online learning platform i Combined into user portrait feature matrix F∈R N×D .

[0014] As a preferred embodiment of the first aspect, the specific implementation method of step S2 is as follows:

[0015] S21: Consider N users in the online learning platform as a node t in the graph i , the corresponding user portrait features are used as the user features f of the corresponding node i , and then construct N nodes t in the user relationship graph i The set T = {t1,…t N}; Define the distance between nodes as the Euclidean distance between their respective user features;

[0016] S22: Use the K-nearest neighbor algorithm to establish edge connections between all N nodes in the user relationship graph, where each node t in the user relationship graph i All of them need to find the K closest nodes in the graph and find the nearest nodes at node t. i Establish edges with these K nodes; all established edges are recorded in the edge set E, and only one record is retained for repeated edges; construct the adjacency matrix A∈R based on the established edge set E N×N , where the element A in the i-th row and j-th column is ij Represents node t i and node t j Is there an edge between them? If so, A ij =1, otherwise A ij =0;

[0017] S23: Determine the set of adjacent nodes of any node in the user relationship graph according to the adjacency matrix A, where for node t i , find the index of the column whose element is 1 from the i-th row of A, that is, all the columns with node t i The index of the adjacent node will be the same as the index of node t i The set of adjacent nodes is defined as node t i The adjacent set N i , the node t i The adjacent set N i Contains node t i itself;

[0018] S24: Perform graph convolution operation on the basis of the user relationship graph. The graph convolution is divided into L layers. The input of the first layer of graph convolution is the user portrait feature matrix F, and the input features of each of the remaining layers of graph convolution are the output features of the previous layer of graph convolution. For any k+1th layer of graph convolution, it aggregates the features of the adjacent set of each node in the graph in turn, and re-outputs the user features of each node:

[0019]

[0020] In the formula, is the node t i User features output in the k+1th layer of graph convolution, is the node t j User features output in the k-th layer of graph convolution, where k = 0 |N i | is the potential of the adjacent set N i , equal to the number of nodes contained in the adjacent set;

[0021] S25: For each node t i , the initial user features And the features of all L-layer graph convolution outputs Average and get each node t i Corresponding group characteristics

[0022]

[0023] As a preferred embodiment of the first aspect, in step S3, for the group features of all nodes that have been obtained, the K-Means clustering algorithm is used to cluster the features, so as to divide the users corresponding to the nodes into T types. The specific implementation steps are as follows:

[0024] S31: From all N population characteristics Randomly select T as the initial centers of the cluster;

[0025] S32: For any group feature other than the initial center, calculate the Euclidean distance between it and the T cluster centers one by one, and classify each group feature into the cluster category where the nearest cluster center is located;

[0026] S33: for each cluster category processed by S32, recalculate the respective cluster center according to all group characteristics in the category;

[0027] S34: Repeat S32 to S33 until all cluster centers converge, and finally divide all users corresponding to group characteristics into T user types {c1, c2, …c T}Typical user groups.

[0028] As a preferred embodiment of the first aspect, the specific implementation method of step S4 is as follows:

[0029] S41: Construct a reinforcement learning model for performing a grouping task on the online learning platform; the goal of the grouping task is to divide N users on the platform into G groups, each with H members; the reinforcement learning model is used to target the users who have been divided into T types {c1, c2, ...c T}, predict the user category required by each group; the state s and action a of the reinforcement learning model are defined as the input X and output Y of the model respectively; input X∈R G×H×(D+T) Contains the features of all N group members in G groups. A feature corresponding to a group member of user i consists of two parts. The first part is the D-dimensional group feature of user i obtained in S2, and the second part is the T-dimensional unique hot encoding of the type to which user i belongs after clustering in S3. The output is Y∈R G×T Represents the probability that each group needs to join each type of member;

[0030] S42: After the reinforcement learning model defines the state and action, multiple input and output iterations are performed according to S421 to S424 to complete a round of grouping, wherein the process of S421 to S424 is as follows:

[0031] S421: Initialize the input X to an all-zero tensor, indicating that no grouping action has been performed;

[0032] S422: The reinforcement learning model obtains output Y according to input X, and performs the operation of allocating new group members to each group according to output Y and the number of remaining users in each cluster category. In the process of allocating new group members, firstly, the most needed user types of G groups are obtained according to output Y, and then the T types are allocated in order from the largest to the smallest number of groups in need, wherein: if m of the G groups need c t type of person, the remaining c t There are n users of the same type. If m≤n, then any m c n The users of the same type are assigned to these m groups, and the remaining groups will no longer be considered for assignment c n type of users; if m>n, the allocated users are insufficient, then these n c n The users of the same type are assigned to the n groups with the highest probability in the output Y. The remaining mn groups continue to assign users of other types and determine the specific user type by comparing the probability values ​​in the output Y. If c n All users of this type have been assigned. You will need to skip assigning users to c in the future. n Assignment of type users;

[0033] S423: After executing a round of S422 operation, each group is assigned a new member, and the group feature corresponding to the new member and the T-dimensional unique hot encoding of the type to which the new member belongs are used to replace the element value at the corresponding position in the input X, thereby obtaining a new input X;

[0034] S424: After S422 to S423 are iterated and repeated H times, all N users on the platform are divided into G groups, each with H members, and the grouping is completed;

[0035] S43: After completing a round of grouping in S42, the reinforcement learning model is trained using a policy gradient method; where the parameter of the reinforcement learning model is θ, the parameter is updated in the following manner:

[0036]

[0037]

[0038] in is the feedback expectation under the reinforcement learning model with parameter θ, which is approximated by Monte Carlo sampling. The total number of samplings is S, and one sampling is one round of grouping, which contains H samplings in total; Indicates that at the hth sampling in the nth round of grouping, in state Take action The probability of R(τ n ) represents a series of grouping actions τ taken in the nth round of grouping n The feedback obtained after is obtained from the actual feedback results of users on the grouping results on the online learning platform; α is the learning rate;

[0039] S44: After the reinforcement learning model is trained, its model parameters are fixed for actual prediction and grouping of users on the online learning platform.

[0040] As a preferred embodiment of the first aspect above, the reinforcement learning model is composed of a fully connected neural network.

[0041] As a preferred embodiment of the first aspect, the reinforcement learning model regularly collects online data on the platform and updates model parameters when being used to perform actual predictive grouping of users on the online learning platform.

[0042] In a second aspect, the present invention provides a reinforcement learning grouping device based on typical characteristics of a user group, comprising:

[0043] The portrait feature extraction module is used to extract and filter the portrait information of the target users in the online learning platform, and to combine the personal information portrait of each target user and the interactive behavior portrait between the user and the platform as the user portrait feature;

[0044] The group feature extraction module is used to build a user relationship graph based on the user portrait features of all users. Each node in the graph corresponds to a target user. The graph convolutional neural network is then used to extract and aggregate features of the user relationship graph to obtain the group features corresponding to each node in the graph.

[0045] The clustering module is used to cluster the group characteristics of all nodes in the user relationship graph to form multiple types of typical user groups;

[0046] The grouping module is used to predict the typical types of groups based on the group features obtained in the group feature extraction module and the typical user groups clustered in the clustering module using the reinforcement learning model, so that the users in each typical user group are gradually divided into different groups, and finally the optimal grouping of all target users on the platform is obtained.

[0047] In a third aspect, the present invention provides a computer-readable storage medium, characterized in that a computer program is stored on the storage medium, and when the computer program is executed by a processor, a reinforcement learning grouping method based on typical characteristics of a user group as described in any of the schemes of the first aspect above is implemented.

[0048] In a fourth aspect, the present invention provides a reinforcement learning grouping device based on typical characteristics of a user group, comprising a memory and a processor;

[0049] The memory is used to store computer programs;

[0050] The processor is used to implement the reinforcement learning grouping method based on typical characteristics of a user group as described in any of the solutions of the first aspect above when executing the computer program.

[0051] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0052] The advantages of the present invention are that it makes full use of the group relationship between individual users and users for feature modeling, reduces the dimension of individual user features to the typical group feature space for grouping, and takes into account the characteristics of huge user scale and explosive combination in the user group grouping problem in the online learning platform, and uses the Q-learning reinforcement learning method to solve it. The method is suitable for grouping student users in discussion, practice, and cooperation scenarios in large-scale online learning, and has high grouping efficiency, good accuracy, and strong interpretability. The present invention has very important practical application value for user grouping teaching practice in large-scale online learning platforms. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 Flowchart of a reinforcement learning grouping method based on typical characteristics of user groups. DETAILED DESCRIPTION

[0054] In order to make the above-mentioned purpose, features and advantages of the present invention more obvious and easy to understand, the specific implementation mode of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in each embodiment of the present invention can be combined accordingly without conflicting with each other.

[0055] Before describing in detail, several concepts mentioned in the present invention are defined as follows:

[0056] The online learning platform in the present invention refers to a platform for learners to study courses online through the network, which may also be referred to as the platform below. Each learner is a user on the platform, and the users who need to be motivated on the platform are called target users. The user's registration information on the platform is the personal information filled in by the user according to the registration mechanism of the platform, such as gender, region, education level, etc. At the same time, during the process of learning on the platform, users will constantly interact with the platform, such as conducting course learning activities, interacting with the platform, etc., and these data are recorded in the form of behavioral information generated by the interaction between the user and the platform.

[0057] like Figure 1 As shown, in a preferred embodiment of the present invention, a reinforcement learning grouping method based on typical characteristics of a user group is provided, and the steps are as follows:

[0058] S1: Extract and filter the target user’s profile information on the online learning platform, and combine the personal information profile of each target user and the interaction behavior profile between the user and the platform as the user profile feature.

[0059] As a specific implementation of this embodiment, the specific implementation method of the above step S1 is as follows:

[0060] S11: Extracting the target user's portrait information from the database of the online learning platform, wherein the portrait information includes a personal information portrait and an interaction behavior portrait between the user and the platform.

[0061] S12: For each user i’s portrait information, remove irrelevant features, retain features related to the grouping task, and digitize the retained portrait information dimensions to real number space. Then, concatenate the portrait information of all dimensions together to obtain the D-dimensional user portrait feature f i ; User portrait features f of all N target users on the online learning platform i Combined into user portrait feature matrix F∈R N×D .

[0062] In addition, if the quality of information extracted from the platform is not high, feature engineering processing is required in advance. The specific form of feature engineering processing needs to be determined according to the data situation. It can generally be achieved by removing outliers, filling missing values ​​and feature screening. Feature screening requires dimensionality reduction of all feature dimensions and eliminating feature dimensions that are not related to user grouping.

[0063] S2: Build a user relationship graph based on the user portrait features of all users. Each node in the graph corresponds to a target user. Then use a graph convolutional neural network to extract and aggregate features of the user relationship graph to obtain the group features corresponding to each node in the graph.

[0064] As a specific implementation of this embodiment, the specific implementation method of the above step S2 is as follows:

[0065] S21: Consider N users in the online learning platform as a node t in the graph i , the corresponding user portrait features are used as the user features f of the corresponding node i , and then construct N nodes t in the user relationship graph i The set T = {t1,…t N}; Define the distance between nodes as the Euclidean distance between their respective user features.

[0066] Specifically, the N nodes t in the established graph i Denoted as T = {t1,…t N}, that is, each user corresponds to each node in the graph one by one, then the distance between nodes can be defined as:

[0067]

[0068] In the formula, d(t i ,t j ) is the node t i and t j The distance between i ,f j ) is the node t i and t j The characteristic f i and f j The distance between i With f j In R D Euclidean distance in space;

[0069] S22: Use the K-nearest neighbor algorithm to establish edge connections between all N nodes in the user relationship graph, where each node t in the user relationship graph i All of them need to find the K closest nodes in the graph and find the nearest nodes at node t. i Establish edges with these K nodes; all established edges are recorded in the edge set E, and only one record is retained for repeated edges; construct the adjacency matrix A∈R based on the established edge set E N×N , where the element A in the i-th row and j-th column is ij Represents node t i and node t j Is there an edge between them? If so, A ij =1, otherwise A ij =0.

[0070] For a node t iWhen looking for the K closest nodes in a graph, we essentially find the nearest K nodes in F = [f1,…,f n ,…f N ],1≤n≤N for a single user’s feature f n ∈R D , find the K features most similar to it {f n1 ,…,f nk}, where most similar means the distance defined above is the smallest.

[0071] S23: According to the adjacency matrix A, the adjacent node set of any node in the user relationship graph can be determined, where for node t i , find the index of the column whose element is 1 from the i-th row of A, that is, all the columns with node t i The index of the adjacent node will be the same as the index of node t i The set of adjacent nodes is defined as node t i The adjacent set N i , the node t i The adjacent set N i Contains node t i Self. According to the adjacent set N i At node t n With {t n1 ,…t nk} establish edges for each node in the edge set E, and record all edges into the edge set E. Note that only one record is retained for repeated edges in the edge set E.

[0072] S24: Perform graph convolution operation on the basis of the user relationship graph. The graph convolution is divided into L layers. The input of the first layer of graph convolution is the user portrait feature matrix F, and the input features of each of the remaining layers of graph convolution are the output features of the previous layer of graph convolution. For any k+1th layer of graph convolution, it aggregates the features of the adjacent set of each node in the graph in turn, and re-outputs the user features of each node:

[0073]

[0074] In the formula, is the node t i User features output in the k+1th layer of graph convolution, is the node t j User features output in the k-th layer of graph convolution, where in particular, k = 0 |N i | is the potential of the adjacent set N i , which is equal to the number of nodes contained in the adjacent set. The above formula shows that in order to obtain Need to node t i The adjacent set N i The characteristics of all nodes in Take average.

[0075] The total number of layers L of the above graph convolution can be optimized according to the actual excitation effect. In a preferred embodiment, L can be optimized to 3 layers.

[0076] S25: For each node t i , the initial user features And the features of all L-layer graph convolution outputs Average and get each node t i Corresponding group characteristics

[0077]

[0078] In the formula, is the node t i The output user features at the kth layer, is the node t i group characteristics.

[0079] S3: Cluster the group characteristics of all nodes in the user relationship graph to form multiple types of typical user groups.

[0080] As a specific implementation of this embodiment, in the above step S3, for the group features of all nodes that have been obtained, the K-Means clustering algorithm is used to cluster the features, so as to divide the users corresponding to the nodes into T types. The specific implementation steps are as follows:

[0081] S31: From all N population characteristics Randomly select T as the initial centers of the cluster.

[0082] S32: For any group feature other than the initial center, calculate the Euclidean distance (defined as above) between it and the T cluster centers one by one, and classify each group feature into the cluster category where the cluster center closest to it is located.

[0083] S33: For each cluster category processed by S32, recalculate the respective cluster center according to all group characteristics in the category.

[0084] S34: Repeat S32 to S33 until all cluster centers converge, and finally divide all users corresponding to group characteristics into T user types {c1, c2, …c T}Typical user groups.

[0085] The convergence of the cluster center can be judged based on the translation distance ∈ before and after the corresponding cluster center is updated. When the update error, that is, the center distance before and after the update, is less than ∈, it means that the clustering has basically converged.

[0086] S4: Based on the group characteristics obtained in S2 and the typical user groups obtained by clustering in S3, the reinforcement learning model is used to predict the typical types of the group, so that the users in each typical user group are gradually divided into different groups, and finally the optimal grouping of all target users on the platform is obtained.

[0087] As a specific implementation of this embodiment, the specific implementation method of the above step S4 is as follows:

[0088] S41: Construct a reinforcement learning model for performing grouping tasks on users on the online learning platform. The goal of the grouping task is to divide N users on the platform into G groups, with H members in each group. The reinforcement learning model is used to group the users into T types {c1, c2, …c T}, predict the user category required for each group, so as to facilitate the grouping operation. In the present invention, the above reinforcement learning model can be composed of a fully connected neural network. The state s and action a of the reinforcement learning model are defined as the input X and output Y of the model respectively; the input sample X∈R G×H×(D+T) Contains the features of all N members in G groups. A feature corresponding to a member of user i consists of two parts. The first part is the D-dimensional group feature of user i obtained in S2 The second part is the T-dimensional unique hot encoding of the category to which user i belongs after clustering in S3 (obtained by encoding the corresponding category serial number), and the output is Y∈R G×T Represents the probability that each group needs to join each type of member, and the output can be used to predict the grouping action that should be taken.

[0089] It should be noted that the above input X is only defined for it, but it needs to be initialized to an all-zero tensor during the initial training of the model. During the training process, the features of all N members in the G groups are continuously filled in for update.

[0090] S42: After the reinforcement learning model defines the state and action, multiple input and output iterations are performed according to S421 to S424 to complete a round of grouping, wherein the process of S421 to S424 is as follows:

[0091] S421: Initialize the input X to an all-zero tensor, indicating that no grouping action has been performed;

[0092] S422: The reinforcement learning model obtains output Y according to input X, and performs the operation of allocating new members to each group according to output Y and the number of remaining users in each cluster category. In the process of allocating new members, firstly, the most needed user types of each of the G groups are obtained according to output Y, and then the T types are allocated in order from the largest to the smallest number of groups in need. The already allocated groups and types are no longer taken into consideration. In the allocation, if m of the G groups need c in a certain allocation, t type of person, the remaining c t There are n users of the same type. If m≤n, then any m c n Type of users are assigned to these m groups, and the remaining groups will no longer consider c n The corresponding probability is that the remaining groups will no longer consider the allocation c n type of users; if m>n, the allocated users are insufficient, then these n c n The users of the same type are assigned to the n groups with the highest probability in the output Y. The remaining mn groups continue to assign users of other types and determine the specific user type by comparing the probability values ​​in the output Y. If c n All users of this type have been assigned, so c will no longer be considered in all steps after this step. n The corresponding probability, that is, the subsequent need to skip c n Assignment of type users;

[0093] S423: After executing a round of S422 operation, each group is assigned a new member, and the group feature corresponding to the new member and the T-dimensional unique hot encoding of the type to which the new member belongs are used to replace the element value at the corresponding position in the input X, thereby obtaining a new input X;

[0094] S424: After S422 to S423 are iterated and repeated H times, all N users on the platform are divided into G groups, each with H members, and the grouping is completed;

[0095] S43: After completing a round of grouping in S42, the reinforcement learning model is trained using a policy gradient method; where the parameter of the reinforcement learning model is θ, the parameter is updated in the following manner:

[0096]

[0097]

[0098] in is the feedback expectation under the reinforcement learning model with parameter θ, which is approximated by Monte Carlo sampling. The total number of samplings is S, and one sampling is one round of grouping, which contains H samplings in total; Indicates that at the hth sampling in the nth round of grouping, in state Take action The probability of R(τ) is predicted by the reinforcement learning model and is therefore related to θ; n ) represents a series of grouping actions τ taken in the nth round of grouping n The feedback obtained after the grouping results are obtained from the actual feedback results of users on the online learning platform, which is a known quantity. The above formula shows that we need to find The gradient of the model parameters is updated, and the learning rate adopted is α.

[0099] S44: After the reinforcement learning model is trained, its model parameters are fixed for actual prediction and grouping of users on the online learning platform.

[0100] It should be noted that, in order to facilitate model training, after grouping users, the online learning platform needs to collect feedback from users on whether they are satisfied with the grouping. The collected feedback can be further used as R(τ n ) is used to train the model.

[0101] In addition, when the above reinforcement learning model is used to perform actual prediction and grouping of users on the online learning platform, it can regularly collect online data on the platform and update the model parameters, so as to continuously use the new data generated on the online learning platform to retrain the model and improve the prediction accuracy of the model. In other words, the training of the above reinforcement learning model can be continuously iterative, that is, after the reinforcement learning model completes a new round of training, the learnable parameters in the model can be temporarily fixed, and then the reinforcement learning model is used to score the users on the online learning platform, and then the evaluation feedback data of the users on the grouping is collected, and the model is continuously trained using the evaluation feedback data of the users on the grouping over a period of time, so that the user grouping on the online learning platform continuously meets the user's demand for the expected grouping.

[0102] Based on the method shown in S1 to S4 in the above embodiment, the method is applied to a specific example to demonstrate its effect. The specific process is as described above and will not be repeated here. The specific parameter settings and the achieved effect are mainly demonstrated below.

[0103] Example

[0104] The present invention is described in detail below by taking the user grouping process of some online learning platforms as an example, and the specific steps are as follows:

[0105] 1) According to the aforementioned step S1, the user's portrait data is obtained, including information portrait and behavior portrait. The following is an example of a user's specific information portrait: gender: male, age: 19, region: Zhejiang Province, income: 0 / month, marital status: no, occupation: student. The following is an example of the user's behavior portrait: course learning time: 200 days, number of courses learned: 3, number of online interactions: 20.

[0106] Then, the acquired user portrait data is processed. First, irrelevant features are removed. For the course learning grouping, region, income, and marital status are almost irrelevant to the grouping, so they are removed from the user portrait. Then the features of the portrait are digitized. An example of the digitization of this user is given below. Gender: 1, age: 0.19, occupation: 0.1, course learning time: 200, number of courses learned: 3, number of online interactions: 20. This is just an example, and the specific digitization method is not unique. After digitization, the initial portrait feature vector of the user is obtained [1, 0.19, 0.1, 200, 3, 20], corresponding to D1 = 6 in S1. Finally, the portrait feature vectors of all users are combined into a matrix, that is, the initial portrait feature F in S1. Assuming that there are N = 10 users to be grouped, then F∈R 10×6 .

[0107] 2) According to the aforementioned S2 step, establish a user relationship graph for the initial user portrait features and perform graph convolution. Assuming that the graph is established using K nearest neighbors and K is set to 2, each user will have at least two neighbors. Assuming that the portrait feature of a user (corresponding node) t1 is [1, 0.19, 0.1, 200, 3, 20], t1's adjacent set N1 = {t1, t2, t3}, it means that there are two users adjacent to him in the user relationship graph, namely t2 and t3. Assume that their portrait features are [0, 0.18, 0.2, 211, 3, 23] and [1, 0.21, 0.1, 189, 3, 19]. When performing graph convolution, the first graph convolution calculation process for t1 is: For convenience of representation, the calculation precision is two decimal places. Assume that the graph convolution has L=3 layers, and the image features of t1 obtained three times are [0.67, 0.19, 0.13, 200, 3, 20.67], [0.32, 0.11, 0.15, 190.23, 4.2, 20.11], [0.54, 0.22, 0.34, 188.98, 3, 18.32], after three graph convolutions, the group features of the user are: That is, the original image features and the image features of the three-time graph convolution are averaged.

[0108] 3) According to the above S3 step, use the K-Means method to cluster the final user features. Assuming that the number of centers is T = 3, first randomly select the final portrait features of 3 users as the 3 centers, then classify the portrait features of each user and assign them to the center closest to it, and then for each class, find the average feature of the class as the new center. Repeat this process until the category center no longer changes significantly. In this way, 3 stable centers will be obtained, and each user has a type, set to {a, b, c}, and the corresponding number of people is {3, 5, 2}.

[0109] 4) According to the above step S4, the reinforcement learning model is trained using the policy gradient algorithm.

[0110] There are 10 users who need to be divided into groups, with 2 people in each group, for a total of 5 groups. 10}, where t1~t3 belong to type a, one-hot encoding [1,0,0], t4~t8 belong to type b, one-hot encoding [0,1,0], t9~t 10 Belongs to type c, one-hot encoding [0,0,1]. The initial input X∈R 5×2×(6+3) , all its elements are 0. After inputting the network, the output Y∈R 5×3. The demand probabilities of each group for each type are [0.6, 0.2, 0.2], [0.6, 0.1, 0.3], [0.5, 0.3, 0.2], [0.7, 0.2, 0.1], [0.2, 0.5, 0.3], and the types with the highest probabilities are a, a, a, a, b, respectively. Therefore, according to the demand, the order of a→b→c is followed. For type a, the probabilities of groups 1, 2, 3, and 4 are the highest, so there are 4 groups that select users of type a, but in fact there are only 3 people of type a, and these 3 people are assigned to groups 1, 2, and 4, because the probability of group 3, 0.5, is less than the probabilities of the other 3 groups, 0.6 or 0.7. For type b, first, the probability value of each group is deleted. The groups and types that have been assigned are no longer considered. The remaining ones are the b and c probabilities corresponding to groups 3 and 5 [0.3, 0.2], [0.5, 0.3]. Both groups need type b, so any two people in b are assigned to groups 3 and 5. At this time, each group has a new member, so there is no need to involve type c in this assignment. The current grouping status is {t1}{t2}{t4}{t3}{t5}. The group characteristics and type characteristics of t1 are combined to obtain a 6 3 = 9-dimensional feature vector [0.63, 0.18, 0.18, 194.8, 3.3, 19.78, 1, 0, 0]. Use this vector to update the value of the first member position of the first group of X, and similarly for other groups. Input the updated X into the network again and get the new output Y, [0.1, 0.5, 0.4], [0.3, 0.5, 0.2], [0.0, 0.1, 0.9], [0.6, 0.3, 0.1], [0.2, 0.3, 0.5]. Because a is no longer occupied, remove the probability of a and get [0.5, 0.4], [0.5, 0.2], [0.1, 0.9], [0.3, 0.1], [0.3, 0.5]. The types required by the 5 groups are b, b, c, b, c, so they are assigned in the order of b→c. For type b, groups 1, 2, and 4 need it. At this time, there are 3 people left in group b, and the 3 people are assigned to groups 1, 2, and 4. For type c, after removing groups 1, 2, 4 and b, only the probability value of c remains, which means that all remaining groups need c, so the two people of c are assigned to groups 3 and 5. So far, the grouping status is {t1, t6}{t2, t7}{t4, t9}{t3, t8}{t5, t 10}, the grouping ends.

[0111] In the training phase, it is necessary to estimate the feedback expectation and calculate the gradient. In the above round of grouping, there are two steps. In the first step, because groups 1, 2, and 4 are assigned a, their corresponding probabilities of a are multiplied. Since the three groups are randomly assigned, they also need to be multiplied by That is 0.17×0.6×0.6×0.7, and the 3rd and 5th groups are assigned b, so we multiply the probabilities of b corresponding to them and consider the random assignment. That is 0.5×0.3×0.5, so Similarly, the second step, Assume that only one round of grouping is performed as sampling, that is, S = 1, and the feedback R(τ 1 )=5, then the gradient of the feedback expectation is calculated as Then back propagation updates the network parameters, completing one step of training, and iterates multiple times until the network converges.

[0112] Experiments show that the grouping method provided by the present invention is suitable for grouping students in discussion, practice, and cooperation scenarios in large-scale online learning, and has high grouping efficiency, good accuracy, and strong interpretability. The present invention has very important practical application value for user grouping teaching practice in large-scale online learning platforms.

[0113] In another embodiment of the present invention, based on the same inventive concept, a reinforcement learning grouping device based on typical characteristics of a user group is provided, which includes:

[0114] The portrait feature extraction module is used to extract and filter the portrait information of the target users in the online learning platform, and to combine the personal information portrait of each target user and the interactive behavior portrait between the user and the platform as the user portrait feature;

[0115] The group feature extraction module is used to build a user relationship graph based on the user portrait features of all users. Each node in the graph corresponds to a target user. The graph convolutional neural network is then used to extract and aggregate features of the user relationship graph to obtain the group features corresponding to each node in the graph.

[0116] The clustering module is used to cluster the group characteristics of all nodes in the user relationship graph to form multiple types of typical user groups;

[0117] The grouping module is used to predict the typical types of groups based on the group features obtained in the group feature extraction module and the typical user groups clustered in the clustering module using the reinforcement learning model, so that the users in each typical user group are gradually divided into different groups, and finally the optimal grouping of all target users on the platform is obtained.

[0118] The modules in the above reinforcement learning grouping device respectively correspond to S1 to S4 of the above embodiments, so the specific implementation methods thereof can also refer to the above embodiments, which will not be described in detail.

[0119] It should be noted that, according to the embodiments disclosed in the present invention, the specific implementation functions of various modules in the above-mentioned reinforcement learning grouping device can be implemented by a written computer software program, and the computer program contains program codes for executing corresponding methods.

[0120] In another embodiment of the present invention, based on the same inventive concept, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the reinforcement learning grouping method based on typical characteristics of a user group as described in S1 to S4 above is implemented.

[0121] In another embodiment of the present invention, based on the same inventive concept, a computer device is provided, which includes a memory and a processor;

[0122] The memory is used to store computer programs;

[0123] The processor is used to implement the reinforcement learning grouping method based on typical characteristics of user groups as described in S1 to S4 above when executing the computer program.

[0124] It is understandable that the above storage medium may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk storage. Optionally, the memory may also be at least one storage device located away from the above processor.

[0125] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0126] It should be noted that the computer device can be any physical machine with a GPU, a CPU, and a smart network card slot, including personal computers (PCs) and servers.

[0127] The above-described embodiment is only a preferred solution of the present invention, but it is not intended to limit the present invention. A person skilled in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. A reinforcement learning grouping method based on typical characteristics of user groups, characterized in that: Here are the steps: S1: Extract and filter the target user’s profile information on the online learning platform, and combine the personal information profile of each target user and the interaction behavior profile between the user and the platform as the user profile feature; S2: Build a user relationship graph based on the user portrait features of all users. Each node in the graph corresponds to a target user. Then use a graph convolutional neural network to extract and aggregate features of the user relationship graph to obtain the group features corresponding to each node in the graph. S3: Cluster the group characteristics of all nodes in the user relationship graph to form multiple types of typical user groups; S4: Based on the group characteristics obtained in S2 and the typical user groups obtained by clustering in S3, the reinforcement learning model is used to predict the group typical types, so that the users in each typical user group are gradually divided into different groups, and finally the optimal grouping of all target users on the platform is obtained; The specific implementation method of step S4 is as follows: S41: Construct a reinforcement learning model for performing a grouping task on the online learning platform; the goal of the grouping task is to divide N users on the platform into G groups, each with H members; the reinforcement learning model is used to target the users who have been divided into T types {c1, c2, ...c T }, predict the user category required by each group; the state s and action a of the reinforcement learning model are defined as the input X and output Y of the model respectively; input X∈R G×H×(D+T) Contains the features of all N group members in G groups. A feature corresponding to a group member of user i consists of two parts. The first part is the D-dimensional group feature of user i obtained in S2, and the second part is the T-dimensional unique hot encoding of the type to which user i belongs after clustering in S3. The output is Y∈R G×T Represents the probability that each group needs to join each type of member; S42: After the reinforcement learning model defines the state and action, multiple input and output iterations are performed according to S421 to S424 to complete a round of grouping, wherein the process of S421 to S424 is as follows: S421: Initialize the input X to an all-zero tensor, indicating that no grouping action has been performed; S422: The reinforcement learning model obtains output Y according to input X, and performs the operation of allocating new group members to each group according to output Y and the number of remaining users in each cluster category. In the process of allocating new group members, firstly, the most needed user types of G groups are obtained according to output Y, and then the T types are allocated in order from the largest to the smallest number of groups in need, wherein: if m of the G groups need c t type of person, the remaining c t There are n users of the same type. If m≤n, then any m c n The users of the same type are assigned to these m groups, and the remaining groups will no longer be considered for assignment c n type of users; if m>n, the number of users allocated is insufficient, then these n c n The users of the same type are assigned to the n groups with the highest probability in the output Y. The remaining mn groups continue to assign users of other types and determine the specific user type by comparing the probability values ​​in the output Y. If c n All users of this type have been assigned. You will need to skip assigning users to c in the future. n Allocation of user types; S423: After executing a round of S422 operation, each group is assigned a new member, and the group feature corresponding to the new member and the T-dimensional unique hot encoding of the type to which the new member belongs are used to replace the element value at the corresponding position in the input X, thereby obtaining a new input X; S424: After S422 to S423 are iterated and repeated H times, all N users on the platform are divided into G groups, each with H members, and the grouping is completed; S43: After completing a round of grouping in S42, the reinforcement learning model is trained using a policy gradient method; where the parameter of the reinforcement learning model is θ, the parameter is updated in the following manner: in is the feedback expectation under the reinforcement learning model with parameter θ, which is approximated by Monte Carlo sampling. The total number of samplings is S, and one sampling is one round of grouping, which contains H samplings in total; Indicates that at the hth sampling in the nth round of grouping, in state Take action The probability of R(τ n ) represents a series of grouping actions τ taken in the nth round of grouping n The feedback obtained after is obtained from the actual feedback results of users on the grouping results on the online learning platform; α is the learning rate; S44: After the reinforcement learning model is trained, its model parameters are fixed for actual prediction and grouping of users on the online learning platform.

2. The method for grouping users by reinforcement learning based on typical characteristics of user groups according to claim 1, characterized in that: The specific implementation method of step S1 is as follows: S11: extracting the target user's portrait information from the database of the online learning platform, wherein the portrait information includes the personal information portrait and the interaction behavior portrait between the user and the platform; S12: For each user i’s portrait information, retain the features related to the grouping task, and digitize the retained portrait information dimensions to the real number space, and then concatenate the portrait information of all dimensions together to obtain the D-dimensional user portrait feature f i ; User profile features f of all N target users on the online learning platform i Combined into user portrait feature matrix F∈R N ×D .

3. The reinforcement learning grouping method based on typical characteristics of user groups according to claim 1, characterized in that: The specific implementation method of step S2 is as follows: S21: Consider N users in the online learning platform as a node t in the graph i , the corresponding user portrait features are used as the user features f of the corresponding node i , and then construct N nodes t in the user relationship graph i The set T = {t1,…t N }; Define the distance between nodes as the Euclidean distance between their respective user features; S22: Use the K-nearest neighbor algorithm to establish edge connections between all N nodes in the user relationship graph, where each node t in the user relationship graph i All of them need to find the K closest nodes in the graph and find the nearest nodes at node t. i Establish edges with these K nodes; all established edges are recorded in the edge set E, and only one record is retained for repeated edges; construct the adjacency matrix A∈R based on the established edge set E N×N , where the element A in the i-th row and j-th column is ij Represents node t i and node t j Is there an edge between them? If so, A ij =1, otherwise A ij =0; S23: Determine the set of adjacent nodes of any node in the user relationship graph according to the adjacency matrix A, where for node t i , find the index of the column whose element is 1 from the i-th row of A, that is, all the columns with node t i The index of the adjacent node will be the same as the index of node t i The set of adjacent nodes is defined as node t i The adjacent set N i , the node t i The adjacent set N i Contains node t i itself; S24: Perform graph convolution operation on the basis of the user relationship graph. The graph convolution is divided into L layers. The input of the first layer of graph convolution is the user portrait feature matrix F, and the input features of each of the remaining layers of graph convolution are the output features of the previous layer of graph convolution. For any k+1th layer of graph convolution, it aggregates the features of the adjacent set of each node in the graph in turn, and re-outputs the user features of each node: In the formula, f i (k+1) is the node t i User features output in the k+1th layer of graph convolution, f j (k) is the node t j User features output in the k-th layer of graph convolution, where k = 0 f j (0) =f j ;|N i | is the potential of the adjacent set N i , equal to the number of nodes contained in the adjacent set; S25: For each node t i , the initial user feature f i (0) And the features {f i (1) ,f i (2) ,f i (3) ,…,f i (L) } to get the average of each node t i The corresponding group characteristics f i * :

4. The method for grouping users by reinforcement learning based on typical characteristics of user groups according to claim 1, characterized in that: In step S3, for the group features of all nodes that have been obtained, the K-Means clustering algorithm is used to cluster the features, so as to divide the users corresponding to the nodes into T types. The specific implementation steps are as follows: S31: From all N population features f i * Randomly select T as the initial centers of the cluster; S32: For any group feature other than the initial center, calculate the Euclidean distance between it and the T cluster centers one by one, and classify each group feature into the cluster category where the nearest cluster center is located; S33: for each cluster category processed by S32, recalculate the respective cluster center according to all group characteristics in the category; S34: Repeat S32 to S33 until all cluster centers converge, and finally divide all users corresponding to group characteristics into T user types {c1, c2, …c T }Typical user groups.

5. The method for grouping users by reinforcement learning based on typical characteristics of user groups according to claim 1, characterized in that: The reinforcement learning model is composed of a fully connected neural network.

6. The method for grouping users by reinforcement learning based on typical characteristics of user groups according to claim 1, characterized in that: When the reinforcement learning model is used to perform actual prediction grouping on users on the online learning platform, online data on the platform is regularly collected and model parameters are updated.

7. A reinforcement learning grouping device based on typical characteristics of a user group, used to implement the reinforcement learning grouping method based on typical characteristics of a user group as claimed in any one of claims 1 to 6, characterized in that: include: The portrait feature extraction module is used to extract and filter the portrait information of the target users in the online learning platform, and to combine the personal information portrait of each target user and the interactive behavior portrait between the user and the platform as the user portrait feature; The group feature extraction module is used to build a user relationship graph based on the user portrait features of all users. Each node in the graph corresponds to a target user. The graph convolutional neural network is then used to extract and aggregate features of the user relationship graph to obtain the group features corresponding to each node in the graph. The clustering module is used to cluster the group characteristics of all nodes in the user relationship graph to form multiple types of typical user groups; The grouping module is used to predict the typical types of groups based on the group features obtained in the group feature extraction module and the typical user groups clustered in the clustering module using the reinforcement learning model, so that the users in each typical user group are gradually divided into different groups, and finally the optimal grouping of all target users on the platform is obtained.

8. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the reinforcement learning grouping method based on typical characteristics of a user group as described in any one of claims 1 to 6 is implemented.

9. A reinforcement learning grouping device based on typical characteristics of user groups, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is used to implement the reinforcement learning grouping method based on typical characteristics of a user group as described in any one of claims 1 to 6 when executing the computer program.

Images