A course recommendation method, device, computer equipment and storage medium

Abstract

The present application discloses a course recommendation method, device, computer equipment and storage medium, the method comprising determining the current state of the target user according to the first historical course browsing data of the target user; obtaining the candidate course categories that meet the screening conditions according to the current state and the pre-trained target reinforcement learning network model, wherein the target reinforcement learning network model uses the course category to which the online course belongs as the action space, and the output quantity of the output action space is the same as the total quantity of the course category; and selecting a set number of target courses from the online courses corresponding to each candidate course category and pushing them to the target user. Using this method, the processing form of the reinforcement learning network model is adopted to achieve the effect that the online courses recommended to the user can bring long-term benefits to the online teaching platform; at the same time, the dimensionality reduction processing of the action space output in the reinforcement learning network model is realized, which ensures the effective recommendation of the online courses to the user end and improves the user experience.

Description

Technical Field

[0001] The present application relates to the technical field of information recommendation, and in particular to a course recommendation method, device, computer equipment and storage medium. Background Art

[0002] With the rapid development and popularization of Internet technology, digital online learning methods have been increasingly accepted by the public, and the number of online courses for users to learn on online teaching platforms has also increased explosively. Faced with such a huge amount of information, it is difficult for users to quickly search for courses that they are interested in or want to learn.

[0003] At present, the platform often solves the above problems by actively or passively recommending courses to users. Most of the existing course recommendations are achieved by modeling the correlation of historical operation data of users on the online teaching platform, so as to predict the courses that users are interested in and recommend them. This modeling method only considers the short-term preferences and benefits of users, ignoring the long-term benefits of the entire platform, and cannot match the interests of the platform. The existing recommendation methods that consider long-term benefits are not applicable to large-scale online course recommendations. Summary of the invention

[0004] In view of this, the embodiments of the present application provide a course recommendation method, apparatus, computer device and storage medium, which can achieve effective recommendation for large-scale online courses on the basis of ensuring the long-term benefits of the platform.

[0005] In a first aspect, an embodiment of the present application provides a course recommendation method, comprising:

[0006] Determining the current state of the target user according to the first history course browsing data of the target user;

[0007] According to the current state and the pre-trained target reinforcement learning network model, candidate course categories that meet the screening conditions are obtained, wherein the target reinforcement learning network model uses the course category to which the online course belongs as an action space, and the output quantity of the output action space is the same as the total quantity of the course category;

[0008] A set number of target courses are selected from the online courses corresponding to each of the candidate course categories and pushed to the target user.

[0009] Furthermore, the step of dividing the course categories to which the online courses belong includes:

[0010] Acquire the second historical course browsing data of each selected user from the message queue, and form a course browsing sequence corresponding to each of the users;

[0011] Taking each of the course browsing sequences as a sentence to be processed, obtaining the course vector of each online course through a word vector partitioning model, and forming a course vector set;

[0012] The course vector set is clustered to obtain the output number of clusters, and the cluster center vector of each cluster is correspondingly determined as a course category.

[0013] Furthermore, determining the current status of the target user according to the historical course browsing data of the target user includes:

[0014] Performing word segmentation processing on the first historical course browsing data of the target user within a set time period to determine a browsed course vector corresponding to the browsed course of the target user;

[0015] The current state of the target user is determined by taking the average vector of each browsed course vector.

[0016] Furthermore, the candidate course categories that meet the screening conditions are obtained according to the current state and the pre-trained target reinforcement learning network model, including:

[0017] Input the current state into the target reinforcement learning network model, and output the output number of candidate vectors as the action space through the target reinforcement learning network model, wherein each candidate vector identifies a course category;

[0018] For each course category, the cumulative reward value of the course category is determined by combining the current state and the current network parameters of the target reinforcement learning network model through a given cumulative reward value model;

[0019] The course categories are ranked according to the cumulative return value, and the course category at the top first set ranking is used as the candidate course category.

[0020] Furthermore, the selecting a set number of target courses from the online courses corresponding to each of the candidate course categories and pushing them to the target user includes:

[0021] For each candidate course category, obtain the cluster center vector of the candidate course category;

[0022] Determine the distance value between each course vector in the cluster cluster associated with the cluster center vector and the cluster center vector;

[0023] Rank the course vectors according to the distance value, and use the course vectors in the top two set rankings as the courses to be recommended;

[0024] Target courses that meet fine-grained screening conditions are selected from the courses to be recommended and pushed to the target users respectively.

[0025] Furthermore, the training steps of the target reinforcement learning network model include:

[0026] The two reinforcement learning network models with the same network structure but different network parameters are respectively recorded as the real-time training network model and the initial reinforcement learning network model;

[0027] According to each course category identified by each cluster center vector and the second historical course browsing data of each selected user, a training sample set for model training is constructed, wherein each training sample in the training sample set includes: a first state sequence of the user's current state, a target cluster center vector, an instantaneous reward value, and a second state sequence of the next state;

[0028] The loss function is fitted according to the output results of each training sample under the real-time training network model and the initial reinforcement learning network model, and the target reinforcement learning network model is obtained through reverse learning of the fitted loss function.

[0029] Furthermore, the loss function is fitted according to the output results of each training sample under the real-time training network model and the initial reinforcement learning network model, and the target reinforcement learning network model is obtained through reverse learning of the fitted loss function, including:

[0030] For each training sample, determining the current cumulative reward value of each action space vector output by the first state sequence under the real-time training network model, and determining the maximum current cumulative reward value;

[0031] Determining a standard cumulative reward value of the first state sequence relative to the target cluster center vector under the initial reinforcement learning network model;

[0032] Fit the loss function according to the maximum current cumulative return value and the standard cumulative return value corresponding to each training sample;

[0033] The network parameters of the real-time training network model are updated according to the fitted loss function, and when the number of updates meets the parameter replacement cycle, the network parameters of the initial reinforcement learning network model are replaced with the network parameters of the real-time training network model;

[0034] The initial reinforcement learning network model after parameter replacement is determined as the target reinforcement learning network model.

[0035] In a second aspect, an embodiment of the present application provides a course recommendation device, comprising:

[0036] An information determination module, used to determine the current state of the target user according to the first history course browsing data of the target user;

[0037] A candidate determination module, wherein the target is to obtain candidate course categories that meet the screening conditions according to the current state and the pre-trained target reinforcement learning network model, wherein the target reinforcement learning network model uses the course category to which the online course belongs as the action space, and the output quantity of the output action space is the same as the total quantity of the course category;

[0038] The target recommendation module is used to select a set number of target courses from the online courses corresponding to each candidate course category and push them to the target user.

[0039] In a third aspect, an embodiment of the present application further provides a computer device, including: a memory and one or more processors;

[0040] The memory is used to store one or more programs;

[0041] When the one or more programs are executed by the one or more processors, the one or more processors implement the course recommendation method as described in the first aspect above.

[0042] In a fourth aspect, an embodiment of the present application further provides a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to execute the course recommendation method as described in the first aspect.

[0043] The above-mentioned course recommendation method, device, computer equipment and storage medium, the method first determines the current state of the target user according to the first historical course browsing data of the target user, and then obtains the candidate course categories that meet the screening conditions according to the target reinforcement learning network model pre-trained according to the current state set, wherein the action space output in the reinforcement learning is identified by the course category to which the online course belongs, and the output number of the action space is equal to the total number of course categories; finally, a certain amount of target courses can be selected from the candidate course categories including the online courses and pushed to the target user. The above-mentioned technical scheme of this embodiment mainly adopts the processing form of the reinforcement learning network model to achieve the effect that the online courses recommended to the target user can bring long-term benefits to the online education platform; at the same time, by reducing the dimension of the action space output in the reinforcement learning network model, that is, by ensuring that the output number of the output action space is only the same as the number of course categories possessed by the online course, the problem that reinforcement learning cannot adapt to large-scale data processing is solved, thereby realizing the effective recommendation of online courses to the user end and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0045] Figure 1 A flowchart of a course recommendation method provided in Example 1 of the present application;

[0046] Figure 2 A flowchart of a course recommendation method provided in Example 2 of the present application;

[0047] Figure 3 A structural block diagram of a course recommendation device provided in Example 3 of the present application;

[0048] Figure 4 A schematic diagram of the structure of a computer device provided in Example 4 of the present application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical scheme and advantages of the present application clearer, the embodiments of the present application will be further described in detail below in conjunction with the accompanying drawings. It should be clear that the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0050] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0051] In the description of the present application, it should be understood that the terms "first", "second", "third", etc. are only used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances. In addition, in the description of the present application, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0052] Embodiment 1

[0053] Figure 1This is a flow chart of a course recommendation method provided in Embodiment 1 of the present application, which is applicable to recommending online courses to users in an online teaching platform. The method can be executed by a course recommendation device, which can be implemented by hardware and / or software and is generally integrated in a computer device.

[0054] It should be noted that when online teaching is used as an application scenario of this embodiment, the computer device integrating the method provided by this embodiment can be regarded as a platform server for online teaching. Generally, there are often thousands of online courses for users to learn on the online teaching platform, presenting a large amount of online courses. Only using traditional reinforcement learning methods can solve the problem of long-term benefits, but considering that reinforcement learning outputs a candidate set as recommended information, when the scale of the original data used to determine the candidate set is large, it is not possible to guarantee the effective execution of reinforcement learning in information recommendation.

[0055] The course recommendation method provided in this embodiment can effectively solve the problem that online courses are too large to be recommended through reinforcement learning.

[0056] like Figure 1 As shown, a course recommendation method provided in this embodiment 1 specifically includes the following steps:

[0057] S101. Determine the current state of the target user according to the first history course browsing data of the target user.

[0058] In this embodiment, users with learning needs need to register and log in to enter the online teaching platform. The online teaching platform can record the user information of each registered user. Generally, the online teaching platform can recommend online courses to each online user. In this embodiment, each user who enters the online teaching interface through the login operation can be regarded as a target user, and for each target user, the corresponding course recommendation can be achieved through the method provided in this embodiment.

[0059] In this embodiment, the historical course browsing data can be specifically understood as the data generated when the user browses the displayed interfaces within a period of time (such as one day, one week, or even one month) after entering the online teaching platform. Specifically, it can be the data generated when browsing courses, such as which courses have been browsed, how many times a course has been browsed within a period of time, etc., where different courses can be distinguished by course identification numbers. For the convenience of distinction, this embodiment records the historical course browsing data obtained from the target user as the first historical course browsing data.

[0060] In this embodiment, the current state can be specifically understood as the operational state of the target user relative to the online course learning before performing the next action operation when the online course learning is used as the application environment. For example, finishing browsing of course A is equivalent to a state of the user.

[0061] Specifically, this step can determine the current state by analyzing the first history course browsing data corresponding to the target user. It can be known that the first history course browsing data includes the target user's operation information relative to the online course within a period of time. This step can regard these operation information as a sentence, and then analyze and obtain the target user's keyword information within this period of time, such as the operated online course ID, and then process and summarize these keyword information to form vector information that can represent the current state.

[0062] S102: Obtain candidate course categories that meet screening conditions according to the current state and the pre-trained target reinforcement learning network model.

[0063] In this embodiment, this step is equivalent to the operation of determining the candidate courses required for course recommendation through the reinforcement learning network model. It should be noted that reinforcement learning can be specifically understood as being able to continuously select actions to be executed according to the current state of the environment in an application environment in the form of intelligent learning to ensure that the number of rewards obtained after executing the selected actions is maximized. The specific execution of selecting actions to be executed in each state and ensuring the maximum number of rewards obtained in the end can be achieved through the reinforcement learning network model.

[0064] The reinforcement learning network model preferably used in this embodiment is a deep reinforcement learning (Deep Q-Learning, DQN) network model, and the reinforcement learning network model formed after learning through the set training samples is recorded as the target reinforcement learning network model. In general, the input data of the reinforcement learning network model is a current state value, and the output data is the action space data related to the application environment, and each action space represents the data information of a candidate execution action. The output number of the action space output by the target reinforcement learning network model is in principle equivalent to the number of executable actions in the application environment, but this mode is more suitable for scenarios including small-scale executable actions.

[0065] In this embodiment, each online course included in the online teaching platform is equivalent to a candidate action that can be executed by a user in principle. Because the number of online courses is large, it is impossible to directly use each online course as each executable candidate action output in the target reinforcement learning network model. This embodiment considers dividing online courses into course categories, and uses each course category as the action space output by the target reinforcement learning network model. That is, the target reinforcement learning network model uses the course category to which the online course belongs as the action space, and the output number of the output action space is the same as the total amount of the course category.

[0066] Specifically, this step can use the current state of the target user as an input data of the target reinforcement learning network model, and can output multiple different action space data information, and the data information of each action space represents a course category in the online teaching platform. Afterwards, each course category output can be regarded as a candidate set, from which candidate course categories that meet the conditions can be screened out. Among them, the screening conditions can be set by the cumulative reward values ​​corresponding to each action space output by the target reinforcement learning network model. In general, for the setting of the screening conditions, the greedy algorithm strategy is often used to determine the action space corresponding to the optimal cumulative reward value as the result of the coarse-grained screening of this step. This embodiment takes into account that the optimal solution determined based on the greedy algorithm strategy cannot guarantee the diversity of the screening results. It is preferred to consider using a non-greedy algorithm to ensure the diversity of the coarse-grained screening results of this step, such as considering multiple course categories with high cumulative reward values ​​as candidate course categories.

[0067] It should be noted that this embodiment can cluster online courses according to the course browsing habits of users through clustering processing, and the cluster center vector of each cluster after clustering can be regarded as data information of a course category.

[0068] S103: Select a set number of target courses from the online courses corresponding to the candidate course categories and push them to the target users.

[0069] In this embodiment, each candidate course category has one or more network courses, and these network courses can be used as candidate courses for course recommendation. Exemplarily, this step can filter out a certain amount of target courses from the candidate course set formed by each candidate course category through a certain screening strategy and push them to the target user.

[0070] Specifically, the target courses can be selected by analyzing the candidate courses included in the candidate course category and selecting the online courses with high importance or weight as the target courses. This step can perform the above target course selection operation on each candidate course category, and the selected target courses are pushed to the target user in no particular order, so that the relevant information of each target course is displayed on the client of the target user.

[0071] A course recommendation method provided in Example 1 of the present application mainly adopts the processing form of a reinforcement learning network model to achieve the effect that the online courses recommended to target users can bring long-term benefits to the online education platform; at the same time, by reducing the dimension of the action space output in the reinforcement learning network model, that is, by ensuring that the output number of the output action space is only the same as the number of course categories of the online course, the problem that reinforcement learning cannot adapt to large-scale data processing is solved, thereby achieving effective recommendation of online courses to the user end and improving user experience.

[0072] As an optional embodiment of the embodiment of the present application, based on the above embodiment, the step of dividing the course category to which the online course belongs may include:

[0073] It is understandable that the premise of determining the candidate course categories through the target reinforcement learning network model in this embodiment is to predetermine which course categories the user currently tends to browse in the online teaching platform. Therefore, this optional embodiment provides a specific implementation of classifying courses according to the user's course browsing behavior data.

[0074] a1) Acquire the second historical course browsing data of each selected user from the message queue, and form a course browsing sequence corresponding to each user.

[0075] The message queue can be specifically understood as a cache queue that caches the behavior data of each user generated when the user interacts with the backend. In this step, the historical course browsing data corresponding to each registered user or each user selected to participate in the course category division within a certain period of time can be obtained from the set message queue, which is recorded as the second historical course browsing data in this embodiment.

[0076] This step can form sequence information containing only the course ID of each online course browsed by the user by analyzing and extracting the second historical course browsing data of each user, which is recorded as the course browsing sequence corresponding to the user. The course browsing sequence corresponding to each user is equivalent to a sentence that can be segmented.

[0077] b1) taking each of the course browsing sequences as a sentence to be processed, obtaining the course vector of each online course through a word vector partitioning model, and forming a course vector set.

[0078] The word vector partitioning model is preferably a word2vec related model for generating word vectors. In this step, each course browsing sequence can be regarded as a sentence to be processed as input data of the word vector partitioning model, and a vector set output by the word vector partitioning model can be obtained. Each vector in the vector set represents a course vector of an online course, and the vector set containing each course vector is preferably recorded as a course vector set.

[0079] c1) performing clustering processing on the course vector set to obtain the output number of clusters, and determining the cluster center vector of each cluster as a course category.

[0080] In this embodiment, the course vector set can be clustered by the K-means clustering algorithm, and the K value of the formed cluster can be determined by the inflection point method, and the determined K value can be used as the output quantity of the action space output by the target reinforcement learning network model in this embodiment. The process of determining the K value by the inflection point method can be described as:

[0081] Search for all possible values ​​of K within a certain range, and for each possible value, use the possible value to perform clustering processing to obtain the corresponding clustering results, and then use the error square sum calculation formula to collect the clustering results under each K value to calculate the error square sum. Among them, the error square sum calculation formula is expressed as:

[0082] Wherein, k is the maximum value of all possible values. In this embodiment, the K value is set starting from 1. Specifically, i is used to represent the change of the K value. Wherein, C is the clustering cluster set corresponding to the clustering result when the K value is selected as i. Each p represents a point in space, and m i Represents a cluster center when the K value is selected as i.

[0083] Through the above formula, a square error sum can be obtained for each K value. Each square error sum is connected, and the K value with the largest slope change on the connection line can be taken as the optimal K value.

[0084] After determining the optimal K value, this embodiment can use the K value as the output quantity of this embodiment, and can obtain the output quantity clusters, and the cluster center vector in each cluster can be regarded as the vector information representing a course category. Thus, the division of course categories required by this embodiment is achieved.

[0085] It should be noted that another prerequisite for determining the candidate course category through the target reinforcement learning network model in this embodiment is that it is also necessary to ensure that the adopted target reinforcement learning network model is a pre-trained network model. This embodiment also provides another optional embodiment to realize the training of the reinforcement learning network model.

[0086] Specifically, as another optional embodiment of the embodiment of the present application, based on the above optional embodiment, this embodiment can express the training steps of the target reinforcement learning network model as follows:

[0087] a2) Two reinforcement learning network models with the same network structure but different network parameters are respectively recorded as the real-time training network model and the initial reinforcement learning network model.

[0088] Through relevant analysis of reinforcement learning, it can be seen that two neural network models with the same network structure are needed in the training process of reinforcement learning, but the network parameters of the two neural network models are not the same. One of them can be recorded as a real-time training network model that needs to be trained in real time, and the other can be recorded as an initial reinforcement learning network model that has undergone some learning and can be used in actual application scenarios, but still needs to be continuously updated.

[0089] b2) constructing a training sample set for model training based on each course category identified by each cluster center vector and the second historical course browsing data of each selected user.

[0090] In this implementation, in order to ensure that the reinforcement learning network model obtained after training can match the course recommendation application scenario of this embodiment, it is necessary to set the training sample set required for model training based on the course recommendation application scenario of this embodiment. In this step, the cluster center vector identifying each course category when performing the above-mentioned course category division can be obtained, and the second historical course browsing data used when performing the above-mentioned course category division can also be obtained.

[0091] By analyzing the browsing data of the second history course corresponding to each user, training samples can be constructed, wherein each training sample in the training sample set includes: a first state sequence of the user's current state, a target cluster center vector, an instantaneous reward value, and a second state sequence of the next state.

[0092] Specifically, in order to form a training sample, this embodiment determines what information should be included in a training sample from the perspective of the parameters required in the reinforcement learning scenario. In general, the main parameters in the reinforcement learning scenario include the current state of the environment, the actions that the user can perform (such as which course to browse), the next state of the environment after the user performs the action, and the instantaneous reward generated after the user performs the action. By browsing the second historical course data, you can know which courses the user has browsed in the historical time period, thereby forming a state sequence that the user has in the application environment, and then determine the parameter information of each parameter in the training sample.

[0093] Exemplarily, assuming that the analysis of the second historical course browsing data determines that the user browsed four courses, Course A, Course B, Course C, and Course D, in sequence over a period of time, this embodiment can construct a user's current status data based on the first three courses, and construct the user's next status data based on the four courses.

[0094] Continuing with the above description, in this embodiment, the corresponding course vector information can be determined for each course, thereby forming a first state sequence representing the current state of the user based on the course vector information of the first three courses, such as the average vector obtained by averaging the course vector information of the first three courses is regarded as the first state sequence. At the same time, a second state sequence representing the next state of the user can also be formed based on the course vector information of the above four courses, and similarly, the course vector information of the above four courses can be summed to obtain it.

[0095] After the fourth course (Course D) is known, it is equivalent to knowing the action space to be executed when the user switches from the current state to the next state. This embodiment does not directly use the vector data of Course D as the action space, but first determines the cluster to which Course D belongs, and then uses the cluster center vector of the cluster to which it belongs as the action space information executed when the user switches from the current state to the next state, that is, the target cluster center vector in the training sample required by this embodiment.

[0096] Similarly, after the user switches from the current state to the next state, an instantaneous reward value can be immediately fed back according to the action performed by the user. Therefore, the fed back instantaneous reward value is also equivalent to a parameter information in a training sample.

[0097] In this step, a corresponding training sample may be determined for each user in the manner described above.

[0098] c2) fitting a loss function according to the output results of each training sample under the real-time training network model and the initial reinforcement learning network model, and obtaining a target reinforcement learning network model through reverse learning of the fitted loss function.

[0099] It can be known that the training process of the network model is equivalent to the process of reverse learning and adjusting the network parameters in the network model by comparing the actual output value obtained after inputting the input data in the training sample into the network model to be trained with the standard output value in the training sample in some way. Among them, the method of comparing the actual output value with the standard output value in some way is mainly achieved by fitting the loss function.

[0100] Based on this, this step can use the current state in the training sample (i.e., the first state sequence) as the input data for implementing the training network model, and use the action space with the largest cumulative reward value in the output action space as the actual output value, combined with the target cluster center vector given in the training sample as the standard output value to fit the loss function.

[0101] In the loss function fitting implementation of this embodiment, it is mainly based on the maximum cumulative reward value corresponding to the actual output value and the cumulative reward value corresponding to the standard output value under the initial reinforcement learning network model. In the execution of training and learning again, the network parameters of the real-time training network model can be adjusted by the fitted loss function, and then the network parameters can be adjusted after the conditions for adjusting the parameters of the initial reinforcement learning network model are met. Finally, the initial reinforcement learning network model after the network parameters are adjusted can be used as the currently available target reinforcement learning network model.

[0102] Furthermore, in this embodiment, the above step c2) can be specifically implemented as the following steps:

[0103] It should be noted that the process of determining the target reinforcement learning network model requires the participation of each of the above-mentioned training samples, and each of the steps provided below in this optional embodiment needs to be executed for each training sample.

[0104] c21) for each training sample, determining the current cumulative reward value of each action space vector output by the first state sequence under the real-time training network model, and determining the maximum current cumulative reward value.

[0105] In this step, its specific implementation mainly includes: first, inputting the first state sequence in the training sample as input data into the real-time training network model, and obtaining each action space vector output by the operation of the real-time training network model (equivalent to the cluster center vector corresponding to each course category formed by clustering division).

[0106] After that, the current cumulative reward value of each action space vector can be obtained, and the maximum current cumulative reward value can be determined by comparing the current cumulative reward values. Among them, the current cumulative reward value can be obtained by calculating a known reward value determination function. The Bellman equation is used to determine the target value required for the loss function fitting. Among them, the Bellman equation can be expressed as:

[0107] Among them, Y represents the target value required for fitting the loss function, Rt+1 represents the instantaneous return value in the training sample, γ is a pre-set parameter, and Q(S t+1 ,a,θ t ) means that when the network parameter is θ tThe cumulative reward value corresponding to the transition of an action space from the current state to the next state obtained under the real-time training network model, Then when the network parameter is θ t The cumulative return value determined according to the target action space determined above under the initial reinforcement learning network model of '.

[0108] c22) determining a standard cumulative reward value of the first state sequence relative to the target cluster center vector under the initial reinforcement learning network model.

[0109] The specific implementation of this step includes: inputting the first state sequence as input data into the initial reinforcement learning network model, finding the cluster center vector in the training sample from the output action space vector, and thereby obtaining its corresponding standard cumulative reward value.

[0110] c23) Fit the loss function according to the maximum current cumulative reward value and the standard cumulative reward value corresponding to each training sample.

[0111] In this embodiment, the Bellman equation may be used to determine the target value required for loss function fitting.

[0112] The Bellman equation can be expressed as:

[0113]

[0114] The calculation of the Bellman equation is relative to each training sample, Y represents the target value required for fitting the loss function, and R t+1 represents the instantaneous reward value in the training sample, γ is a pre-set parameter, Q(S t+1 ,a,θ t ) means that when the network parameter is θ t The action space obtained under the real-time training network model consists of the current state S t Transition to next state S t+1 The current cumulative return value corresponding to the time, based on each current cumulative return value, the maximum cumulative return value can be determined. Then when the network parameter is θ t The standard cumulative reward value determined according to the target action space determined above under the initial reinforcement learning network model of '. Among them, the target action space is equivalent to the action space corresponding to the maximum cumulative reward value mentioned above, and the target action space is often the target cluster center vector in the training sample.

[0115] After that, the specific loss function value can be determined according to the given loss function, where the loss function can be expressed as:

[0116] Among them, Q(St+1 ,a,θ t Indicates that the network parameter is θ t The target action space obtained under the real-time training network model is converted from the previous state to the current state S t The previous cumulative return value corresponding to the time, Y is the target value determined above, n is the number of training samples, and the above expression mainly fits the mean square error between the actual value and the target value Y of the real-time training network model.

[0117] c24) updating the network parameters of the real-time training network model according to the fitted loss function, and when the number of updates meets the parameter replacement cycle, replacing the network parameters of the initial reinforcement learning network model with the network parameters of the real-time training network model.

[0118] Specifically, the mean square error can be used to perform reverse training of the real-time training network model to adjust the network parameters to update the network model. In addition to updating the network parameters of the real-time training network model in real time, this step also counts the number of updates in real time. When the cumulative value of the number of updates reaches the parameter replacement cycle, the network parameters of the real-time training network model at that moment need to be used to replace the network parameters of the initial reinforcement learning network model, so as to achieve the update of the initial reinforcement learning network model.

[0119] In this embodiment, the preferred parameter replacement cycle is a number of updates accumulated from 0 to a value, and the value can be determined based on historical experience, such as 50 times.

[0120] c25) determining the initial reinforcement learning network model after parameter replacement as the target reinforcement learning network model.

[0121] The above optional embodiment of this embodiment specifically provides the implementation of course category division of online courses, and the training implementation of the reinforcement learning network model with the course category formed by the division as the action space dimension. The course category division can effectively reduce the scale of the online courses to be recommended, thereby reducing the action space dimension in reinforcement learning and ensuring the effective application of reinforcement learning in large-scale data recommendation scenarios.

[0122] Embodiment 2

[0123] Figure 2A flow chart of a course recommendation method provided in Example 2 of the present application. This embodiment is based on the above embodiment. In this embodiment, the current state of the target user can be determined based on the historical course browsing data of the target user. Specifically, the following is described: word segmentation processing is performed on the first historical course browsing data of the target user within a set time period to determine the browsed course vectors of the browsed courses corresponding to the target user; and the current state of the target user is determined by taking the average vector of each of the browsed course vectors.

[0124] At the same time, this embodiment can also obtain candidate course categories that meet the screening conditions based on the current state and the pre-trained target reinforcement learning network model as follows: input the current state into the target reinforcement learning network model, and output the output number of candidate vectors as the action space through the target reinforcement learning network model, wherein each candidate vector identifies a course category; for each course category, determine the cumulative reward value of the course category through a given cumulative reward value model, combined with the current state and the current network parameters of the target reinforcement learning network model; rank the course categories according to the cumulative reward value, and take the course category in the first set position as the candidate course category.

[0125] In addition, this embodiment can also obtain candidate course categories that meet the screening conditions based on the current state and the pre-trained target reinforcement learning network model as follows: for each candidate course category, obtain the cluster center vector of the candidate course category; determine the distance value between each course vector in the cluster cluster associated with the cluster center vector and the cluster center vector; rank each of the course vectors according to the distance value, and use the course vectors in the top second set ranking as the course to be recommended; select the target course that meets the fine-grained screening conditions from each of the courses to be recommended and push them to each of the target users respectively.

[0126] like Figure 2 As shown, a course recommendation method provided in Example 2 of the present application specifically includes the following operations:

[0127] S201. Perform word segmentation processing on the first historical course browsing data of the target user within a set time period to determine a browsed course vector corresponding to the browsed course of the target user.

[0128] Exemplarily, the word vector related model word2vec can also be used to implement analysis and processing, thereby obtaining the course vectors of all online courses that the user has browsed within the set time period, which is recorded as the browsed course vector in this step.

[0129] S202: Determine the current state of the target user by taking the average vector of each browsed course vector.

[0130] This step is equivalent to an implementation of the current state, that is, the average vector representing the current state of the target user is obtained by calculating the average value of each of the browsed course vectors.

[0131] S203: Input the current state into the target reinforcement learning network model, and output the output number of candidate vectors as the action space through the target reinforcement learning network model.

[0132] This step is equivalent to the specific application of the target reinforcement learning network model. It can output multiple action spaces corresponding to the current state input. Each action space is represented in vector form. Each vector is recorded as a candidate vector, where each candidate vector identifies a course category. That is, an output action space identifies a course category. Therefore, the number of outputs is equivalent to the K value determined when the course category is divided based on the clustering algorithm.

[0133] S204. For each course category, determine the cumulative reward value of the course category through a given cumulative reward value model, combined with the current state and the current network parameters of the target reinforcement learning network model.

[0134] It can be known that each course category represented by the cluster center vector can be obtained through the above-mentioned division of course categories, and this step is equivalent to the operation of each course category. The specific determination of the cumulative return value in this step can preferably be calculated using the cumulative return value function described above. The information required to be known is the candidate vector (cluster center vector) corresponding to the course category, the current network parameters of the target reinforcement learning network model, and the next state corresponding to the reinforcement learning process.

[0135] S205. Rank the course categories according to the cumulative return value, and select the course category that is in the first set ranking as a candidate course category.

[0136] Through the above steps, the cumulative return value corresponding to each course category can be determined. In this step, the cumulative return values ​​can be ranked, thereby selecting multiple course categories with high rankings as candidate course categories, so as to ensure the diversity of the coarse-grained screening results obtained. Among them, the first setting noun can be preferably 2, that is, the course categories ranked in the top two.

[0137] The following S206 to S209 give the specific implementation of fine-grained screening of target courses.

[0138] S206. For each candidate course category, obtain a cluster center vector of the candidate course category.

[0139] It can be known that this step and the following S207 and S208 are operations relative to each candidate course category. A candidate course category is also equivalent to the cluster center of a cluster cluster. This step can obtain the cluster center vector with the center, that is, the candidate vector of each candidate course category relative to the target reinforcement learning network model.

[0140] S207: Determine the distance value between each course vector in the cluster associated with the cluster center vector and the cluster center vector.

[0141] In this embodiment, a cluster contains at least one online course belonging to the cluster center, and each online course is represented by a corresponding course vector. This step can calculate the distance value from each course vector to the cluster center vector.

[0142] S208. Rank the course vectors according to the distance value, and use the course vectors that are in the top two set rankings as the courses to be recommended.

[0143] This step can rank the distance values ​​obtained above, thereby obtaining a plurality of top-ranked course vectors. This step preferably selects the top 20 courses as the recommended courses, that is, the second set noun is preferably 20.

[0144] S209: Select target courses that meet fine-grained screening conditions from the courses to be recommended and push them to the target users respectively.

[0145] This step can summarize the recommended courses corresponding to each candidate course category, and can re-rank the recommended courses according to the given fine-grained screening conditions, thereby selecting an appropriate number of online courses as target courses. Among them, the fine-grained screening conditions can be selected according to the specific application scenario, and after selection, the ranking reference dimension for ranking the recommended courses can be determined.

[0146] In one implementation mode given in this embodiment, 4 online courses can be randomly selected from the recommended courses corresponding to each candidate course category and pushed to the target user as target courses. The pushed target courses can be displayed on the homepage interface of the user client for the user to select and browse.

[0147] A course recommendation method provided in the second embodiment of the present invention specifies the process of determining the current state of the target user, and also specifies the process of determining the candidate course categories, and the process of screening the target course. The method provided in this embodiment mainly adopts the processing form of the reinforcement learning network model to achieve the effect that the online courses recommended to the target users can bring long-term benefits to the online education platform; at the same time, by reducing the dimension of the action space output in the reinforcement learning network model, that is, by ensuring that the output number of the output action space is only the same as the number of course categories possessed by the online course, the problem that reinforcement learning cannot adapt to large-scale data processing is solved, thereby achieving effective recommendation of online courses to the user end and improving the user experience.

[0148] Embodiment 3

[0149] Figure 3 This is a structural block diagram of a course recommendation device provided in Example 3 of the present application. The device is suitable for recommending online courses in an online teaching platform to users. The device can be implemented by hardware and / or software and is generally integrated in a computer device. Figure 3 As shown, the device includes: an information determination module 31, a candidate determination module 32 and a target recommendation module 33.

[0150] An information determination module 31 is used to determine the current state of the target user according to the first history course browsing data of the target user;

[0151] The candidate determination module 32 aims to obtain candidate course categories that meet the screening conditions according to the current state and the pre-trained target reinforcement learning network model, wherein the target reinforcement learning network model uses the course category to which the network course belongs as the action space, and the output quantity of the output action space is the same as the total quantity of the course category;

[0152] The target recommendation module 33 is used to select a set number of target courses from the online courses corresponding to each of the candidate course categories and push them to the target user.

[0153] A course recommendation device provided in the third embodiment mainly adopts the processing form of the reinforcement learning network model to achieve the effect that the online courses recommended to the target users can bring long-term benefits to the online education platform; at the same time, by reducing the dimension of the action space output in the reinforcement learning network model, that is, by ensuring that the output number of the output action space is only the same as the number of course categories of the online course, the problem that reinforcement learning cannot adapt to large-scale data processing is solved, thereby realizing effective recommendation of online courses to the user end and improving the user experience.

[0154] Furthermore, the device may also include: a course classification module,

[0155] The course classification module can be used specifically for:

[0156] Acquire the second historical course browsing data of each selected user from the message queue, and form a course browsing sequence corresponding to each of the users;

[0157] Taking each of the course browsing sequences as a sentence to be processed, obtaining the course vector of each online course through a word vector partitioning model to form a course vector set;

[0158] The course vector set is clustered to obtain the output number of clusters, and the cluster center vector of each cluster is correspondingly determined as a course category.

[0159] Furthermore, the information determination module 31 may be specifically used for:

[0160] Performing word segmentation processing on the first historical course browsing data of the target user within a set time period to determine a browsed course vector corresponding to the browsed course of the target user;

[0161] The current state of the target user is determined by taking the average vector of each browsed course vector.

[0162] Furthermore, the candidate determination module 32 may be specifically used for:

[0163] Input the current state into the target reinforcement learning network model, and output the output number of candidate vectors as the action space through the target reinforcement learning network model, wherein each candidate vector identifies a course category;

[0164] For each course category, the cumulative reward value of the course category is determined by combining the current state and the current network parameters of the target reinforcement learning network model through a given cumulative reward value model;

[0165] The course categories are ranked according to the cumulative return value, and the course category at the top first set ranking is used as the candidate course category.

[0166] Furthermore, the target recommendation module 33 can be specifically used for:

[0167] For each candidate course category, obtain the cluster center vector of the candidate course category;

[0168] Determine the distance value between each course vector in the cluster cluster associated with the cluster center vector and the cluster center vector;

[0169] Rank the course vectors according to the distance value, and use the course vectors in the top two set rankings as the courses to be recommended;

[0170] Target courses that meet fine-grained screening conditions are selected from the courses to be recommended and pushed to the target users respectively.

[0171] Furthermore, the device may also include a model training module, wherein the model training module may include:

[0172] An information initialization unit, used to record two reinforcement learning network models with the same network structure and different network parameters as a real-time training network model and an initial reinforcement learning network model respectively;

[0173] A sample determination unit, configured to construct a training sample set for model training according to each course category identified by each cluster center vector and the second historical course browsing data of each selected user, wherein each training sample in the training sample set includes: a first state sequence of the user's current state, a target cluster center vector, an instantaneous reward value, and a second state sequence of the next state;

[0174] The target acquisition unit is used to fit the loss function according to the output results of each training sample under the real-time training network model and the initial reinforcement learning network model, and obtain the target reinforcement learning network model through reverse learning of the fitted loss function.

[0175] Furthermore, the target acquisition unit can be specifically used for:

[0176] For each training sample, determining the current cumulative reward value of each action space vector output by the first state sequence under the real-time training network model, and determining the maximum current cumulative reward value;

[0177] Determining a standard cumulative reward value of the first state sequence relative to the target cluster center vector under the initial reinforcement learning network model;

[0178] Fit the loss function according to the maximum current cumulative return value and the standard cumulative return value corresponding to each training sample;

[0179] The network parameters of the real-time training network model are updated according to the fitted loss function, and when the number of updates meets the parameter replacement cycle, the network parameters of the initial reinforcement learning network model are replaced with the network parameters of the real-time training network model;

[0180] The initial reinforcement learning network model after parameter replacement is determined as the target reinforcement learning network model.

[0181] Embodiment 4

[0182] Figure 4This is a schematic diagram of the structure of a computer device provided in Embodiment 4 of the present application. The computer device includes: a processor 40, a memory 41, a display screen 42, an input device 43, and an output device 44. The number of processors 40 in the computer device can be one or more. Figure 4 A processor 40 is taken as an example. The number of memories 41 in the computer device can be one or more. Figure 4 A memory 41 is taken as an example. The processor 40, memory 41, display screen 42, input device 43 and output device 44 of the computer device can be connected through a bus or other means. Figure 4 In the embodiment, the computer device may be a computer, a notebook or a smart tablet.

[0183] The memory 41, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the computer device described in any embodiment of the present invention (for example, the information determination module 31, the candidate determination module 32 and the target recommendation module 33 in the course recommendation device). The memory 41 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the device, etc. In addition, the memory 41 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 41 may further include a memory remotely arranged relative to the processor 40, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0184] The display screen 42 may be a display screen 42 with a touch function, which may be a capacitive screen, an electromagnetic screen, or an infrared screen. Generally speaking, the display screen 42 is used to display data according to the instructions of the processor 40, and is also used to receive a touch operation acting on the display screen 42 and send a corresponding signal to the processor 40 or other devices.

[0185] The input device 43 can be used to receive input digital or character information, and generate key signal input related to user settings and function control of the display device. It can also be a camera for acquiring images and a sound pickup device for acquiring audio data. The output device 44 can include audio devices such as speakers. It should be noted that the specific composition of the input device 43 and the output device 44 can be set according to actual conditions.

[0186] The processor 40 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 41, that is, implements the above-mentioned course recommendation method.

[0187] The computer device provided above can be used to execute the course recommendation method provided in any of the above embodiments, and has corresponding functions and beneficial effects.

[0188] Embodiment 5

[0189] Embodiment 5 of the present invention further provides a storage medium containing computer executable instructions, wherein the computer executable instructions are used to execute a course recommendation method when executed by a computer processor, including:

[0190] Determining the current state of the target user according to the first history course browsing data of the target user;

[0191] According to the current state and the pre-trained target reinforcement learning network model, candidate course categories that meet the screening conditions are obtained, wherein the target reinforcement learning network model uses the course category to which the online course belongs as an action space, and the output quantity of the output action space is the same as the total quantity of the course category;

[0192] A set number of target courses are selected from the online courses corresponding to each of the candidate course categories and pushed to the target user.

[0193] Of course, the storage medium containing computer executable instructions provided in an embodiment of the present invention is not limited to the operation of the course recommendation method as described above, but can also execute related operations in the course recommendation method provided in any embodiment of the present invention, and has corresponding functions and beneficial effects.

[0194] Through the above description of the implementation method, the technicians in the relevant field can clearly understand that the present application can be implemented with the help of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application can be essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a robot, a personal computer, a server, or a network device, etc.) to execute the course recommendation method described in any embodiment of the present application.

[0195] It is worth noting that in the above-mentioned course recommendation device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application.

[0196] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0197] Note that the above are only preferred embodiments of the present application and the technical principles used. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. A course recommendation method, characterized in that: include: Determining the current state of the target user according to the first history course browsing data of the target user; According to the current state and the pre-trained target reinforcement learning network model, candidate course categories that meet the screening conditions are obtained, wherein the target reinforcement learning network model uses the course category to which the online course belongs as an action space, and the output quantity of the output action space is the same as the total quantity of the course category; Selecting a set number of target courses from the online courses corresponding to each of the candidate course categories and pushing them to the target user; The step of dividing the course category to which the online course belongs includes: Acquire the second historical course browsing data of each selected user from the message queue, and form a course browsing sequence corresponding to each of the users; Taking each of the course browsing sequences as a sentence to be processed, obtaining the course vector of each online course through a word vector partitioning model, and forming a course vector set; Performing clustering processing on the course vector set to obtain the output number of clusters, and determining the cluster center vector of each cluster as a course category; Determining the current status of the target user according to the historical course browsing data of the target user includes: Performing word segmentation processing on the first historical course browsing data of the target user within a set time period to determine a browsed course vector corresponding to the browsed course of the target user; The current state of the target user is determined by taking the average vector of each browsed course vector.

2. The method according to claim 1, characterized in that The step of obtaining candidate course categories that meet the screening conditions based on the current state and the pre-trained target reinforcement learning network model includes: Input the current state into the target reinforcement learning network model, and output the output number of candidate vectors as the action space through the target reinforcement learning network model, wherein each candidate vector identifies a course category; For each course category, the cumulative reward value of the course category is determined by combining the current state and the current network parameters of the target reinforcement learning network model through a given cumulative reward value model; The course categories are ranked according to the cumulative return value, and the course category at the top first set ranking is used as the candidate course category.

3. The method according to claim 1, characterized in that: The step of selecting a set number of target courses from the online courses corresponding to the candidate course categories and pushing them to the target user includes: For each candidate course category, obtain the cluster center vector of the candidate course category; Determine the distance value between each course vector in the cluster cluster associated with the cluster center vector and the cluster center vector; Rank the course vectors according to the distance value, and use the course vectors in the top two set rankings as the courses to be recommended; Target courses that meet fine-grained screening conditions are selected from the courses to be recommended and pushed to the target users respectively.

4. The method according to claim 1, characterized in that The training steps of the target reinforcement learning network model include: The two reinforcement learning network models with the same network structure but different network parameters are respectively recorded as the real-time training network model and the initial reinforcement learning network model; According to each course category identified by each cluster center vector and the second historical course browsing data of each selected user, a training sample set for model training is constructed, wherein each training sample in the training sample set includes: a first state sequence of the user's current state, a target cluster center vector, an instantaneous reward value, and a second state sequence of the next state; The loss function is fitted according to the output results of each training sample under the real-time training network model and the initial reinforcement learning network model, and the target reinforcement learning network model is obtained through reverse learning of the fitted loss function.

5. The method according to claim 4, characterized in that The loss function is fitted according to the output results of each training sample under the real-time training network model and the initial reinforcement learning network model, and the target reinforcement learning network model is obtained through reverse learning of the fitted loss function, including: For each training sample, determining the current cumulative reward value of each action space vector output by the first state sequence under the real-time training network model, and determining the maximum current cumulative reward value; Determining a standard cumulative reward value of the first state sequence relative to the target cluster center vector under the initial reinforcement learning network model; Fit the loss function according to the maximum current cumulative return value and the standard cumulative return value corresponding to each training sample; The network parameters of the real-time training network model are updated according to the fitted loss function, and when the number of updates meets the parameter replacement cycle, the network parameters of the initial reinforcement learning network model are replaced with the network parameters of the real-time training network model; The initial reinforcement learning network model after parameter replacement is determined as the target reinforcement learning network model.

6. A course recommendation device, characterized in that: include: An information determination module, used to determine the current state of the target user according to the first history course browsing data of the target user; A candidate determination module, wherein the target is to obtain candidate course categories that meet the screening conditions according to the current state and the pre-trained target reinforcement learning network model, wherein the target reinforcement learning network model uses the course category to which the online course belongs as the action space, and the output quantity of the output action space is the same as the total quantity of the course category; A target recommendation module, used to select a set number of target courses from the online courses corresponding to each candidate course category and push them to the target user; The device further comprises: a course classification module, The course classification module is specifically used for: Acquire the second historical course browsing data of each selected user from the message queue, and form a course browsing sequence corresponding to each of the users; Taking each of the course browsing sequences as a sentence to be processed, obtaining the course vector of each online course through a word vector partitioning model, and forming a course vector set; Performing clustering processing on the course vector set to obtain the output number of clusters, and determining the cluster center vector of each cluster as a course category; The information determination module is specifically used for: Performing word segmentation processing on the first historical course browsing data of the target user within a set time period to determine a browsed course vector corresponding to the browsed course of the target user; The current state of the target user is determined by taking the average vector of each browsed course vector.

7. A computer device, characterized in that: include: memory and one or more processors; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

8. A storage medium containing computer executable instructions, characterized in that: The computer executable instructions are used to perform the method according to any one of claims 1 to 5 when executed by a computer processor.

Images