Multi-model denoising method and system for reinforcement learning recommendation system

Abstract

The invention discloses a multi-model denoising method and system for a reinforcement learning recommendation system, and the method comprises the steps: obtaining noisy interaction data, and carrying out the prediction of the noisy interaction data through employing a generalized matrix decomposition model and other two matrix decomposition models as prediction models; predicting model parameters are optimized by minimizing KL divergence of fitting real user preference distribution among a plurality of predicting models and maximizing likelihood of observation data under given real user preference conditions; the optimized prediction model serves as a teacher model, and the teacher model is trained through the noise reduction positive sample noise and the noise reduction negative sample noise to obtain a de-noised teacher model; constructing a student model, and defining a target loss function; according to the method, the denoising algorithm is used for training the denoising user model, the denoising user model is used for interacting with an online reinforcement learning strategy, feedback better conforming to the real preference of the user is provided, and the noise robustness and the recommendation performance of the reinforcement learning recommendation method are improved.

Description

Technical Field

[0001] The present application relates to the technical field of recommendation systems, and more specifically to a multi-model denoising method and system for a reinforcement learning recommendation system. Background Art

[0002] Recommendation system is one of the core technologies in modern information services and is widely used in e-commerce, social media, content distribution and other scenarios.

[0003] Reinforcement learning is an important branch of machine learning. Through the interaction between the agent and the environment, it learns how to choose the best action in different situations to maximize long-term rewards. It is widely used in fields such as robot control, game AI, multi-step decision-making and resource optimization, and has promoted the development of intelligent decision-making systems.

[0004] Existing denoising methods for recommendation systems are generally based on resampling or reweighting. The performance of the resampling method is heavily dependent on the selection of the sampling distribution and the variance of the sample. The reweighting method cannot accurately distinguish between noise and clean samples. Some real samples may also produce high losses in the initial training stage due to insufficient model learning. Another type of method is to denoise by combining additional information (such as related behaviors after the user clicks). This method relies on additional knowledge, has poor flexibility, and is only suitable for specific scenarios. There is a lack of work on alleviating noise problems in the reinforcement learning recommendation scenario. Summary of the invention

[0005] In this embodiment, a multi-model denoising method, system, electronic device and storage medium for a reinforcement learning recommendation system are provided to solve the problem of alleviating noise in the reinforcement learning recommendation scenario in the related art.

[0006] In a first aspect, an embodiment of the present invention provides a multi-model denoising method for a reinforcement learning recommendation system, and the multi-model denoising method for a reinforcement learning recommendation system includes: Obtain noisy interaction data, and use the generalized matrix factorization model and two other matrix factorization models as prediction models to predict the noisy interaction data; The prediction model parameters are optimized by minimizing the KL divergence of the distribution of real user preferences fitted between multiple prediction models and maximizing the likelihood of observed data given the real user preferences. The optimized prediction model is used as the teacher model, and the teacher model is trained by denoising the positive sample noise and denoising the negative sample noise to obtain a denoising teacher model; Construct a student model and define the target loss function, calculate the soft target of the denoising teacher model, and use the KL divergence to measure the difference between the output distribution of the denoising teacher model and the output distribution of the student model, so that the student model can learn the soft labels provided by the denoising teacher model; The student model parameters are updated based on the target loss function, and the student model is iteratively trained as the denoising user model.

[0007] In an optional embodiment, obtaining noisy data includes noisy binary data, and the binary data is used to infer a real user preference matrix and its corresponding denoising teacher model.

[0008] In an optional embodiment, the prediction model parameters are optimized by minimizing the KL divergence of the real user preference distributions fitted between multiple prediction models while maximizing the likelihood of the observed data given the real user preference conditions, including: Constructing an objective function for minimizing the KL divergence between different prediction models and maximizing the likelihood of observed data given real user preferences; For any two matrix decomposition models, calculate the KL divergence between their output probability distributions; Extend the KL divergence calculation to model pairs, add the KL divergence between all model pairs and use it as part of the objective function; Define the likelihood function according to Bayes' theorem and maximize the likelihood; The combination of minimizing KL divergence and maximizing likelihood forms a comprehensive objective function, which is combined with the optimization algorithm to update the prediction model parameters.

[0009] In an optional embodiment, the optimized prediction model is used as a teacher model, and the teacher model is trained by denoising positive sample noise and denoising negative sample noise to obtain a denoising teacher model, including: Denoise positive sample noise, remove the noise in the positive sample, for each positive sample, calculate its loss and adjust the weight as needed, and fix the negative sample noise part when the teacher model performs denoising positive sample noise training; Denoise negative sample noise, remove the noise in the negative samples, for each negative sample, calculate its loss and adjust the weight as needed, and fix the positive sample noise part when the teacher model performs denoised negative sample noise training.

[0010] In an optional embodiment, a student model is constructed and a target loss function is defined, including: Create and initialize a student model, and define a loss function composition, wherein the loss function includes soft target supervision and hard target supervision; The soft target supervision comes from the soft target supervision of the teacher model, which is used for the difference between the output distribution of the teacher model and the output distribution of the student model; the hard target supervision refers to the supervision of the student model itself by the real label; the cross entropy loss function is used to evaluate the difference between the prediction result of the student model and the real label; The objective loss function consists of two parts: soft target supervision and hard target supervision, and is weighted and balanced through hyperparameters.

[0011] In an optional embodiment, calculating the soft target of the teacher model includes: For each input sample, calculate the predicted logits values ​​of all denoising teacher models; The logits values ​​of all denoising teacher models are averaged to get the final teacher model output.

[0012] In an optional embodiment, calculating the soft target of the teacher model further includes: The temperature coefficient is introduced to adjust the softmax function to amplify the probability of the negative class.

[0013] Compared with the prior art, the multi-model denoising method of the reinforcement learning recommendation system of the present invention has the following beneficial effects: The present invention obtains multiple teacher models by minimizing the KL divergence between the real user preference distributions modeled by multiple recommendation models and maximizing the likelihood of data observations. The target model is obtained by multi-model knowledge distillation. During the process, the target model integrates the parameters of the teacher model and removes the noise of some teacher outputs. At the same time, the soft label can contain more information, so that the student model can better capture the hidden information in the data. The denoising user model is trained using the above denoising algorithm to interact with the online reinforcement learning strategy, provide feedback that is more in line with the user's real preferences, and improve the noise robustness and recommendation performance of the reinforcement learning recommendation method.

[0014] In a second aspect, an embodiment of the present invention provides a multi-model denoising system for a reinforcement learning recommendation system, comprising: The initialization module is configured to: obtain noisy interaction data, and use the generalized matrix factorization model and the other two matrix factorization models as prediction models to predict the noisy interaction data; A prediction model optimization module is configured to optimize prediction model parameters by minimizing the KL divergence of the distribution of real user preferences fitted between multiple prediction models while maximizing the likelihood of observed data given the real user preferences; The denoising teacher model training module is configured as follows: the optimized prediction model is used as the teacher model, and the teacher model is trained by denoising positive sample noise and denoising negative sample noise to obtain the denoising teacher model; The knowledge distillation module is configured to: construct a student model and define a target loss function, calculate the soft target of the denoising teacher model, and use KL divergence to measure the difference between the output distribution of the denoising teacher model and the output distribution of the student model so that the student model can learn the soft labels provided by the denoising teacher model; The target model training module is configured to: update the student model parameters based on the target loss function, and iteratively train the student model as a denoising user model.

[0015] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a processor, a communication interface, a memory and a bus, wherein the processor, the communication interface and the memory communicate with each other through the bus, and the processor can call logic instructions in the memory to execute the steps of the method provided in the first aspect.

[0016] In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the multi-model denoising method for a reinforcement learning recommendation system as described in the first aspect.

[0017] Compared with the prior art, the beneficial effects of the multi-model denoising system, electronic device and storage medium of the reinforcement learning recommendation system of the present invention are the same as the multi-model denoising method of the reinforcement learning recommendation system described in the first aspect, so they will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 This is a flow chart of a multi-model denoising method for a reinforcement learning recommendation system according to an embodiment of the present invention; Figure 2 It is a schematic diagram of a multi-model denoising method of a reinforcement learning recommendation system in an embodiment of the present invention; Figure 3 It is a structural block diagram of a multi-model denoising system of a reinforcement learning recommendation system in an embodiment of the present invention; Figure 4 1 is a structural block diagram of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0020] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0021] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with general skills in the technical field to which this application belongs. The words "one", "a", "the", "these" and the like in this application do not indicate a quantitative limitation, and they may be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" may mean: A exists alone, A and B exist at the same time, and B exists alone. Generally, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0022] In order to better understand the technical solution of the present invention, we first give the symbolic representation and task definition of the recommendation system denoising task, and briefly introduce the solution of the present invention. Let u∈U and i∈I represent users and items respectively, where U and I are the user set and item set respectively. ˜R∈R |U|×|I| is a binary matrix used to represent the interaction data affected by noise. ui represents the (u,i)th item in the matrix ˜R, which is defined as: if user u interacts with item i, then ;otherwise Due to the existence of noise samples, the true preference matrix R is different from the interaction data (binary matrix) ˜R affected by the noise.

[0023] With R∈R |U|×|I| represents the true preference matrix, where r ui represents the true preference of user u for item i. What can be observed is the binary matrix ˜R affected by noise, while the true preference matrix R is not available. Formally, assume Follows a Bernoulli distribution: ; in, Describes the user's positive preference for the item (i.e. =1, we can get (u,i) approximation). The parameters θ define the model f. Model (Generalized Matrix Factorization Model) It is used to generate the final recommendation results. Specifically, the output value of the model (generalized matrix decomposition model) after processing logits through the sigmoid function is expressed as (u,i) is also = 1. Generally speaking, the model It is necessary to have a higher model expression ability. The present invention uses the generalized matrix factorization model (GMF). Since the denoised signal comes from the consistency between the prediction results of different models, two other Bernoulli distributions are introduced. μ1 (u,i) and g μ2 (u,i) parameterization. In the present invention, g μ1 and g μ2 A relatively simple model is used - Matrix Factorization (MF). For ease of introduction, the following content will use g μ (u,i) to represent g μ1 (u,i) and g μ2 The introduction of multiple models can make the denoising teacher model more robust, and these models will participate in the subsequent multi-model knowledge distillation. The present invention can be extended to more models or replace the generalized matrix decomposition model. and matrix factorization model g μ。 The above generalized matrix factorization model and matrix factorization model g μ It is to obtain multiple denoising teacher models through training. For positive noise samples and negative noise samples, we further assume that given the true preference Under the condition of The distribution of is: ; ; in, (u,i) and (u,i) are respectively determined by the parameters The h and h′ models in this paper are implemented using the MF model (h and h′ models are models used to describe the consistency between real user preferences and observed data). The task of this paper is to infer the real user preference matrix R and its corresponding denoising teacher model (generalized matrix decomposition model) given the binary data ˜R affected by noise. and matrix factorization model g μ .

[0024] In an embodiment of the present invention, a multi-model denoising method for a reinforcement learning recommendation system is provided. Figure 1 is a flow chart of the multi-model denoising method of the reinforcement learning recommendation system of the present invention. Figure 1 and Figure 2 As shown, the process includes the following steps: S100, obtaining noisy interaction data, and using a generalized matrix factorization model (GMF) and two other matrix factorization models (MF) as prediction models to predict the noisy interaction data; It should be noted that the noisy data obtained includes noisy binary data, and the binary data is used to infer the real user preference matrix and its corresponding denoising teacher model.

[0025] Experiments show that different prediction models (recommendation models) have very similar results when predicting clean samples, but have large differences on noisy samples. That is, different prediction models tend to fit different parts of noisy samples, but clean samples are what all models try to fit.

[0026] It should be further explained that a recommendation model is a technology specially designed for recommendation systems. Its core task is to predict the goods, content or other items that users may be interested in based on their historical behavior, preferences and contextual information, and recommend these items to users. The terms prediction model and recommendation model are sometimes used interchangeably, but they usually refer to different aspects of the same concept. Recommendation model: This refers to the entire system or framework that integrates various components and technologies to generate recommendations. Prediction models more specifically refer to mathematical or statistical models used to estimate user preferences for items.

[0027] S200, optimizing the prediction model parameters by minimizing the KL divergence of the real user preference distribution fitted between multiple prediction models and maximizing the likelihood of the observed data under the given real user preference condition; Specifically, the prediction model parameters are optimized by minimizing the KL divergence of the real user preference distribution fitted between multiple prediction models and maximizing the likelihood of the observed data given the real user preference conditions, including: Constructing an objective function for minimizing the KL divergence between different prediction models and maximizing the likelihood of observed data given real user preferences; For any two matrix decomposition models, calculate the KL divergence between their output probability distributions; Extend the KL divergence calculation to model pairs, add the KL divergence between all model pairs and use it as part of the objective function; To ensure the consistency of all models, add the KL divergence between all model pairs and use it as part of the objective function, and strive to minimize this part.

[0028] Define the likelihood function according to Bayes' theorem and maximize the likelihood; The combination of minimizing KL divergence and maximizing likelihood forms a comprehensive objective function, which is combined with the optimization algorithm to update the prediction model parameters.

[0029] The present invention uses supervisory signals from multiple different models (referring to outputs or prediction results from multiple different models, which are used as guiding information when training the target model) to improve the final target model (i.e., the student model), and uses the consistency of the model's predictions on clean samples to make the target model have a better denoising effect.

[0030] For example, p(R) represents the real user preference bernoulli(η), and The corresponding expression is bernoulli ( ) and bernoulli( ).because and are close to the real user preference R, so according to the above analysis, they should maintain a small KL divergence, the formula is: ; in, express and The KL divergence between Indicates that for Sampling from a distribution calculate The expected value of the difference.

[0031] Since there is no supervision signal of clean sample R, it is necessary to introduce supervision signal from the damaged data observation ~R, which can be approximated by Bayesian theorem: ; in Represents corrupted data observations The probability of Indicates that given a corrupted data observation Under the condition of , the conditional probability that the real user prefers R is, It means that under the condition of given real user preference R, the corrupted data observation The conditional probability of .

[0032] Combining the above two formulas and shifting the terms will give the final optimization formula; .

[0033] At this time, optimizing the left side of the above formula means maximizing the likelihood of data observation given the real user preferences, while minimizing the KL divergence between the two models (both of which approximate the real user preference distribution). This is equivalent to training the network with noisy data ~R (generalized matrix factorization model) , and then pass the information to the network (matrix factorization model) g μ Because the matrix decomposition model g μ It is relatively simple, so this network (matrix decomposition model) can only fit the clean part of the data, so it does not affect the learning of another network on the clean samples, and at the same time it can pull back another network on the noisy samples, which plays a denoising role.

[0034] The final optimization objective function can be expressed as: ; because and The gradient of the noise is different, which will also have a slight impact on the performance. Finally, the denoising objective function is formulated as: ; in, is a hyperparameter, and [0,1].

[0035] For the objective function If this is calculated directly, the result will not be satisfactory, because the calculation involves the update of three models, which will cause interference between the models and affect model learning. The specific training method is as follows: S300, using the optimized prediction model as a teacher model, and training the teacher model by denoising positive sample noise and denoising negative sample noise to obtain a denoising teacher model; The optimized prediction model is used as the teacher model, and the teacher model is trained by denoising the positive sample noise and denoising the negative sample noise to obtain the denoising teacher model, including: Denoise positive sample noise, remove the noise in the positive sample, for each positive sample, calculate its loss and adjust the weight as needed, and fix the negative sample noise part when the teacher model performs denoising positive sample noise training; Denoise negative sample noise, remove the noise in the negative samples, for each negative sample, calculate its loss and adjust the weight as needed, and fix the positive sample noise part when the teacher model performs denoised negative sample noise training.

[0036] Specifically, in order to solve the problem that the updates of the three models mentioned above will cause interference between the models and affect model learning, the present invention divides the training process of the denoising teacher model into two steps, namely: denoising positive sample noise and denoising negative sample noise, which are used to remove the noise in the positive sample and the noise in the negative sample respectively. These two steps will be iteratively optimized during the training process, and only one step will be performed each time, effectively avoiding the situation where the models interfere with each other during training.

[0037] ; .

[0038] in, Representation Model The prediction function is used to predict the rating of user u for item i. Representation Model The prediction function is used to predict whether user u has interaction with item i. is a large positive hyperparameter that replaces is a large positive hyperparameter that replaces .

[0039] S400, constructing a student model, defining a target loss function, calculating a soft target of a denoising teacher model, and using KL divergence to measure the difference between the output distribution of the denoising teacher model and the output distribution of the student model, so that the student model learns the soft label provided by the denoising teacher model; It should be noted that traditional supervised learning usually uses hard labels (0 or 1), while in the knowledge distillation process, the predicted probability distribution of the denoised teacher model is provided to the student model as a soft label. These soft labels contain more information than hard labels because they not only indicate which category is most likely, but also provide the relative likelihood of all categories, which helps the student model better learn the knowledge of the teacher model.

[0040] Specifically, construct a student model and define the target loss function, including: Create and initialize a student model, and define a loss function composition, wherein the loss function includes soft target supervision and hard target supervision; The soft target supervision comes from the soft target supervision of the teacher model, which is used for the difference between the output distribution of the teacher model and the output distribution of the student model; the hard target supervision refers to the supervision of the student model itself by the real label; the cross entropy loss function is used to evaluate the difference between the prediction result of the student model and the real label; The objective loss function consists of two parts: soft target supervision and hard target supervision, and is weighted and balanced through hyperparameters.

[0041] Furthermore, the soft target of the teacher model is calculated, including: For each input sample, calculate the predicted logits values ​​of all denoising teacher models; The logits values ​​of all denoising teacher models are averaged to get the final teacher model output.

[0042] Compute the soft target for the teacher model, also including: A temperature coefficient is introduced to adjust the softmax function to amplify the probability of the negative class.

[0043] Using the multiple teacher models trained in the previous step, we can guide the student model training to obtain the recommendation model with good denoising ability. The target loss function L of this multi-model knowledge distillation step consists of two parts, namely the "soft target" supervision L from the teacher model kd (“soft target” refers to the probability distribution output by the teacher model for each training example) and the student model itself is supervised by the true label y (Using the cross-entropy loss function Cross-Entropy, CE), the target loss function L is as follows: ; Among them, x is the input data, W is the student model parameter, and α and β are two hyperparameters used to balance the two losses.

[0044] Represents the "soft target" supervision from the two teacher models. The result zt of the teacher model is obtained by averaging the results zt1 and zt2 of the two teacher models. The neural network (model architecture used for recommendation systems and denoising tasks, especially those teacher models and student models involved in the multi-model knowledge distillation process) uses a softmax layer to realize the conversion from logits to probabilities. The softmax function is: ; Where pi is the probability of the i-th class, represents the logits of the i-th category, express However, when the softmax function is used directly as the "soft target", the high-precision denoising teacher model prediction will be very accurate, and the value of its negative label will be basically 0, which will become consistent with the contribution of the hard target to the loss function. In order to make the soft target contain more information, the temperature coefficient T is introduced, and the softmax function becomes the following form: ; Therefore, the temperature coefficient T can play a role similar to label smoothing, amplifying the probability of negative classes and adding more information. The closeness between the output distribution of the denoising teacher model and the output distribution of the student model is generally connected using KL divergence. Part of the loss function is the "soft target" supervision received from the denoising teacher model. The formula is as follows: .

[0045] in It means that at temperature T, the teacher model is The predicted probability distribution of It represents the input of the student model at temperature T The predicted probability distribution of represents the predicted probability of the teacher model for category i at temperature T, represents the predicted probability of analogy i by the student model at temperature T.

[0046] S500, updating the student model parameters based on the target loss function, and iteratively training the student model as a denoising user model.

[0047] It should be noted that the training of online reinforcement learning algorithms requires real-time reception of feedback signals from each interaction, and the quality of the decision is evaluated through the feedback signals, so as to update the strategy through continuous trial and error. Therefore, online reinforcement learning algorithms usually need to interact with user models that simulate real user feedback for training. The performance of the reinforcement learning recommendation model is highly dependent on the quality of the feedback signal given by the user model. The present invention creatively uses the above-mentioned denoising algorithm to construct a denoised user model, thereby introducing denoising signals in the reinforcement learning recommendation system.

[0048] The specific construction method of the denoising user model is to use the user features, item features and noisy interaction data in the static data set as input, and use cross-model consistency and multi-model knowledge distillation to train the denoising user model. The trained user model is used to interact with the reinforcement learning strategy and give feedback that is more in line with the user's true preferences, thereby improving the performance of the reinforcement learning recommendation method by improving noise robustness.

[0049] The present invention obtains a denoised user model that models the user's real interests and denoises the noise in the interactive data. The model can receive recommendations given by the reinforcement learning recommendation strategy and give feedback that is more in line with the real results, helping the reinforcement learning strategy to better iterate and optimize. Therefore, the present invention is applicable to a variety of reinforcement learning algorithms and data sets, and has good scalability and flexibility.

[0050] The present invention obtains multiple teacher models by minimizing the KL divergence between the real user preference distributions modeled by multiple recommendation models and maximizing the likelihood of data observations. The target model is obtained by multi-model knowledge distillation. During the process, the target model integrates the parameters of the teacher model and removes the noise of some teacher outputs. At the same time, the soft label can contain more information, so that the student model can better capture the hidden information in the data. The denoising user model is trained using the above denoising algorithm to interact with the online reinforcement learning strategy, provide feedback that is more in line with the user's real preferences, and improve the noise robustness and recommendation performance of the reinforcement learning recommendation method.

[0051] The embodiment of the present invention also provides a multi-model denoising system for a reinforcement learning recommendation system, which is used to implement the above method embodiment, and will not be repeated hereafter. The terms "module", "unit", "subunit", etc. used below can implement a combination of software and / or hardware with predetermined functions. Although the system described in the following embodiments is preferably implemented in software, the implementation of hardware or a combination of software and hardware is also possible and conceivable.

[0052] like Figure 3 As shown, Figure 3 : is a structural block diagram of a multi-model denoising system of a reinforcement learning recommendation system in the present invention, the system comprising: The initialization module is configured to: obtain noisy interaction data, and use the generalized matrix factorization model and the other two matrix factorization models as prediction models to predict the noisy interaction data; A prediction model optimization module is configured to optimize prediction model parameters by minimizing the KL divergence of the distribution of real user preferences fitted between multiple prediction models while maximizing the likelihood of observed data given the real user preferences; The denoising teacher model training module is configured as follows: the optimized prediction model is used as the teacher model, and the teacher model is trained by denoising positive sample noise and denoising negative sample noise to obtain the denoising teacher model; The knowledge distillation module is configured to: construct a student model and define a target loss function, calculate the soft target of the denoising teacher model, and use KL divergence to measure the difference between the output distribution of the denoising teacher model and the output distribution of the student model so that the student model can learn the soft labels provided by the denoising teacher model; The target model training module is configured to: update the student model parameters based on the target loss function, and iteratively train the student model as a denoising user model.

[0053] Figure 4 A structural block diagram of an electronic device provided by an embodiment of the present invention, such as Figure 4As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the following method: Obtain noisy interaction data, and use the generalized matrix factorization model and two other matrix factorization models as prediction models to predict the noisy interaction data; The prediction model parameters are optimized by minimizing the KL divergence of the distribution of real user preferences fitted between multiple prediction models and maximizing the likelihood of observed data given the real user preferences. The optimized prediction model is used as the teacher model, and the teacher model is trained by denoising the positive sample noise and denoising the negative sample noise to obtain a denoising teacher model; Construct a student model and define the target loss function, calculate the soft target of the denoising teacher model, and use the KL divergence to measure the difference between the output distribution of the denoising teacher model and the output distribution of the student model, so that the student model can learn the soft labels provided by the denoising teacher model; The student model parameters are updated based on the target loss function, and the student model is iteratively trained as the denoising user model.

[0054] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0055] An embodiment of the present invention further provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method provided in each of the above embodiments is implemented, for example, including: Obtain noisy interaction data, and use the generalized matrix factorization model and two other matrix factorization models as prediction models to predict the noisy interaction data; The prediction model parameters are optimized by minimizing the KL divergence of the distribution of real user preferences fitted between multiple prediction models and maximizing the likelihood of observed data given the real user preferences. The optimized prediction model is used as the teacher model, and the teacher model is trained by denoising the positive sample noise and denoising the negative sample noise to obtain a denoising teacher model; Construct a student model and define the target loss function, calculate the soft target of the denoising teacher model, and use the KL divergence to measure the difference between the output distribution of the denoising teacher model and the output distribution of the student model, so that the student model can learn the soft labels provided by the denoising teacher model; The student model parameters are updated based on the target loss function, and the student model is iteratively trained as the denoising user model.

[0056] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0057] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-model denoising method for a reinforcement learning recommendation system, characterized in that: The multi-model denoising method of the reinforcement learning recommendation system includes: Obtain noisy interaction data, and use the generalized matrix factorization model and two other matrix factorization models as prediction models to predict the noisy interaction data; The prediction model parameters are optimized by minimizing the KL divergence of the distribution of real user preferences fitted between multiple prediction models and maximizing the likelihood of observed data given the real user preferences. The optimized prediction model is used as the teacher model, and the teacher model is trained by denoising the positive sample noise and denoising the negative sample noise to obtain a denoising teacher model; Construct a student model and define the target loss function, calculate the soft target of the denoising teacher model, and use the KL divergence to measure the difference between the output distribution of the denoising teacher model and the output distribution of the student model, so that the student model can learn the soft labels provided by the denoising teacher model; The student model parameters are updated based on the target loss function, and the student model is iteratively trained as the denoising user model.

2. The multi-model denoising method for a reinforcement learning recommendation system according to claim 1, characterized in that: The noisy data is obtained including noisy binary data, and the binary data is used to infer a real user preference matrix and its corresponding denoising teacher model.

3. The multi-model denoising method for the reinforcement learning recommendation system according to claim 1, characterized in that: The prediction model parameters are optimized by minimizing the KL divergence of the real user preference distribution fitted between multiple prediction models and maximizing the likelihood of the observed data given the real user preference conditions, including: Constructing an objective function for minimizing the KL divergence between different prediction models and maximizing the likelihood of observed data given real user preferences; For any two matrix decomposition models, calculate the KL divergence between their output probability distributions; Extend the KL divergence calculation to model pairs, add the KL divergence between all model pairs and use it as part of the objective function; Define the likelihood function according to Bayes' theorem and maximize the likelihood; The combination of minimizing KL divergence and maximizing likelihood forms a comprehensive objective function, which is combined with the optimization algorithm to update the prediction model parameters.

4. The multi-model denoising method for a reinforcement learning recommendation system according to claim 1, characterized in that: The optimized prediction model is used as the teacher model, and the teacher model is trained by denoising the positive sample noise and denoising the negative sample noise to obtain the denoising teacher model, including: Denoise positive sample noise, remove the noise in the positive sample, for each positive sample, calculate its loss and adjust the weight as needed, and fix the negative sample noise part when the teacher model performs denoising positive sample noise training; Denoise negative sample noise, remove the noise in the negative samples, for each negative sample, calculate its loss and adjust the weight as needed, and fix the positive sample noise part when the teacher model performs denoised negative sample noise training.

5. The multi-model denoising method for a reinforcement learning recommendation system according to claim 1, characterized in that: Construct a student model and define the target loss function, including: Create and initialize a student model, and define a loss function composition, wherein the loss function includes soft target supervision and hard target supervision; The soft target supervision comes from the soft target supervision of the teacher model, which is used for the difference between the output distribution of the teacher model and the output distribution of the student model; the hard target supervision refers to the supervision of the student model itself by the real label; the cross entropy loss function is used to evaluate the difference between the prediction result of the student model and the real label; The objective loss function consists of two parts: soft target supervision and hard target supervision, and is weighted and balanced through hyperparameters.

6. The multi-model denoising method for a reinforcement learning recommendation system according to claim 1, characterized in that: Compute the soft targets for the teacher model, including: For each input sample, calculate the predicted logits values ​​of all denoising teacher models; The logits values ​​of all denoising teacher models are averaged to get the final teacher model output.

7. The multi-model denoising method for a reinforcement learning recommendation system according to claim 6, characterized in that: Compute the soft target for the teacher model, also including: The temperature coefficient is introduced to adjust the softmax function to amplify the probability of the negative class.

8. A multi-model denoising system for a reinforcement learning recommendation system, characterized in that: include: The initialization module is configured to: obtain noisy interaction data, and use the generalized matrix factorization model and the other two matrix factorization models as prediction models to predict the noisy interaction data; A prediction model optimization module is configured to optimize prediction model parameters by minimizing the KL divergence of the distribution of real user preferences fitted between multiple prediction models while maximizing the likelihood of observed data given the real user preferences; The denoising teacher model training module is configured as follows: the optimized prediction model is used as the teacher model, and the teacher model is trained by denoising positive sample noise and denoising negative sample noise to obtain the denoising teacher model; The knowledge distillation module is configured to: construct a student model and define a target loss function, calculate the soft target of the denoising teacher model, and use KL divergence to measure the difference between the output distribution of the denoising teacher model and the output distribution of the student model so that the student model can learn the soft labels provided by the denoising teacher model; The target model training module is configured to: update the student model parameters based on the target loss function, and iteratively train the student model as a denoising user model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the multi-model denoising method for the reinforcement learning recommendation system according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the multi-model denoising method for a reinforcement learning recommendation system according to any one of claims 1 to 7 are implemented.

Images