A method and system for detecting abnormal recommendation data based on reinforcement learning

Abstract

The application discloses a kind of based on reinforcement learning's abnormal recommended data detection method and system, mainly focus on multi-modal abnormal data identification in recommendation system, rely on reinforcement learning algorithm, for semi-supervised abnormal data, a method for simultaneously identifying "consistent anomaly" and "inconsistent anomaly" is proposed.The method is identified by the method proposed in the application before being transmitted into the recommendation system after interacting with the recommendation system and collecting user data, and the abnormal data therein is removed, and the normal data is transmitted to the recommendation system.The results are fed back to the user after being processed by the recommendation system.The application can easily integrate information of different modalities to achieve better results.

Description

TECHNICAL FIELD

[0001] The application relates to an abnormal recommendation data detection method and system based on reinforcement learning, which is used for detecting abnormal data in a recommendation system and belongs to the technical field of recommendation system data processing. BACKGROUND

[0002] With the development of science and technology, the amount of data has increased compared to the past, and the modalities of data have become increasingly diverse. Traditional anomaly detection can process data of the same modality, but cannot cope with the complex real environment today. Therefore, multi-modal anomaly detection is born. In a recommendation system, the sources of data are diverse, generally mainly including pictures and texts. How to process and identify abnormal data plays an important role in the recommendation system. By identifying abnormal data before the data enters the recommendation system, the performance of the recommendation system can be greatly accelerated, and the cost of human risk control processing can be saved.

[0003] Traditional primary anomaly detection algorithms for data from a single source only consider data from two perspectives, normal data and abnormal data. Different algorithms have different definitions of abnormal data. Some algorithms assume that there is a cluster structure in the data, so the so-called abnormal data is an outlier outside the cluster structure. Some other algorithms cannot solve the problem of data anomaly without cluster structure, and use dimension transformation. For example, some data may be chaotic in Euclidean space and cannot be distinguished from abnormal data, but when they are mapped in non-Euclidean space, there may be a certain structure to help distinguish them. Manifold learning algorithms can be regarded as this kind.

[0004] Meanwhile, compared with traditional single-modal anomaly detection which only needs to distinguish abnormal data into abnormal data and non-abnormal data, multi-modal data has two different types of anomalies. One can be called "inconsistent anomaly", which is inconsistent between different modalities. The other can be called "consistent anomaly", which describes the same abnormal data in different modalities. However, previous multi-modal anomaly detection work assumes that there is a certain clustering structure in real-world abnormal data, which limits the development of the algorithm and makes it unable to process non-clustered data anomaly detection. Secondly, previous multi-modal anomaly detection algorithms need to model "consistent anomaly" and "inconsistent anomaly" respectively, which makes it impossible to consider both types of anomalies as a whole. Finally, there is certain expert knowledge in the real world that can assist the algorithm in distinguishing, and most of the data is in an unannotated state, so it is closer to semi-supervised learning.

[0005] The existing solutions to multi-modal anomaly data detection mainly have the following problems: 1) Different types of multi-modal anomaly detection methods make corresponding assumptions on data, have strong abnormal pattern assumptions, such as distribution-based methods and clustering-based methods, and this type of method often cannot produce satisfactory results when the assumption does not hold. 2) Existing multi-modal anomaly detection methods are often based on complex mathematical models, cannot utilize existing neural network models, and are relatively disadvantaged in processing picture and text modal information. 3) Existing methods rarely consider point anomalies and collective anomalies at the same time, resulting in poor performance of static anomaly detection model parameters in dynamic scenarios. SUMMARY

[0006] The present application provides an anomaly recommendation data detection method and system based on reinforcement learning, which uses different neural networks to extract the key dimension information of multiple modalities of original data, adopts a bilinear pooling method for fusion between modalities, and obtains better results compared with the traditional simple vector addition and simple splicing method.

[0007] Technical scheme: An anomaly recommendation data detection method based on reinforcement learning mainly focuses on multi-modal anomaly data recognition in a recommendation system, relies on a reinforcement learning algorithm, and proposes a method for simultaneously recognizing "consistent anomalies" and "inconsistent anomalies" for semi-supervised anomaly data. The method interacts with the recommendation system, identifies the abnormal data after collecting user data and before transmitting the data to the recommendation system, removes the abnormal data, and transmits the normal data to the recommendation system. The results are fed back to the user after being processed by the recommendation system.

[0008] The main process is as follows:

[0009] (1) Collect existing user data in the recommendation system, including user ID, browsing record corresponding item list, timestamp and other meta information required by the recommendation system. Partially annotate the user data, and divide the data into annotated data set containing abnormal data D a and unannotated data set D u ; the collected user data is provided to the model for subsequent training.

[0010] (2) Use the user data in (1) to learn and train to generate a corresponding recognition model.

[0011] In the process of generating the identification model, the Agent network is used to identify whether the multi-modal data is abnormal, the Teacher network is used to give rewards for the action and data situation output by the Agent network at the last moment, the Select network is used to select the data at the next moment, and the Fusion network is used to fuse the multi-modal data. The specific training steps are as follows:

[0012] Step 1: initialize the Agent network, the Teacher network, the Select network, and the Fusion network.

[0013] Step 2: the Agent network accepts t the state at the moment and the reward and outputs the action , wherein the state represents the data vector obtained after multi-modal fusion of the data in the labeled data set D a and the unlabeled data set D u collected by the recommendation system, the reward represents the evaluation of the Teacher network on the action output by the Agent network, and the action represents the probability that the state is abnormal data.

[0014] Step 3: save the state , the reward , and the output action of the Agent network as a triple , , ;

[0015] Step 4: the Select network accepts the action and returns the selected next data, and pushes the selected data to the Fusion network;

[0016] Step 5: the Fusion network receives the data of different modalities pushed by the Select network at the moment t +1 , extracts the feature information of the modalities through the neural network M 1 and M 2 ;

[0017] Step 6: use the bilinear pooling to fuse the feature information M 1 and M 2 to form a single modal data, i.e. t+1 time state s t+1 ;

[0018] Step 7: The Teacher network accepts the action of the last time , and the state of the next time output by the Fusion network s t+1 , generates the reward for the action of the last time ;

[0019] Step 8: Repeat steps 3-7, and take the triple , , generated by each repetition process as a piece of training data. Stop after collecting a certain number (such as 2048) of training data.

[0020] Step 9: Use the collected training data to train the model, optimize the parameters of each neural network, and take the discounted cumulative reward value of the generated trajectory as the optimization target:

[0021] .

[0022] Step 10: Repeat steps 3-9 until the specified number of training rounds is met or the optimization target is reached, and obtain the trained Agent network as the identification model.

[0023] (3) Multi-modal abnormal data recognition agent deployment.

[0024] Deploy the above trained Agent network into a real recommendation system, and use real online data to identify abnormal data. Through online data, the accuracy, recall rate, etc. of the agent for real multi-modal abnormal data recognition can be continuously detected. If the corresponding indicators decrease and do not meet the requirements, repeat process (2) to retrain and perform process (3) to deploy.

[0025] An abnormal recommendation data detection system based on reinforcement learning, comprising: a user data collection module, an identification model training module, a deployment module, and a model evaluation module;

[0026] The user data collection module collects existing user data in the recommendation system, and performs partial labeling on the user data. The data is divided into an abnormal data labeled data set D a and an unlabeled data set D u ; the collected user data is used to train the identification model by the identification model training module.

[0027] ​The recognition model training module learns and trains by using user data, uses an Agent network to identify whether multi-modal data is abnormal, uses a Teacher network to give rewards for the action and data of the agent at the previous moment, uses a Select network to select data at the next moment, and uses a Fusion network to fuse multi-modal data.

[0028] The deployment module deploys the trained Agent network into a real recommendation system and identifies abnormal data by using real online data.

[0029] The model evaluation module continuously detects the accuracy and recall rate of the agent in identifying real multi-modal abnormal data by using online data of the interaction between the Agent network and the recommendation system, and if the corresponding indicators decrease and do not meet the requirements, the recognition model training module is repeatedly executed and the deployment module is executed.

[0030] A computer device includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor executes the computer program to implement the abnormal recommendation data detection method based on reinforcement learning.

[0031] A computer readable storage medium stores a computer program for executing the abnormal recommendation data detection method based on reinforcement learning.

[0032] Advantages: Compared with the prior art, the abnormal recommendation data detection method and system based on reinforcement learning can detect abnormal information for any multi-modal data without considering the specific data type and data structure style, and without making any assumptions for specific data. The application expands the limitations of previous similar algorithm applications and improves the precision brought by training to obtain better performance benefits. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 is the method principle diagram of the embodiment of the application;

[0034] Figure 2 is the model training flowchart of the embodiment of the application. DETAILED DESCRIPTION

[0035] The application will be further illustrated by specific embodiments. It should be understood that these embodiments are only used to illustrate the application and not to limit the scope of the application. After reading the application, those skilled in the art can modify various equivalent forms of the application, which fall within the scope defined by the claims of the application.

[0036] An abnormal recommendation data detection method based on reinforcement learning, comprising:

[0037] (1) Collect existing user data in the recommendation system, including user ID, browsing record corresponding item list, timestamp and other meta information required by the recommendation system in the last period of time required by the recommendation system. Partially annotate the user data, and divide the data into annotated data set containing abnormal data D a and unannotated data set D u ; For the collected user data, subsequent training is provided to the model.

[0038] (2) Use the user data in (1) to learn and train to generate a corresponding identification model.

[0039] In the process of generating the identification model, the Agent network is used to identify whether the multi-modal data is abnormal, the Teacher network is used to give rewards according to the action and data of the Agent network output at the last moment, the Select network is used to select the data at the next moment, and the Fusion network is used to fuse the multi-modal data. The specific training steps are as follows:

[0040] Step 1: initialize the Agent network, the Teacher network, the Select network, and the Fusion network.

[0041] Step 2: the Agent network accepts t moment state and reward and outputs action , wherein the state indicates that the data in the annotated data set D a and the unannotated data set D u collected by the recommendation system are fused to obtain a data vector, the reward indicates the evaluation of the Teacher network on the action output by the Agent network, and the action indicates the probability of the state being abnormal data.

[0042] Step 3: save the state and reward , output action accepted by the Agent network as a triple , , ;

[0043] Step 4: the Select network accepts action , return the selected next data and push the selected data to the Fusion network;

[0044] Step 5: The Fusion network extracts the feature information of the modal through the neural network by receiving the data of different modal pushed by the Select network at time t +1 M 1 and M 2 ;

[0045] Step 6: Use the bilinear pooling to fuse the feature information M 1 and M 2 to form a single modal data, i.e. t the state at time s t+1 ;

[0046] Step 7: The Teacher network accepts the action at the last time and the state at the next time output by the Fusion network s t+1 to generate the reward for the action at the last time ;

[0047] Step 8: Repeat steps 3-7, and take the triple , , generated by each repetition process as a piece of training data. Stop after collecting a certain number of training data (e.g. 2048 pieces).

[0048] Step 9: Use the collected training data to train the model, optimize the parameters of each neural network, and take the discounted cumulative reward value of the generated trajectory as the optimization target:

[0049] .

[0050] Step 10: Repeat steps 3-9 until the specified number of training rounds is met or the optimization target is reached, and get the trained Agent network.

[0051] (3) Multi-modal abnormal data recognition agent deployment.

[0052] The trained Agent network is deployed into a real recommendation system to identify abnormal data using real online data. Through online data, the accuracy, recall rate, and other indicators of the agent for real multi-modal abnormal data identification can be continuously detected. If the corresponding indicators decrease to the extent that they do not meet the requirements, the process (2) is repeated to retrain, and the process (3) is performed to deploy.

[0053] An abnormal recommendation data detection system based on reinforcement learning includes a user data collection module, an identification model training module, a deployment module, and a model evaluation module. The user data collection module collects existing user data in the recommendation system, partially labels the user data, and divides the data into an abnormal data labeled data set and an unlabeled data set. D a and an unlabeled data set D u The identification model training module trains the identification model using the collected user data. The identification model training module uses the user data for learning and training, uses the Agent network to identify whether the multi-modal data is abnormal, uses the Teacher network to give rewards based on the actions of the Agent network at the previous time and the data situation, uses the Select network to select the data at the next time, and uses the Fusion network to fuse the multi-modal data. The deployment module deploys the trained Agent network into a real recommendation system to identify abnormal data using real online data. The model evaluation module continuously detects the accuracy, recall rate, and other indicators of the Agent network for real multi-modal abnormal data identification through online data interaction between the Agent network and the recommendation system. If the corresponding indicators decrease to the extent that they do not meet the requirements, the identification model training module is repeated, and the deployment module is executed.

[0054] In a real recommendation scenario, the user side collects user data from the recommendation system through the user data collection module or the user data collection process. The collected data from the recommendation system is first sent to the identification model, and after identifying the abnormal data, the data without problems is transmitted to the recommendation system part. Finally, the recommendation system feeds back the results to the user.

[0055] As Figure 1As shown, two networks, Select network and Fusion network, are used to cope with the state selection of the next moment and the fusion of the obtained multi-modal data, and the two networks are collectively referred to as Environment network; the rule form of the pre-experience can be used to constrain how the Select network selects the next moment data according to the last moment action; the Select network is used to convert the non-Markov process into a Markov process, and the Fusion network is used to realize multi-modal abnormal data pushing; in the Teacher network, the action of the last moment and the state selected by the Environment network in the next moment are obtained, and the evaluation reward for the action of the last moment is output. The Teacher network can select those reward functions customized according to prior knowledge, or use a neural reward network learned by data as its ontology. In the Agent network, it can be judged whether the current multi-modal data is abnormal through the reinforcement learning method.

[0056] The Agent network aims to learn an optimal agent facing the multi-modal anomaly detection situation, which can obtain the maximum reward and, therefore, in the case of the strategy of the Agent network being π, the expected reward that can be obtained is as follows:

[0057]

[0058] In the above formula, θ represents the parameters of the Agent network, τ represents the collected data sample, R(τ) represents the reward of the trajectory τ, p θ τ ) represents the probability of the trajectory τ, and the expected reward formula is the expected return starting from the state s , taking action a , and then following the behavior strategy π=P ( a | s ). Our expectation is to adjust the strategy π of the Agent network so that the expected reward can be maximized. In order to achieve this purpose, there are many policy gradient related methods that can be implemented, and the proximal policy optimization PPO algorithm is selected. It learns the parameters θ by iteratively maximizing the following loss function:

[0059]

[0060] In the formula, s represents the state of the current moment , θ θ represents the Agent network parameters, θk represents the parameters of the kth round of Agent network, ​Qk(s, a) denotes the value of the advantage function computed by the Agent network in the k-th round under the policy

[0061] The state sampling function Select network is a key module in the simulation environment, which consists of two functions g a and g n to achieve the balance of exploration and exploitation of the full dataset D . Among them, g a is a function that uniformly samples D a from the labeled dataset s t . This provides each abnormal data with the same opportunity to be exploited by the Agent network. On the other hand, g n is a sampling function specifically for normal data, which samples D u from the unlabeled dataset s t+1 according to the proximity of the current state. In the interaction process between the Agent network-Environment network, g a and g n are used simultaneously: with probability p, perform g a , and with probability 1-p, perform g n . This way allows the Agent network to explore large data while fully utilizing the labeled multi-modal abnormal data. In this work, p=0.5 can be considered as equal utilization of the two kinds of data.

[0062] The Teacher network is defined as a function that produces a reward signal to our Agent network according to its performance in detecting known abnormalities:

[0063]

[0064] ​​​This indicates that the agent network only receives a positive reward when it correctly identifies known multimodal anomalies as "anomalies." If the agent network can correctly identify normal observations, it will not receive a reward. This is because in real-world data, anomalous data often constitutes a minority, while normal data is the vast majority. Giving a positive reward to normal data might lead the agent network to tend to identify all data as normal. Finally, if the agent network produces false negatives or false positives, it will be penalized with a negative reward. Therefore, It can explicitly encourage agent networks to make full use of anomalous datasets. D a .

[0065] Figure 2 The diagram illustrates the training process of the identification model. The algorithm is trained on multimodal data collected from users of the recommendation system. After training, an optimal Agent network is obtained, capable of identifying multimodal anomaly data. Using this trained Agent network as the identification model, the recommendation system can detect data information in real time, determine whether it is anomaly data, and transmit normal data to the recommendation system. The algorithm uses an Agent network to identify whether user data in the recommendation system is multimodal anomaly data, a Teacher network to score the actions and data output by the Agent network during training, a Select network to select the data to be input into the Agent network in the next step, and a Fusion network to fuse the user data from the recommendation system.

[0066] like Figure 2 The specific training steps for the recognition model shown are as follows:

[0067] Step 101: Initialize the Agent Network Teacher Network Select network and Fusion Network Using existing pre-trained neural network models for specific modalities. Value function network Buffer D is used to temporarily store data generated during training.

[0068] Step 102: Reset network gradients, synchronize all network parameters, and reset training buffer D.

[0069] Step 103: Fusion network Transmit the multimodal data selected by the Select network from the dataset in the previous moment. dt , the result is saved as s t .

[0070] Step 104: transmit the state to the Teacher network s t and the action a t , the result is saved as .

[0071] Step 105: transmit the state to the Agent network s t and the reward , the result is saved as the action a t+1 .

[0072] Step 106: transmit the action to the Select network a t+1 , the Select network will select the data at the next time point from the existing data set, and mark it as d t+1

[0073] Step 107: save the state s t , the action a t and the reward in the training Buffer D.

[0074] Step 108: repeat steps 103-107 until the required amount of data is collected.

[0075] Step 109: calculate the advantage function, the formula is as follows, wherein s is the state at the current time s t , a is the action at the current time a t , represents the state at the next time , r represents the reward obtained at the current time , and the Q function is calculated by the existing state s, action a and value function network V:

[0076]

[0077] wherein,

[0078]

[0079]

[0080] Step 110: using PPO algorithm, update the training strategy, whose optimization goal is:

[0081]

[0082]

[0083] wherein ε is a very small hyperparameter, which roughly represents the tolerance of the distance between the new strategy and the old strategy, and s is the current state s t , a is the action at the current time a t , s' represents the state at the next time s t+1 r represents the reward obtained at the current time The clip function takes three parameters as input, and outputs the first parameter if it is within the range of the second parameter and the third parameter, otherwise the value is clipped, and the upper and lower limits are the values of the third parameter and the second parameter. represents the probability of action a when the neural network parameter is θ and the state is s represents the probability of action a when the neural network parameter is θ k and the state is s. represents the probability of action a when the neural network parameter is θ k , the state is s, and the action is a.

[0084] Step 111: repeat steps 109~110 until the optimization requirement is met or the specified number of rounds is reached.

[0085] Obviously, those skilled in the art should understand that the steps of the abnormal recommended data detection method based on reinforcement learning of the embodiments of the application described above or the modules of the abnormal recommended data detection system based on reinforcement learning can be realized by general computing devices, which can be concentrated on a single computing device or distributed on a network composed of multiple computing devices. Alternatively, they can be realized by program codes executable by computing devices, so that they can be stored in storage devices and executed by computing devices, and in some cases, the steps shown or described here can be executed in different order, or they can be manufactured into individual integrated circuit modules, or multiple modules or steps among them can be manufactured into a single integrated circuit module to realize. Thus, the embodiments of the application are not limited to any specific combination of hardware and software.

Claims

1. A method for detecting abnormal recommendation data based on reinforcement learning, characterized in that, Comprise: 1) Collect existing user data in the recommendation system; 2) Use user data for learning training to generate a corresponding identification model; In the process of generating the identification model, the Agent network is used to identify whether the multi-modal data is abnormal, the Teacher network is used to give rewards for the action and data situation output by the Agent network at the previous moment, and the Select network is used to select the data at the next moment; The Fusion network is used to fuse the multi-modal data; 3) Multi-modal abnormal data identification agent deployment; Deploy the trained Agent network to the real recommendation system, and use online data to identify abnormal data; through online data, the indicators of the agent for multi-modal abnormal data identification are continuously detected, and if the corresponding indicators decrease to not meet the requirements, the identification model is retrained and the Agent network is redeployed; Collect existing user data in the recommendation system, including user ID, browsing record corresponding item list, timestamp and other meta information required by the recommendation system in the recent period of time required by the recommendation system; Partially annotate the user data, divide the data into annotated data sets containing abnormal data D a And unlabeled data sets D u ; For the collected user data, provide the identification model for model training; The identification model training steps are as follows: Step 1: initialize the Agent network, the Teacher network, the Select network, and the Fusion network; Step 2: Agent network accepts t Time state And reward And outputs action Wherein state Represents the data vector obtained after multi-modal fusion of the labeled data set collected by the recommendation system D a And the data in the unlabeled data set D u Reward Represents the evaluation of the action output by the Agent network by the Teacher network, and action Represents the probability that the state is abnormal data; Step 3: The state received by the Agent network and the reward , the output action , are saved as a triple , , > Step 4: Select network accepts action Return the next data selected and push the selected data to the Fusion network. Step 5: The Fusion network receives data from the Select network. t Data from different modalities pushed at time +1 is used to extract modal feature information through a neural network. M 1 and M 2 ; Step 6: Feature information is fused using bilinear pooling to form a single modality data, i.e. M 1 and M 2 is fused to form a single modality data, i.e. t +1 time state s t+1 ; Step 7: The Teacher network accepts the action from the previous time step , and the state from the next time step output by the Fusion network s t+1 , produces a reward for the action from the previous time step ;​ Step 8: Repeat Step 3 ~ Step 7, and the triplets generated by each repetition of the process are stored as a training data , , > as a training data, and the process is stopped after a predetermined number of training data is collected. Step 9: use the collected training data to train the model and optimize the neural network parameters, and take the discounted cumulative reward value of the generated trajectory as the optimization target: ； Step 10: repeat steps 3~9 until the specified training number is met or the optimization target is reached, and obtain the trained Agent network as the identification model.

2. A system for implementing the method of claim 1 for detecting abnormal recommendation data based on reinforcement learning, characterized in that, Comprise: Collect user data module, identification model training module, deployment module, and model evaluation module; The collecting user data module collects existing user data in a recommendation system, performs partial labeling on the user data, and divides the data into a labeled data set containing abnormal data D a and an unlabeled data set D u The user data collected is used to train an identification model by the identification model training module The identification model training module uses user data for learning training, uses the Agent network to identify whether the multi-modal data is abnormal, uses the Teacher network to give rewards for the action and data situation of the agent at the previous moment; and uses the Select network to select the data at the next moment; The Fusion network is used to fuse the multi-modal data; The deployment module deploys the trained Agent network to the real recommendation system to identify abnormal data using real online data; The model evaluation module continuously detects the indicators of the agent for multi-modal abnormal data identification through the online data of the Agent network interacting with the recommendation system, and if the corresponding indicators decrease to not meet the requirements, the identification model training module is repeatedly executed, and the deployment module is executed.

3. A computer device, characterized by: The computer device comprises a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor executes the above computer program to realize the abnormal recommendation data detection method based on reinforcement learning in any one of claims 1.

4. A computer-readable storage medium, characterized in that: The computer readable storage medium stores a computer program for executing the abnormal recommendation data detection method based on reinforcement learning in any one of claims 1.

Images