Training method of recommendation network, content recommendation method, device and electronic equipment

Abstract

Embodiments of the present disclosure provide a training method of a recommendation network, a content recommendation method and device, an electronic device, a computer readable storage medium and product, and relate to the technical field of artificial intelligence and the technical field of content recommendation. The method comprises: obtaining a plurality of sample environments comprising object features of sample objects and content features of candidate contents; iteratively training a first strategy network and a second strategy network based on the plurality of sample environments, determining a comprehensive reward of the first strategy network based on respective iteration rewards of the first strategy network and the second strategy network obtained after each iteration; updating network parameters of the first strategy network and the second strategy network according to the comprehensive reward of the first strategy network until an iteration end condition is met, and taking the obtained first strategy network as a recommendation network. The embodiments of the present disclosure realize obtaining the recommendation network based on self-game reinforcement learning, and can recommend precise interested contents for users in actual application.

Description

Technical Field

[0001] This disclosure relates to the fields of content recommendation technology and artificial intelligence technology. Specifically, this disclosure relates to a training method for a recommendation network, a content recommendation method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] Content recommendation utilizes technologies such as machine learning to proactively display content that users might be interested in during their browsing and interaction with a product, thereby enhancing the user experience. Reinforcement learning is a machine learning method that obtains feedback through interaction with the environment, continuously updating its strategy to achieve the optimal decision-making goal.

[0003] In recent years, researchers have begun exploring the introduction of reinforcement learning into recommender systems, optimizing recommendation performance through dynamic strategy adjustments. However, current reinforcement learning-based recommender systems still have some shortcomings, including less than ideal recommendation results and an inability to accurately recommend content of interest to users. Summary of the Invention

[0004] This disclosure provides a method for training a recommendation network, a content recommendation method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product, which can solve the problem that existing technologies cannot accurately recommend content of interest to users. The technical solution provided by this disclosure is as follows: According to a first aspect of the present disclosure, a method for training a recommendation network is provided, the method comprising: Obtain the sample environment, including the object features of the sample objects and the content features of the candidate content; Based on the sample environment, the first policy network and the second policy network are iteratively trained to obtain the iterative rewards corresponding to the first policy network and the second policy network at the current iteration training time; the first policy network predicts the recommended candidate content when recommending to the sample object from the candidate content; the second policy network predicts the excluded candidate content when recommending to the sample object from the candidate content. The comprehensive reward of the first policy network is determined based on the iterative rewards corresponding to the first policy network and the second policy network, respectively. The network parameters of the first policy network are updated based on the comprehensive reward of the first policy network until the iteration termination condition is met, and the trained first policy network is used as the recommendation network.

[0005] As an optional embodiment, the method further includes: Obtain a preset reward value, and determine the comprehensive reward of the second strategy network based on the reward value and the comprehensive reward of the first strategy network; The network parameters of the second policy network are updated based on the comprehensive reward of the second policy network to obtain the second policy network for the next iteration of training.

[0006] As an optional implementation, for each training iteration, the iteration rewards corresponding to the first policy network and the second policy network at the current iteration are obtained in the following way: Perform at least one training operation on the sample environment, and obtain the iterative reward of the first policy network based on the training reward of each training operation of the first policy network, and obtain the iterative reward of the second policy network based on the training reward of each training operation of the second policy network: The current state of the current training operation is input into the first policy network and the second policy network, respectively. The first policy network predicts recommended candidate content from the candidate content, and the second policy network predicts excluded candidate content from the candidate content. Based on the recommendation value of the recommended candidate content, the training reward of the first policy network for the current training operation is determined, and based on the recommendation value of the excluded candidate content, the training reward of the second policy network for the current training operation is determined. The content features of the recommended candidate content and the content features of the excluded candidate content are deleted from the current state to obtain the current state of the next training operation. The current state of the first training operation of each sample environment is the sample environment.

[0007] As an optional embodiment, determining the training reward for the current training operation of the first policy network based on the recommendation value of the recommended candidate content includes: Based on the recommendation value of the recommended candidate content, the content reward for the current training operation of the first policy network is determined; The types of recommended candidate content and historical recommended content for the current training operation are determined, and the content adjustment value of the content reward for the current training operation of the first policy network is determined; wherein, the historical recommended content includes the recommended candidate content of each training operation before the current training operation; The content reward for the current training operation of the first policy network is adjusted based on the content adjustment value to obtain the training reward for the current training operation of the first policy network.

[0008] As an optional embodiment, determining the first reward for the current operation of the first policy network based on the recommendation value of the recommended candidate content and the recommendation value of the excluded candidate content includes: Based on the recommendation value of the recommended candidate content, the content reward for the current training operation of the first policy network is determined; For each training operation, a second reward adjustment value is determined based on the training reward of the second policy network for the current training operation. The content reward of the current training operation of the first policy network is adjusted based on the reward adjustment value, and the training reward of the current training operation of the first policy network is determined based on the adjusted content reward.

[0009] As an optional embodiment, the iterative reward of the first policy includes the training reward of each training operation of the first policy network, and the iterative reward of the second policy network includes the training reward of each training operation of the second policy network. The step of determining the comprehensive reward of the first policy network based on the iterative rewards corresponding to the first policy network and the second policy network includes: For each training operation, the comprehensive reward of the first policy network for the current training operation is determined based on the training reward of the first policy network for the current training operation and the training reward of the second policy network for the current training operation. The comprehensive reward of the first policy network is obtained based on the comprehensive reward of each training operation of the first policy network.

[0010] As an optional embodiment, the number of sample environments is multiple, and updating the network parameters of the first policy network based on the comprehensive reward of the first policy network includes: The network parameters of the first policy network are updated based on the comprehensive reward of the first policy network corresponding to the multiple sample environments. Each iteration of training also includes: Identify the first number of sample environments in which the overall reward of the first policy network is greater than the overall reward of the second policy network. If the ratio of the first quantity to the total number of sample environments is greater than a preset value, then the first policy network on which the current iteration of training is based is added to the policy pool; The recommendation network includes each first policy network that is added to the policy pool when the iteration ends.

[0011] As an optional embodiment, when the first policy network is being trained for the first time, the second policy network is determined in the following manner: Obtain the multiple opponent policy networks corresponding to the first policy network; Obtain multiple test environments; For each test environment, the test environment is input into each opponent policy network, and the recommended candidate content is obtained by predicting the test environment through each opponent policy network; For each adversary policy network, the performance of the adversary policy network is determined by predicting the recommended value of the excluded candidate content for each test environment. The opponent's policy network with the highest performance among all opponent policy networks is selected as the second policy network.

[0012] As an optional embodiment, for each adversary policy network, determining the performance of the adversary policy network by predicting the recommendation value of the recommended candidate content corresponding to each test environment includes: For each test environment, the test environment is input into the first policy network, and the first policy network is used to predict the test environment to obtain recommended candidate content. For each test environment, the total reward of the opponent policy network and the total reward of the first policy network are determined based on the recommendation value of the excluded candidate content predicted by the opponent policy network and the recommendation value of the recommended candidate content predicted by the first policy network. A second number of initial environments are identified in which the total reward of the first policy network corresponding to the multiple test environments is greater than the total reward of the second policy network. The performance of the adversary policy network is determined based on the ratio of the second number to the total number of test environments, wherein the ratio is positively correlated with the performance.

[0013] According to a second aspect of the present disclosure, a content recommendation method is provided, the method comprising: Obtain the current environment, including the object characteristics of the target object and the content characteristics of the candidate content; The current environment is input into the recommendation network to obtain the recommendation candidate content output by the recommendation network, and the recommendation candidate content is recommended to the target object. The recommendation network is trained using the method described in any one of the first aspects of the claim.

[0014] According to a third aspect of the present disclosure, a method for training a policy network is provided, the method comprising: Obtain the sample environment, including the object features of the sample objects and the content features of the candidate content; Based on the sample environment, the first policy network and the second policy network are iteratively trained to obtain the iterative rewards corresponding to the first policy network and the second policy network at the current iteration training time; the first policy network predicts the recommended candidate content when recommending to the sample object from the candidate content; the second policy network predicts the excluded candidate content when recommending to the sample object from the candidate content. The comprehensive reward of the second policy network is determined based on the iterative rewards corresponding to the first policy network and the second policy network, respectively. The network parameters of the second policy network are updated based on the comprehensive reward of the second policy network until the iteration termination condition is met, and the trained second policy network is used as the filtering network.

[0015] According to a fourth aspect of the present disclosure, a training apparatus for a recommendation network is provided, the apparatus comprising: The first acquisition module is used to acquire the sample environment, including the object features of the sample object and the content features of the candidate content; The first training module is used to iteratively train the first policy network and the second policy network based on the sample environment to obtain the iteration reward corresponding to the first policy network and the second policy network at the current iteration training time; the first policy network predicts the recommended candidate content when recommending to the sample object from the candidate content; the second policy network predicts the excluded candidate content when recommending to the sample object from the candidate content. The first reward determination module is used to determine the comprehensive reward of the first policy network based on the iterative rewards corresponding to the first policy network and the second policy network respectively. The first network parameter update module is used to update the network parameters of the first policy network according to the comprehensive reward of the first policy network until the iteration termination condition is met, and the trained first policy network is used as the recommendation network.

[0016] According to a fifth aspect of the present disclosure, a content recommendation apparatus is provided, the apparatus comprising: The second acquisition module is used to acquire the current environment, including the object characteristics of the target object and the content characteristics of the candidate content; The recommendation module is used to input the current environment into the recommendation network, obtain the recommendation candidate content output by the recommendation network, and recommend the recommendation candidate content to the target object; wherein, the recommendation network is trained using the method described in any of the first aspects.

[0017] According to a sixth aspect of the present disclosure, a training apparatus for a policy network is provided, the apparatus comprising: The third acquisition module is used to acquire the sample environment, including the object features of the sample object and the content features of the candidate content; The second training module is used to iteratively train the first policy network and the second policy network based on the sample environment to obtain the iterative rewards corresponding to the first policy network and the second policy network at the current iteration training time; the first policy network predicts the recommended candidate content when recommending to the sample object from the candidate content; the second policy network predicts the excluded candidate content when recommending to the sample object from the candidate content. The second reward determination module is used to determine the comprehensive reward of the second policy network based on the iterative rewards corresponding to the first policy network and the second policy network respectively. The second network parameter update module is used to update the network parameters of the first policy network according to the comprehensive reward of the second policy network until the iteration termination condition is met, and the trained second policy network is used as the filtering network.

[0018] According to a seventh aspect of the present disclosure, an electronic device is provided, the electronic device including a memory, a processor, and a computer program stored in the memory, the processor executing the computer program to implement the above-described training method for a data processing recommendation network, an inference recommendation method, and a training method for a policy network.

[0019] According to an eighth aspect of the present disclosure, a computer-readable storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements a training method for a data processing recommendation network, an inference recommendation method, and a training method for a policy network.

[0020] According to a ninth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described training method for a data processing recommendation network, an inference recommendation method, and a training method for a policy network.

[0021] The beneficial effects of the technical solutions provided in this disclosure are: By iteratively training the first and second policy networks based on the sample environment, we obtain the recommended candidate content predicted by the first policy network when recommending candidate content to the sample object in each iteration, and the excluded candidate content predicted by the second policy network from the candidate content. This allows us to view the predictions of the first policy network as actions performed by the first policy network and the predictions of the second policy network as actions performed by the second policy network. We then obtain the rewards of the first and second policy networks in the game. Based on the iterative rewards of the first and second policy networks, we generate a comprehensive reward for the first policy network. This comprehensive reward includes not only the reward obtained when the first policy network outputs recommended candidate content but also the reward obtained when the second policy network outputs excluded candidate content. The network parameters of the first policy network are updated using this comprehensive reward, enabling the first policy network to train under the influence of the actions performed by the second policy network, thus gradually improving its recommendation level during training. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments of this disclosure will be briefly introduced below.

[0023] Figure 1 A schematic diagram of a system architecture provided for an embodiment of this disclosure; Figure 2 A flowchart illustrating a training method for a recommendation network provided in an embodiment of this disclosure; Figure 3 A schematic diagram of a process for determining a comprehensive reward provided in an embodiment of this disclosure; Figure 4 A diagram illustrating the structure of a training system for a recommendation network provided in this embodiment of the disclosure; Figure 5 This disclosure provides a schematic diagram of training in each iteration. Figure 6 This is a schematic diagram of a process for determining a comprehensive reward, provided as an embodiment of the present disclosure. Figure 7 A schematic diagram of a process for determining a winning network provided in an embodiment of this disclosure; Figure 8 A schematic diagram illustrating the determination of a second strategy network according to an embodiment of this disclosure; Figure 9 A schematic diagram of a process for determining the network performance of an adversary strategy, provided as an embodiment of this disclosure; Figure 10A flowchart illustrating a training method for a recommendation network provided in an embodiment of this disclosure; Figure 11 A flowchart illustrating a content recommendation method provided in an embodiment of this disclosure; Figure 12 A flowchart illustrating a method for training a policy network according to an embodiment of this disclosure; Figure 13 A schematic diagram of a training system structure provided in this embodiment of the present disclosure; Figure 14 This is a schematic diagram of an application scenario provided by an embodiment of the present disclosure; Figure 15 A schematic diagram of the structure of a training device for a recommendation network provided in an embodiment of this disclosure; Figure 16 This is a schematic diagram of the structure of a content recommendation device provided in an embodiment of the present disclosure; Figure 17 A schematic diagram of the structure of a training device for a policy network provided in an embodiment of this disclosure; Figure 18 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0024] The embodiments of this disclosure are described below with reference to the accompanying drawings. It should be understood that the embodiments described below with reference to the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of this disclosure, and do not constitute a limitation on the technical solutions of the embodiments of this disclosure.

[0025] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the terms “comprising” and “including” as used in embodiments of this disclosure mean that the corresponding feature can be implemented as the presented feature, information, data, step, operation, element, and / or component, but do not exclude implementation as other features, information, data, step, operation, element, component, and / or combinations thereof supported by the art. It should be understood that when we say that an element is “connected” or “coupled” to another element, the one element can be directly connected or coupled to the other element, or it can mean that the one element and the other element are connected through an intermediate element. Furthermore, “connected” or “coupled” as used herein can include wireless connection or wireless coupling. The term “and / or” as used herein indicates at least one of the items defined by the term; for example, “A and / or B” can be implemented as “A,” or as “B,” or as “A and B.” When describing multiple (two or more) items, if the relationship between the multiple items is not explicitly defined, the multiple items can refer to one, several or all of the multiple items. For example, the description of "parameter A includes A1, A2, A3" can be implemented as parameter A includes A1 or A2 or A3, or it can be implemented as parameter A includes at least two of the three items A1, A2 and A3.

[0026] In this disclosure, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0027] To make the objectives, technical solutions, and advantages of this disclosure clearer, the embodiments of this disclosure will be described in further detail below with reference to the accompanying drawings.

[0028] First, the technical terms used in this disclosure will be introduced and explained: Reinforcement Learning (RL) is a machine learning method that uses a trial-and-error feedback mechanism to enable an agent to continuously optimize its decisions. The agent can be a deep learning network; in this embodiment, the agent is referred to as a policy network. In reinforcement learning, the environment is the external world that the agent interacts with. The agent learns how to maximize cumulative rewards by performing actions in the environment, which defines the action space the agent can take, the observed state space, and the reward mechanism.

[0029] Existing single-agent reinforcement learning methods have several significant limitations in practical applications. First, because the agent interacts only with a fixed environment and lacks diverse feedback, it is prone to getting trapped in an "information cocoon," resulting in overly restrictive policies that struggle to adapt to complex dynamic environments. Second, single-agent reinforcement learning is prone to overfitting during training, meaning the agent becomes overly reliant on specific patterns in the training environment and fails to generalize to a wider range of scenarios. Furthermore, due to the lack of sufficient data or interaction information in the initial stages, the agent struggles to quickly establish effective policies, leading to poor performance in cold-start scenarios.

[0030] The environment (Env) is the external world in which the agent exists. It defines the states that the agent can observe, the actions it can perform, and the consequences of its actions. The environment is a core component of a reinforcement learning system, determining the agent's learning objectives and interaction rules. In this embodiment, the environment is the interaction space between the user and candidate content during the runtime of the recommendation system.

[0031] A recommender system is an information filtering system that analyzes user feedback, preferences, and interests to provide a personalized list of items (such as products, movies, music, articles, etc.) to help users discover content that may be of interest. The core goal of recommender systems is to reduce information overload, improve user decision-making efficiency amidst massive amounts of information, and enhance user experience. Currently, most recommender systems utilize technologies based on deep neural networks, such as collaborative filtering and reinforcement learning.

[0032] Collaborative filtering is used for user-to-user collaborative filtering: by calculating the similarity between users, it recommends content liked by other similar users. Item-to-item collaborative filtering: based on content similarity, it recommends other content similar to the user's historical preferences. Deep neural networks (DNNs) can be combined with collaborative filtering, using deep neural networks to learn user and content features into embedding vectors, and making recommendations based on similarity in the embedding space. However, collaborative filtering relies on a large amount of relevant user data and performs poorly for new content and new users (i.e., the cold start problem).

[0033] Reinforcement learning recommendation systems, such as Deep Q-Learning and Actor-Critic methods, train agents based on user feedback to continuously optimize strategies. However, in practical applications, they suffer from training instability, and the strategies are prone to overfitting the interests of specific user groups, resulting in insufficient diversity of recommended content.

[0034] The training method, content recommendation method, apparatus, electronic device, computer-readable storage medium, and computer program product of the recommendation network disclosed herein are intended to solve at least one of the above-mentioned technical problems of the prior art.

[0035] It should be noted that in the optional embodiments of this disclosure, data related to object information (such as age, gender, etc.) requires the object's permission or consent when the embodiments of this disclosure are applied to specific products or technologies. Furthermore, the collection, use, and processing of this data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. In other words, if the embodiments of this disclosure involve data related to an object, it must be obtained with the object's authorization and consent, the authorization and consent of relevant departments, and in accordance with the relevant laws, regulations, and standards of the country and region. If the embodiments involve personal information, the acquisition of all personal information requires the individual's consent. If sensitive information is involved, the separate consent of the information subject is required. The embodiments also need to be implemented with the object's authorization and consent.

[0036] The following description of several exemplary embodiments illustrates the technical solutions of this disclosure and the technical effects produced by these solutions. It should be noted that the following embodiments can be referenced, learned from, or combined with each other. Identical terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0037] Figure 1 This is a schematic diagram of a system architecture provided in an embodiment of the present disclosure. The system architecture includes a terminal 10, a content recommendation server 20, a network training server 30, an environment library 40, and an application server 50. The content recommendation server 20 can be a server for a target application. This server is used to recommend content of interest to the user based on the user's recommendation request. The target application can be an e-commerce application, a video media application, a music media application, or a job recruitment application, etc., which require a recommendation system to make targeted recommendations to the user based on their characteristics.

[0038] Terminal 10 can deploy an application client for the target application corresponding to application server 20. Users can browse the content in the application client through terminal 10. Users can send content recommendation requests to content recommendation server 20 through terminal 10. For example, users can trigger the terminal to send a recommendation request to content recommendation server by entering keywords of the content they want to search for in the search bar of the application client.

[0039] The content recommendation server 20 deploys a recommendation network trained by the network training server. When the content recommendation server receives a recommendation request, it can filter the content in the content database based on the keywords or search terms in the recommendation request, select candidate content that matches the recommendation request, and input the selected candidate content into the environment generation server 40. The environment generation server 40 includes a feature extraction module, which is used to extract features from the user's attribute information to obtain the user's features, extract features from the candidate content to obtain the features of the candidate content, and generate the user's current environment based on the user's features and the features of the candidate content. The current environment is then sent to the content recommendation server. After obtaining the user's current environment, the content recommendation server inputs the user's current environment into the recommendation network to obtain recommended content, and displays the recommended content to the user. There can be multiple recommended contents.

[0040] A network training server 30 is used to train a recommendation network. During training, the network training server can generate sample environments through an environment generation server and train the recommendation network based on these sample environments. Specifically, during training, the network training server can obtain a first policy network and a second policy network to participate in the training process of the first policy network. By inputting the sample environment into the first and second policy networks, the server obtains the user-interested recommendation content predicted step-by-step by the first policy network for that sample environment, and the user-uninterested recommendation content predicted step-by-step by the second policy network for that sample environment. The server compares the ability of the second policy network to output uninterested recommendation content with the ability of the first policy network to output interested recommendation content to determine the reward for each step of the prediction process for the first policy network. Based on all the rewards corresponding to the sample environment, the server obtains the loss value of the first policy network, thereby optimizing the first policy network.

[0041] In this system architecture, during training, the network training server samples an additional second-policy network to participate in the training process of the first-policy network. The stronger the ability of the second-policy network to output recommended content that the user is not interested in, the smaller the reward for the first-policy network, and thus the greater the training loss. This allows the second-policy network to incentivize the first-policy network to optimize, so that the final recommendation network can more accurately recommend content that users are interested in.

[0042] The aforementioned servers (content recommendation server 20, network training server 30, and environment generation server 40) can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Communication between content recommendation server 20 and network training server 30, between content recommendation server 20 and terminal 10, between content recommendation server 20 and environment generation server 40, and between network training server 30 and environment generation server 40 can be achieved via wired or wireless networks.

[0043] Optionally, the aforementioned wireless or wired networks use standard communication technologies and / or protocols. The network is typically the Internet, but can be any network, including but not limited to any combination of local area networks (LANs), metropolitan area networks (MANs), wide area networks (WANs), mobile, wired or wireless networks, private networks, or virtual private networks.

[0044] In some embodiments, the server undertakes the main computing work and the terminal undertakes the secondary computing work; or, the server undertakes the secondary computing work and the terminal undertakes the main computing work; or, the server and the terminal collaborate on computing using a distributed computing architecture.

[0045] The technical solutions of the present disclosure and their effects are described below through several embodiments. It should be noted that the following embodiments can be referenced, learned from, or combined with each other. Identical terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0046] This disclosure provides a method for training a recommendation network, which can be executed by the aforementioned network training server, such as... Figure 2 As shown, the method includes: S201. Obtain the sample environment, which includes the object features of the sample objects and the content features of the candidate content.

[0047] The training method for the recommendation network provided in this embodiment can be applied to the field of content recommendation. The content can be in various forms, such as videos, music, news information, pictures, articles, and e-commerce products. The following description uses video as an example.

[0048] In video recommendation platforms, users can register to obtain an account and then log in to browse videos. The sample objects in this embodiment can be users of the video recommendation platform. User data can be obtained through various methods, and feature extraction can be performed on this data to obtain user characteristics. In this embodiment, the number of sample environments can be one or more.

[0049] It can obtain attribute data such as age, gender, and region from the information provided by users during registration; it can obtain feedback data such as browsing history, viewing time, likes, comments, and shares from user logs on content recommendation platforms; and it can also obtain feedback data from users when browsing videos, including user ratings and comments, to determine user interests and preferences.

[0050] After obtaining relevant user data, the data can be preprocessed, including requesting data from the user data and removing noise and outliers. It can also include time-series processing of the user data to obtain time-series behavioral data, ensuring that the behavioral data is sorted by timestamp so that subsequent deep learning models can capture the user's temporal characteristics, such as obtaining the user's short-term and long-term interests.

[0051] Specifically, to extract users' short-term and long-term interests from their time-series behavioral data, a recurrent neural network (RNN) model or a sequence transformation model (Transformer) can be used to extract features from the user's time-series behavioral data to obtain the user's object features.

[0052] Optionally, to transform user-related data into more compact feature vectors, feature learning can be performed on the user-related data through clustering analysis or embedding learning. Clustering analysis is used to identify user types and group users into different clusters based on behavioral similarity. Clustering algorithms such as K-Means and Density-Based Clustering (DBSCAN) can be used to divide users into different clusters based on behavioral similarity, with the centroid of each cluster representing a type of behavioral feature. Embedding learning is used to map user-related data into a continuous vector space through low-dimensional embeddings. This can be achieved by training a neural network model (e.g., an autoencoder) to learn a low-dimensional representation of behavioral data. An autoencoder is an unsupervised learning method that learns an effective embedding representation by compressing input data to a low-dimensional representation and then decoding it back to the original data, obtaining a low-dimensional embedding vector as the user's feature representation.

[0053] Optionally, the features learned by the aforementioned deep learning models can be fused. This involves integrating object features obtained from RNNs, Transformers, clustering analysis, embedding learning, etc., to form a comprehensive object feature vector, which serves as the user's object feature. It should be noted that before extracting features from the user's relevant data, the model can be trained unsupervised to improve the accuracy of the extracted features in reflecting the user's interests.

[0054] In this embodiment of the disclosure, for each candidate content, relevant data of the candidate content can be obtained, and feature extraction can be performed on the relevant data of the candidate content to obtain the content features of the candidate content. The relevant data of the candidate content includes the metadata of the candidate content and the corresponding interaction data. The metadata of the candidate content may include text data such as the title, tags, description, and keywords of the candidate content, used to reflect the theme, type, and sentiment of the content. The interaction data corresponding to the candidate content includes statistical data such as the click-through rate, number of favorites, comments, and number of shares of the content, used to reflect the popularity of the candidate content.

[0055] Optionally, when extracting features from the relevant data of the candidate content, features can be extracted from the metadata and from the interaction data separately. By extracting features from the metadata, semantic information can be extracted to generate text embedding vectors for the candidate content. Specifically, existing natural language processing models can be used to encode the text of the content. This natural language processing model can be a BERT model or a Word2Vec model, where the BERT model can be a pre-trained BERT model to avoid training from an initial model, which would be inefficient.

[0056] Meanwhile, by extracting features from the interactive data, interactive information can be added to the feature vectors of candidate content.

[0057] When extracting features from the interaction data of candidate content, different weights can be assigned to different types of interaction information. For example, click-through rate reflects the attractiveness of content better than the number of favorites, and the weight of click-through rate can be greater than the weight of favorites. Finally, the user interaction information is merged into a single feature by weighted summation.

[0058] Optionally, after obtaining the text features of the content data metadata and the interaction features of the interaction data, multimodal feature fusion can be used to fuse the text features and interaction features to generate the final content feature vector. Specific fusion methods can include at least one of feature concatenation, weighted fusion, and feature engineering. It should be noted that feature engineering involves further processing the concatenated feature vector using feature engineering techniques (such as PCA, standardization, regularization, etc.) to improve the model's training performance.

[0059] Optionally, since natural language processing models such as BERT and Word2Vec have been pre-trained on large-scale text datasets, these pre-trained models can be used directly to extract text features. In order to enable the model to better capture content features, supervised learning can be used to fine-tune the model.

[0060] Through the above process, a feature vector of candidate content is finally generated. This feature vector integrates the semantic information of the candidate content and the interaction information between multiple users and the content.

[0061] In this embodiment of the disclosure, in practical applications, the content recommendation platform can determine corresponding candidate content based on the user's input search keywords, or it can filter matching candidate content based on the user's relevant data. During the training process, in order to generate a sample environment, a search keyword can be generated for the user based on their search history to determine the corresponding candidate content, or the corresponding candidate content can be selected directly based on the user's relevant data. For each user, a sample environment can be generated by combining the user's object features and the content features of the candidate content.

[0062] Optionally, the object features of multiple users can be aggregated, and each aggregated object feature can be used as a sample object feature. This allows the generated sample environment to represent the object features of a certain type of user, effectively reducing the size of the training data, improving training efficiency, and mitigating data sparsity issues. Especially when relevant data for some users is limited, clustering helps generate more representative feature vectors for each user, thus compensating for insufficient data. Optionally, when determining candidate content, search keywords can be generated based on the search history of this type of user, or relevant candidate content can be directly filtered based on the user's relevant data.

[0063] The number of clusters can be pre-set or automatically determined by analyzing the object features of multiple users. Optionally, the features of multiple sample objects obtained after clustering can be weighted and fused to generate a reference sample object. This reference sample object represents newly registered users and helps improve recommendation performance during cold starts. Optionally, a sample object can correspond to multiple different sets of candidate content, and the object features of the sample object and the content features of each set of candidate content constitute a sample environment.

[0064] S202. Based on the sample environment, iteratively train the first policy network and the second policy network to obtain the iterative rewards corresponding to the first policy network and the second policy network at the current iteration training.

[0065] The first strategy network predicts recommended candidate content from the candidate content when recommending to the sample object; the second strategy network predicts excluded candidate content from the candidate content when recommending to the sample object.

[0066] In this embodiment of the disclosure, the first policy network and the second policy network can be obtained through pre-training, for example, by collecting object features of multiple users and content features of content that has been recommended to users in the past, using user feedback information as labels, and performing supervised training to obtain the first policy network and the second policy network.

[0067] In this embodiment, the first policy network and the second policy network can be viewed as agent roles in reinforcement learning. The first policy network can be seen as a recommender agent, used to predict content that the user is interested in from the current state. The second policy network can be seen as an adversary agent, used to predict content that the user is not interested in from the current state. The first and second policy networks can be deep learning networks, such as feedforward neural networks (FNN), convolutional neural networks (CNN), and recurrent neural networks (RNN) for prediction. Each first policy network is based on the same network architecture, only the network parameters differ, and each second policy network is based on the same network architecture, only the network parameters differ. The first and second policy networks can be implemented based on the same network architecture or different network architectures. For example, the first policy network can be a feedforward neural network, and the second policy network can be a convolutional neural network. Alternatively, the first and second policy networks can be feedforward neural networks with different network parameters.

[0068] In this embodiment of the disclosure, the sample environment includes object features of sample objects and content features of candidate content, and the first policy network and the second policy network can be iteratively trained based on the sample environment.

[0069] The first strategy network and the second strategy network can output predicted candidate content based on the sample environment, respectively. The first strategy network can output recommended candidate content based on the sample environment, and the second strategy network can output excluded candidate content based on the sample environment. The first strategy network is used to predict content that the user is interested in from the candidate content, i.e., the recommended candidate content in this embodiment of the disclosure. The second strategy network is used to predict content that the user is not interested in from the candidate content, i.e., the excluded candidate content in this embodiment of the disclosure. Therefore, when recommending content to the user, excluded candidate content can be predicted based on the second strategy network, and excluded candidate content can be filtered out from the candidate content before recommendations are made to the user.

[0070] In this embodiment of the disclosure, for each training iteration, the iteration reward of the first policy network in that iteration can be determined based on the recommended candidate content predicted by the first policy network in that iteration, and the iteration reward of the second policy network in that iteration can be determined based on the excluded candidate content predicted by the second policy network in that iteration. Each training iteration can be viewed as a game between the first and second policy networks based on the sample environment. The first policy network's prediction based on the sample environment to obtain the recommended candidate network can be seen as the action performed by the first policy network in this game, and the second policy network's prediction based on the sample environment to obtain the excluded candidate content can be seen as the action performed by the second policy network in this game. The iteration rewards of the two networks are determined according to the actions they each perform.

[0071] Optionally, the iterative reward of the first policy network can be determined based on the recommendation value of the candidate content predicted by the first policy network; the iterative reward of the second policy network can be determined based on the recommendation value of the excluded candidate content predicted by the second policy network. The recommendation value of the candidate content reflects the degree to which other users like the candidate content. The higher the recommendation value of the candidate content, the greater the iterative reward of the first policy network; the higher the recommendation value of the excluded candidate content, the smaller the iterative reward of the second policy network.

[0072] As an optional embodiment, the recommendation value of each candidate content is determined in the following way: Obtain multiple recommendation feedback messages when the candidate content is recommended to the recommended object; The recommendation value of the candidate content is determined based on the recommendation feedback information.

[0073] In this embodiment of the disclosure, in order to determine the recommendation value of candidate content, feedback information generated when candidate content was recommended in the past can be used to estimate the degree of interest of sample objects in the candidate content.

[0074] For each candidate content, multiple recommendation feedback pieces related to that content can be pre-acquired. These feedback pieces can include performance evaluation data for the content, such as click-through rate, number of favorites, likes, comments, and shares. This feedback information can be obtained through log systems, online monitoring, or user surveys and can be stored in the recommendation system's database for subsequent calculation of recommendation value. Optionally, weight parameters can be set for different feedback pieces, and the weighted sum of all feedback pieces can be used as the recommendation value of the candidate content.

[0075] As an optional embodiment, if the value of the iterative reward is within a preset range, then when the recommendation value is obtained based on the recommendation feedback information of the candidate content, the recommendation value can be processed to the preset range, such as [0,10]. That is, the recommendation value of each candidate content is a number within the preset range (including both endpoints). At this time, the recommendation value of the recommended candidate content can be used as the iterative reward of the first policy network, and the difference between the maximum value within the preset range and the recommendation value of the excluded candidate content can be used as the iterative reward of the second policy network.

[0076] In this embodiment of the disclosure, for the current iteration, after the first policy network predicts the candidate content to be recommended to the user, its iteration reward in the current iteration can be determined based on the recommendation value of the candidate content. After the second policy network predicts the candidate content to be excluded from the user's recommendation, its iteration reward in the current iteration can be determined based on the recommendation value of the excluded candidate content. Based on the recommendation value of the candidate content, the user's corresponding feedback information (e.g., clicking or not clicking the content) is simulated if the candidate content is recommended to the user in this sample scenario. Based on the recommendation value of the excluded candidate content, the user's corresponding feedback information is simulated if the excluded candidate content is recommended to the user in this sample scenario.

[0077] like Figure 3 As shown, Figure 3 This is a schematic flowchart illustrating a process for determining a comprehensive reward according to an embodiment of this disclosure. By inputting a sample environment into a first policy network and a second policy network, recommended candidate content output by the first policy network and excluded candidate content output by the second policy network are obtained, respectively. Based on the recommendation value of the recommended candidate content, an iterative reward for the first policy network is obtained. Based on the recommendation value of the excluded candidate content, an iterative reward for the second policy network is obtained. Based on the iterative rewards of the first and second policy networks, a comprehensive reward for the first policy network is obtained.

[0078] S203. Determine the comprehensive reward of the first policy network based on the iterative rewards corresponding to the first policy network and the second policy network respectively.

[0079] In this embodiment of the disclosure, in each iteration of training, the first policy network and the second policy network each correspond to an iteration reward. A comprehensive reward can be obtained based on the iteration rewards corresponding to the first policy network and the second policy network. This comprehensive reward is used to update the network parameters of the first policy network, and is therefore referred to as the comprehensive reward of the first policy network.

[0080] Specifically, the iterative reward of the first policy network can be adjusted based on the iterative reward of the second policy network to obtain the comprehensive reward of the first policy network. Based on the prediction effect of the second policy network, the first policy network is given corresponding negative feedback. Specifically, the better the prediction effect of the second policy network, that is, the greater the iterative reward of the second policy network, the greater the negative feedback is given to the first policy network.

[0081] Optionally, a reward adjustment value can be generated based on preset adjustment weights and the iterative reward of the second strategy network. The iterative reward of the first strategy network is then adjusted based on this reward adjustment value to obtain the comprehensive reward of the first strategy network. The reward adjustment value can be the product of the preset adjustment weights and the iterative reward of the second strategy network. Optionally, the comprehensive reward of the first strategy network can be determined by the following formula:

[0082] Where α and β are preset weights, which can be pre-set. α can be set to 1, and β can be set to 0.2. The specific values ​​can also be adjusted according to the training effect. X1 is the iteration reward of the first policy network, and X2 is the iteration reward of the second policy network. It can be seen that the total reward of the first policy network obtained in this embodiment is affected by the second policy network, and the higher the iteration reward of the second policy network, the lower the overall reward of the first policy network.

[0083] S204. Update the network parameters of the first policy network according to the comprehensive reward of the first policy network until the iteration termination condition is met, and use the trained first policy network as the recommendation network.

[0084] In this embodiment of the disclosure, the network parameters of the first policy network can be updated based on the overall reward of the first policy network. The network parameters of the first policy network can be updated by maximizing the overall reward of the first policy network.

[0085] This is an option; you can calculate the corresponding training loss based on the first policy network and update the network parameters by minimizing the training loss.

[0086] The iteration termination condition may include: The convergence of the comprehensive reward of the first policy network is defined as follows: the change in the comprehensive reward value is lower than the set threshold over multiple consecutive training cycles; or the number of training rounds reaches the set maximum number of training rounds.

[0087] By iteratively training the first and second policy networks based on the sample environment, we obtain the recommended candidate content predicted by the first policy network when recommending candidate content to the sample object in each iteration, and the excluded candidate content predicted by the second policy network from the candidate content. This allows us to view the predictions of the first policy network as actions performed by the first policy network and the predictions of the second policy network as actions performed by the second policy network. We then obtain the rewards of the first and second policy networks in the game. Based on the iterative rewards of the first and second policy networks, we generate a comprehensive reward for the first policy network. This comprehensive reward includes not only the reward obtained when the first policy network outputs recommended candidate content but also the reward obtained when the second policy network outputs excluded candidate content. The network parameters of the first policy network are updated using this comprehensive reward, enabling the first policy network to train under the influence of the actions performed by the second policy network, thus gradually improving its recommendation level during training.

[0088] like Figure 4 As shown, Figure 4 The figure shows a training system architecture diagram for a recommendation network provided in this embodiment of the present disclosure. The system architecture of this recommendation network includes an input module, a feature modeling module, a self-play reinforcement learning module, and an output module. The input module is used to acquire relevant data of candidate content and relevant data of users. The input module inputs the acquired data into the feature modeling module. The feature modeling module performs content feature modeling on the candidate content based on the content-related data and feature modeling on the users based on the user-related data.

[0089] After feature modeling is completed, the content features and user object features are input into the self-game reinforcement learning module. The self-game reinforcement learning module constructs an environment based on the content features and user object features. The recommender policy network and the opponent policy network each make their own prediction actions based on the environment. According to a preset reward mechanism, the recommender policy network and the opponent policy network are optimized. Finally, the trained recommender network is used as the recommendation policy. This recommender policy network is the first policy network in this embodiment, and the opponent policy network is the second policy network in this embodiment.

[0090] During optimization, networks with better performance in either the recommender's or opponent's policy network can be added to a policy pool. The environments corresponding to these performances are stored in the experience replay module for subsequent training of the recommender and opponent networks based on these replay environments. Optionally, the recommender's policy network can be added to the recommender's policy pool, and the opponent's policy network to the opponent's policy pool. The optimal recommender policy can be periodically selected from the policy pools and deployed in the recommendation system. When encountering new user scenarios, different policies from the policy pools can be randomly selected for self-game training to maintain recommendation diversity and improve recommendation performance in cold start scenarios. Furthermore, recommender policy networks can be periodically selected from the policy pools for evaluation. Through canary releases, recommendation performance can be tested in real-world environments, and the deployed policy networks can be dynamically updated based on actual recommendation results.

[0091] As an optional embodiment of this disclosure, the training method for the recommendation network further includes: Obtain a preset reward value, and determine the comprehensive reward of the second strategy network based on the reward value and the comprehensive reward of the first strategy network; The network parameters of the second policy network are updated based on the comprehensive reward of the second policy network to obtain the second policy network for the next iteration of training.

[0092] In this embodiment of the disclosure, when training the first policy network, the second policy network can also be trained, so that while the first policy network is being optimized, the second policy network is also being optimized. The first policy network needs to continuously play a game with the second policy network and continuously optimize its own decisions, thereby more accurately identifying the content that users are truly interested in, improving the accuracy of recommendations and user experience, and achieving the effect of counter-optimization.

[0093] In this embodiment, after determining the comprehensive reward of the first policy network, the comprehensive reward of the second policy network can be determined based on a preset reward value. The reward value can be the maximum value (right endpoint) of a preset range, i.e., when the preset range is [0, 10], the preset value is 10. The comprehensive reward of the second policy network is calculated based on the preset reward value and the comprehensive reward of the first policy network; this is the sum of the comprehensive reward of the second policy network and the reward of the first policy network. This allows for the measurement of the prediction performance of the second policy network and the first policy network during the current training iteration.

[0094] Optionally, the total reward of the second strategy network can be the difference between the reward value and the total reward of the first strategy network. For example, if the reward value is 10 and the total reward of the first strategy network is 7, then the total reward of the second strategy network is 3.

[0095] like Figure 5 As shown, Figure 5 This disclosure provides a schematic diagram of each iteration of training. The sample environment is input into a first policy network and a second policy network, and the iteration rewards of the first policy network and the second policy network are obtained respectively. The comprehensive reward of the first policy network is obtained based on the iteration reward of the first policy network, and the comprehensive reward of the second policy network is obtained based on the comprehensive reward of the first policy network. The network parameters of the first policy network are adjusted according to the comprehensive reward of the first policy network, and the network parameters of the second policy network are adjusted according to the comprehensive reward of the second policy network.

[0096] As can be seen, the first strategy network and the second strategy network in this embodiment form a game relationship during the training process. The first strategy network continuously optimizes the prediction ability of recommended candidate content, while the second strategy network optimizes the prediction ability of excluded candidate content, which can effectively prevent the first strategy network from getting stuck in local optima.

[0097] The embodiments disclosed herein are based on a training method that optimizes self-game strategies. The first strategy network and the second strategy network are trained simultaneously. The second strategy network can be regarded as the opponent strategy network of the first strategy network. Both the first strategy network and the second strategy network can continuously improve their prediction performance during the continuous game process.

[0098] As an optional embodiment of this disclosure, for each training iteration, the iteration rewards corresponding to the first policy network and the second policy network at the current iteration are obtained in the following manner: Perform at least one training operation on the sample environment, and obtain the iterative reward of the first policy network based on the training reward of each training operation of the first policy network, and obtain the iterative reward of the second policy network based on the training reward of each training operation of the second policy network: The current state of the current training operation is input into the first policy network and the second policy network, respectively. The first policy network predicts recommended candidate content from the candidate content, and the second policy network predicts excluded candidate content from the candidate content. Based on the recommendation value of the recommended candidate content, the training reward of the first policy network for the current training operation is determined, and based on the recommendation value of the excluded candidate content, the training reward of the second policy network for the current training operation is determined. The content features of the recommended candidate content and the content features of the excluded candidate content are deleted from the current state to obtain the current state of the next training operation. The current state of the first training operation of each sample environment is the sample environment.

[0099] In the embodiments of this disclosure, the first policy network and the second policy network can interact with the sample environment multiple times, with each interaction serving as a training operation. For each training operation, the first policy network predicts a candidate content to recommend to the user, and the second policy network predicts an exclusion candidate content to recommend to the user.

[0100] In each training operation, the first policy network predicts recommended candidate content from the candidate content and obtains the training reward for the current training operation of the first policy network. The second policy network predicts excluded candidate content from the candidate content and obtains the training reward for the current training operation of the second policy network.

[0101] Generally, in real-world applications, a user's level of interest in content can be determined through user feedback on the content. This feedback can include whether the user clicked on the recommended content, the proportion of time the user spent browsing the recommended content to the total browsing time, whether the user saved the content, and if the content is a product, whether the user placed an order for the product. However, when training offline, it is impossible to obtain the sample objects' feedback on the recommended candidate content and the excluded candidate content. Therefore, in this embodiment of the disclosure, the sample objects' level of interest in the recommended content can be determined through the content's own recommendation value.

[0102] Optionally, the recommended feedback information can include timely feedback information and long-term feedback information. Instant feedback information can include click-through rate, number of favorites, and whether a purchase was made after clicking. Long-term feedback information has a long-term effect, such as increased user retention and average browsing time. Different weights can be assigned to instant and long-term feedback information to obtain the recommendation value of the candidate content and achieve diversified rewards.

[0103] In this embodiment of the disclosure, the recommendation value of each candidate sample in the sample environment can be determined when the sample environment is acquired to improve training efficiency. Alternatively, the recommendation value of the content can be determined after the first policy network and the second policy network predict their respective content, thereby reducing the problem of the large amount of computation required to predetermine the recommendation value of all candidate content.

[0104] The available methods for determining the recommendation value of each candidate content have been introduced earlier and will not be repeated here.

[0105] In this embodiment of the disclosure, at least one training operation is performed on the sample environment. By taking the sample environment as the current state of the first training operation, the corresponding current state is input into the first policy network and the second policy network each time the training operation is performed, so that the first policy network and the second policy network predict the current state respectively, and obtain the recommended candidate content output by the first policy network and the excluded candidate content output by the second policy network.

[0106] The training operation includes: obtaining the current state of the current operation; inputting the current state of the current operation into the first policy network and the second policy network to obtain the recommended candidate content output by the first policy network and the excluded candidate content output by the second policy network, respectively; obtaining the training reward of the first policy network for the current training operation based on the recommendation value of the recommended candidate content; obtaining the training reward of the second policy network for the current training operation based on the recommendation value of the excluded candidate content; then deleting the content features of the recommended candidate content and the content features of the excluded candidate content in the current state to obtain the next state; and using the next state as the current state for the next training operation.

[0107] In this embodiment of the disclosure, for each training operation, after determining the recommended candidate content and the excluded candidate content, the content features of the recommended candidate content and the excluded candidate content are deleted from the current state to avoid dependence on these processed contents in subsequent recommendation processes. By deleting these content features, a new current state is generated, which will be used as input for the next training operation, thereby obtaining the trajectory of the first policy network's step-by-step prediction in each sample environment.

[0108] In this embodiment of the disclosure, by performing at least one training operation during each iteration of training, an action sequence generated by a first policy network based on the sample environment and an action sequence generated by a second policy network based on the sample environment are obtained. The action sequence corresponding to the first policy network is a sequence composed of multiple recommended candidate contents gradually predicted by the first policy network after performing multiple training operations on the sample environment, and the action sequence corresponding to the second policy network is a sequence composed of multiple excluded candidate contents gradually predicted by the second policy network after performing multiple training operations on the sample environment.

[0109] In each action sequence, the corresponding predicted content is used as an action, and each action corresponds to a reward. Since the first policy network is used to predict content that the user is interested in, the reward for the corresponding action is higher if the recommended candidate content is content that the user is very interested in. Since the second policy network is used to predict content that the user is not interested in, the reward for the corresponding action is higher if the excluded candidate content is content that the user is not interested in.

[0110] As an optional embodiment, during each iteration, the iteration rewards of the first policy network in each training operation can be weighted and fused to obtain the iteration reward of the first policy network, and the iteration rewards of the second policy network in each training operation can be weighted and fused to obtain the iteration reward of the second policy network.

[0111] As an optional embodiment of this disclosure, determining the training reward of the first policy network for the current training operation based on the recommendation value of the recommended candidate content includes: Based on the recommendation value of the recommended candidate content, the content reward for the current training operation of the first policy network is determined; The types of recommended candidate content and historical recommended content for the current training operation are determined, and the content adjustment value of the content reward for the current training operation of the first policy network is determined; wherein, the historical recommended content includes the recommended candidate content of each training operation before the current training operation; The content reward for the current training operation of the first policy network is adjusted based on the content adjustment value to obtain the training reward for the current training operation of the first policy network.

[0112] In this embodiment, the content type includes the category, topic, or attribute to which the content belongs, such as news, video, or product categories. Historical recommended content can be content recommended by the first policy network during training operations prior to the current training operation. To enable the first policy network to explore different types of candidate content and prevent overfitting during training, the content reward can be increased when the type of the recommended candidate content output by the first policy network differs from the type of historical recommendations.

[0113] In this embodiment of the disclosure, the content reward of the first policy network in the current training operation can be generated based on the recommendation value of the recommended candidate content obtained by the first policy network in the current training operation. When the type of the recommended candidate content output by the first policy network is the same as the type of the historical recommended content, the content reward can be reduced as the training reward of the first policy network in the current training operation.

[0114] Optionally, the semantic similarity between the types of recommended candidate content and the types of historical recommended content can be determined based on their semantic features. The type differences between recommended candidate content and historical recommended content can then be determined based on the semantic similarity, and a content adjustment value can be determined.

[0115] If the type of the current recommended content is highly consistent with the type of historical recommended content, the content adjustment value is small; conversely, if the type of the current recommended content differs significantly from the type of historical recommended content, the content adjustment value is large. The content adjustment value can be obtained using the following formula: Content adjustment value = semantic similarity

[0116] in, , Using preset parameters, it can be seen that when the semantic similarity is less than When the content adjustment value is greater than 0, the content reward is adjusted upwards. When the semantic similarity is greater than 0... If the content adjustment value is less than 0, the content reward will be adjusted downwards. It can take the value 0.5. The value is 0.4.

[0117] Finally, the content reward can be adjusted based on the content adjustment value to obtain the iteration reward for the current training operation of the first policy network. This adjustment avoids the first policy network recommending only one type of content, thus improving the diversity of the content recommended by the first policy network.

[0118] Through the above methods, the embodiments of this disclosure can dynamically adjust the reward mechanism during training operations, and can make personalized content recommendations based on the historical behavior and preferences of different users. At the same time, the recommendation strategy can be further optimized through reward adjustments, thereby improving the recommendation effect and user experience.

[0119] Optionally, the content reward for the current training operation of the second strategy network can be determined based on the recommendation value of the excluded candidate content. Determine the type of excluded candidate content and the type of historical recommended content for the current training operation, and determine the content adjustment value of the content reward for the current training operation of the second policy network; The content reward for the current training operation of the second policy network is adjusted based on the content adjustment value to obtain the training reward for the current training operation of the second policy network.

[0120] As an optional embodiment of this disclosure, the iterative reward of the first policy includes the training reward of each training operation of the first policy network, and the iterative reward of the second policy network includes the training reward of each training operation of the second policy network. The step of determining the comprehensive reward of the first policy network based on the iterative rewards corresponding to the first policy network and the second policy network includes: For each training operation, the comprehensive reward of the first policy network for the current training operation is determined based on the training reward of the first policy network for the current training operation and the training reward of the second policy network for the current training operation. The comprehensive reward of the first policy network is obtained based on the comprehensive reward of each training operation of the first policy network.

[0121] In this embodiment of the disclosure, during each iteration of training, the iteration reward of the first policy network may include the training reward of the first policy network during each training operation, and the iteration teaching of the second policy network may include the training reward of the second policy network during each training operation.

[0122] The training reward of the first policy network in the current training operation can be adjusted based on the training reward of the second policy network in the current training operation to obtain the comprehensive reward of the first policy network in the current training operation. For details on the adjustment method, please refer to the previous section on adjusting the iterative reward of the first policy network based on the iterative reward of the second policy network to obtain the comprehensive reward of the first policy network.

[0123] like Figure 6 As shown, Figure 6 This embodiment of the present disclosure provides a schematic diagram of a process for determining a comprehensive reward. The process involves inputting the current state corresponding to the current training operation into a first policy network and a second policy network to obtain recommended candidate content output by the first policy network and excluded candidate content output by the second policy network. Based on the value evaluation of the recommended candidate content, the content reward for the current training operation of the first policy network is obtained. Based on the value evaluation of the excluded candidate content, the content reward for the current training operation of the second policy network is obtained. The content reward for the current training operation of the first policy network is adjusted based on historical recommended content to obtain the training reward for the current training operation of the first policy network. The content reward for the current training operation of the second policy network is adjusted based on historical recommended content to obtain the training reward for the current training operation of the second policy network. Finally, the training reward for the current training operation of the first policy network is adjusted based on the training reward for the current training operation of the second policy network to obtain the comprehensive reward for the current training operation of the first policy network. Furthermore, the comprehensive reward for the current training operation of the second policy network can also be determined based on the difference between a preset reward value and the comprehensive reward for the current training operation of the first policy network.

[0124] Through the above methods, the embodiments of this disclosure can dynamically adjust the reward mechanism during training operations, and can make personalized content recommendations based on the historical behavior and preferences of different users. At the same time, the recommendation strategy can be further optimized through reward adjustments, thereby improving the recommendation effect and user experience.

[0125] In this embodiment of the disclosure, the combined reward of the first policy network in each training operation can be used as the combined reward of the first policy network. That is, the iterative reward of the first policy network, the iterative reward of the second policy network, and the combined reward of the first policy network can all be regarded as a reward set including the corresponding rewards in multiple training operations.

[0126] Optionally, the network parameters of the first policy network can be updated based on the comprehensive reward of the first policy network. This can be done by updating the network parameters of the first policy network based on the comprehensive reward of the first policy network during each training operation.

[0127] As an optional embodiment of this disclosure, the number of sample environments is multiple, and the network parameters of the first policy network are updated according to the comprehensive reward of the first policy network, including: The network parameters of the first policy network are updated based on the comprehensive reward of the first policy network corresponding to multiple sample environments; Each iteration of training also includes: Identify the first number of sample environments in which the overall reward of the first policy network is greater than the overall reward of the second policy network. If the ratio of the first quantity to the total number of sample environments is greater than a preset value, then the first policy network on which the current iteration of training is based will be added to the policy pool. The recommendation network includes each first policy network added to the policy pool when the iteration ends.

[0128] In this embodiment of the disclosure, the number of sample environments can be multiple, and the sample objects corresponding to different sample environments can be the same or different. Specifically, when updating the network parameters of the first policy network based on the comprehensive reward of the first policy network corresponding to multiple sample environments, the network parameters of the first policy network can be updated based on the comprehensive reward of the first policy network corresponding to each of the multiple sample environments during each training operation.

[0129] After determining the first reward corresponding to each sample environment under each training operation in this embodiment of the disclosure, the required data can be determined according to a pre-determined loss function, and the corresponding data can be acquired during the execution of the training operation. This data may include state, action, reward, and next state.

[0130] Optionally, for each sample environment, the current state, first reward, content representation of recommended candidate content, and next state corresponding to each training operation can be recorded as a quadruple. For a sample environment, the number of training operations corresponds to the number of quadruples. From the perspective of generation time, multiple quadruples constitute a trajectory corresponding to the sample environment. In this embodiment, the loss function can be calculated based on the trajectory corresponding to each sample environment. Specifically, a policy gradient-based algorithm can be used to optimize the policy of the first policy network. This policy gradient algorithm can be PPO, and the loss function of PPO includes policy loss (Actor Loss) and value function loss (Critic Loss). The training loss can be calculated by sampling the state, action, reward, and next state.

[0131] In this embodiment, the first policy network and the second policy network interact multiple times in the sample environment, which can be viewed as a game between the first policy network and the second policy network regarding the sample environment. If there are multiple sample environments, then the first policy network and the second policy network engage in multiple games.

[0132] Optionally, the win rate of the first strategy network can be determined based on the total number of games played between the first and second strategy networks, and the number of wins achieved by the first strategy network. When the win rate of the first strategy network exceeds a preset value, the first strategy network is stored in the strategy pool, i.e., its network parameters are stored. In practical applications, the first strategy network can be retrieved from the strategy pool as the recommendation network.

[0133] In each training iteration, which includes multiple games, the winning network for each game can be determined based on the combined rewards of the first policy network and the second policy network. The combined reward of the first policy network can be a weighted sum of the combined rewards of all training operations of the first policy network, and the combined reward of the second policy network can be a weighted sum of all training operations of the second policy network. The combined reward of each training operation of the second policy network can be the difference between a preset reward value and the combined reward of the first policy network for that training operation. In this case, the winning network for that game can be determined based on the magnitude of the combined rewards of the first and second policy networks.

[0134] Optionally, the overall reward of the first strategy network may include the overall reward of each training operation of the first strategy network, and the overall reward of the second strategy network may include the overall reward of each training operation of the second strategy network. In this case, the winning network of the game can be determined based on the magnitude of the overall reward of the first strategy network and the overall reward of the second strategy network for each training operation. For example, in a game consisting of 5 training operations, if the overall reward of the first strategy network in a training operation is greater than that of the second strategy network in a training operation 3 times, then the first strategy network is the winning network for that game. Optionally, the winning network of the game can also be determined based on the total overall reward of the first strategy network at each training operation and the total overall reward of the second strategy network at each training operation, or based on the magnitude of the overall reward of the first strategy network and the overall reward of the second strategy network in the last training operation.

[0135] If the win rate of the first policy network is greater than the preset value, it indicates that the first policy network in the current iteration of training has performed well and can recommend candidate content effectively. In this case, the first policy network used in the current training operation is added to the policy pool. This way, the high-quality first policy network will be retained for subsequent training and recommendation operations.

[0136] like Figure 7 As shown, Figure 7 A flowchart illustrating the determination of a winning network provided in this disclosure includes: S501. Set the sample environment as the current state for the first training operation; S502, Input the current state into the first policy network and the second policy network respectively; S503, Obtain recommended candidate content and excluded candidate content; S504. Based on the recommendation value of recommended candidate content and excluded candidate content, determine the first reward (the comprehensive reward of the current training operation of the first policy network) and the second reward (the comprehensive reward of the current training operation of the second policy network), as well as the next state; determine whether the termination conditions are met, where the termination conditions include the first termination condition and the second termination condition. The first termination condition is that the number of training operations is executed reaches a preset value, and the second termination condition is that the difference between the first reward and the second reward is greater than a preset value. If the termination conditions are met, further determine whether the first termination condition is met. If the termination conditions are not met, use the next state as the current state of the next training operation. If the first termination condition is met, execute S505. If the first termination condition is not met, execute S506. S505. Determine the combined reward of the first strategy network and the combined reward of the second strategy network. S506. If the first termination condition is met, the win rate network is determined based on the size of the first total reward and the second total reward. If the second termination condition is met, the strategy network with the larger reward is determined as the win rate network.

[0137] In this embodiment of the disclosure, by setting two termination conditions for performing training operations, the obvious superiority or inferiority of the first policy network and the second policy network can be seen when the second termination condition is met, thereby improving training efficiency by terminating the training operation earlier.

[0138] This embodiment of the disclosure can dynamically select the best-performing policy network and add it to the policy pool based on the performance of different policy networks during training. In this way, only superior networks are retained, avoiding the long-term existence of inferior policies and improving the stability and accuracy of the recommendation system. By pooling multiple policy networks, over-reliance on a single policy can be avoided, thereby improving the diversity and robustness of the recommendation system and enabling it to adapt to different users and environments.

[0139] When the training termination condition is met, all the first policy networks added to the policy pool can be used as recommendation networks and sequentially deployed to the actual recommendation task. In this way, multiple policy networks in the policy pool will be used for subsequent recommendation tasks, ensuring that the recommended content is based on the results of high-quality policy networks.

[0140] As an optional embodiment of this disclosure, when the first policy network is being trained for the first time, the second policy network is determined in the following manner: Obtain the multiple opponent policy networks corresponding to the first policy network; Obtain multiple test environments; For each test environment, the test environment is input into each opponent policy network, and the recommended candidate content is obtained by predicting the test environment through each opponent policy network; For each adversary policy network, the performance of the adversary policy network is determined by predicting the recommended value of the excluded candidate content for each test environment. The opponent's policy network with the highest performance among all opponent policy networks is selected as the second policy network.

[0141] In embodiments of this disclosure, when the first policy network is trained for the first time, multiple adversary policy networks corresponding to the first policy network can be obtained first. These adversary policy networks may be previously trained or be policy networks that have been trained to a certain extent and obtained from external sources.

[0142] The adversarial policy network is opposed to the first policy network. The adversarial policy network is used to predict content that the user is not interested in, while the first policy network is used to predict content that the user is interested in. The adversarial policy network can be used to conduct adversarial training against the first policy network to help improve the recommendation ability of the first policy network.

[0143] Specifically, multiple test environments can be prepared to simulate different real-world application scenarios, thereby helping to evaluate the performance of each adversary policy network. For each test environment, by inputting it into each adversary policy network, the exclusion candidate content of each adversary policy network in each environment is obtained. Thus, the performance can be determined based on the recommendation value of the exclusion candidate content predicted by each adversary policy network in different test environments.

[0144] like Figure 8 As shown, Figure 8 This diagram illustrates a method for determining a second policy network according to an embodiment of this disclosure. Performance testing of the adversary policy network can be performed using multiple test environments. Specifically, the performance of the adversary policy network can be evaluated based on the recommendation value of the excluded candidate content output by each adversary policy network. In the diagram, since adversary policy network 3 has the highest performance, it is selected as the second policy network.

[0145] In this embodiment, by evaluating the performance of each adversary policy network, the adversary policy network with the best performance is selected and used as the second policy network. The second policy network is then used for adversarial training against the first policy network, helping the first policy network to better learn the recommended policy.

[0146] In this embodiment, through game-playing with multiple adversary policy networks, the first policy network can progressively optimize the selection of recommended content during adversarial training, effectively avoiding overfitting and improving the generalization ability of the recommendation system. By selecting the best-performing adversary policy network as the second policy network, the system can effectively improve its ability to identify low-value content and reduce the generation of invalid or low-quality recommended content. By introducing multiple adversary policy networks and evaluating their performance, the system can select among diverse policies, thereby enhancing the adaptability and robustness of the recommendation strategy and improving the overall performance and user experience of the system.

[0147] As an optional embodiment of this disclosure, for each adversary policy network, determining the performance of the adversary policy network based on the predicted recommendation value of the recommended candidate content corresponding to each test environment includes: For each test environment, the test environment is input into the first policy network, and the first policy network is used to predict the test environment to obtain recommended candidate content. For each test environment, the total reward of the opponent policy network and the total reward of the first policy network are determined based on the recommendation value of the excluded candidate content predicted by the opponent policy network and the recommendation value of the recommended candidate content predicted by the first policy network. A second number of initial environments are identified in which the total reward of the first policy network corresponding to the multiple test environments is greater than the total reward of the second policy network. The performance of the adversary policy network is determined based on the ratio of the second number to the total number of test environments, wherein the ratio is positively correlated with the performance.

[0148] In embodiments of this disclosure, the performance of each adversary policy network can be evaluated to determine its effectiveness in adversarial training against the first policy network. Specifically, for each adversary policy network, it can be subjected to adversarial training against the first policy network to determine the performance of that adversary policy network.

[0149] In this embodiment of the disclosure, similar to the sample environment, for each test environment, the first policy network and the second policy network can interact with the test sample environment multiple times. The total reward of the first policy network can be the comprehensive reward of the first policy network with respect to the test environment during the interaction process, and the total reward of the second policy network can be the comprehensive reward of the second policy network with respect to the test environment. Specifically, the method for determining the winning network corresponding to the test environment can refer to the method for determining the win rate network corresponding to the sample environment. like Figure 9 As shown, Figure 9 This is a flowchart illustrating a method for determining the performance of an adversary policy network according to an embodiment of this disclosure. Taking adversary policy network 1 as an example, this disclosure can obtain a winning network for each test environment by training a first policy network and the adversary policy network on a test environment. Then, the performance of adversary policy network 1 is determined based on its win rate relative to the first policy network. Therefore, embodiments of this disclosure can use the win rate of the adversary policy network relative to the first policy network to measure the performance of the adversary policy network.

[0150] The performance of the adversary policy network is further evaluated by calculating the ratio of the second number to the total number of test environments. A higher ratio indicates that the adversary policy network performs poorly in a larger proportion of environments, while the first policy network performs better. A smaller ratio indicates that the adversary policy network performs worse than the first policy network in more test environments, and its performance is poor. Conversely, a larger ratio indicates that the adversary policy network performs better.

[0151] In this embodiment of the disclosure, by evaluating multiple test environments, the performance of each adversary policy network under different scenarios can be comprehensively analyzed and its performance quantified, so as to select the adversary policy network with better performance as the second policy network.

[0152] Optionally, for each opponent policy network, the ratio of the second quantity to the total quantity in the test environment is used as the win rate between the opponent policy network and the first policy network. The win rate is recorded in the win rate policy table so as to determine the corresponding second policy network when the first policy network is trained next time.

[0153] In this embodiment, the performance of multiple adversary policy networks is evaluated, and the best-performing adversary policy network is selected as the second policy network based on its performance in different testing environments. This process includes comparing the win rates of multiple adversary policy networks to ensure that the most challenging policy network is selected for adversarial training, thereby improving the learning effect of the first policy network. The adversary policy network continuously challenges the recommender policy network, forcing the recommender policy network to find more effective strategies to cope with various user feedback during training. This increases the adaptability of the recommendation system to dynamic changes in user interests, resulting in a robust model training process.

[0154] In evaluating multiple opponent policy networks, the win rate of each network can be calculated based on its performance across multiple test environments. Specifically, for each opponent policy network, the system calculates the ratio of the second network to the total number of networks in the test environments, and selects the opponent policy network with the largest ratio as the second policy network.

[0155] To further optimize the selection of the second policy network in the next training iteration, the win rate between each opponent policy network and the first policy network is recorded in a win rate strategy table. This win rate refers to the win rate of each opponent policy network relative to the first policy network in the testing environment, i.e., the ratio. The win rate strategy table is a table recording the win rate between each opponent policy network and the first policy network. By recording the win rate of each opponent policy network, the most suitable opponent policy network can be dynamically selected for training based on historical data during future training processes.

[0156] Specifically, the win rate of each opponent policy network is recorded in the win rate policy table, and this data is used to determine which opponent policy network should be used as the second policy network to compete against the first policy network in the next training session.

[0157] When conducting the next training session, a second policy network can be selected to compete against the first policy network based on the records in the win rate policy table.

[0158] The strongest adversary policy network in historical training can be selected to ensure that the second policy network provides an effective challenge, thereby enhancing the learning ability of the first policy network. In this case, the selection of the second policy network is based on its win rate performance in historical evaluations, further enhancing the effectiveness of adversarial training, avoiding random selection, and ensuring the targeted and efficient nature of the training.

[0159] In this embodiment, by selecting the opponent's policy network that performs best against the first policy network for training, the challenging nature of adversarial training is ensured, thereby accelerating the learning process of the first policy network. By dynamically recording the win rate of the opponent's policy network, the most suitable second policy network can be selected more intelligently, allowing each training session to be optimized based on historical experience, thus improving training efficiency and effectiveness. By continuously selecting different opponent policy networks for adversarial training against the first policy network, the adaptability and robustness of the recommendation system can be enhanced, ensuring that it can provide high-quality recommendation results in complex and ever-changing real-world environments.

[0160] The following is a complete embodiment of the model training method provided in this application, such as... Figure 10 As shown, Figure 10 A flowchart illustrating a training method for a recommendation network provided in this embodiment includes: S601. Obtain multiple sample environments, each of which includes the object features of a sample object and the content features of multiple candidate contents. S602. Based on the multiple sample environments, the policy network to be trained is continuously trained until the training termination condition is met. The policy network to be trained includes a first policy network. Each first policy network added to the policy pool when the training termination condition is met is used as a recommendation network. S602 includes: S701. Perform at least one training operation for each of the sample environments; S702. Based on the first reward of each training operation corresponding to each sample environment, determine the training loss of the first policy network, and adjust the network parameters of the first policy network according to the training loss to obtain the first policy network on which the next training operation is based. The first reward is the comprehensive reward for each training operation of the first policy network; S703. Based on the second reward of each training operation corresponding to each sample environment, determine the training loss of the second policy network, and adjust the network parameters of the second policy network according to the training loss to obtain the second policy network on which the next training operation is based. The second reward is the combined reward for each training operation of the second policy network. S704. Determine the first number of initial environments in which the first total reward is greater than the second total reward. If the ratio of the first number to the total number of initial environments is greater than a preset value, then add the first policy network on which the current training operation is based to the policy pool. The first total reward is the sum of the first rewards corresponding to each training operation in this sample environment, and the second total reward is the sum of the second rewards corresponding to each training operation in this sample environment. S701 includes: S801. Input the current state of the current operation into the first policy network and the second policy network respectively. Predict recommended candidate content from multiple candidate content through the first policy network and predict excluded candidate content from multiple candidate content through the second policy network. S802. Obtain recommendation feedback information of the recommended candidate content, and determine the recommendation value of the recommended candidate content based on the recommendation feedback information of the recommended candidate content; S803. Obtain recommendation feedback information for excluded candidate content, add recommendation feedback information for each excluded candidate content, and determine the recommendation value of the excluded candidate content. S804. The recommendation value of the recommended candidate content is used as the third reward (iteration reward) of the first policy network in the current training operation; the difference between the preset reward value and the recommendation value of the excluded candidate content is used as the fourth reward (iteration reward) of the second policy network in the current operation. S805. Based on the fourth reward of the current operation of the second strategy network, determine the reward adjustment value, and adjust the third reward based on the first adjustment value to obtain the content reward; S806. Determine the type of recommended candidate content and the type of historical recommended content for the current operation, and determine the reward adjustment value; S807. Adjust the reward for this content based on the reward adjustment value to obtain the first reward for the current operation; S808. Determine the second reward based on the difference between the total reward and the first reward; S809. Remove the content features of recommended candidate content and excluded candidate content from the current state to obtain the next state; S810. Determine whether the termination condition is met. If the termination condition is met, stop the training operation. If the termination condition is not met, use the next state as the current state for the next training operation and continue to execute S801. The termination condition includes a first termination condition and a second termination condition. The first termination condition is reaching the preset number of training operations, and the second termination condition is that the difference between the first reward and the second reward is greater than a preset value.

[0161] like Figure 11 As shown in the embodiments of this disclosure, a content recommendation method is also provided, the method comprising: S901. Obtain the current environment, including the object characteristics of the target object and the content characteristics of the candidate content.

[0162] In this embodiment of the disclosure, the target object can send a recommendation request to the content recommendation server via a terminal. After receiving the request, the content recommendation server can determine the corresponding candidate content based on the request. For example, the request may include keywords, which can be words entered by the target object on the terminal. After determining the candidate content, feature extraction can be performed on the relevant data of the target object and the relevant data of the candidate content to obtain the current environment of the target object. Optionally, pre-extracted object features of the target object and content features of the candidate content can be obtained from a feature database.

[0163] S902. Input the current environment into the recommendation network to obtain the recommended candidate content output by the recommendation network, and recommend the recommended candidate content to the target object.

[0164] The recommendation network is obtained using the aforementioned training method for recommendation networks.

[0165] The recommendation network can be trained using the training method of the recommendation network provided in any embodiment of this application. The specific model training process can be found in the above content, and will not be repeated here.

[0166] like Figure 12 As shown in the embodiments of this disclosure, a method for training a policy network is also provided, the method comprising: S1001. Obtain the sample environment, including the object features of the sample object and the content features of the candidate content; S1002. Based on the sample environment, iteratively train the first policy network and the second policy network to obtain the iterative rewards corresponding to the first policy network and the second policy network at the current iteration training. The first strategy network predicts recommended candidate content from the candidate content when recommending to the sample object; the second strategy network predicts excluded candidate content from the candidate content when recommending to the sample object. S1003. Determine the comprehensive reward of the second policy network based on the iterative rewards corresponding to the first policy network and the second policy network respectively; S1004. Update the network parameters of the second policy network according to the comprehensive reward of the second policy network until the iteration termination condition is met, and use the trained second policy network as the filtering network.

[0167] In this embodiment, a method for training a second policy network, which operates on the same principle as training a first policy network, is provided. This trained second policy network is used as a filtering network. When recommending content to users, content that is not of interest to the user can be filtered out based on this filtering network. Specifically, the method may include the following steps: Obtain the current environment, including the object characteristics of the target object and the content characteristics of the candidate content; The current environment is input into the filtering network, and the filtered network outputs excluded candidate content. The candidate content that is not excluded from the candidate content is recommended to the target object.

[0168] In this embodiment of the disclosure, a filtering network and a recommendation network can be combined to recommend content to users. Optionally, the current environment can be filtered out to exclude candidate content through the filtering network. Then, the content features of the candidate content other than the excluded candidate content and the object features of the target object are used as the updated environment. The updated environment is then input into the recommendation network to obtain recommended candidate content to recommend to the target object.

[0169] Based on the same training logic as the recommendation network, the following optional methods are also included during the training of the filtering network.

[0170] As an optional embodiment, the method further includes: Obtain a preset reward value, and determine the comprehensive reward of the first strategy network based on the reward value and the comprehensive reward of the second strategy network; The network parameters of the first policy network are updated based on the comprehensive reward of the first policy network to obtain the first policy network for the next iteration of training.

[0171] As an optional implementation, for each training iteration, the iteration rewards corresponding to the second policy network and the first policy network at the current iteration are obtained in the following way: Perform at least one of the following training operations on the sample environment, and obtain the iterative reward of the second policy network based on the training reward of each training operation of the second policy network; obtain the iterative reward of the first policy network based on the training reward of each training operation of the first policy network: The current state of the current training operation is input into the second policy network and the first policy network, respectively. The second policy network predicts recommended candidate content from the candidate content, and the first policy network predicts excluded candidate content from the candidate content. Based on the recommendation value of the recommended candidate content, the training reward of the second policy network for the current training operation is determined, and based on the recommendation value of the excluded candidate content, the training reward of the first policy network for the current training operation is determined. The content features of the recommended candidate content and the content features of the excluded candidate content are deleted from the current state to obtain the current state of the next training operation. The current state of the first training operation for each sample environment is the sample environment.

[0172] As an optional embodiment, determining the training reward for the current training operation of the second policy network based on the recommendation value of the recommended candidate content includes: Based on the recommendation value of the candidate content, the content reward for the current training operation of the second policy network is determined; Determine the type of recommended candidate content for the current training operation and the type of historical recommended content, and determine the content adjustment value of the content reward for the current training operation of the second policy network; wherein, the historical recommended content includes the recommended candidate content of each training operation before the current training operation; The content reward for the current training operation of the second policy network is adjusted based on the content adjustment value to obtain the training reward for the current training operation of the second policy network.

[0173] As an optional embodiment, determining the first reward for the current operation of the second strategy network based on the recommendation value of the recommended candidate content and the recommendation value of the excluded candidate content includes: Based on the recommendation value of the candidate content, the content reward for the current training operation of the second policy network is determined; For each training operation, the reward adjustment value of the content reward of the second policy network for the current training operation is determined based on the training reward of the first policy network for the current training operation. Based on the reward adjustment value, the content reward of the current training operation of the second policy network is adjusted, and the training reward of the current training operation of the second policy network is determined based on the adjusted content reward.

[0174] As an optional embodiment, the iterative reward of the first policy includes the training reward of each training operation of the second policy network, and the iterative reward of the first policy network includes the training reward of each training operation of the first policy network. The comprehensive reward of the second policy network is determined based on the iterative rewards corresponding to the second policy network and the first policy network, including: For each training operation, the comprehensive reward of the second policy network for the current training operation is determined based on the training reward of the second policy network for the current training operation and the training reward of the first policy network for the current training operation. The comprehensive reward of the second policy network is obtained based on the comprehensive reward of each training operation of the second policy network.

[0175] As an optional embodiment, the number of sample environments is multiple, and the network parameters of the second policy network are updated according to the comprehensive reward of the second policy network, including: The network parameters of the second policy network are updated based on the comprehensive reward of the second policy network corresponding to the multiple sample environments. Each iteration of training also includes: Identify the first number of sample environments in which the comprehensive reward of the second policy network is greater than the comprehensive reward of the first policy network. If the ratio of the first quantity to the total number of sample environments is greater than a preset value, then the second policy network on which the current iteration of training is based is added to the policy pool; The recommendation network includes each second policy network added to the policy pool when the iteration termination condition is met.

[0176] As an optional embodiment, when the second policy network is being trained for the first time, the first policy network is determined in the following manner: Obtain multiple recommender policy networks corresponding to the second policy network; Obtain multiple test environments; For each test environment, the test environment is input into each recommender policy network, and the recommended candidate content is obtained by predicting the test environment through each recommender policy network; For each recommender policy network, the performance of the recommender policy network is determined by predicting the recommendation value of the excluded candidate content for each test environment. The recommender policy network with the highest performance among all recommender policy networks is designated as the first policy network.

[0177] As an optional embodiment, for each recommender policy network, the performance of the recommender policy network is determined based on the predicted recommendation value of the candidate content corresponding to each test environment, including: For each test environment, the test environment is input into the second policy network, and the second policy network is used to predict the test environment to obtain recommended candidate content; For each test environment, the total reward of the recommender strategy network and the total reward of the second strategy network are determined based on the recommendation value of the excluded candidate content predicted by the recommender strategy network and the recommendation value of the recommended candidate content predicted by the second strategy network. A second number of initial environments are identified in which the total reward of the second policy network in the multiple test environments is greater than the total reward of the first policy network. The performance of the recommender policy network is determined based on the ratio of the second number to the total number of test environments, wherein the ratio is positively correlated with the performance.

[0178] like Figure 13 As shown, Figure 13This is a schematic diagram of a training system structure provided in an embodiment of this disclosure. In this system structure, the first policy network and the second policy network can be trained independently. For the first policy network, based on the win rate of the first policy network corresponding to other opponent policy networks, the opponent policy network with the lowest win rate of the first policy network can be obtained from the opponent policy pool as the second policy network. Then, by performing multiple target operations on the sample environment, the winning network corresponding to each sample environment is determined, and the win rate of the first policy network and the second policy network in this operation is determined. If the win rate of the first policy network is greater than a preset threshold, the first policy network is placed into the recommender policy pool. The sampling result is obtained through target operations, and the sampling result is the first reward corresponding to the first policy network in each sample environment. The training loss of the first policy network is determined according to the sampling result, and the first policy network is adjusted according to the training loss. It is then determined whether the training termination condition is met. Here, the training termination condition is whether the number of iterations has been reached. If the training termination condition is met, the training of the first policy network is terminated. If the training termination condition is not met, the performance of the first policy network and the parameter-tuned first policy network can be compared, and the policy network with better performance can be used as the next first policy network.

[0179] By selecting a different second policy network each time during the training of the first policy network, and without training the second policy network during the training process itself, the efficiency of training the first policy network is improved. The training process of the second policy network is the same as that of the first policy network. That is, during the training of the second policy network, a recommender policy network is determined from the recommender policy pool based on the win rate each time, which serves as the first policy network. The specific training method is the same as that of the first policy network, and will not be repeated here.

[0180] like Figure 14 As shown, Figure 14 This is a schematic diagram of an application scenario provided by an embodiment of this disclosure. In this application scenario, a user can represent a user group. Users can use different terminals to browse content recommended by the application. Taking the application as a video application as an example, a user clicking the search button on the terminal or directly browsing the content pushed by the application can trigger a request to the content recommendation server to recommend video content. The content recommendation server can request a policy network from the policy pool management server. The policy pool management server can provide the content recommendation server with the highest-performing policy network from the recommender policy pool. As the recommendation network, the content recommendation server also needs to request the current environment from the environment simulation server to obtain the current environment, and then make predictions based on the recommendation network to obtain recommended candidate content.

[0181] The policy pool management server includes recommender policies and adversary policies. The recommender policy predicts content the user is interested in based on the environment, while the adversary policy predicts content the user is not interested in based on the environment. The recommender and adversary policy pools actually store the network parameters of the policy networks. All recommender policy networks in the recommender policy pool are based on the same deep learning network, differing only in their network parameters. Similarly, all adversary policy networks in the adversary policy pool can also be based on the same deep learning network, again differing only in their network parameters. Furthermore, the recommender and adversary policy networks can also be based on the same deep learning network. The policy pool management server can include an agent for managing the recommender policy pool (called the recommender agent) and an agent for managing the adversary policy pool (called the adversary agent).

[0182] In this embodiment of the disclosure, the environment simulation server is used to generate sample environments, the user's current environment, and, if performance testing is required, a test environment. The process of generating the environment requires the joint participation of a database, a data acquisition server, and a feature extraction server. The database is used to store content uploaded by content creators and relevant user data. The data acquisition server can collect data based on the data stored in the database to obtain the data required for generating the environment. The feature extraction server further extracts features from the data collected by the data acquisition server so that the environment simulation server can generate the corresponding environment based on the user's characteristics and the characteristics of the content.

[0183] This disclosure provides a training apparatus for a recommendation network, such as... Figure 15 As shown, the training device for the recommendation network may include: a first acquisition module 601, a first training module 602, a first reward determination module 603, and a first network parameter update module 604, wherein, The first acquisition module 601 is used to acquire the sample environment, including the object features of the sample object and the content features of the candidate content; The first training module 602 is used to iteratively train the first policy network and the second policy network based on the sample environment to obtain the iteration reward corresponding to the first policy network and the second policy network at the current iteration training; the first policy network predicts the recommended candidate content when recommending to the sample object from the candidate content; the second policy network predicts the excluded candidate content when recommending to the sample object from the candidate content. The first reward determination module 603 is used to determine the comprehensive reward of the first policy network based on the iterative rewards corresponding to the first policy network and the second policy network respectively. The first network parameter update module 604 is used to update the network parameters of the first policy network according to the comprehensive reward of the first policy network until the iteration end condition is met, and the trained first policy network is used as the recommendation network.

[0184] As an optional embodiment, the first reward determination module is further configured to: Obtain a preset reward value, and determine the comprehensive reward of the second strategy network based on the reward value and the comprehensive reward of the first strategy network; The first network parameter update module is also used for: The network parameters of the second policy network are updated based on the comprehensive reward of the second policy network to obtain the second policy network for the next iteration of training.

[0185] As an optional implementation, for each training iteration, the iteration rewards corresponding to the first policy network and the second policy network at the current iteration are obtained in the following way: Perform at least one training operation on the sample environment, and obtain the iterative reward of the first policy network based on the training reward of each training operation of the first policy network, and obtain the iterative reward of the second policy network based on the training reward of each training operation of the second policy network: The current state of the current training operation is input into the first policy network and the second policy network, respectively. The first policy network predicts recommended candidate content from the candidate content, and the second policy network predicts excluded candidate content from the candidate content. Based on the recommendation value of the recommended candidate content, the training reward of the first policy network for the current training operation is determined, and based on the recommendation value of the excluded candidate content, the training reward of the second policy network for the current training operation is determined. The content features of the recommended candidate content and the content features of the excluded candidate content are deleted from the current state to obtain the current state of the next training operation. The current state of the first training operation of each sample environment is the sample environment.

[0186] As an optional embodiment, when the first reward determination module determines the training reward of the first policy network for the current training operation based on the recommendation value of the recommended candidate content, it is specifically used for: Based on the recommendation value of the recommended candidate content, the content reward for the current training operation of the first policy network is determined; The types of recommended candidate content and historical recommended content for the current training operation are determined, and the content adjustment value of the content reward for the current training operation of the first policy network is determined; wherein, the historical recommended content includes the recommended candidate content of each training operation before the current training operation; The content reward for the current training operation of the first policy network is adjusted based on the content adjustment value to obtain the training reward for the current training operation of the first policy network.

[0187] As an optional embodiment, when the first reward determination module determines the first reward for the current operation of the first policy network based on the recommendation value of the recommended candidate content and the recommendation value of the excluded candidate content, it is specifically used for: Based on the recommendation value of the recommended candidate content, the content reward for the current training operation of the first policy network is determined; For each training operation, the reward adjustment value of the content reward of the first policy network for the current training operation is determined based on the training reward of the second policy network for the current training operation. The content reward of the current training operation of the first policy network is adjusted based on the reward adjustment value, and the training reward of the current training operation of the first policy network is determined based on the adjusted content reward.

[0188] As an optional embodiment, the iterative reward of the first policy includes the training reward of each training operation of the first policy network, and the iterative reward of the second policy network includes the training reward of each training operation of the second policy network. The first reward determination module determines the comprehensive reward of the first policy network based on the iterative rewards corresponding to the first policy network and the second policy network, specifically for: For each training operation, the comprehensive reward of the first policy network for the current training operation is determined based on the training reward of the first policy network for the current training operation and the training reward of the second policy network for the current training operation. The comprehensive reward of the first policy network is obtained based on the comprehensive reward of each training operation of the first policy network.

[0189] As an optional embodiment, the number of sample environments is multiple. When the first network parameter update module updates the network parameters of the first policy network according to the comprehensive reward of the first policy network, it is specifically used for: The network parameters of the first policy network are updated based on the comprehensive reward of the first policy network corresponding to the multiple sample environments. Each iteration of training also includes: Identify the first number of sample environments in which the overall reward of the first policy network is greater than the overall reward of the second policy network. If the ratio of the first quantity to the total number of sample environments is greater than a preset value, then the first policy network on which the current iteration of training is based is added to the policy pool; The recommendation network includes each first policy network added to the policy pool when the iteration termination condition is met.

[0190] As an optional embodiment, when the first policy network is being trained for the first time, the second policy network is determined in the following manner: Obtain the multiple opponent policy networks corresponding to the first policy network; Obtain multiple test environments; For each test environment, the test environment is input into each opponent policy network, and the recommended candidate content is obtained by predicting the test environment through each opponent policy network; For each adversary policy network, the performance of the adversary policy network is determined by predicting the recommended value of the excluded candidate content for each test environment. The opponent's policy network with the highest performance among all opponent policy networks is selected as the second policy network.

[0191] As an optional embodiment, for each adversary policy network, determining the performance of the adversary policy network by predicting the recommendation value of the recommended candidate content corresponding to each test environment includes: For each test environment, the test environment is input into the first policy network, and the first policy network is used to predict the test environment to obtain recommended candidate content. For each test environment, the total reward of the opponent policy network and the total reward of the first policy network are determined based on the recommendation value of the excluded candidate content predicted by the opponent policy network and the recommendation value of the recommended candidate content predicted by the first policy network. A second number of initial environments are identified in which the total reward of the first policy network corresponding to the multiple test environments is greater than the total reward of the second policy network. The performance of the adversary policy network is determined based on the ratio of the second number to the total number of test environments, wherein the ratio is positively correlated with the performance.

[0192] This disclosure provides a content recommendation device, such as... Figure 16 As shown, the content recommendation device may include: a second acquisition module 701 and a recommendation module 702, wherein, The second acquisition module 701 is used to acquire the current environment, including the object characteristics of the target object and the content characteristics of the candidate content; The recommendation module 702 is used to input the current environment into the recommendation network, obtain the recommendation candidate content output by the recommendation network, and recommend the recommendation candidate content to the target object; wherein, the recommendation network is trained using the aforementioned recommendation network training method.

[0193] This disclosure provides a method for training a policy network, such as... Figure 17 As shown, the content recommendation device may include: a third acquisition module 801 and a second training module 802, a second reward determination module 803 and a second network parameter update module 804, wherein, The third acquisition module 801 is used to acquire the sample environment, including the object features of the sample object and the content features of the candidate content; The second training module 802 is used to iteratively train the first policy network and the second policy network based on the sample environment to obtain the iterative rewards corresponding to the first policy network and the second policy network at the current iteration training time; the first policy network predicts the recommended candidate content when recommending to the sample object from the candidate content; the second policy network predicts the excluded candidate content when recommending to the sample object from the candidate content. The second reward determination module 803 is used to determine the comprehensive reward of the second policy network based on the iterative rewards corresponding to the first policy network and the second policy network respectively. The second network parameter update module 804 is used to update the network parameters of the first policy network according to the comprehensive reward of the second policy network until the iteration termination condition is met, and the trained second policy network is used as the filtering network.

[0194] The apparatus of this disclosure embodiment can execute the method provided in this disclosure embodiment, and its implementation principle is similar, and it has corresponding technical effects. The actions performed by each module in the apparatus of each embodiment of this disclosure correspond to the steps in the method of each embodiment of this disclosure. For a detailed functional description of each module of the apparatus, please refer to the description in the corresponding method shown above, and it will not be repeated here.

[0195] The apparatus of this disclosure embodiment can execute the method provided in this disclosure embodiment, and its implementation principle is similar, and it has corresponding technical effects. The actions performed by each module in the apparatus of each embodiment of this disclosure correspond to the steps in the method of each embodiment of this disclosure. For a detailed functional description of each module of the apparatus, please refer to the description in the corresponding method shown above, and it will not be repeated here.

[0196] This disclosure provides an electronic device including a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps of the method provided in any optional embodiment of this disclosure. Compared with existing technologies, this method achieves the following: By iteratively training the first and second policy networks based on the sample environment, the recommended candidate content predicted by the first policy network when recommending candidate content to the sample object is obtained in each iteration, and the excluded candidate content predicted by the second policy network from the candidate content when recommending candidate content to the sample object is obtained. This allows the predictions of the first policy network to be regarded as actions executed by the first policy network, and the predictions of the second policy network to be regarded as actions executed by the second policy network. The rewards of the first and second policy networks in the game are obtained. Based on the iterative rewards of the first and second policy networks, a comprehensive reward of the first policy network is generated. This comprehensive reward of the first policy network includes not only the reward obtained when recommending candidate content output by the first policy network, but also the reward obtained when excluding candidate content output by the second policy network. The network parameters of the first policy network are updated through the comprehensive reward of the first policy network, enabling the first policy network to be trained under the influence of the actions executed by the second policy network, and gradually improving its recommendation level during the training process.

[0197] In one alternative embodiment, an electronic device is provided, such as Figure 18 As shown, Figure 18 The illustrated electronic device 4000 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of this disclosure.

[0198] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with this disclosure. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0199] Bus 4002 may include a pathway for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 18 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0200] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium capable of carrying or storing computer programs and capable of being read by a computer, without limitation herein.

[0201] The memory 4003 is used to store computer programs that execute embodiments of the present disclosure, and is controlled by the processor 4001 to execute them. The processor 4001 is used to execute the computer programs stored in the memory 4003 to implement the steps shown in the foregoing method embodiments.

[0202] Among them, electronic devices may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (such as in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 18 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0203] This disclosure provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps and corresponding content of the aforementioned method embodiments.

[0204] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium, a computer-readable medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0205] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, can implement the steps and corresponding content of the aforementioned method embodiments.

[0206] The terms “first,” “second,” “third,” “fourth,” “1,” “2,” etc. (if present) in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in a sequence other than that shown in the figures or text.

Claims

1. A method for training a recommendation network, characterized in that, The method includes: Obtain the sample environment, including the object features of the sample objects and the content features of the candidate content; Based on the sample environment, the first policy network and the second policy network are iteratively trained to obtain the iterative rewards corresponding to the first policy network and the second policy network at the current iteration training time; the first policy network predicts the recommended candidate content when recommending to the sample object from the candidate content; the second policy network predicts the excluded candidate content when recommending to the sample object from the candidate content. The comprehensive reward of the first policy network is determined based on the iterative rewards corresponding to the first policy network and the second policy network, respectively. The network parameters of the first policy network are updated based on the comprehensive reward of the first policy network until the iteration termination condition is met, and the trained first policy network is used as the recommendation network.

2. The method according to claim 1, characterized in that, The method further includes: Obtain a preset reward value, and determine the comprehensive reward of the second strategy network based on the reward value and the comprehensive reward of the first strategy network; The network parameters of the second policy network are updated based on the comprehensive reward of the second policy network to obtain the second policy network for the next iteration of training.

3. The method according to claim 1, characterized in that, For each training iteration, the iteration rewards for the first and second policy networks at the current iteration are obtained in the following way: Perform at least one training operation on the sample environment, and obtain the iterative reward of the first policy network based on the training reward of each training operation of the first policy network, and obtain the iterative reward of the second policy network based on the training reward of each training operation of the second policy network: The current state of the current training operation is input into the first policy network and the second policy network, respectively. The first policy network predicts recommended candidate content from the candidate content, and the second policy network predicts excluded candidate content from the candidate content. Based on the recommendation value of the recommended candidate content, the training reward of the first policy network for the current training operation is determined, and based on the recommendation value of the excluded candidate content, the training reward of the second policy network for the current training operation is determined. The content features of the recommended candidate content and the content features of the excluded candidate content are deleted from the current state to obtain the current state of the next training operation. The current state of the first training operation of each sample environment is the sample environment.

4. The method according to claim 3, characterized in that, The step of determining the training reward for the current training operation of the first policy network based on the recommendation value of the recommended candidate content includes: Based on the recommendation value of the recommended candidate content, the content reward for the current training operation of the first policy network is determined; The types of recommended candidate content and historical recommended content for the current training operation are determined, and the content adjustment value of the content reward for the current training operation of the first policy network is determined; wherein, the historical recommended content includes the recommended candidate content of each training operation before the current training operation; The content reward for the current training operation of the first policy network is adjusted based on the content adjustment value to obtain the training reward for the current training operation of the first policy network.

5. The method according to claim 3 or 4, characterized in that, The step of determining the first reward for the current operation of the first policy network based on the recommendation value of the recommended candidate content and the recommendation value of the excluded candidate content includes: Based on the recommendation value of the recommended candidate content, the content reward for the current training operation of the first policy network is determined; For each training operation, the reward adjustment value of the content reward of the first policy network for the current training operation is determined based on the training reward of the second policy network for the current training operation. The content reward of the current training operation of the first policy network is adjusted based on the reward adjustment value, and the training reward of the current training operation of the first policy network is determined based on the adjusted content reward.

6. The method according to any one of claims 3-5, characterized in that, The iterative reward of the first strategy includes the training reward of each training operation of the first strategy network, and the iterative reward of the second strategy network includes the training reward of each training operation of the second strategy network. The step of determining the comprehensive reward of the first policy network based on the iterative rewards corresponding to the first policy network and the second policy network includes: For each training operation, the comprehensive reward of the first policy network for the current training operation is determined based on the training reward of the first policy network for the current training operation and the training reward of the second policy network for the current training operation. The comprehensive reward of the first policy network is obtained based on the comprehensive reward of each training operation of the first policy network.

7. The method according to claim 2, characterized in that, The number of sample environments is multiple, and the step of updating the network parameters of the first policy network based on the comprehensive reward of the first policy network includes: The network parameters of the first policy network are updated based on the comprehensive reward of the first policy network corresponding to multiple sample environments; Each iteration of training also includes: Identify the first number of sample environments in which the overall reward of the first policy network is greater than the overall reward of the second policy network. If the ratio of the first quantity to the total number of sample environments is greater than a preset value, then the first policy network on which the current iteration of training is based is added to the policy pool; The recommendation network includes each first policy network added to the policy pool when the iteration termination condition is met.

8. The method according to claim 1, characterized in that, When the first policy network is being trained for the first time, the second policy network is determined in the following way: Obtain the multiple opponent policy networks corresponding to the first policy network; Obtain multiple test environments; For each test environment, the test environment is input into each opponent policy network, and the recommended candidate content is obtained by predicting the test environment through each opponent policy network; For each adversary policy network, the performance of the adversary policy network is determined by predicting the recommended value of the excluded candidate content for each test environment. The opponent's policy network with the highest performance among all opponent policy networks is selected as the second policy network.

9. The method according to claim 8, characterized in that, For each adversary policy network, determining the performance of the adversary policy network by predicting the recommendation value of the recommended candidate content corresponding to each test environment includes: For each test environment, the test environment is input into the first policy network, and the first policy network is used to predict the test environment to obtain recommended candidate content. For each test environment, the total reward of the opponent policy network and the total reward of the first policy network are determined based on the recommendation value of the excluded candidate content predicted by the opponent policy network and the recommendation value of the recommended candidate content predicted by the first policy network. A second number of initial environments are identified in which the total reward of the first policy network corresponding to the multiple test environments is greater than the total reward of the second policy network. The performance of the adversary policy network is determined based on the ratio of the second number to the total number of test environments, wherein the ratio is positively correlated with the performance.

10. A content recommendation method, characterized in that, The method includes: Obtain the current environment, including the object characteristics of the target object and the content characteristics of the candidate content; The current environment is input into the recommendation network to obtain the recommendation candidate content output by the recommendation network, and the recommendation candidate content is recommended to the target object. The recommendation network is trained using the method described in any one of claims 1-9.

11. A method for training a policy network, characterized in that, The method includes: Obtain the sample environment, including the object features of the sample objects and the content features of the candidate content; Based on the sample environment, the first policy network and the second policy network are iteratively trained to obtain the iterative rewards corresponding to the first policy network and the second policy network at the current iteration training time; the first policy network predicts the recommended candidate content when recommending to the sample object from the candidate content; the second policy network predicts the excluded candidate content when recommending to the sample object from the candidate content. The comprehensive reward of the second policy network is determined based on the iterative rewards corresponding to the first policy network and the second policy network, respectively. The network parameters of the second policy network are updated based on the comprehensive reward of the second policy network until the iteration termination condition is met, and the trained second policy network is used as the filtering network.

12. A training device for a recommendation network, characterized in that, include: The first acquisition module is used to acquire the sample environment, including the object features of the sample object and the content features of the candidate content; The first training module is used to iteratively train the first policy network and the second policy network based on the sample environment to obtain the iteration reward corresponding to the first policy network and the second policy network at the current iteration training time; the first policy network predicts the recommended candidate content when recommending to the sample object from the candidate content; the second policy network predicts the excluded candidate content when recommending to the sample object from the candidate content. The first reward determination module is used to determine the comprehensive reward of the first policy network based on the iterative rewards corresponding to the first policy network and the second policy network respectively. The first network parameter update module is used to update the network parameters of the first policy network according to the comprehensive reward of the first policy network until the iteration termination condition is met, and the trained first policy network is used as the recommendation network.

13. A content recommendation device, characterized in that, The method includes: The second acquisition module is used to acquire the current environment, including the object characteristics of the target object and the content characteristics of the candidate content; The recommendation module is used to input the current environment into the recommendation network, obtain the recommendation candidate content output by the recommendation network, and recommend the recommendation candidate content to the target object; wherein, the recommendation network is trained using the method described in any one of claims 1-9.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the method according to any one of claims 1-11.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-11.

Images