Information recommendation model training method and device, electronic equipment and storage medium

Abstract

The invention discloses an information recommendation model training method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a virtual user feature model and a virtual user behavior model; obtaining an initial information recommendation agent, interacting the initial information recommendation agent with the virtual user feature model and the virtual user behavior model, and collecting trajectory data of the information recommendation agent; and performing near-end strategy optimization training on the initial information recommendation agent based on the trajectory data of the information recommendation agent to obtain a target information recommendation model. According to the technical scheme, the reinforcement learning technology is matched with the virtual user feature model and the virtual user behavior model to perform iterative training of the information agent, so that the information recommendation model can perform accurate recommendation in a real scene, and the accuracy of information recommendation is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a training method, device, electronic device and storage medium for an information recommendation model. Background Art

[0002] With the development of artificial intelligence technology, reinforcement learning has been widely used in scenarios such as shopping recommendations, video recommendations, and game strategy recommendations.

[0003] Currently, there is a problem of inaccurate information recommendation in social media (Weibo, WeChat, etc.) through reinforcement learning. Summary of the invention

[0004] The present invention provides a training method, device, electronic device and storage medium for an information recommendation model to improve the information recommendation accuracy of the information recommendation model.

[0005] According to one aspect of the present invention, a method for training an information recommendation model is provided, comprising:

[0006] Obtaining a virtual user feature model and a virtual user behavior model;

[0007] Acquire an initial information recommendation agent, interact the initial information recommendation agent with a virtual user feature model and a virtual user behavior model, and collect trajectory data of the information recommendation agent;

[0008] Based on the trajectory data of the information recommendation agent, the initial information recommendation agent is trained for proximal strategy optimization to obtain a target information recommendation model.

[0009] According to another aspect of the present invention, there is provided a training device for an information recommendation model, comprising:

[0010] A virtual model acquisition module is used to acquire a virtual user feature model and a virtual user behavior model;

[0011] A trajectory data collection module is used to obtain an initial information recommendation agent, interact the initial information recommendation agent with a virtual user feature model and a virtual user behavior model, and collect trajectory data of the information recommendation agent;

[0012] The proximal strategy optimization training module is used to perform proximal strategy optimization training on the initial information recommendation agent based on the trajectory data of the information recommendation agent to obtain a target information recommendation model.

[0013] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0014] at least one processor;

[0015] and a memory communicatively coupled to the at least one processor;

[0016] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the training method of the information recommendation model described in any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the information recommendation model training method described in any embodiment of the present invention when executed.

[0018] The technical solution of the embodiment of the present invention obtains the initial information recommendation agent by obtaining the virtual user feature model and the virtual user behavior model, and interacts the initial information recommendation agent with the virtual user feature model and the virtual user behavior model to collect the trajectory data of the information recommendation agent, and then performs proximal strategy optimization training on the initial information recommendation agent based on the trajectory data of the information recommendation agent to obtain the target information recommendation model. The above technical solution uses reinforcement learning technology with the virtual user feature model and the virtual user behavior model to iteratively train the information agent, so that the information recommendation model can make accurate recommendations in real scenarios, thereby improving the accuracy of information recommendation.

[0019] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 is a flow chart of a training method for an information recommendation model provided according to Embodiment 1 of the present invention;

[0022] Figure 2 is a flow chart of a training method for an information recommendation model provided according to Embodiment 2 of the present invention;

[0023] Figure 3 is a flowchart of a training method for an information recommendation model provided according to Embodiment 3 of the present invention;

[0024] Figure 4 is a structural diagram of a training device for an information recommendation model provided according to a fourth embodiment of the present invention;

[0025] Figure 5 It is a structural schematic diagram of an electronic device for implementing the training method of the information recommendation model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. The acquisition, storage, use, processing, etc. of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0028] Embodiment 1

[0029] Figure 1 This is a flowchart of a training method for an information recommendation model provided in the first embodiment of the present invention. This embodiment is applicable to the case of information recommendation decision-making based on social media. The method can be executed by a training device for an information recommendation model. The training device for an information recommendation model can be implemented in the form of hardware and / or software. The training device for an information recommendation model can be configured in an electronic device such as a terminal and / or a server. Figure 1 As shown, the method includes:

[0030] S110: Acquire a virtual user feature model and a virtual user behavior model.

[0031] In the embodiments of the present invention, the virtual user feature model refers to a model that can generate user feature data, which can be used as the status of the information recommendation agent. The user feature data can include but are not limited to the age, gender, interests, and occupation of social media users. The virtual user behavior model refers to a model that can generate user behavior data, which can be used as the actions of the information recommendation agent. The user behavior data can include but are not limited to the actions of users clicking to view, turning pages, liking, and stepping on during the social media application process.

[0032] Exemplarily, the pre-built or real-time generated virtual user feature model and virtual user behavior model may be retrieved from a preset storage path of the electronic device, which is not specifically limited here.

[0033] S120, obtaining an initial information recommendation agent, interacting the initial information recommendation agent with a virtual user feature model and a virtual user behavior model, and collecting trajectory data of the information recommendation agent.

[0034] In an embodiment of the present invention, trajectory data refers to experience data for learning and training of an information recommendation agent, and may include multiple user feature data generated by a virtual user feature model, multiple user behavior data generated by a virtual user behavior model, and multiple reward information.

[0035] Exemplarily, the policy network and value function network of the information recommendation agent are initialized to obtain an initial information recommendation agent, and then the initial information recommendation agent interacts with a virtual user feature model and a virtual user behavior model to collect trajectory data of the information recommendation agent, and the trajectory data of the information recommendation agent is stored in a shared experience pool for access and learning by the information recommendation agent.

[0036] S130. Perform proximal strategy optimization training on the initial information recommendation agent based on the trajectory data of the information recommendation agent to obtain a target information recommendation model.

[0037] In an embodiment of the present invention, the policy network of the initial information recommendation agent can be updated based on the trajectory data of the information recommendation agent through a proximal policy optimization algorithm until the training stop condition is met to obtain a target information recommendation model.

[0038] After obtaining the target information recommendation model, the target information recommendation model can be deployed and applied to realize online real-time information recommendation.

[0039] The technical solution of the embodiment of the present invention obtains the initial information recommendation agent by obtaining the pre-built virtual user feature model and virtual user behavior model, interacts the initial information recommendation agent with the virtual user feature model and virtual user behavior model, collects the trajectory data of the information recommendation agent, and then performs proximal strategy optimization training on the initial information recommendation agent based on the trajectory data of the information recommendation agent to obtain the target information recommendation model. The above technical solution uses reinforcement learning technology with virtual user feature model and virtual user behavior model to iteratively train the information agent, so that the information recommendation model can make accurate recommendations in real scenarios, thereby improving the accuracy of information recommendation.

[0040] Embodiment 2

[0041] Figure 2 A flowchart of a training method for an information recommendation model provided in Embodiment 2 of the present invention, wherein the method of this embodiment can be combined with the various optional schemes in the training method for the information recommendation model provided in the above embodiments. The training method for the information recommendation model provided in this embodiment is further optimized. Optionally, the acquisition of a pre-built virtual user feature model and a virtual user behavior model includes: building a virtual user feature model based on a generative adversarial network; building a virtual user behavior model based on imitation learning.

[0042] like Figure 2 As shown, the method includes:

[0043] S210, constructing a virtual user feature model based on a generative adversarial network;

[0044] S220. Construct a virtual user behavior model based on imitation learning.

[0045] S230, obtaining an initial information recommendation agent, interacting the initial information recommendation agent with a virtual user feature model and a virtual user behavior model, and collecting trajectory data of the information recommendation agent.

[0046] S240: Perform proximal strategy optimization training on the initial information recommendation agent based on the trajectory data of the information recommendation agent to obtain a target information recommendation model.

[0047] In the embodiment of the present invention, the virtual user feature model constructed based on the generative adversarial network can more accurately capture the personalized characteristics of social media users, thereby improving the accuracy of recommended content. In addition, the user behavior model learned through imitation learning can enable the information recommendation agent to adapt to changes in user interests and behaviors in real time, thereby improving the dynamic adaptability of information recommendation.

[0048] In some optional embodiments, a virtual user feature model is constructed based on a generative adversarial network, including: obtaining random noise data, inputting the random noise data into a generator to obtain virtual user feature data; obtaining real user feature data; updating the model parameters of the discriminator by a gradient ascent method based on the virtual user feature data and the real user feature data; updating the model parameters of the generator by a gradient descent method based on the virtual user feature data; until a training stop condition is met to obtain a virtual user feature model.

[0049] In an embodiment of the present invention, the generative adversarial network includes a generator and a discriminator. For the discriminator, the error between the virtual user feature data and the real user feature data can be calculated, and the model parameters of the discriminator can be updated by the gradient ascent method. For the generator, the generator can generate virtual user feature data, and then input the virtual user feature data to the discriminator for discrimination and obtain the corresponding error, and then calculate the error gradient and back-propagate the gradient through the gradient descent method to update the weight of the generator.

[0050] Exemplarily, the training steps for constructing a virtual user feature model based on a generative adversarial network include:

[0051] S211. Sample m random noise data, input the m random noise data into the generator G, and obtain virtual user feature data.

[0052] S212: Obtain m real user feature data.

[0053] S213, based on the virtual user feature data and the real user feature data, the model parameters of the discriminator are updated by the gradient ascent method, and the formula for updating the model parameters of the discriminator is:

[0054]

[0055] Among them, z (i) represents the i-th random noise data, G(z (i) ) represents the virtual user feature data generated by the i-th random noise data, x (i) represents the i-th real user feature data, D(·) represents the output of the discriminator, θ d represents the discriminator.

[0056] S214. Based on the virtual user feature data, the model parameters of the generator are updated by the gradient descent method. The formula of the model parameters of the generator is:

[0057]

[0058] Among them, θ g Represents a generator.

[0059] In some optional embodiments, a virtual user behavior model is constructed based on imitation learning, including: obtaining a model training offline data set of the user behavior model, wherein the model training offline data set includes user feature information, recommendation information, and action information; based on the model training offline data set of the user behavior model, imitation learning is performed on the initial virtual user behavior model to obtain a virtual user behavior model.

[0060] In the embodiment of the present invention, the offline data set for model training refers to real user history data, including user feature information, recommendation information and action information. The user feature information may include but is not limited to the age, gender, interests and occupation of social media users. The recommendation information may include but is not limited to the videos, news or literary works recommended by users in the process of social media history application. The recommendation information has corresponding tags, which may be military, international situation, entertainment or outdoor, etc. The action information may include but is not limited to the actions of users clicking to view, turning pages, liking and stepping on in the process of social media history application.

[0061] Exemplarily, the user feature information and the recommendation information corresponding to the user feature information are input into the initial virtual user behavior model, and the data output by the initial virtual user behavior model can be represented as a Gaussian distribution In the form of φ (o) represents the mean of the Gaussian distribution, Represents the variance of the Gaussian distribution. Further, the strategy loss is solved based on the following formula:

[0062] min φ ∑ (o,b)∈D (b-μ φ (o)) 2 ;

[0063] Among them, o represents the set of user feature information and the recommended information corresponding to the user feature information, b represents action information, and D represents the offline data set for model training. Furthermore, after calculating the policy loss, gradient descent can be performed to update the model parameters of the virtual user behavior model until the model training stop condition is met to obtain the virtual user behavior model.

[0064] The technical solution of the embodiment of the present invention can more accurately capture the personalized characteristics of social media users through a virtual user feature model constructed based on a generative adversarial network, thereby improving the accuracy of recommended content. In addition, the user behavior model learned through imitation learning can enable the information recommendation agent to adapt to changes in user interests and behaviors in real time, thereby improving the dynamic adaptability of information recommendation. The information agent is iteratively trained using reinforcement learning technology in combination with a virtual user feature model and a virtual user behavior model, so that the information recommendation model can make accurate recommendations in real scenarios, thereby improving the accuracy of information recommendation.

[0065] Embodiment 3

[0066] Figure 3 A flowchart of a training method for an information recommendation model provided in Embodiment 3 of the present invention, the method of this embodiment can be combined with the various optional schemes in the training method for the information recommendation model provided in the above embodiments. The training method for the information recommendation model provided in this embodiment has been further optimized. Optionally, the initial information recommendation agent interacts with the virtual user feature model and the virtual user behavior model to collect the trajectory data of the information recommendation agent, including: obtaining the current user feature data through the virtual user feature model; determining the recommendation information corresponding to the current user feature data based on the current user feature data and the initial information recommendation agent; determining the reward information corresponding to the current user feature data and the next user feature data based on the current user feature data, the recommendation information corresponding to the current user feature data and the virtual user behavior model, and cyclically determining the recommendation information and reward information corresponding to the next user feature data until the interaction ends; constructing the trajectory data of the information recommendation agent based on all the collected user feature data, recommendation information and reward information in the interaction.

[0067] like Figure 3 As shown, the method includes:

[0068] S310: Obtain a virtual user feature model and a virtual user behavior model.

[0069] S320, obtaining initial information and recommending an intelligent agent.

[0070] S330: Acquire current user feature data through the virtual user feature model.

[0071] S340. Based on the current user feature data and the initial information recommendation agent, determine the recommendation information corresponding to the current user feature data; based on the current user feature data, the recommendation information corresponding to the current user feature data and the virtual user behavior model, determine the reward information corresponding to the current user feature data and the next user feature data, and cyclically determine the recommendation information and reward information corresponding to the next user feature data until the interaction ends.

[0072] S350: constructing trajectory data of an information recommendation agent based on all user feature data, recommendation information, and reward information collected during the interaction.

[0073] In the embodiment of the present invention, the current user feature data can be generated through the virtual user feature model, and the current user feature data is used as the situation. The initial information recommendation agent obtains the next action according to the situation, that is, obtains the recommendation information corresponding to the current user feature data. Further, the current user feature data and the recommendation information corresponding to the current user feature data can be input, and the virtual user behavior model can generate user behavior, and then determine the reward information and the next user feature data according to the user behavior. The recommendation information and reward information corresponding to the next user feature data are determined by looping until the interaction ends. The trajectory data of the information recommendation agent can be constructed based on all the user feature data, recommendation information and reward information collected in the interaction.

[0074] Among them, the relationship between user behavior and reward information can be pre-configured, for example, the reward information corresponding to viewing is +2, the reward information corresponding to liking is +5, the reward information corresponding to turning the page is -2, and the reward information corresponding to stepping on is -5, etc.

[0075] S360: Perform proximal strategy optimization training on the initial information recommendation agent based on the trajectory data of the information recommendation agent to obtain a target information recommendation model.

[0076] Optionally, the initial information recommendation agent is trained for proximal strategy optimization based on the trajectory data of the information recommendation agent to obtain a target information recommendation model, including: generating a target playback cache based on the trajectory data of the information recommendation agent; and the initial information recommendation agent is trained for proximal strategy optimization based on the target playback cache to obtain a target information recommendation model.

[0077] In the embodiment of the present invention, the target playback cache refers to a collection of multiple information recommendation strategies, which may include user feature data, recommendation information, action value function and advantage function.

[0078] Specifically, an action value function can be determined based on user feature data and recommendation information in the trajectory data of the information recommendation agent, and then an advantage function can be determined based on user feature data and recommendation information in the trajectory data of the information recommendation agent; a target playback cache can be constructed based on user feature data, recommendation information, action value function and advantage function. Furthermore, the target playback cache can be sampled based on a preset number of batches to obtain model training sampling data; based on the model training sampling data, the network parameters of the initial information recommendation agent are updated using a gradient descent method with adaptive time estimation until a training stop condition is met, thereby obtaining a target information recommendation model.

[0079] Exemplarily, the training steps of the target information recommendation model may include:

[0080] S1000, initializing the strategy network and value function network of the information recommendation agent to obtain an initial information recommendation agent.

[0081] S2000: Initialize the playback buffer.

[0082] S3000: within a preset step length, user feature data s can be generated through a virtual user feature model t , the initial information recommendation agent is based on the user feature data s t Generate recommendation information t .

[0083] S4000, based on user feature data t 、Recommended informationa t And the user behavior feedback generated by the virtual user behavior model obtains the reward information r t and the next user feature data s t+1 .

[0084] S5000, loop S3000 and S4000 until the step length ends, and obtain the trajectory data of the information recommendation agent T represents the trajectory length.

[0085] S6000, determine the action value function Determine the advantage function

[0086] S7000, storage data Go to the playback cache and get the target playback cache.

[0087] S8000. Disrupt the order of related data in the target playback buffer and renumber them.

[0088] S9000, set the sampling times to j=0.

[0089] S10000, sample a preset batch number B from the target playback buffer to D j ,

[0090]

[0091] S11000, using D j Get the gradients Δψ and Δω, and use the Adam gradient descent method to update ψ and ω:

[0092]

[0093]

[0094] Among them, ω represents the parameters of the policy network, ψ represents the parameters of the value function network, and f(·) represents the proximal policy optimization algorithm.

[0095] S12000, cycle S10000-S11000 until Otherwise, j is increased by 1, and the loop continues to repeat steps S11000 and S12000.

[0096] S13000, loop S8000-S12000 until the set number of steps is reached.

[0097] S14000, loop S3000-S13000, and iterate to the set number of rounds.

[0098] S15000, save the policy network of the information recommendation agent, complete the training, and obtain the target information recommendation model.

[0099] The technical solution of the embodiment of the present invention determines the recommendation information corresponding to the current user feature data by recommending an intelligent agent based on the current user feature data and the initial information, and then determines the reward information corresponding to the current user feature data and the next user feature data based on the current user feature data, the recommendation information corresponding to the current user feature data and the virtual user behavior model, and cyclically determines the recommendation information and reward information corresponding to the next user feature data until the interaction ends, so that the information recommendation intelligent agent can self-adjust and evolve in the constantly changing information environment, thereby improving the accuracy of information recommendation.

[0100] Embodiment 4

[0101] Figure 4 This is a schematic diagram of the structure of a training device for an information recommendation model provided in Embodiment 4 of the present invention. Figure 4 As shown, the device comprises:

[0102] A virtual model acquisition module 410 is used to acquire a virtual user feature model and a virtual user behavior model;

[0103] A trajectory data collection module 420 is used to obtain an initial information recommendation agent, interact the initial information recommendation agent with a virtual user feature model and a virtual user behavior model, and collect trajectory data of the information recommendation agent;

[0104] The proximal strategy optimization training module 430 is used to perform proximal strategy optimization training on the initial information recommendation agent based on the trajectory data of the information recommendation agent to obtain a target information recommendation model.

[0105] The technical solution of the embodiment of the present invention obtains the initial information recommendation agent by obtaining the virtual user feature model and the virtual user behavior model, and interacts the initial information recommendation agent with the virtual user feature model and the virtual user behavior model to collect the trajectory data of the information recommendation agent, and then performs proximal strategy optimization training on the initial information recommendation agent based on the trajectory data of the information recommendation agent to obtain the target information recommendation model. The above technical solution uses reinforcement learning technology with the virtual user feature model and the virtual user behavior model to iteratively train the information agent, so that the information recommendation model can make accurate recommendations in real scenarios, thereby improving the accuracy of information recommendation.

[0106] In some optional implementations, the virtual model acquisition module 410 includes:

[0107] A virtual user feature model building unit, used to build a virtual user feature model based on a generative adversarial network;

[0108] The virtual user behavior model building unit is used to build a virtual user behavior model based on imitation learning.

[0109] In some optional implementations, the virtual user feature model building unit may also be specifically used to:

[0110] Get random noise data;

[0111] Inputting the random noise data into a generator to obtain virtual user feature data;

[0112] Obtain real user feature data;

[0113] Based on the virtual user feature data and the real user feature data, the model parameters of the discriminator are updated by the gradient ascent method; based on the virtual user feature data, the model parameters of the generator are updated by the gradient descent method; until the training stop condition is met, a virtual user feature model is obtained.

[0114] In some optional implementations, the virtual user behavior model building unit may also be specifically used to:

[0115] Acquire a model training offline data set of a user behavior model, wherein the model training offline data set includes user feature information, recommendation information, and action information;

[0116] Based on the model training offline data set of the user behavior model, the initial virtual user behavior model is subjected to imitation learning to obtain a virtual user behavior model.

[0117] In some optional implementations, the trajectory data collection module 420 may also be used to:

[0118] Acquiring current user feature data through the virtual user feature model;

[0119] Based on the current user feature data and the initial information recommendation agent, determine the recommendation information corresponding to the current user feature data;

[0120] Based on the current user feature data, the recommendation information corresponding to the current user feature data, and the virtual user behavior model, determine the reward information corresponding to the current user feature data and the next user feature data, and cyclically determine the recommendation information and reward information corresponding to the next user feature data until the interaction ends;

[0121] The trajectory data of the information recommendation agent is constructed based on all the user feature data, recommendation information and reward information collected during the interaction.

[0122] In some optional implementations, the proximal strategy optimization training module 430 includes:

[0123] a playback cache generation unit, configured to generate a target playback cache based on the trajectory data of the information-recommended agent;

[0124] The replay cache training unit is used to perform proximal strategy optimization training on the initial information recommendation agent based on the target replay cache to obtain a target information recommendation model.

[0125] In some optional implementations, the playback cache generation unit is specifically configured to:

[0126] Determine an action value function based on the user feature data in the trajectory data of the information recommendation agent and the recommendation information;

[0127] Determine an advantage function based on the user feature data in the trajectory data of the information recommendation agent and the recommendation information;

[0128] The target playback cache is constructed based on user feature data, recommendation information, action value function and advantage function.

[0129] In some optional implementations, the playback cache training unit is specifically used to:

[0130] Based on a preset batch number, sampling the target playback buffer to obtain model training sampling data;

[0131] Based on the model training sampling data, the network parameters of the initial information recommendation agent are updated using the gradient descent method with adaptive time estimation until the training stop condition is met, thereby obtaining a target information recommendation model.

[0132] The information recommendation model training device provided in the embodiment of the present invention can execute the information recommendation model training method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0133] Embodiment 5

[0134] Figure 5 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0135] like Figure 5 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The I / O interface 15 is also connected to the bus 14.

[0136] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0137] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 performs the various methods and processes described above, such as a training method for an information recommendation model, which includes:

[0138] Obtaining a virtual user feature model and a virtual user behavior model;

[0139] Acquire an initial information recommendation agent, interact the initial information recommendation agent with a virtual user feature model and a virtual user behavior model, and collect trajectory data of the information recommendation agent;

[0140] Based on the trajectory data of the information recommendation agent, the initial information recommendation agent is trained for proximal strategy optimization to obtain a target information recommendation model.

[0141] In some embodiments, the training method of the information recommendation model may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the training method of the information recommendation model described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the training method of the information recommendation model in any other appropriate manner (e.g., by means of firmware).

[0142] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0143] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0144] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0145] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0146] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0147] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0148] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0149] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A training method for an information recommendation model, characterized in that: include: Obtaining a virtual user feature model and a virtual user behavior model; Acquire an initial information recommendation agent, interact the initial information recommendation agent with a virtual user feature model and a virtual user behavior model, and collect trajectory data of the information recommendation agent; Based on the trajectory data of the information recommendation agent, the initial information recommendation agent is trained for proximal strategy optimization to obtain a target information recommendation model.

2. The method according to claim 1, characterized in that The step of obtaining a virtual user feature model and a virtual user behavior model includes: Construct a virtual user feature model based on generative adversarial networks; Constructing virtual user behavior model based on imitation learning.

3. The method according to claim 2, characterized in that The method of constructing a virtual user feature model based on a generative adversarial network includes: Get random noise data; Inputting the random noise data into a generator to obtain virtual user feature data; Obtain real user feature data; Based on the virtual user feature data and the real user feature data, the model parameters of the discriminator are updated by the gradient ascent method; based on the virtual user feature data, the model parameters of the generator are updated by the gradient descent method; until the training stop condition is met, a virtual user feature model is obtained.

4. The method according to claim 2, characterized in that: The construction of a virtual user behavior model based on imitation learning includes: Acquire a model training offline data set of a user behavior model, wherein the model training offline data set includes user feature information, recommendation information, and action information; Based on the model training offline data set of the user behavior model, the initial virtual user behavior model is subjected to imitation learning to obtain a virtual user behavior model.

5. The method according to claim 1, characterized in that The step of interacting the initial information recommendation agent with the virtual user feature model and the virtual user behavior model to collect trajectory data of the information recommendation agent includes: Acquiring current user feature data through the virtual user feature model; Based on the current user feature data and the initial information recommendation agent, determine the recommendation information corresponding to the current user feature data; Based on the current user feature data, the recommendation information corresponding to the current user feature data, and the virtual user behavior model, determine the reward information corresponding to the current user feature data and the next user feature data, and cyclically determine the recommendation information and reward information corresponding to the next user feature data until the interaction ends; The trajectory data of the information recommendation agent is constructed based on all the user feature data, recommendation information and reward information collected during the interaction.

6. The method according to claim 1, characterized in that The performing proximal strategy optimization training on the initial information recommendation agent based on the trajectory data of the information recommendation agent to obtain a target information recommendation model includes: generating a target playback cache based on the information recommending trajectory data of the agent; The initial information recommendation agent is trained for proximal strategy optimization based on the target playback cache to obtain a target information recommendation model.

7. The method according to claim 6, characterized in that The generating a target playback cache based on the trajectory data of the recommended agent according to the information includes: Determine an action value function based on the user feature data in the trajectory data of the information recommendation agent and the recommendation information; Determine an advantage function based on the user feature data in the trajectory data of the information recommendation agent and the recommendation information; Construct a target playback cache based on user feature data, recommendation information, action value function, and advantage function; Accordingly, the initial information recommendation agent is trained for proximal strategy optimization based on the target playback cache to obtain a target information recommendation model, including: Based on a preset batch number, sampling the target playback buffer to obtain model training sampling data; Based on the model training sampling data, the network parameters of the initial information recommendation agent are updated using the gradient descent method with adaptive time estimation until the training stop condition is met, thereby obtaining a target information recommendation model.

8. A training device for an information recommendation model, characterized in that: include: A virtual model acquisition module is used to acquire a virtual user feature model and a virtual user behavior model; A trajectory data collection module is used to obtain an initial information recommendation agent, interact the initial information recommendation agent with a virtual user feature model and a virtual user behavior model, and collect trajectory data of the information recommendation agent; The proximal strategy optimization training module is used to perform proximal strategy optimization training on the initial information recommendation agent based on the trajectory data of the information recommendation agent to obtain a target information recommendation model.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the training method of the information recommendation model described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the training method of the information recommendation model described in any one of claims 1 to 7 when executed.

Images