Model training device, method, apparatus and medium

Abstract

The embodiments of the present application relate to the field of artificial intelligence technology, and disclose a model training device, method, apparatus, and medium. Among them, by implementing the model training method proposed in the implementation of the present application, the exploration of the log data environment and the training of the reinforcement learning model can be processed in parallel during the training process of the reinforcement learning model, thereby accelerating the training speed of the reinforcement learning model, and obtaining a target strategy network according to the trained reinforcement learning model, so as to provide decisions for information display through the target strategy network in the future, so that information display opportunities can be fully utilized and the network resource utilization rate of information display can be improved.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, specifically to the field of artificial intelligence technology, and in particular to a model training device, a model training method, an information display method, a model training apparatus, and a computer storage medium. Background Art

[0002] In information display scenarios, such as advertising display scenarios, information providers can place information on the information processing system in two ways: by agreeing on the number of displays and by bidding.

[0003] In the related art, for two types of information delivered in different ways, the information processing system controls the display of the two types of information separately. For example, the information processing system first predicts the information display opportunities in the system, and allocates the predicted information display opportunities to the two types of information delivered in different ways; when an information display opportunity comes, the information processing system selects one information from the information of the corresponding delivery method for display.

[0004] However, the above-mentioned solution of separately controlling the display of the two types of information may result in insufficient utilization of information display opportunities in the system, thereby resulting in poor utilization of network resources for information display.

[0005] It can be seen that how to improve the utilization rate of network resources for information display has become a problem that needs to be solved urgently. Summary of the invention

[0006] The embodiments of the present application provide a model training device, method, apparatus and medium, which can parallelly process the exploration of the log data environment and the training of the reinforcement learning model during the training process of the reinforcement learning model, which is conducive to accelerating the training speed of the reinforcement learning model.

[0007] On the one hand, the present application implements a model training device, in which a first processing module for maintaining a strategy network and a second processing module for maintaining a reinforcement learning model are deployed, wherein:

[0008] The first processing module is used to obtain target sample information from the log data environment through the maintained policy network;

[0009] The second processing module is used to obtain training data, and perform a first iterative training on the reinforcement learning model through the training data, and write the network optimization parameters obtained after the first iterative training into the network parameter sharing area, wherein the network optimization parameters refer to the parameters obtained after the first iterative training of the policy network included in the reinforcement learning model; the training data is a plurality of target sample information;

[0010] The first processing module is further used to obtain the network optimization parameters from the network parameter sharing area, and update the parameters of the policy network maintained by the first processing module to the network optimization parameters; and obtain the target sample information from the log data environment again through the updated policy network;

[0011] The second processing module is further used to obtain training data, and perform second iterative training on the reinforcement learning model after the first iterative training through the training data, and obtain the target reinforcement learning model according to the second iterative training.

[0012] On the other hand, an embodiment of the present application provides a model training method, the method comprising:

[0013] Acquire target sample information from the log data environment through a policy network maintained by the first processing module;

[0014] Calling the second processing module to obtain training data, and performing a first iterative training on the reinforcement learning model through the training data, and writing the network optimization parameters obtained after the first iterative training into the network parameter sharing area, wherein the network optimization parameters refer to the parameters obtained after the first iterative training of the policy network included in the reinforcement learning model; the training data is a plurality of target sample information;

[0015] Acquire the network optimization parameters from the network parameter sharing area through the first processing module, update the parameters of the policy network maintained by the first processing module to the network optimization parameters, and call the updated policy network to acquire target sample information from the log data environment again;

[0016] Calling the second processing module to obtain training data, and performing a second iterative training on the reinforcement learning model after the first iterative training by using the training data;

[0017] A target reinforcement learning model is obtained according to the second iterative training, and a policy network in the target reinforcement learning model is determined as a target policy network.

[0018] On the other hand, an embodiment of the present application provides a model training device, the model training device comprising:

[0019] A sample acquisition unit, configured to acquire target sample information from a log data environment through a policy network maintained by the first processing module;

[0020] A model training unit, configured to call the second processing module to obtain training data, and perform a first iterative training on the reinforcement learning model through the training data, and write the network optimization parameters obtained after the first iterative training into the network parameter sharing area, wherein the network optimization parameters refer to the parameters obtained after the first iterative training of the policy network included in the reinforcement learning model; the training data is a plurality of target sample information;

[0021] The sample acquisition unit is further used to acquire the network optimization parameters from the network parameter sharing area through the first processing module, update the parameters of the policy network maintained by the first processing module to the network optimization parameters, and call the updated policy network to acquire target sample information from the log data environment again;

[0022] The model training unit is further used to call the second processing module to obtain training data, and perform second iterative training on the reinforcement learning model after the first iterative training through the training data, and obtain a target reinforcement learning model according to the second iterative training;

[0023] A processing unit is used to determine the policy network in the target reinforcement learning model as a target policy network.

[0024] Correspondingly, an embodiment of the present application also provides another model training device, which includes a communication interface, a processor and a storage device; the storage device is used to store program instructions; the processor is used to call the program instructions and execute the above-mentioned model training method.

[0025] Correspondingly, an embodiment of the present application also provides a computer storage medium, which stores program instructions, and when the program instructions are executed, they are used to implement the above-mentioned model training method.

[0026] Accordingly, according to one aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. The processor of the model training device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the model training device performs the model training method provided above.

[0027] In an embodiment of the present application, the model training device can process the exploration of the log data environment and the training of the reinforcement learning model in parallel during the training process of the reinforcement learning model, thereby accelerating the training speed of the reinforcement learning model, and obtaining the target policy network according to the trained reinforcement learning model, so as to facilitate the subsequent provision of decisions for information display through the target policy network, so that information display opportunities can be fully utilized and the network resource utilization rate of information display can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0029] Figure 1 is a schematic diagram of status data provided by an embodiment of the present application;

[0030] Figure 2 It is a flowchart of an existing reinforcement learning model training provided in an embodiment of the present application;

[0031] Figure 3 It is a structural diagram of an information processing system provided by an embodiment of the present application;

[0032] Figure 4 It is a flowchart of a model training method provided in an embodiment of the present application;

[0033] Figure 5 It is a structural diagram of a distributed reinforcement learning framework provided in an embodiment of the present application;

[0034] Figure 6 It is a schematic diagram of a flow chart of information display provided in an embodiment of the present application;

[0035] Figure 7 It is a structural diagram of a system framework provided by an embodiment of the present application;

[0036] Figure 8 is a corresponding relationship diagram between inventory and advertisements provided in an embodiment of the present application;

[0037] Fig. 9 It is a structural schematic diagram of a model training device provided in an embodiment of the present application;

[0038] Fig.10 It is a structural diagram of a model training device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0040] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0041] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0042] Among them, machine learning (ML) is a multi-disciplinary interdisciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0043] Reinforcement Learning (RL), also known as reinforcement learning, evaluation learning or enhanced learning, is one of the paradigms and methodologies of machine learning. It is used to describe and solve the problem of how an agent can maximize rewards or achieve specific goals through learning strategies during its interaction with the environment.

[0044] Reinforcement learning is developed from theories such as animal learning and parameter perturbation adaptive control. Its basic principles are:

[0045] If an agent's behavior strategy leads to a positive reward (reinforcement signal) in the environment, then the agent's tendency to use this behavior strategy in the future will be strengthened. The agent's goal is to find the optimal strategy in each discrete state to maximize the expected discounted reward.

[0046] Reinforcement learning regards learning as a trial-and-error evaluation process. The agent selects an action for the environment. After the environment accepts the action, its state changes and a reinforcement signal (reward or punishment) is generated and fed back to the agent. The agent selects the next action based on the reinforcement signal and the current state of the environment. The principle of selection is to increase the probability of receiving positive reinforcement (reward). The selected action not only affects the immediate reinforcement value, but also affects the state of the environment at the next moment and the final reinforcement value.

[0047] With the research and progress of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, etc., as well as the information display field involved in this solution. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0048] In the field of information display, the information to be displayed is usually different categories of information. When information display is required, how to select a target information from different categories of information for display so that the information display opportunities in the system can be fully utilized, thereby improving the network resource utilization rate of information display, has become a research hotspot. Among them, different categories of information to be displayed can include first type information and second type information; the first type information is information with display requirements, and the second type information is information that competes for display opportunities through resource expenditure. Taking the advertising display system as an example, the first type of information mentioned above can be contract advertising, and the second type of information can be bidding advertising.

[0049] In one implementation scenario, the advertiser signs a contract with the media party, and the media party plays a certain amount (predetermined amount) of advertisements to the advertiser's specified type of users within a specified time. If the contract is reached, the advertiser pays the media a certain fee. If the amount of playback does not meet the standard, the media needs to compensate the advertiser; if the amount of playback exceeds the predetermined amount, no additional fees will be charged. Advertisements exposed in this way are called contract ads.

[0050] In another implementation scenario, advertisers will give a bid for requests with the same targeting. Multiple advertisers will bid for the same request, and the advertiser with the highest bid will compete to win the exposure of this request. Ads exposed in this way are called bidding ads.

[0051] In one embodiment, in order to improve the network resource utilization rate of information display, a proportional integral differential (PID) algorithm solution and a reinforcement learning solution may be used, wherein:

[0052] The PID algorithm scheme mixes different categories of information to be displayed, and determines the score of each information to be displayed through real-time online feedback, so as to select the information with the highest score as the target information for display. This scheme relies too much on the timeliness of feedback. In theory, when there is no feedback delay, it can achieve very good results, but the actual system delay is very large. There are many factors that cause delay, such as system processing time, such as other tasks crowding out computing resources. Generally speaking, the feedback delay is more than 1 minute, which causes a decrease in effect. Secondly, the PID algorithm scheme does not directly provide the optimal solution by calculating the overall benefit, and there is a lot of room for improvement.

[0053] The reinforcement learning solution can solve the problems existing in the PID algorithm solution. Reinforcement learning refers to the exploration of the log data environment (which can be referred to as the environment in the embodiment of the present application) by the Agent, giving the score of each strategy in the current state, and finally learning to achieve the optimal strategy. For example, in the mixed arrangement scenario of two different advertisements involved in the embodiment of the present application (hereinafter referred to as the mixed advertisement scenario), the log data environment refers to: the sample set composed of the refined arrangement queue corresponding to each historical information display request, and the reward function value (Reward) is the overall system revenue. Among them, the refined arrangement queue corresponding to any historical information display request can be understood as a queue composed of all advertisements that match any historical information display request, and the refined arrangement queue can contain both contract advertisements and bidding advertisements.

[0054] In one possible implementation, the reinforcement learning model can be a model based on the Actor-Critic (AC) algorithm (hereinafter referred to as the AC model), which includes an actor network (i.e., a policy network (Actor)) and an evaluator network (i.e., a value network (Critic)). The reinforcement learning model can generate multiple Agents, each of which independently explores the environment and asynchronously returns Reward to update a shared network. Among them, the policy network is responsible for giving actions in the current state, and the value network is responsible for predicting the total revenue that can be obtained from the current state to the end time (or it can also be understood as long-term benefit). Among them, the above-mentioned Agent in the embodiment of the present application can refer to the first processing module, and the first processing module is a module deployed with a first process, and each first processing module independently explores the environment through the first process deployed by itself.

[0055] The key to reinforcement learning lies in three points: 1. Action definition; 2. State data definition; 3. Reward setting. Among them:

[0056] action

[0057] The action in the advertisement mixing scenario described in this patent is to score the advertisement in the current state.

[0058] Status data

[0059] The state data reflects the reason why the agent takes action. The state data must be able to sufficiently characterize the current environment so that there is sufficient distinction between different states. The state data includes at least one of information-level data, overall data, and traffic dimension feature data.

[0060] The information level data includes at least one of: an identifier of the corresponding information, an identifier of the corresponding information display position, a playback volume of the corresponding information, a playback volume requirement of the corresponding information, a playback speed of the corresponding information, and an upper limit of the playback volume of the corresponding information.

[0061] The overall data includes: at least one of the overall missing rate of the first type of information in the system, the average click rate of the first type of information in the system, the average click rate of the second type of information in the system, and the average resource expenditure of the second type of information in the system.

[0062] The traffic dimension feature includes at least one of: region data matched by the corresponding information display request, gender data matched by the corresponding information display request, and age data matched by the corresponding information display request.

[0063] The present application embodiment only uses the information contained in the above information-level data, overall data and traffic dimension feature data as an example for illustration. The above information-level data, overall data and traffic dimension feature data include but are not limited to the data listed above. For example, in the advertising mixed arrangement scenario, the above status data can be as follows: Figure 1 As shown, Figure 1 The advertising features are the above information data, the overall features are the above overall data, and the PAC features are the above traffic dimension features. PAC refers to the dimension of each request. For example, if a request is for a 30-year-old male in Shanghai, then the PAC feature corresponding to the request is (Shanghai, male, 30 years old).

[0064] Reward

[0065] Reward design is the core part of reinforcement learning. A good reward should reflect the task goal and be easy to converge. In the mixed advertising scenario involved in the embodiment of the present application, the overall revenue of the advertising display system includes three parts: contract guarantee, contract click rate and bidding Ecmp. Among them, the contract guarantee is represented by the overall shortage rate of the first type of information in the log data environment, the contract click rate is represented by the average click rate score of the first type of information in the log data environment, and the bidding Ecmp is represented by the average resource expenditure score of the second type of information in the log data environment. Ecmp is the expected revenue per thousand exposures, which is the revenue after an advertisement is displayed one thousand times.

[0066] Among them, the overall shortage rate of the first type of information in the log data environment can be obtained by the number of plays and the number of plays that should be played of each first type of information; the average click rate of the first type of information can be obtained by taking the average of the predicted click rates of the first type of information; the average resource payment of the second type of information can be obtained by taking the average of the predicted resource payment of the first type of information. The resource payment can be the bid of the bidding advertisement for the display opportunity corresponding to each information display request.

[0067] It is understandable that the essence of mixed arrangement is to treat the benefits of different categories of information to be displayed equally, and different categories of information to be displayed compete with each other to achieve the global optimality of the system. Taking the field of advertising display as an example, the essence of mixed arrangement is to treat the benefits of contract advertising and bidding advertising equally, and contract advertising and bidding advertising compete with each other to achieve the global optimality of the system. In this case, an algorithm that gives the best advertising selection strategy according to the environment is needed, and the idea of ​​exploration and game is most suitable for reinforcement learning. Reinforcement learning explores the environment (contract bidding competition environment) through the agent, and gives the solution when the overall benefit is the highest, which can further improve the overall benefit, so that the information display opportunities in the system can be fully utilized, thereby improving the network resource utilization rate of information display. However, there is a fatal problem with the reinforcement learning solution. The training of the reinforcement learning model needs to follow serial steps, and the training speed is slow.

[0068] For example, taking the AC model as an example, the reinforcement learning model includes an actor network (i.e., a strategy network (Actor)) and an evaluator network (i.e., a value network (Critic)), see Figure 2 , the specific training steps include:

[0069] S10: The agent continuously explores the environment through the initialized policy network, generates a sample each time, and writes it into the sample pool until the sample pool (the sample pool size is generally set to 10,000) is full.

[0070] S11: Randomly extract a batch of samples from the sample pool and train the Actor and Critic networks.

[0071] S12: The agent explores the environment through the trained Actor network, generates a sample, writes it into the sample pool, and randomly replaces one of the samples.

[0072] The above steps S11-S12 are repeated until the model converges.

[0073] From the above content, it can be seen that the current reinforcement learning model training scheme is a serial scheme. It cannot train while exploring the environment, and cannot explore the environment while training, so the training efficiency of the model is low.

[0074] In order to solve the problem of low efficiency in reinforcement learning model training, this application proposes an information processing system and a model training method based on the information processing system. The model training method can be executed by a model training device, which is a server or a terminal device. As a feasible method, see Figure 3 , the information processing system may include an information display client and a server. The server may be a background server of the information display client (for example, a Web server, an application server, a data processing server), which is used to provide information display decision services, enhanced model training and other services to the information display client to support the operation of the client; it may include any of the following: a Web server, an application server, a data processing server, and the server here may refer to a physical server or a server cluster. The above-mentioned information display client is used to provide an information display interface to the user; it may include at least one of the following: an APP (Application), a web page (such as a web page running in a browser in a terminal device to be displayed to the user for browsing, an H5 web page, etc.), etc. In this case, the model training method can be executed by a server in the information processing system (that is, the above-mentioned model training device that executes the model training method is a server in the information processing system).

[0075] In actual application scenarios, the server can train the reinforcement learning model (the reinforcement learning model includes a policy network and a value network) through the model training method proposed in the embodiment of the present application until the reinforcement model learning model converges, obtains the target reinforcement learning model, and determines the policy network in the target reinforcement learning model as the target policy network. Subsequently, the server can receive information display requests from information display clients, and select one information from two different types of information as the target information for display through the target policy network obtained by reinforcement learning, thereby realizing mixed control of the two types of information, so that the information display opportunities in the system can be fully utilized, thereby improving the network resource utilization rate of information display.

[0076] Alternatively, as another feasible way, the information processing system may also include only a terminal device. In this case, the model training method proposed in the embodiment of the present application may be executed by a terminal device (that is, the above-mentioned model training device that executes the model training method is a terminal device in the information processing system). The terminal device may refer to a terminal device with a network connection function and an information display application installed. For example, the terminal device may be a smart phone, a tablet computer, an e-book reader, smart glasses, a smart watch, an MP3 player (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Compression Standard Audio Layer 3), an MP4 (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Compression Standard Audio Layer 4) player, a laptop or desktop computer, etc. Optionally, the terminal device may also be referred to as a user device, a portable terminal, a laptop terminal, a desktop terminal, etc.

[0077] In an embodiment of the present application, the above-mentioned information display application may include any application that provides information recommendation locations, such as, including but not limited to video playback applications, video live broadcast applications, news applications, reading applications, music playback applications, social applications, game applications, communication applications, browser applications, and applications that come with the terminal system, etc.

[0078] In one embodiment, the model training device is deployed with a first processing module for maintaining a policy network, a second processing module for maintaining a reinforcement learning model, and a central processing module, wherein the reinforcement learning model includes a policy network and a value network, wherein:

[0079] The first processing module is used to obtain the target sample information corresponding to the historical information display request from the log data environment through the maintained policy network; wherein the log data environment includes display sample information corresponding to each of the multiple historical information display requests, and the display sample corresponding to the display sample information includes first type information and second type information; in the mixed advertising scenario, the above-mentioned information processing system may refer to the advertising display system, the first type information may be a contract advertisement, the second type information may be a bidding advertisement, and any display sample information corresponding to any historical information display request includes: the advertisement displayed by the advertising system in response to any of the historical information display requests (i.e., the display sample), the overall revenue corresponding to displaying the advertisement, and when responding to any of the historical information display requests, the current status data of each advertisement in the refined ranking queue that matches the any of the historical information display requests. Exemplarily, the status data may include the following: Figure 1 Various data shown.

[0080] The target sample information corresponding to the above historical information display request is obtained from the log data environment by the first processing module through the policy network maintained by itself. In a specific implementation, the first processing module can continuously explore the log data environment through the policy network maintained by itself, and determine the target sample information corresponding to each historical information display request from the target data environment, and one historical information display request corresponds to one target sample information. The following takes the target historical information display request (i.e., any historical information display request) as an example to illustrate the implementation method of the first processing module exploring the log data environment and obtaining the target sample information from the log data environment, including:

[0081] The target display sample information corresponding to the target historical information display request is obtained from the log data environment, and the target display sample information is processed through the strategy network maintained by the first processing module to obtain the sample scoring strategy for the target historical information display request. Among them, the target display sample information includes: the target historical sample displayed in response to the target historical information display request, the overall revenue corresponding to the display of the target historical sample, and the status data of each candidate sample in the refined ranking queue that matches the target historical information display request; the sample scoring strategy indicates the display score of the target historical sample and each candidate sample under the target historical information display request. In the advertising display scenario, assuming that the target historical information display request is a historical advertising request, the target display sample information includes: the target historical advertisement displayed in response to the historical advertising request, the overall revenue brought by the display of the target historical advertisement, and the status data of each candidate advertisement that can be displayed in response to the historical advertising request.

[0082] Furthermore, after determining the sample scoring strategy, the target sample (for example, the candidate sample with the highest display score is selected as the target sample) and the display score of the target sample can be determined from the multiple candidate samples matching the target historical information display request according to the sample scoring strategy, and then the target sample information corresponding to the above-mentioned target historical information display request is generated according to the target sample and the display score of the target sample. The target sample information may include: the target sample, the display score of the target sample, the overall revenue corresponding to displaying the target sample, and the status data of each advertisement in the refined ranking queue corresponding to the target historical information display request based on which the strategy network makes the above-mentioned sample scoring strategy.

[0083] The second processing module is used to obtain training data, perform first iterative training on the reinforcement learning model through the training data, and write the network optimization parameters obtained after the first iterative training into the network parameter sharing area, where the network optimization parameters refer to the parameters obtained after the first iterative training of the policy network included in the reinforcement learning model, wherein the training data is a plurality of target sample information (which can be understood as a batch of target sample information);

[0084] The first processing module is also used to obtain network optimization parameters from the network parameter sharing area, update the parameters of the policy network maintained by the first processing module to the network optimization parameters, and obtain target sample information from the log data environment again through the updated policy network.

[0085] Furthermore, the second processing module is also used to continue to obtain training data, and perform second iterative training on the reinforcement learning model after the first iterative training through the training data, and obtain the target reinforcement learning model according to the second iterative training.

[0086] In one embodiment, the first iterative training and the second iterative training may both refer to M times of iterative training, for example, the first iterative training may refer to the first M times of iterative training, and the second iterative training may refer to the second M times of iterative training, wherein M is an integer greater than 0 and may be pre-set according to experimental measurement data, for example, 1000, and M is used to characterize that for every M times of iterative training, the network optimization parameters (i.e., the network parameters of the policy network in the reinforcement learning model) are written into the network parameter sharing area once.

[0087] In a specific implementation, during the training of the reinforcement learning model by the model training device, the first processing module can continuously explore the log data environment through the policy network maintained by itself, and obtain the target sample information from the log data environment; the second processing module can continuously obtain multiple target sample information as training data, and iteratively train a complete reinforcement learning model maintained by itself according to the training data. After each iterative training M times, the network optimization parameters obtained by the current M iterative training are written into the network parameter sharing area until the reinforcement learning model converges, and then the model training is stopped. Before model training, the first processing module can continuously obtain the above-mentioned network optimization parameters from the network parameter sharing area, update the network parameters of the policy network maintained by itself according to the network optimization parameters, and explore the log data environment again according to the updated policy network, and obtain the target sample information from the log data environment, and repeat this cycle until the model stops training and the target reinforcement learning model is obtained.

[0088] The central processing module is used to determine the policy network in the target reinforcement learning model as the target policy network, and the target policy network is used to determine the display score of each candidate information matching the information display request, and the display score is used to indicate the probability that the corresponding candidate information is pushed to the information display position corresponding to the information display request for display. Specifically, when an information display request is subsequently received (the information display request is used to request to display information at a specified information display position), multiple candidate information matching the information display request can be obtained, and the display score of each candidate information is determined through the target policy network, and the target information is obtained based on the display score of each candidate information (for example, the candidate information with the highest display score is determined as the target information), and the target information is pushed to the specified information display position for display. In this way, the target information can be selected from the candidate information for display, so that the information display opportunities in the system can be fully utilized, thereby improving the network resource utilization rate of information display.

[0089] See also Figure 4 As shown, the model training method performed by the above-mentioned model training device may include the following steps S401-S405:

[0090] S401: Acquire target sample information corresponding to a historical information display request from a log data environment through a policy network maintained by a first processing module.

[0091] In one embodiment, the first processing module also maintains a corresponding sample sharing area, and the model training device can continuously explore the log data environment through the strategy network maintained by the first processing module, obtain target sample information from the log data environment, and write it into the sample shared memory maintained by itself once every time K target sample information is explored. Wherein, K is an integer greater than 0, which can be pre-set based on experimental test data, for example, 2000, and the specific value of K is used to characterize the number of target sample information written into the sample shared memory at one time.

[0092] In one embodiment, the model training device may write the target sample information into the sample sharing area corresponding to the first processing module in a cyclic writing manner. When the sample sharing area is full, the oldest target sample information is overwritten from the head.

[0093] S402: Call the second processing module to obtain training data, and perform first iterative training on the reinforcement learning model through the training data, and write the network optimization parameters obtained after the first iterative training into the network parameter sharing area, where the network optimization parameters refer to the parameters obtained after the first iterative training of the policy network included in the reinforcement learning model. The training data here refers to the multiple target sample information mentioned above.

[0094] See also Figure 2It can be seen that in the training process of the traditional reinforcement learning model (the reinforcement learning module includes a policy network and a value network), a policy network independently explores the log data environment, obtains samples (i.e., the historical information display request corresponds to the target sample information) and writes them into the sample pool. Subsequently, a batch of samples can be extracted from the sample pool to train the policy network and value network in the reinforcement learning model. Take one training as an example to illustrate the training process of the traditional reinforcement learning model: each training can input a sample corresponding to a target historical information display request into the reinforcement learning model, and the policy network in the reinforcement learning model can process the sample corresponding to the historical information display request and determine the target sample scoring strategy, which is used to indicate the display score of each candidate information matching the historical information display request. Furthermore, the value network can evaluate the above-mentioned target sample scoring strategy output by the policy network and return the evaluation result to the policy network. The policy network can update its own network parameters based on the evaluation result. For the policy network, it is necessary to use the Loss (loss function) to update the network parameters of the policy network in the direction of the Loss decrease.

[0095] The above Loss can be found in Formula 1.1:

[0096]

[0097] Among them, x is the current state, a is the action taken in the current state, π is the strategy. In the embodiment of the present application, taking the sample input for the current training as the target sample information corresponding to a target historical information display request as an example, the current state refers to the state data of each candidate information that matches the target historical information display request when the target historical information display request is received; the target action taken in the current state refers to the action of scoring each candidate information, and the strategy refers to the above-mentioned target sample scoring strategy, which is used to indicate the display score of each candidate information.

[0098] In formula 1.1, Q π (x,a) represents the future long-term benefit after taking the target action in the current state, and r(x,a) represents the immediate benefit after taking the target action in the current state.

[0099] From the above content, we can know that each time the traditional value network is trained, it is necessary to calculate the Q value corresponding to the input sample of the current training (for example, the Q π (x, a) and so on). However, in the embodiment of the present application, the model training device may include multiple first processing modules (see Figure 5As shown), each first processing module is used to maintain a separate policy network, and each first processing module can independently execute the above-mentioned steps of obtaining the target sample information corresponding to the historical information display request from the log data environment, that is, each first processing module can independently explore the log data environment and obtain samples therefrom. In this process, although each first processing module explores the same log data environment, due to slight differences in the execution time of each first processing module exploring the log data environment or other factors, the sample distributions explored by each first processing module will be different (for example, different samples are explored for the same historical information display request). In this case, when the samples are subsequently obtained as training data to train the reinforcement learning model, it may happen that the samples obtained twice come from two different first processing modules. If the above-mentioned Loss is used to train the value network in reinforcement learning, it may cause policy oscillations and difficulty in model training.

[0100] In practice, it is found that although there are differences in the sample distribution obtained by asynchronous exploration of each first processing module in the embodiment of the present application, the overall distribution of the samples is consistent after integrating each first processing module. Therefore, in order to overcome the problems existing in training the value network through the above-mentioned Loss, the embodiment of the present application can no longer consider the value Q during the training process of the value network, but the overall distribution of the value.

[0101] Specifically, in the embodiment of the present application, the training data required for one training includes multiple target sample information, and the first iteration training of the reinforcement learning model is explained by the training data: the target sample information in the training data can be processed by the policy network in the reinforcement learning model to obtain the target sample scoring strategy. Furthermore, the evaluation result and value distribution result of the target sample scoring strategy are determined by the value network in the reinforcement learning model, and the network parameters of the policy network in the reinforcement learning model are updated according to the evaluation results, and the network parameters of the value network in the reinforcement learning model are updated according to the value distribution results.

[0102] The specific method of updating the network parameters of the value network in the reinforcement learning model according to the value distribution result (hereinafter referred to as value distribution) includes: calculating the distance d between the value distribution and the target value distribution, and calculating the function value of the target loss function Loss1 of the value network according to the distance. For example, the specific calculation formula of Loss1 can be found in Formula 1.2:

[0103]

[0104] in, The value distribution determined for the value network, Q here π (x,a)=EZπ (x, a), Z represents the distribution of scoring strategies for each target sample, and the others remain unchanged; (Z w (x,a) is the target value distribution mentioned above, which can also be understood as the optimal value distribution, which is predetermined.

[0105] Furthermore, after determining the function value of the above-mentioned objective loss function Loss1, the network parameters of the value network can be updated according to the decreasing direction of Loss1.

[0106] In one embodiment, a memory identifier may be set for the sample sharing area maintained by the first processing module, and the memory identifier includes a readable identifier (e.g., 1) and a read-forbidden identifier (e.g., 0). Specifically, if the sample sharing area is writing data or the remaining storage space is greater than a specified threshold, the memory identifier of the sample sharing area is a read-forbidden identifier; if the sample sharing area has completed writing data and the remaining space of the sample sharing area is 0 (i.e., indicating that the sample sharing area is full), the memory identifier of the sample sharing area is a readable identifier. The above-mentioned specified threshold is pre-set according to experimental data, for example, it may be 0, and if the remaining storage space of the sample sharing area is greater than 0, it indicates that the sample sharing area is not full.

[0107] In one embodiment, the first processing module includes multiple first processing modules, each of which is used to maintain a corresponding policy network and sample sharing area. The specific implementation method of calling the second processing module to obtain training data is:

[0108] The second processing module is called to poll each sample sharing area. If it is detected that the memory identifier of any sample sharing area is a readable identifier, any sample sharing area is determined as a readable sample sharing area. Further, multiple target sample information can be obtained from the readable sample sharing area, and the multiple target sample information can be determined as training data.

[0109] S403: Obtain network optimization parameters from the network parameter sharing area through the first processing module, update the parameters of the policy network maintained by the first processing module to the network optimization parameters, and call the updated policy network to obtain target sample information from the log data environment again.

[0110] In one embodiment, the model training device can continuously pull the network optimization parameters (which can be understood as the latest network parameters of the policy network) in the network parameter sharing area through the first processing module, update the policy network maintained by itself through the network optimization parameters, and continue to explore the log data environment through the updated policy network, and obtain new target sample information from the log data environment, and repeat this cycle until the model stops training and the target reinforcement learning model is obtained.

[0111] S404: Call the second processing module to obtain training data, and perform second iterative training on the reinforcement learning model after the first iterative training through the training data.

[0112] S405: Obtain the target reinforcement learning model according to the second iterative training, and determine the policy network in the target reinforcement learning model as the target policy network, the target policy network is used to determine the display score of each candidate information matching the information display request, and the display score is used to indicate the probability that the corresponding candidate information is pushed to the information display position corresponding to the information display request for display. In a specific implementation, the above-mentioned first iterative training and second iterative training can both refer to M iterative trainings. In the process of training the reinforcement learning model by the model training device, the first processing module can continuously explore the log data environment through the policy network maintained by itself, and obtain the target sample information corresponding to the historical information display request from the log data environment. The second processing module can continuously obtain multiple target sample information as training data, and iteratively train a complete reinforcement learning model maintained by itself according to the training data. After each M-time iterative training, the network optimization parameters obtained by the current M-time iterative training are written into the network parameter sharing area until the reinforcement learning model converges, then the model training is stopped to obtain the target reinforcement learning model.

[0113] In one embodiment, the above network parameter sharing area and sample sharing area can be regarded as shared memory. Shared memory is a one-dimensional array of fixed type and only supports C language basic types (Tint, Float, Char, etc.). Whether it is a sample or a network parameter of Tensorflow, it needs to be processed and encoded into a specified format before it can be stored in the shared memory. This processing process is called serialization; similarly, when pulling data, it also needs to be reversed, processed and parsed, and converted into data that can be processed by the first processing module and the second processing module. Among them, Tensorflow is a deep learning training package that contains some mathematical calculation functions. In this case, in the embodiment of the present application, the data written into the network parameter sharing area and each sample sharing area are all serialized data. The specific implementation method of writing the target sample information into the target sample sharing area corresponding to the first processing module in a cyclic writing manner is: the target sample information is serialized to convert the target sample information into an array matching the data type corresponding to the sample sharing area, and the array is written into the sample sharing area corresponding to the first processing module in a cyclic writing manner. Exemplarily, the data type can be a C language basic type. Furthermore, when the subsequent model training device obtains multiple target sample information from the readable sample sharing area through the second processing module, it can obtain multiple arrays from the readable sample sharing area, deserialize the multiple arrays, and obtain multiple target sample information.

[0114] It can be known from the above content that in the embodiment of the present application, during the training of the reinforcement learning model, the model training device can call the policy network maintained by itself through the first processing module to continuously explore the log data environment and obtain the target sample information from the log data environment; through the second processing module, continuously obtain multiple target sample information as training data, and iteratively train a complete reinforcement learning model maintained by itself according to the training data, and write the network optimization parameters obtained by the current M iterations of training into the network parameter sharing area for each iterative training M times, until the reinforcement learning model converges, then stop the model training. Before model training, the model training device can continuously obtain the above network optimization parameters from the network parameter sharing area through the first processing module, update the network parameters of the policy network maintained by itself according to the network optimization parameters, and explore the log data environment again according to the updated policy network, obtain the target sample information from the log data environment, and repeat this cycle until the model stops training, obtains the target reinforcement learning model, and determines the policy network in the target reinforcement learning model as the target policy network. By adopting this model training method, the exploration of the log data environment and the training of the reinforcement learning model can be processed in parallel during the training process, which is conducive to improving the training speed of the reinforcement learning model, so as to obtain the target strategy network more quickly to provide the optimal solution for information display, provide decisions for the subsequent information display, and make full use of the information display opportunities in the system, thereby improving the network resource utilization rate of information display.

[0115] In order to better understand the specific implementation of the above model training method, the model training method is applied to Figure 5 The distributed reinforcement learning framework shown in FIG. 1 is described, and the distributed framework includes a first processing module, a second processing module, and a central processing module. Among them:

[0116] First processing module:

[0117] The module deployed with the first process is not unique. Multiple modules can be started, usually 5 modules, to maintain a separate Actor network (i.e., policy network), log data environment, and Tensorflow context. The first processing module inherits the Actor network parameters of the second processing module. Initially, the parameters are randomly initialized. The first processing module continuously explores the environment through the Actor network maintained by itself, and writes the samples (i.e., the target sample information) into the corresponding S_Memory (i.e., the sample sharing area). When writing, the flag position is 0, and it is set to 1 after it is full and finished. Each time it is finished, it will pull the latest Actor network parameters from ANP_Memory (i.e., the network parameter sharing area), update the Actor network maintained by itself, and then continue to explore the environment with the latest parameters. Among them, the Tensorflow context refers to an environment maintained after Tensorflow runs, which is automatically generated.

[0118] Second processing module:

[0119] The module with the second process is deployed, which is unique and cannot be started multiple times. It maintains a complete reinforcement learning model (Actor network and Critic network), log data environment, and tensorflow context. The second processing module is only responsible for training the network and completing the serial structure (see Figure 1 ) in the training step, two shared memory structures are maintained, one is S_Memory and the other is ANP_Memory. The second processing module continuously polls the S_Memory maintained by the Agent. Once it can be read, it reads the data for training. Then it continuously pushes the trained Actor network parameters to ANP_Memory.

[0120] S_Memory:

[0121] The sample sharing area can also be understood as a sample pool. Each first processing module corresponds to a separate sample pool, and a sample pool is jointly maintained by its corresponding first processing module and second processing module. There is a flag (i.e. the above-mentioned memory identifier) ​​as a lock. It will be set to 1 each time the storage is completed, and will be set to 0 when writing or not full.

[0122] ANP_Memory:

[0123] The network parameter sharing area is responsible for maintaining the latest Actor network parameters. The second processing module writes the latest Actor network parameters, and the first processing module reads and updates its own Actor network parameters.

[0124] Serialization, deserialization:

[0125] The above-mentioned S_Memory and ANP_Memory are both shared memories. Shared memory is a one-dimensional array of a fixed type that only supports basic C language types (Int, Float, Char, etc.). Both samples and Tensorflow network parameters need to be processed and encoded into a specified format before they can be stored in the shared memory. This processing process is called serialization. Similarly, when pulling data, it also needs to be de-encoded, processed and parsed to become data that can be processed by the first processing module and the second processing module.

[0126] In the specific implementation, the distributed training steps are as follows:

[0127] Central processing module (module with main process deployed):

[0128] Get the playback data, calculate the current playable data, and initialize all shared memory and parameters based on this data (such as Actor network parameters and Critic network, etc.

[0129] Instantiate the first process and the second process, and start N (N is an integer greater than 0) first processes and 1 second process. In a specific implementation, a first process class and a second process class may be created, and the first process class may be written into N first processes, and the second process class may be written into 1 second process, thereby completing the startup of N first processes and 1 second process.

[0130] When the reinforcement learning model converges, a trained Actor network (i.e., target policy network) is obtained. Subsequently, when an information display request is received, multiple candidate information matching the information display request can be obtained, and the optimal information display decision can be made through the target policy network, and the display score of each candidate information can be output.

[0131] First processing module:

[0132] The environment is explored continuously, and every K samples are written into the shared memory in a circular way. When it is full, the oldest sample is overwritten from the head. Before it is full, the shared memory flag is set to 0.

[0133] Pull the latest network parameters to update your Actor network.

[0134] Second processing module:

[0135] Poll the S_Memory corresponding to each first processing module. If a S_Memory is found to be readable, a batch of data will be randomly read as training data.

[0136] The Actor and Critic networks are trained and updated by randomly reading training data. The total benefit under the current parameters is calculated in each round, and the optimal parameters are saved. The latest Actor network parameters are written to the shared memory every M trainings until the model converges.

[0137] Among them, the above N represents the number of first processes started, K is the number of samples written at a time, and M is how many times the latest Actor network parameters are written to ANP_Memory after training. Generally, N=5, K=2000, and M=1000.

[0138] See also Figure 6 As shown, based on the target policy network obtained by the above model training method, the embodiment of the present application proposes an information display method, which can be executed by the above model training device or by other devices with information display function. The method may include the following steps:

[0139] S601: Obtain an information display request, where the information display request is used to request to display information at a designated information display position.

[0140] In a possible implementation, the information display request is a request sent by an information display client to a corresponding server when an information display position appears in the information processing system, so as to request information displayed on the information display position from the server.

[0141] S602: Acquire multiple candidate information matching the information display request. Specifically, the multiple candidate information may include first type information and second type information. Taking the advertisement display scenario as an example, the first type information is contract advertisement, and the second type information is bidding advertisement.

[0142] S603: Determine the display score of each candidate information through the target strategy network, and the display score is used to indicate the probability of the corresponding candidate information being pushed to the designated information display position for display. Figure 4 The corresponding model training method is used to train the reinforcement learning model.

[0143] In a specific implementation, the status data of each candidate information can be obtained. The status data of any candidate information can be used to indicate the status of the corresponding information at the current moment, and as the information display continues, all or part of the status data of the information in the information display system will also change accordingly. The target policy network can make information display decisions according to the status data of each candidate information and determine the display score of each candidate information. The display score is used to indicate the probability that the corresponding information is pushed to the designated information display position for display. The display score of each information is the score obtained by evaluating the first type of information and the second type of information on the same dimension.

[0144] S604: Obtain target information based on the display score of each candidate information. In a specific implementation, the candidate information with the highest display score can be selected from the candidate information as the target information. In the advertising display scenario, the first type of information is a contract advertisement, and the second type of information is a bidding advertisement. The score of the bidding advertisement can be determined according to the Ecmp of the bidding advertisement, and the score of the contract advertisement can be determined by the target strategy network. The higher the display score of an advertisement, the greater the probability that the advertisement will be pushed to the designated information display position for display. The embodiment of the present application can use the advertisement with the highest score as the target advertisement, and push the target advertisement to the designated advertisement display position for display.

[0145] S605: Push the target information to the designated information display position for display.

[0146] To summarize, the scheme shown in the embodiment of the present application processes the status data of each information through a pre-trained target strategy network to obtain the target information selected from each information. That is to say, for an information display request, a strategy network obtained by reinforcement learning selects one information from two different types of information as the target information for display, thereby realizing mixed control of the two types of information, so that the information display opportunities in the system can be fully utilized, thereby improving the network resource utilization rate of information display.

[0147] Take the solution shown in the embodiment of the present application as an example, which is applied to an advertisement display scenario. Figure 7 As shown, it shows a framework diagram of a mixed display of advertisements involved in an embodiment of the present application. Figure 7 As shown in the figure, the mixing model is in the center, and the system input includes broadcast control parameters, TrackLog exposure data and inventory data. The model gives parameters for bidding targets and contract targets, passes them into the dictionary structure of the feature server (FeatureServer), and finally takes effect in the mixer (mixer).

[0148] Overall, Figure 7 The system framework shown is divided into three parts, namely data processing 71, shuffling model 72, and online system 73. The above parts are introduced one by one below:

[0149] The data processing 71 part includes three modules: data source, data transmission, and data processing, which completes the processing operation from raw data to algorithm input, which is described as follows:

[0150] Inventory data comes from the inventory estimation service, which uses past data to make detailed predictions about the future, accurate to the mapping of each access request (Page View, PV) to each advertisement, and can reflect the inventory volume of each order on a given day. The bipartite graph is calculated based on inventory data. Two data can be obtained through the bipartite graph: the playback probability of the contract advertisement and the playback curve of the day. The former provides a reference for the contract guarantee, and the latter provides the contract's occupied space.

[0151] The supply side is inventory data, expressed through attribute dimensions; the demand side is advertising data, expressed through directional attribute dimensions. The system generates a connection between inventory and advertising by associating the dimensions on the supply side with the directional information on the demand side. Among them, a corresponding relationship diagram between inventory and advertising involved in the embodiment of the present application can be as follows: Figure 8 shown.

[0152] There are two types of logs: one is request-level data track_log, and the other is exposure-level data joined_exposure.

[0153] Through track_log, we can obtain the refined ranking queue of each request. Through the refined ranking queue within a time period and the expected revenue per thousand impressions (Expected Cost Per Mile, ECPM) of all ads in the queue, predicted click-through rate, filtering conditions, support strategies and other data, the reinforcement learning algorithm of the learning device can simulate the online competition environment through this data. If the length of the time period (Δt) is small enough, it can be assumed that the distribution of bidding contracts in the first Δt is the same as or similar to the distribution of bidding contracts in the last Δt.

[0154] Through joined_exposure, we can get which ad is actually exposed for each request, as well as the corresponding billing and ecpm information. The reinforcement learning algorithm can obtain feedback on online advertising through this data.

[0155] The playback control of the contract is affected by a variety of playback control parameters, such as rate (the probability of entering the sorting queue), theta (playback probability), etc., which are key information to assist in adjusting the contract to ensure quality.

[0156] The online system 73 has two parts, one is FeatureServer, referred to as fs, fs is a server (which can be equivalent to the server in the above information processing system), the scores of each advertisement obtained in this application will be transmitted to fs with other parameters (Theta, Rate), and fs will wait for the request of Mixer after integration. The other is Mixer, which is a complex system. The part related to the embodiment of this application is the mixing module. When an ad display request arrives, Mixer will receive the bidding and contract ad queues, and then request the display scores of each ad in it from fs, and obtain the final displayed ad.

[0157] In a possible implementation of an embodiment of the present application, the target strategy network in the mixed scheduling model can process the status data of each advertisement in the bidding and contract advertising queue corresponding to the advertising display request, output the display score of each advertisement, and provide it to the fs of the online system. The mixer pulls the display score of each advertisement from the fs to obtain the final displayed advertisement.

[0158] From the above content, it can be seen that in the advertising display scenario, in the embodiment of the present application, the display of information delivered in two different ways (such as contract advertising and bidding advertising) can be mixed and controlled through the target strategy network. The application scenarios of the above scheme can be as follows:

[0159] The server in the information processing system trains the reinforcement learning model through the above model training method to obtain a target strategy network. Subsequently, when an information display client in the information processing system has an advertising display opportunity, the information display client sends a request to the server to obtain the advertisement; after receiving the request, the server obtains multiple advertisements matching the request from the contract advertisements and the bidding advertisements to form a precision ranking queue; then, the server processes the status data of the advertisements in the precision ranking queue through the above target strategy network to obtain an advertisement with the highest overall revenue from the precision ranking queue, and pushes it to the information display client for display.

[0160] An embodiment of the present application further provides a computer storage medium, in which program instructions are stored. When the program instructions are executed, they are used to implement the corresponding methods described in the above embodiments.

[0161] See also Fig. 9 , is a structural diagram of a model training device in an embodiment of the present application. The model training device in an embodiment of the present application can be set in the above-mentioned model training device, or it can be a computer program (including program code) running in the model training device.

[0162] In one implementation of the device of the embodiment of the present application, the device includes the following structure.

[0163] A sample acquisition unit 90, configured to acquire target sample information from the log data environment through the policy network maintained by the first processing module;

[0164] The model training unit 91 is used to call the second processing module to obtain training data, and perform a first iterative training on the reinforcement learning model through the training data, and write the network optimization parameters obtained after the first iterative training into the network parameter sharing area, wherein the network optimization parameters refer to the parameters obtained after the first iterative training of the policy network included in the reinforcement learning model; the training data is a plurality of target sample information;

[0165] The sample acquisition unit 90 is further configured to acquire the network optimization parameters from the network parameter sharing area through the first processing module, update the parameters of the policy network maintained by the first processing module to the network optimization parameters, and call the updated policy network to acquire target sample information from the log data environment again;

[0166] The model training unit 91 is further configured to call the second processing module to obtain training data, and perform a second iterative training on the reinforcement learning model after the first iterative training using the training data, and obtain a target reinforcement learning model according to the second iterative training;

[0167] The processing unit 92 is used to determine the policy network in the target reinforcement learning model as the target policy network.

[0168] In one embodiment, the first processing module also maintains a corresponding sample sharing area, and the sample acquisition unit 90 is also used to set a memory identifier for the sample sharing area, and the memory identifier includes a readable identifier and a read-forbidden identifier, wherein if the sample sharing area is writing data or the remaining storage space is greater than a specified threshold, the memory identifier of the sample sharing area is a read-forbidden identifier; if the sample sharing area has completed writing data and the remaining space of the sample sharing area is 0, the memory identifier of the sample sharing area is a readable identifier.

[0169] In one embodiment, the first processing module includes a plurality of modules, each of which is used to maintain a corresponding policy network and sample sharing area, and the model training unit 91 is specifically used to:

[0170] Calling the second processing module to poll each sample sharing area, if it is detected that the memory identifier of any sample sharing area is a readable identifier, then determining the any sample sharing area as a readable sample sharing area;

[0171] Acquire a plurality of target sample information from the readable sample sharing area;

[0172] The plurality of target sample information is determined as training data.

[0173] In one embodiment, after acquiring the target sample information from the log data environment, the sample acquisition unit 90 is further configured to write the target sample information into the sample sharing area corresponding to the first processing module in a cyclic writing manner.

[0174] In one embodiment, the data written into the network parameter sharing area and each sample sharing area are all serialized data, and the sample acquisition unit 90 is further specifically used for:

[0175] Serializing the target sample information to convert the target sample information into an array matching a data type corresponding to the sample sharing area;

[0176] The array is written into the sample sharing area corresponding to the first processing module in a circular writing manner.

[0177] In one embodiment, the model training unit 91 is further specifically used to obtain multiple arrays from the readable sample sharing area, deserialize the multiple arrays, and obtain multiple target sample information.

[0178] In one embodiment, the log data environment includes display sample information corresponding to each of multiple historical information display requests, and the display samples corresponding to the display sample information include first type information and second type information; the first type information is information with display number requirements, and the second type information is information that competes for display opportunities through resource investment.

[0179] In one embodiment, the sample acquisition unit 90 is specifically configured to:

[0180] Acquire target display sample information corresponding to the target history information display request from the log data environment;

[0181] Processing the target display sample information through the strategy network maintained by the first processing module to obtain a sample scoring strategy for the target history information display request;

[0182] Determine a target sample and a display score of the target sample from a plurality of candidate samples matching the target history information display request according to the sample scoring strategy;

[0183] According to the target sample and the display score of the target sample, target sample information corresponding to the target history information display request is generated.

[0184] In one embodiment, the processing unit 92 is further configured to:

[0185] Obtaining an information display request, wherein the information display request is used to request to display information at a specified information display position;

[0186] Acquire multiple candidate information matching the information display request;

[0187] Determine a display score of each candidate information through the target strategy network, wherein the display score is used to indicate the probability of the corresponding information being pushed to the designated information display position for display;

[0188] Acquire target information based on the display score of each candidate information;

[0189] The target information is pushed to the designated information display position for display.

[0190] In the embodiments of the present application, the specific implementation of each of the above-mentioned units can refer to the description of the relevant contents in the embodiments corresponding to the aforementioned figures.

[0191] In the embodiment of the present application, the model training device can continuously explore the log data environment through the first processing module to call the policy network maintained by itself during the training of the reinforcement learning model, and obtain the target sample information from the log data environment; continuously obtain multiple target sample information as training data through the second processing module, and iteratively train a complete reinforcement learning model maintained by itself according to the training data, and write the network optimization parameters obtained by the current M iterations of training into the network parameter sharing area for each M iterations of training, until the reinforcement learning model converges, and then stop the model training. Before model training, the model training device can continuously obtain the above-mentioned network optimization parameters from the network parameter sharing area through the first processing module, update the network parameters of the policy network maintained by itself according to the network optimization parameters, and explore the log data environment again according to the updated policy network, and obtain the target sample information from the log data environment, and repeat this cycle until the model stops training, the target reinforcement learning model is obtained, and the policy network in the target reinforcement learning model is determined as the target policy network. By adopting this model training method, the exploration of the log data environment and the training of the reinforcement learning model can be processed in parallel during the training process, which is conducive to improving the training speed of the reinforcement learning model, so as to obtain the target strategy network more quickly to provide the optimal solution for information display and provide decision-making for the subsequent information display.

[0192] See also Fig.10 , is a schematic diagram of the structure of a model training device according to an embodiment of the present application. The model training device according to the embodiment of the present application includes a power supply module and other structures, and includes a processor 10, a storage device 11, and a communication interface 12. The processor 10, the storage device 11, and the communication interface 12 can exchange data, and the processor 10 implements the corresponding model training function.

[0193] The storage device 11 may include a volatile memory, such as a random-access memory (RAM); the storage device 11 may also include a non-volatile memory, such as a flash memory, a solid-state drive (SSD), etc.; the storage device 11 may also include a combination of the above-mentioned types of memory.

[0194] The processor 10 may be a central processing unit (CPU) 10. In one embodiment, the processor 10 may also be a graphics processing unit (GPU) 10. The processor 10 may also be a combination of a CPU and a GPU. In the model training device, multiple CPUs and GPUs may be included as needed to perform corresponding model training.

[0195] The communication interface 12 may include a display (LCD, etc.), a speaker, and the like.

[0196] In one embodiment, the storage device 11 is used to store program instructions. The processor 10 can call the program instructions to implement the various methods involved in the embodiments of the present application.

[0197] In a first possible implementation, the processor 10 of the model training device calls the program instructions stored in the storage device 11 to obtain target sample information from the log data environment through the policy network maintained by the first processing module; calls the second processing module to obtain training data, and performs a first iterative training on the reinforcement learning model through the training data, and writes the network optimization parameters obtained after the first iterative training into the network parameter sharing area, wherein the network optimization parameters refer to the parameters of the policy network included in the reinforcement learning model obtained after the first iterative training; the training data is a plurality of target sample information; obtains the network optimization parameters from the network parameter sharing area through the first processing module, updates the parameters of the policy network maintained by the first processing module to the network optimization parameters, and calls the updated policy network to obtain the target sample information from the log data environment again; calls the second processing module to obtain training data, and performs a second iterative training on the reinforcement learning model after the first iterative training through the training data; obtains the target reinforcement learning model according to the second iterative training, and determines the policy network in the target reinforcement learning model as the target policy network.

[0198] In one embodiment, the first processing module also maintains a corresponding sample sharing area, and the processor 10 is further used to set a memory identifier for the sample sharing area, and the memory identifier includes a readable identifier and a read-forbidden identifier, wherein if the sample sharing area is writing data or the remaining storage space is greater than a specified threshold, the memory identifier of the sample sharing area is a read-forbidden identifier; if the sample sharing area has completed writing data and the remaining space of the sample sharing area is 0, the memory identifier of the sample sharing area is a readable identifier.

[0199] In one embodiment, the first processing module includes a plurality of modules, each of which is used to maintain a corresponding policy network and sample sharing area, and the processor 10 is specifically used to:

[0200] Calling the second processing module to poll each sample sharing area, if it is detected that the memory identifier of any sample sharing area is a readable identifier, then determining the any sample sharing area as a readable sample sharing area;

[0201] Acquire a plurality of target sample information from the readable sample sharing area;

[0202] The plurality of target sample information is determined as training data.

[0203] In one embodiment, after acquiring the target sample information from the log data environment, the processor 10 is further configured to write the target sample information into a sample sharing area corresponding to the first processing module in a cyclic writing manner.

[0204] In one embodiment, the data written into the network parameter sharing area and each sample sharing area are all serialized data, and the processor 10 is further specifically configured to:

[0205] Serializing the target sample information to convert the target sample information into an array matching a data type corresponding to the sample sharing area;

[0206] The array is written into the sample sharing area corresponding to the first processing module in a circular writing manner.

[0207] In one embodiment, the processor 10 is further specifically configured to obtain a plurality of the arrays from the readable sample sharing area, and perform deserialization processing on the plurality of arrays to obtain a plurality of target sample information.

[0208] In one embodiment, the log data environment includes display sample information corresponding to each of multiple historical information display requests, and the display samples corresponding to the display sample information include first type information and second type information; the first type information is information with display number requirements, and the second type information is information that competes for display opportunities through resource investment.

[0209] In one embodiment, the processor 10 is specifically configured to:

[0210] Acquire target display sample information corresponding to the target history information display request from the log data environment;

[0211] Processing the target display sample information through the strategy network maintained by the first processing module to obtain a sample scoring strategy for the target history information display request;

[0212] Determine a target sample and a display score of the target sample from a plurality of candidate samples matching the target history information display request according to the sample scoring strategy;

[0213] According to the target sample and the display score of the target sample, target sample information corresponding to the target history information display request is generated.

[0214] In one embodiment, the processor 10 is further configured to:

[0215] Obtaining an information display request, wherein the information display request is used to request to display information at a specified information display position;

[0216] Acquire multiple candidate information matching the information display request;

[0217] Determine a display score of each candidate information through the target strategy network, wherein the display score is used to indicate the probability of the corresponding information being pushed to the designated information display position for display;

[0218] Acquire target information based on the display score of each candidate information;

[0219] The target information is pushed to the designated information display position for display.

[0220] In the embodiment of the present application, the specific implementation of the above-mentioned processor 10 can refer to the description of the relevant contents in the embodiments corresponding to the aforementioned figures.

[0221] The model training device in the embodiment of the present application can continuously explore the log data environment through the first processing module to call the policy network maintained by itself during the training of the reinforcement learning model, and obtain the target sample information from the log data environment; continuously obtain multiple target sample information as training data through the second processing module, and iteratively train a complete reinforcement learning model maintained by itself according to the training data, and write the network optimization parameters obtained by the current M iterations of training into the network parameter sharing area for each M iterations of training, until the reinforcement learning model converges, and then stop the model training. Before model training, the model training device can continuously obtain the above-mentioned network optimization parameters from the network parameter sharing area through the first processing module, update the network parameters of the policy network maintained by itself according to the network optimization parameters, and explore the log data environment again according to the updated policy network, and obtain the target sample information from the log data environment, and repeat this cycle until the model stops training, and the target reinforcement learning model is obtained, and the policy network in the target reinforcement learning model is determined as the target policy network. By adopting this model training method, the exploration of the log data environment and the training of the reinforcement learning model can be processed in parallel during the training process, which is conducive to improving the training speed of the reinforcement learning model, so as to obtain the target strategy network more quickly to provide the optimal solution for information display and provide decision-making for the subsequent information display.

[0222] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the described program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0223] The above disclosure is only part of the embodiments of the present application, which certainly cannot be used to limit the scope of rights of the present application. Ordinary technicians in this field can understand that all or part of the processes of implementing the above embodiments and making equivalent changes according to the claims of this application are still within the scope of the invention.

Claims

1. A model training device, characterized in that: The model training device is deployed with a first processing module for maintaining a strategy network, a second processing module for maintaining a reinforcement learning model, and a central processing module, wherein: The first processing module is used to obtain target sample information corresponding to the historical information display request from the log data environment through the maintained policy network; The second processing module is used to obtain training data, and perform a first iterative training on the reinforcement learning model through the training data, and write the network optimization parameters obtained after the first iterative training into the network parameter sharing area, wherein the network optimization parameters refer to the parameters obtained after the first iterative training of the policy network included in the reinforcement learning model; the training data is a plurality of target sample information; The first processing module is further used to obtain the network optimization parameters from the network parameter sharing area, and update the parameters of the policy network maintained by the first processing module to the network optimization parameters; and obtain the target sample information from the log data environment again through the updated policy network; The second processing module is further used to obtain training data, and perform second iterative training on the reinforcement learning model after the first iterative training through the training data, and obtain a target reinforcement learning model according to the second iterative training; The central processing module is used to determine the policy network in the target reinforcement learning model as the target policy network, and the target policy network is used to determine the display score of each candidate information matching the information display request, and the display score is used to indicate the probability that the corresponding candidate information is pushed to the information display position corresponding to the information display request for display.

2. A model training method, characterized in that: The method comprises: Obtaining target sample information corresponding to the historical information display request from the log data environment through the policy network maintained by the first processing module; Calling the second processing module to obtain training data, and performing a first iterative training on the reinforcement learning model through the training data, and writing the network optimization parameters obtained after the first iterative training into the network parameter sharing area, wherein the network optimization parameters refer to the parameters obtained after the first iterative training of the policy network included in the reinforcement learning model; the training data is a plurality of target sample information; Acquire the network optimization parameters from the network parameter sharing area through the first processing module, update the parameters of the policy network maintained by the first processing module to the network optimization parameters, and call the updated policy network to acquire target sample information from the log data environment again; Calling the second processing module to obtain training data, and performing a second iterative training on the reinforcement learning model after the first iterative training by using the training data; A target reinforcement learning model is obtained according to the second iterative training, and the policy network in the target reinforcement learning model is determined as the target policy network, and the target policy network is used to determine the display score of each candidate information matching the information display request, and the display score is used to indicate the probability that the corresponding candidate information is pushed to the information display position corresponding to the information display request for display.

3. The method according to claim 2, characterized in that The first processing module also maintains a corresponding sample sharing area, and the method further includes: A memory identifier is set for the sample sharing area, and the memory identifier includes a readable identifier and a read-forbidden identifier. If the sample sharing area is writing data or the remaining storage space is greater than a specified threshold, the memory identifier of the sample sharing area is a read-forbidden identifier; if the sample sharing area has completed writing data and the remaining space of the sample sharing area is 0, the memory identifier of the sample sharing area is a readable identifier.

4. The method according to claim 3, characterized in that The first processing module includes a plurality of modules, each of which is used to maintain a corresponding policy network and sample sharing area, and the calling of the second processing module to obtain training data includes: Calling the second processing module to poll each sample sharing area, if it is detected that the memory identifier of any sample sharing area is a readable identifier, then determining the any sample sharing area as a readable sample sharing area; Acquire a plurality of target sample information from the readable sample sharing area; The plurality of target sample information is determined as training data.

5. The method according to claim 4, characterized in that After obtaining the target sample information from the log data environment, the method further includes: The target sample information is written into the sample sharing area corresponding to the first processing module in a circular writing manner.

6. The method according to claim 5, characterized in that The data written into the network parameter sharing area and each sample sharing area are all serialized data, and the target sample information is written into the target sample sharing area corresponding to the first processing module in a cyclic writing manner, including: Serializing the target sample information to convert the target sample information into an array matching a data type corresponding to the sample sharing area; The array is written into the sample sharing area corresponding to the first processing module in a circular writing manner.

7. The method according to claim 6, characterized in that The acquiring the plurality of target sample information from the readable sample sharing area comprises: Acquire a plurality of the arrays from the readable sample sharing area, perform deserialization processing on the plurality of arrays, and obtain a plurality of target sample information.

8. The method according to claim 2, characterized in that The log data environment includes display sample information corresponding to each of multiple historical information display requests, and the display samples corresponding to the display sample information include first type information and second type information; the first type information is information with display number requirements, and the second type information is information that competes for display opportunities through resource investment.

9. The method according to claim 8, characterized in that The policy network maintained by the first processing module obtains target sample information from the log data environment, including: Acquire target display sample information corresponding to the target history information display request from the log data environment; Processing the target display sample information through the strategy network maintained by the first processing module to obtain a sample scoring strategy for the target history information display request; Determine a target sample and a display score of the target sample from a plurality of candidate samples matching the target history information display request according to the sample scoring strategy; According to the target sample and the display score of the target sample, target sample information corresponding to the target history information display request is generated.

10. The method according to claim 2, characterized in that The method further comprises: Obtaining an information display request, wherein the information display request is used to request to display information at a specified information display position; Acquire multiple candidate information matching the information display request; Determining a display score of each candidate information through the target strategy network, wherein the display score is used to indicate the probability of the corresponding candidate information being pushed to the designated information display position for display; Acquire target information based on the display score of each candidate information; The target information is pushed to the designated information display position for display.

11. The method according to claim 2, characterized in that The reinforcement learning model includes a policy network and a value network, and the first iterative training of the reinforcement learning model using the training data includes: Processing the target sample information in the training data through the strategy network in the reinforcement learning model to obtain the target sample scoring strategy; Determine the evaluation result and value distribution result of the target sample scoring strategy through the value network in the reinforcement learning model; The network parameters of the policy network in the reinforcement learning model are updated according to the evaluation result, and the network parameters of the value network in the reinforcement learning model are updated according to the value distribution result.

12. An information display method, characterized in that: The method comprises: Obtaining an information display request, wherein the information display request is used to request to display information at a specified information display position; Acquire multiple candidate information matching the information display request; Determine a display score for each candidate information through a target strategy network, wherein the display score is used to indicate the probability that the corresponding information is pushed to the designated information display position for display; the target strategy network is obtained by training a reinforcement learning model through the model training method according to any one of claims 2 to 11; Acquire target information based on the display score of each candidate information; The target information is pushed to the designated information display position for display.

13. A model training device, characterized in that: include: A sample acquisition unit, configured to acquire target sample information from a log data environment through a policy network maintained by the first processing module; A model training unit, configured to call the second processing module to obtain training data, and perform a first iterative training on the reinforcement learning model through the training data, and write the network optimization parameters obtained after the first iterative training into the network parameter sharing area, wherein the network optimization parameters refer to the parameters obtained after the first iterative training of the policy network included in the reinforcement learning model; the training data is a plurality of target sample information; The sample acquisition unit is further configured to acquire the network optimization parameters from the network parameter sharing area through the first processing module, update the parameters of the policy network maintained by the first processing module to the network optimization parameters, and call the updated policy network to acquire target sample information from the log data environment again; The model training unit is further used to call the second processing module to obtain training data, and perform second iterative training on the reinforcement learning model after the first iterative training through the training data, and obtain a target reinforcement learning model according to the second iterative training; A processing unit is used to determine the policy network in the target reinforcement learning model as a target policy network.

14. A model training device, characterized in that: The model training device includes a processor and a storage device, which are connected to each other, wherein the storage device is used to store a computer program, the computer program includes program instructions, and the processor is configured to call the program instructions to execute the method according to any one of claims 2-11.

15. A computer storage medium, characterized in that: The computer storage medium stores program instructions, which, when executed, are used to implement the method according to any one of claims 2 to 12.

16. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 2 to 11 is implemented.

Images