Information recommendation model training method, device, computer equipment and storage medium
By obtaining state transfer pairs, usage value and action inference models from offline data sets, the problem of low training efficiency in the existing technology is solved, and the construction and optimization of efficient information recommendation models are realized.
Patent Information
- Application Number
- CN202210108960.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-28
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2042-01-28
AI Technical Summary
Existing information recommendation model training methods require a large amount of online interactive data, resulting in low training efficiency and difficulty in effectively using offline data for model optimization.
By obtaining state transfer pairs from offline datasets, using multiple value reasoning models and action reasoning models for value and action prediction, adjusting model parameters until the training stop condition is reached, thereby building an efficient information recommendation model.
It improves model training efficiency, reduces dependence on online interactive data, improves the accuracy and real-timeness of information recommendations, and reduces energy consumption.
Smart Images

Figure CN114444698B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to an information recommendation model training method, apparatus, computer equipment and storage medium. Background Art
[0002] With the continuous development of science and technology, the current online service platform can provide more convenience for the target object to meet the various needs of the target object. For example, when the online service platform is an online shopping platform, it can display a variety of purchasable goods to the target object. However, with the eruption of Internet service content, the amount of information that can be displayed by the online service platform is also increasing, making it difficult for the target object to determine the content of interest from the numerous information contents. Therefore, it is necessary to use information recommendation models to recommend information to the target object.
[0003] At present, information recommendation models are generally trained based on reinforcement learning, which starts with optimization from a randomly initialized strategy, repeatedly collects target object data and improves performance until the performance of the information recommendation model reaches the target value. However, this learning method requires a large amount of online interactive data, and the need to repeatedly collect target object data also leads to low efficiency in model training. Summary of the invention
[0004] Based on this, it is necessary to provide an information recommendation model training method, device, computer equipment and storage medium that can improve the efficiency of model training in response to the above technical problems.
[0005] A method for training an information recommendation model, the method comprising:
[0006] Acquire a state transition pair corresponding to the action record for the information flow data from the offline data set; the state transition pair is the state change information corresponding to the action record
[0007] Through multiple value reasoning models, respectively predict the value of the state transition pair to obtain the predicted value output by each value reasoning model;
[0008] Performing action prediction on the state transition pair through an action reasoning model to obtain a predicted action corresponding to the state transition pair;
[0009] Based on the predicted value, the predicted action, and the action corresponding to the state transition, each of the value reasoning model and the action reasoning model is adjusted until a training stop condition is reached, and an information recommendation model is determined based on the action reasoning model obtained by stopping the training;
[0010] The information flow data to be recommended is input into the information recommendation model for information recommendation.
[0011] An information recommendation model training device, the device comprising:
[0012] An acquisition module, used to acquire a state transition pair corresponding to an action record for information flow data from an offline data set; the state transition pair is state change information corresponding to the action record;
[0013] The reasoning module is used to perform value prediction on the state transition pair through multiple value reasoning models to obtain the predicted value output by each value reasoning model; perform action prediction on the state transition pair through an action reasoning model to obtain the predicted action corresponding to the state transition pair;
[0014] An adjustment module is used to adjust the value reasoning model and the action reasoning model based on the predicted value, the predicted action and the action corresponding to the state transition until the training stop condition is reached, and determine the information recommendation model based on the action reasoning model obtained by stopping the training; input the information flow data to be recommended into the information recommendation model for information recommendation.
[0015] In one embodiment, the state transition pair includes a preceding state transition pair and a succeeding state transition pair; the acquisition module also includes a data determination module, which is used to determine a target time and a next time adjacent to the target time; obtain a preceding state transition pair corresponding to the target time from an offline data set, and obtain a succeeding state transition pair corresponding to the next time adjacent to the target time from an offline data set.
[0016] In one embodiment, the acquisition module also includes a data determination module, which is used to determine multiple historical times and multiple subsequent times corresponding to the target time; determine the offline data corresponding to each of the historical times from the offline data set, and obtain the target state corresponding to the target time based on the offline data corresponding to each of the historical times; determine the target action adopted in the target state from the offline data set; determine the offline data corresponding to each of the subsequent times from the offline data set, and obtain the successor state corresponding to the target time based on the offline data corresponding to each of the subsequent time periods; and obtain a state transition pair corresponding to the target time by combining the target action, the target state and the successor state.
[0017] In one embodiment, the reasoning module also includes an action reasoning module, which is used to obtain an action reasoning model and a preset action set; through the action reasoning model, the probability value of adopting each action in the preset action set in the state corresponding to the state transition pair is determined; based on the probability value of adopting each action in the preset action set, the predicted action adopted in the state corresponding to the state transition pair is obtained.
[0018] In one embodiment, the state transition pair includes a predecessor state transfer pair and a successor state transfer pair; the prediction value includes a predecessor prediction value corresponding to the predecessor state transfer pair and a successor prediction value corresponding to the successor state transfer pair; the adjustment module is used to adjust the model parameters of each value reasoning model through the predecessor prediction value and the successor prediction value, and adjust the model parameters of the action reasoning model through the predecessor prediction value, the predicted action and the action corresponding to the predecessor state transfer pair; continue to perform value prediction and model parameter adjustment based on the adjusted value reasoning model, and continue to perform action prediction and model parameter adjustment based on the adjusted action reasoning model until the training stop condition is reached, and determine the information recommendation model based on the action reasoning model obtained by stopping the training.
[0019] In one embodiment, the reasoning module also includes a value reasoning module, which is used to obtain a value reasoning model set; the value reasoning model set includes multiple value reasoning models; through each value reasoning model in the value reasoning model set, value reasoning is performed on the preceding state transfer pair in the state transfer pair set to obtain the preceding prediction value output by each value reasoning model; through each value reasoning model in the value reasoning model set, value reasoning is performed on the subsequent state transfer pair in the state transfer pair set to obtain the subsequent prediction value output by each value reasoning model.
[0020] In one embodiment, the reasoning module also includes a parameter adjustment module, which is used to determine the target value based on the subsequent prediction value output by each of the value reasoning models; and adjust the model parameters of each of the value reasoning models according to the differences between the previous prediction values and the target values.
[0021] In one embodiment, the parameter adjustment module is also used to determine the degree of discreteness between the subsequent prediction values output by each of the value reasoning models, and determine the model measurement value based on the degree of discreteness; perform mean operation on the subsequent prediction values output by each of the value reasoning models to obtain the subsequent value mean, and determine the target value based on the model measurement value and the subsequent value mean.
[0022] In one embodiment, the parameter adjustment module is also used to perform a variance operation on the subsequent prediction value output by each of the value inference models to obtain a subsequent variance; obtain a preset training penalty coefficient, and obtain a corresponding model measurement value based on the training penalty coefficient and the subsequent variance.
[0023] In one embodiment, the parameter adjustment module is also used to determine the learning rate corresponding to each value reasoning model in the multiple value reasoning models; for each value reasoning model in the multiple value reasoning models, a first loss function corresponding to the current value reasoning model is obtained according to the difference between the previous predicted value output by the current value reasoning model and the target value; for each value reasoning model in the multiple value reasoning models, the model parameters of the current value model are adjusted by the first loss function and the learning rate corresponding to the current value model.
[0024] In one embodiment, the predicted action is a predicted action corresponding to the preceding state transfer pair; the reasoning module also includes a parameter adjustment module, which is used to take the action in the preceding state transfer pair as a standard action, and obtain an action difference based on the difference between the standard action and the predicted action; obtain a second loss function based on the action difference and the preceding prediction value; and adjust the model parameters of the action reasoning model through the second loss function.
[0025] In one embodiment, the parameter adjustment module is also used to calculate the mean of the preceding prediction values output by each of the value inference models to obtain the mean of the preceding values, and obtain a second loss function based on the action difference and the mean of the preceding values.
[0026] In one embodiment, the information recommendation model training device is also used to obtain information flow data to be recommended and determine the state corresponding to the information flow data to be recommended; determine the recommended action adopted in the state corresponding to the historical information to be processed through the action reasoning model in the information recommendation model; and generate corresponding recommendation information based on the recommended action.
[0027] In one embodiment, the information recommendation model training device is further used to determine the action type to which the recommended action belongs; when the action type to which the recommended action belongs is the target action type, corresponding recommendation information is generated according to the recommended action.
[0028] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps in any one of the information recommendation model training methods provided in the embodiments of the present application are implemented.
[0029] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of any one of the information recommendation model training methods provided in the embodiments of the present application.
[0030] A computer program product or a computer program, which includes computer instructions, which are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the following steps in any one of the information recommendation model training methods provided in the embodiments of the present application.
[0031] The above-mentioned information recommendation model training method, device, computer equipment, storage medium and computer program can obtain state transition pairs from offline data sets, and can predict the value of state transition pairs based on multiple value reasoning models to obtain predicted values, and can predict the action of state transition pairs based on action reasoning models to obtain predicted actions. By obtaining the predicted value and predicted action, each value reasoning model and action reasoning model can be adjusted based on the predicted value, predicted action and the action corresponding to the state transition pair until the training stop condition is reached, and the action reasoning model for stopping training is obtained. By obtaining the action reasoning model for stopping training, an information recommendation model for information recommendation can be obtained based on the action reasoning model. Since the present application can be trained through offline data sets, compared with the traditional method of obtaining a large amount of online interaction data by continuously and repeatedly collecting target object data, and training the model through a large amount of online interaction data, the present application does not need to repeatedly collect target object data, thereby improving the training efficiency of the model. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 A schematic diagram of a flow chart of an information recommendation model training method in one embodiment;
[0033] Figure 2 A training diagram of a value reasoning model in one embodiment;
[0034] Figure 3 is a schematic diagram of training an action reasoning model in one embodiment;
[0035] Figure 4 The corresponding relationship between the prediction value and the training penalty coefficient in one embodiment;
[0036] Figure 5 is a schematic diagram of an information recommendation model in one embodiment;
[0037] Figure 6 A schematic diagram of a training architecture of an information recommendation model in one embodiment;
[0038] Figure 7 It is a flowchart of an information recommendation model training method in a specific embodiment;
[0039] Figure 8 It is a structural block diagram of an information recommendation model training device in one embodiment;
[0040] Fig. 9 It is a structural block diagram of an information recommendation model training device in one embodiment;
[0041] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0042] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0043] The information recommendation model training method is applied to the information recommendation model training system. The recommendation model training system includes a terminal 102 and a server 104. Both the terminal 102 and the server 104 can execute the recommendation model training method of the present application independently, and the terminal 102 and the server 104 can also execute the recommendation model training method of the present application in collaboration. The following is an explanation using the server 104 executing the recommendation model training method of the present application as an example. The server 104 can obtain an offline data set, sample state transition pairs from the offline data set, and train the value reasoning model and the action reasoning model based on the state transition pairs until the training stop condition is reached, thereby obtaining an information recommendation model for information recommendation based on the action reasoning model obtained by stopping the training.
[0044] The server 104 may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc.
[0045] This application relates to the field of artificial intelligence. For example, the above-mentioned recommended model is a machine learning model obtained through training with artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.
[0046] This application also relates to the field of machine learning. For example, the above-mentioned recommendation model is a machine learning model obtained by machine learning. Machine Learning (ML) is a multi-field interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all fields of artificial intelligence. Machine learning and deep learning usually include technologies such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning by teaching.
[0047] It should be noted that the user information (including but not limited to user device information, user personal information, etc.), offline data sets, target object data and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.
[0048] In order to better understand the recommendation model training method in the embodiment of the present application, the overall concept of the present application is introduced below:
[0049] Traditional recommendation models usually generate recommendation information based on target object data, for example, by generating recommendation information through the content browsed by the target object at a certain historical moment. However, generating recommendation information through the content browsed by the target object at a historical moment will produce some vulgar and gimmicky recommendation content, and because traditional recommendation models are difficult to comprehensively consider multiple recommendation motivations, such as the diversity of recommended items, potential conversion rewards, etc., the recommended information is not accurate. The recommendation model obtained by reinforcement learning training in this application can solve the above technical problems. The optimization goal of the recommendation model training method based on reinforcement learning is to maximize the cumulative return of the strategy, which is more in line with the purpose of the recommendation system. The strategy found by reinforcement learning not only considers the immediate reward brought by the action, that is, the reward brought by the recommended item being clicked, converted or collected, but also considers the impact of the recommendation information on the interest of the target object, and finds the optimal strategy within a longer time range.
[0050] However, there are still challenges in implementing a recommendation system based on reinforcement learning. Traditional reinforcement learning methods start with optimization from a randomly initialized strategy, and repeatedly collect trajectories to improve performance. However, this learning method requires a large amount of online interaction data, and the performance in the initial stage of training is low. Therefore, this application trains the recommendation model through an offline data set, avoiding the need to repeatedly collect trajectories and the need for a large amount of online interaction data, thereby improving the training efficiency of the model.
[0051] This application first learns a value reasoning model from offline data that is intended to indicate long-term cumulative benefits, and then integrates the predicted value output by the learned value reasoning model into the action reasoning model as a weight item, so as to train the action reasoning model through the weight item, thereby strengthening the action reasoning model's emphasis on high-value actions. When the trained action reasoning model is obtained, a recommendation model for information recommendation can be obtained based on the trained action reasoning model. Since the value reasoning model obtained through reinforcement learning training cannot directly generate recommended actions, but needs to find the maximum value in the value prediction of the entire action space, it is very time-consuming. Therefore, the value reasoning model in this application is only used to assist the action reasoning model in training, and when the model is used, the value reasoning model is abandoned, and the recommendation model for information recommendation is obtained through the trained action reasoning model. In this way, the real-time performance of the recommended information output by the recommendation model can be improved.
[0052] In one embodiment, Figure 1 As shown, a recommendation model training method is provided, and the method is applied to a computer device as an example for explanation, wherein the computer device may specifically be a terminal 102 or a server 104. The information recommendation model training method includes the following steps:
[0053] Step S102, obtaining a state transition pair corresponding to the action record for the information flow data from the offline data set; the state transition pair is state change information corresponding to the action record.
[0054] The state transition pair refers to a set of data consisting of a state, an action, and a reward function. In one embodiment, the state transition pair can be a multi-tuple, for example, a four-tuple (s t , a t , r, s t+1 ). Among them, s represents the state space, s t refers to the state observed at time t, for example, s t is the state of the target object at time t. a represents the action space, a t refers to the action observed at time t, for example, a t is the action taken by the target object at time t. r is the reward function, which represents the target object in state s t When taking action a t The immediate reward received afterwards. t+1 Refers to taking action a t The state observed later, for example, the target object is in state s t Next, take action a t After that, the state s that can be transferred to t+1 .
[0055] The state transition pair may be a preceding state transition pair or a succeeding state transition pair. The state transition pair set may include a preceding state transition pair and a succeeding state transition pair. There is a time sequence between the preceding state transition pair and the succeeding state transition pair. For example, the preceding state transition pair is (s t , a t , r, s t+1 ), the corresponding successor state transition pair is (s t+1 , a t+1 , r, s t+2 ).
[0056] Information flow data refers to data obtained by collecting the operation information of the target object. For example, with authorization, the browsing, clicking, and favorite information of the target object in an online shopping platform within a period of time can be recorded to obtain information flow data. Action records refer to actions in information flow data, such as click actions and favorite actions of the target object in an online shopping platform within a period of time. In a specific embodiment, the state can be determined based on at least one of the browsing content, click content, and favorite content of the target object in the operation record. The action can specifically be the browsing action, click action, favorite action, etc. of the target object in the operation record.
[0057] Specifically, the computer device may obtain an offline data set, and sample and obtain a state transition pair from the offline data set. The offline data set refers to a collection of multiple offline data, and the offline data set can be obtained by recording the information flow data of multiple target objects. The target object may be a virtual object or a user object. For example, the target object may be an AI virtual object set in a computer device. The computer device may obtain the information flow data of the AI virtual object in a historical time period, and obtain an offline data set based on the information flow data of the AI virtual object in a historical time period. For another example, when the target object is a user object, the computer device may obtain the information flow data of the user object in a historical time period, and obtain an offline data set based on the information flow data of the user object. When the target object is a user object, the offline data set obtained is a data set obtained after authorization by the user.
[0058] In one embodiment, obtaining a state transition pair corresponding to an action record for information flow data from an offline data set includes: obtaining a state transition pair set from the offline data set; the state transition pairs in the state transition pair set include a preceding state transition pair and a succeeding state transition pair having a time sequence;
[0059] The computer device can obtain an offline data set, and sample and obtain a preceding state transfer pair and a succeeding state transfer pair with a time sequence from the offline data set, and use the preceding state transfer pair and the succeeding state transfer pair with a time sequence as a state transfer pair set.
[0060] In one embodiment, the computer device may use the information flow data of the target object as an offline data set.
[0061] In one embodiment, before generating an offline data set through the information flow data of the target object, the terminal can display an authorization interface, so that the target object can grant the computer device the authority to collect browsing information, click information and other information operation records through the authorization interface. Then, the computer device can obtain the target object's browsing information, click information, collection information and other information operation records within a period of time based on the authorization of the target object, and generate a corresponding offline data set based on the acquired information flow data.
[0062] In one embodiment, obtaining a state transition pair corresponding to an action record for information flow data from an offline data set includes: obtaining the offline data set and determining a target time and a next time adjacent to the target time; obtaining a preceding state transition pair corresponding to the target time from the offline data set, and obtaining a subsequent state transition pair corresponding to the next time adjacent to the target time from the offline data set.
[0063] Specifically, the computer device obtains the offline data set and determines the target time and the next time adjacent to the target time. For example, the target time may be determined as t, and the next time adjacent to the target time may be determined as t+1. The target time may be any time before the current time.
[0064] When the target time is determined, the computer device can obtain the state and action corresponding to the target time from the offline data set, for example, obtain the state s corresponding to the target time t t 、Action a t Furthermore, the computer device obtains the state and action corresponding to the next time adjacent to the target time from the offline data set, for example, obtaining the state s corresponding to the target time t+1 t+1 、Action a t+1 The computer device can also determine from the offline data set whether the target object is in state s t+1 When taking action a t+1 The state after t+2 When the status s is obtained t 、Action a t , status t+1 When the computer device is in state s t 、Action a t , status t+1 And the preset reward function r, get the previous state transfer pair corresponding to the target time (s t , a t , r, s t+1 ). When the status s is obtained t+1 、Action a t+1 , status t+2 When the computer device is in state s t+1 、Action a t+1 , status t+2 and the preset reward function r, and obtain the subsequent state transfer pair (s t+1 , a t+1 , r, s t+2 ). The computer device comprehensively obtains the preceding state transition pair and the subsequent state transition pair to obtain a state transition pair set corresponding to the target time.
[0065] In one embodiment, the state transition pair set may be a five-tuple (s t , a t , r, s t+1 , a t+1 ), where (s t , a t , r) is called the previous state transition pair, and the five-tuple (r, st+1 , a t+1 ), called a subsequent state transition pair.
[0066] In one embodiment, the computer device determines the state s t 、Action a t And the preset reward function r, get the previous state transfer pair corresponding to the target time (s t , a t , r). According to the state s t+1 、Action a t+1 and the preset reward function r, and obtain the subsequent state transfer pair (s t+1 , a t+1 , r).
[0067] In one embodiment, the reward function is a binary function of state and action. For example, to determine that the target object is in the current state, when the browsing action is adopted, the reward is determined to be 0.2; when the target object is determined to be in the current state, when the clicking action is adopted, the reward is determined to be 1.
[0068] In one of the embodiments, the offline data set has a correspondence between time and state, and a correspondence between time and action, and the computer device can obtain a state transition set based on the correspondence between time and state, and the correspondence between time and action.
[0069] In the above embodiment, by determining the target time and the next time adjacent to the target time, a preceding state transition pair and a succeeding state transition pair with a time order can be quickly sampled from the offline data set based on the target time and the next time, thereby obtaining a state transition pair set based on the preceding state transition pair and the succeeding state transition pair with a time order.
[0070] In one embodiment, obtaining a state transition pair corresponding to an action record for information flow data from an offline data set includes: determining a plurality of historical times and a plurality of subsequent times corresponding to a target time; determining offline data corresponding to each historical time from the offline data set, and obtaining a target state corresponding to the target time based on the offline data corresponding to each historical time; determining a target action adopted in the target state from the offline data set; determining offline data corresponding to each subsequent time from the offline data set, and obtaining a subsequent state corresponding to the target time based on the offline data corresponding to each subsequent time period; and obtaining a state transition pair corresponding to the target time by combining the target action, the target state, and the subsequent state.
[0071] Specifically, when determining the target time, the computer device may determine multiple historical times corresponding to the target time. For example, when the target time is t, the computer device may determine the past ten time periods [t-10, t-9, ..., t-1], and use the past time periods as historical times. The time length of the time period may be freely set according to demand, for example, the time length may be determined according to the sampling frequency of the offline data, or the time length may be customized. Accordingly, the computer device may also determine multiple subsequent times corresponding to the target time. For example, the computer device may determine the future ten time periods, and use the future time periods as subsequent times. Further, when determining multiple historical times, the computer device may obtain the offline data corresponding to each historical time from the offline data set, and obtain the target state corresponding to the target time according to the offline data corresponding to each historical time. The offline data set includes the corresponding relationship between time and offline data. For example, time (t-10) corresponds to browsing product A, time (t-9) corresponds to clicking product B, time (t-8) corresponds to browsing product C, etc. The offline data may be the object data of the target object collected during the data collection time period.
[0072] Accordingly, when multiple successor times are determined, the computer device may collect offline data corresponding to each successor time from the offline data set, and obtain the successor state corresponding to the target time according to the offline data corresponding to each successor time. Further, the computer device may determine the current action taken by the target object in the current state from the offline data set, for example, determine the target action taken by the target object at the target time, and combine the target action, target state and successor state to obtain the state transition pair corresponding to the target time.
[0073] In one embodiment, when the offline data corresponding to each historical time is obtained, the computer device may splice the offline data corresponding to each historical time to obtain the target state corresponding to the target time. For example, the computer device may splice the browsed content, clicked content, and favorited content in multiple offline data to obtain the target state. Correspondingly, the computer device may also splice the offline data corresponding to each subsequent time to obtain the subsequent state.
[0074] In one of the embodiments, when the target object browses product A at historical time A and collects product C at historical time B, etc., the computer device can splice product A and product C to obtain the target state of the target object at the target time.
[0075] In one embodiment, when determining the target state of the target object at the target time, the computer device may determine the target action taken by the target object in the target state from the offline data set. For example, the computer device may determine from the offline data set that the target object clicked the purchase control of product C at the target time. The target action at this time is clicking the purchase control of product C.
[0076] It is easy to understand that the computer device can generate a preceding state transition pair in the above manner, and can also generate a succeeding state transition pair in the above manner.
[0077] In one embodiment, after obtaining the authorization corresponding to each target object, the computer device can generate an offline data set according to the information flow data corresponding to each target object. For example, the computer device can determine a preset sampling frequency, and determine the browsing information, favorites, and click information of the target object at the corresponding time according to the preset sampling frequency, and obtain the corresponding relationship between time and information according to at least one of the browsing information, favorites, and click information, that is, obtain the corresponding relationship between time and offline data. Therefore, when the target time is determined, the computer device can determine the state and action corresponding to the target time based on the corresponding relationship between time and offline data, and determine the state and action corresponding to the next time.
[0078] In the above embodiment, the target state is obtained through multiple historical times, and the successor state is obtained through multiple subsequent times, so that the determined target state and successor state can include richer state information, thereby improving the accuracy of model training based on the richer state information.
[0079] Step S104, using multiple value reasoning models, respectively perform value prediction on the state transition pairs to obtain the predicted value output by each value reasoning model.
[0080] Specifically, the computer device can input the state transition pair set into each value reasoning model in the value reasoning model set. For each value reasoning model in the value reasoning model set, the value of the state transition pair is predicted by the current value reasoning model to obtain the predicted value output by each value reasoning model.
[0081] The value inference model can be a model related to the parameter θ and including multiple fully connected layers, which can use Bellman Update to update the model parameters. The value inference Q(s, a) is used to predict the expected benefit obtained by taking the action in the state transition pair in the state corresponding to the state transition pair, where the expected benefit is the value. The expected benefit is related to the reward function r. When the target object takes the action in the state transition pair in the state corresponding to the state transition pair, the greater the reward output by the reward function, the greater the expected benefit obtained.
[0082] In one embodiment, the state corresponding to the state transition pair refers to the state with the smallest time in the state transition pair. For example, when the current sequence state transition pair is (s t , a t , r, s t+1 ), the state corresponding to the previous state transition pair is s t ; The current sequence state transition pair is (s t , a t , r), the state corresponding to the previous state transition pair is also s t . Thus, the value reasoning model is t , a t , r, s t+1 ) or (s t , a t , r) When performing value reasoning, it can be determined that in state s t Next, sampling action a t The expectation of the profit that can be obtained when , that is, the previous prediction value is obtained.
[0083] In one embodiment, state transition pairs corresponding to action records for information flow data are obtained from an offline data set, including: obtaining a set of state transition pairs from the offline data set; performing value prediction on the state transition pairs respectively through multiple value reasoning models to obtain prediction values output by each of the value reasoning models, including: obtaining a value reasoning model set; the value reasoning model set includes multiple value reasoning models; performing value reasoning on the preceding state transfer pairs in the state transfer pair set through each value reasoning model in the value reasoning model set to obtain the preceding prediction values output by each value reasoning model; performing value reasoning on the succeeding state transfer pairs in the state transfer pair set through each value reasoning model in the value reasoning model set to obtain the succeeding prediction values output by each value reasoning model.
[0084] Specifically, the present application sets up multiple value reasoning models, and each value reasoning model performs value reasoning on the preceding state transfer pair and the succeeding state transfer pair in the state transfer pair set, and obtains the preceding prediction value and the succeeding prediction value output by each value reasoning model. For example, the i-th value reasoning model can perform value reasoning on the preceding state transfer pair and obtain the preceding prediction value Perform value reasoning on the subsequent state transfer pair to obtain the previous order prediction value Among them, θ is the model parameter of the value reasoning model, and t represents time.
[0085] In one embodiment, reference Figure 2 , when the state corresponding to the previous state transition pair is obtained, that is, when s t When the computer device can t Perform word embedding processing to transform the state s t Convert it into a vector and extract the features of the vector to get s t Furthermore, the value inference model is based on s t The hidden state is used for value reasoning to obtain the previous prediction value Figure 2 A schematic diagram of training a value reasoning model in one embodiment is shown.
[0086] In this embodiment, by setting up multiple value reasoning models, subsequent predicted values can be output based on the multiple value reasoning models to obtain target values used to represent standard values, and then the corresponding value reasoning model can be trained based on the difference between the previous predicted value and the target value.
[0087] Step S106: perform action prediction on the state transition pair through the action reasoning model to obtain a predicted action corresponding to the state transition pair.
[0088] Specifically, the action reasoning model refers to a machine learning model built based on the cross entropy loss function. Through the action reasoning model, the action taken by the target object in the state corresponding to the state transition pair can be determined. At this time, the action determined by the action reasoning model is the predicted action. Among them, similar to the value prediction, the state corresponding to the state transition pair is also the state with the minimum time in the state transition pair.
[0089] In one embodiment, an action prediction is performed on a state transition pair through an action reasoning model to obtain a predicted action to be adopted in a state corresponding to the state transition pair, including: obtaining an action reasoning model and a preset action set; determining, through the action reasoning model, the probability value of adopting each action in the preset action set in the state corresponding to the state transition pair; and obtaining the predicted action to be adopted in the state corresponding to the state transition pair based on the probability value of adopting each action in the preset action set.
[0090] Specifically, the model trainer can pre-set a preset action set, wherein the preset action set includes multiple actions that the target object can take, for example, the preset action set includes browsing actions, click actions, collection actions, etc. that can be taken for the corresponding items. For each action in the action set, the computer device predicts the probability value of the target object adopting the current action in the state corresponding to the state transition pair through the action reasoning model. Furthermore, the computer device obtains the predicted action adopted in the state corresponding to the state transition pair based on the probability value of each action in the predicted action set. For example, the computer device uses the action with the largest probability value as the predicted action.
[0091] In one embodiment, the preset action set may be defined as the labels of all recommended information allowed to appear in the offline data set. For example, when it is allowed to generate corresponding recommendation information for item A and item C, the actions in the preset action set may be clicking item A, collecting item A, browsing item A, clicking item C, collecting item C, and browsing item C.
[0092] In the above embodiment, since reasoning is performed on each action in the preset action set, the embodiment of the present application will perform comprehensive reasoning on each action in the action set, so as to obtain accurate predicted actions based on the reasoning results of the comprehensive reasoning.
[0093] In one embodiment, action prediction is performed on a state transition pair through an action reasoning model to obtain a predicted action corresponding to the state transition pair, including: action reasoning is performed on a preceding state transition pair in a state transition pair set through an action reasoning model to obtain a predicted action adopted in the state corresponding to the preceding state transition pair.
[0094] In one embodiment, reference Figure 3 , the computer device can perform feature extraction processing on the previous state transfer pair to obtain the hidden state of the state corresponding to the previous state transfer pair, and input the hidden state into the action reasoning model to obtain the predicted action output by the action reasoning model. Figure 3 A schematic diagram of training an action reasoning model in one embodiment is shown.
[0095] Step S108, based on the predicted value, predicted action and the corresponding action of the state transition, adjust the value reasoning model and the action reasoning model until the training stop condition is reached, and determine the information recommendation model based on the action reasoning model obtained by stopping the training.
[0096] The action corresponding to the state transition pair may be the action included in the state transition pair.
[0097] Specifically, when the predicted value, predicted action, and state transition corresponding action are obtained, the computer device can adjust the value reasoning model and action reasoning model based on the predicted value, predicted action, and state transition corresponding action until the training stop condition is reached. For example, the computer device can average the predicted value output by each value reasoning model to obtain the predicted value mean, and adjust the model parameters of the corresponding value reasoning model according to the difference between the predicted value and the predicted value mean. And the computer device adjusts the action reasoning model according to the predicted value mean, predicted action, and state transition corresponding action.
[0098] When the training stop condition is reached, the computer device determines the information recommendation model based on the action reasoning model obtained by stopping the training, for example, using the action reasoning model obtained by stopping the training as the information recommendation model.
[0099] Step S110: input the information flow data to be recommended into the information recommendation model for information recommendation.
[0100] Specifically, when a trained information recommendation model is obtained, the computer device can obtain the information flow data to be recommended, and input the information flow data to be recommended into the information recommendation model, so that the information recommendation model outputs corresponding recommendation information. The information flow data to be recommended refers to data obtained based on the operation information of the target object in a historical time period. For example, the information flow data to be recommended can be data obtained based on the content browsed, clicked, or collected by the target object at multiple historical times before the current time.
[0101] The above-mentioned recommendation model training method, device, computer equipment, storage medium and computer program can obtain state transition pairs from offline data sets, and can predict the value of state transition pairs based on multiple value reasoning models to obtain predicted values, and can predict the action of state transition pairs based on action reasoning models to obtain predicted actions. By obtaining multiple predicted values and predicted actions, each value reasoning model and action reasoning model can be adjusted based on the predicted values, predicted actions and actions corresponding to the state transition pairs, until the training stop condition is reached, and the action reasoning model for stopping training is obtained. By obtaining the action reasoning model for stopping training, an information recommendation model for information recommendation can be obtained based on the action reasoning model. Since the present application can be trained through offline data sets, compared with the traditional method of obtaining a large amount of online interaction data by continuously and repeatedly collecting target object data, and training the model through a large amount of online interaction data, the present application does not need to repeatedly collect target object data, thereby improving the training efficiency of the model.
[0102] In one embodiment, the state transition pair includes a preceding state transition pair and a succeeding state transition pair; the predicted value includes a preceding predicted value corresponding to the preceding state transition pair and a succeeding predicted value corresponding to the succeeding state transition pair; based on the predicted value, the predicted action, and the action corresponding to the state transition pair, the model of each value reasoning model and the action reasoning model is adjusted until the training stop condition is reached, and the information recommendation model is determined based on the action reasoning model obtained by stopping the training, including: adjusting the model parameters of each value reasoning model through the preceding predicted value and the succeeding predicted value, adjusting the model parameters of the action reasoning model through the preceding predicted value, the predicted action, and the action corresponding to the preceding state transition pair; continuing to perform value prediction and model parameter adjustment based on the adjusted value reasoning model, continuing to perform action prediction and model parameter adjustment based on the adjusted action reasoning model, until the training stop condition is reached, and determining the information recommendation model based on the action reasoning model obtained by stopping the training. The model parameters of each value reasoning model are adjusted through the preceding predicted value and the succeeding predicted value, and the model parameters of the action reasoning model are adjusted through the preceding predicted value, the predicted action, and the standard action corresponding to the preceding state transition pair.
[0103] Specifically, when the previous prediction value and the subsequent prediction value are obtained, the computer device can generate a target value for representing the standard value through the subsequent prediction value output by each value reasoning model. For each of the multiple value reasoning models, the model parameters of the current value reasoning model are adjusted through the previous prediction value and the target value output by the current value reasoning model. t Perform the recommended action a t Will get an immediate reward r and transfer to the successor state st+1 . When the above process is repeated, a trajectory can be obtained. The optimization goal of the value reasoning model in this application is that the recommendation model obtained through the auxiliary training of the value reasoning model can generate appropriate recommendation actions, thereby maximizing the sum of immediate rewards under the discount factor that can be obtained by the trajectory through the generated recommendation actions. In addition, since the optimization goal of the value reasoning model is no longer simply to maximize the probability of the target object clicking on the recommended information, but on this basis, the value reasoning model is trained by considering the value of the subsequent state to which the current state is transferred after taking the corresponding action. Therefore, the goal of the value reasoning model trained by the successor predicted value is to maximize the long-term satisfaction of the target object.
[0104] The difficulty of training a recommendation model based on reinforcement learning on an offline dataset is that the distribution of data in the offline dataset is narrow and incomplete in the entire state-action space, but the value reasoning model trained on it is required to give predictions on data outside the distribution of the dataset, so that the computer device cannot guarantee the accuracy of the prediction. In order to solve the above technical problems, the present application deploys multiple value reasoning models, and measures the accuracy of the preceding prediction value output by the value reasoning model through multiple subsequent prediction values output by the multiple value reasoning models. Since the accuracy of the preceding prediction value output by the value reasoning model can be measured, training can be performed in the direction of outputting accurate preceding prediction values, thereby solving the over-estimation problem caused by inconsistent distribution of training-test data.
[0105] When the action corresponding to the previous predicted value, predicted action, and previous state transition pair is obtained, the computer device can also obtain the weight item for training the action reasoning model according to the previous predicted value output by each value reasoning model, for example, reference Figure 3 , the computer device can use the mean of the preceding prediction values output by each value reasoning model as a weight item for training the action reasoning model. Furthermore, the computer device determines the action difference between the predicted action and the action corresponding to the preceding state transition, and adjusts the model parameters of the action reasoning model through the weight item and the action difference. Since the preceding prediction value output by the value reasoning model is integrated into the action reasoning model as a weight item, that is, since the action reasoning model is trained through the preceding prediction value, the trained action reasoning model can also increase the importance attached to high-value actions, so that the trained action reasoning model can also output recommended actions with reference to the value of the action. Among them, a high-value action refers to an action that inputs the action into the value reasoning model so that the value reasoning model outputs a value higher than a preset value threshold.
[0106] Furthermore, in order to obtain a high-accuracy value reasoning model and action reasoning model, the computer device may train the value reasoning model and the action reasoning model for multiple cycles, wherein the state transition pair set may include a set of state transition pairs, or may include multiple sets of state transition pairs. A set of state transition pairs refers to a preceding state transition pair and a succeeding state transition pair with a time sequence, for example, (s t , a t , r, s t+1 ) and (s t+1 , a t+1 , r, s t+2 ). The computer device may randomly sample a set of state transition pairs, or use multiple sets of state transition pairs to obtain a set of state transition pairs. In each training cycle, the computer device may adjust the model parameters of the value reasoning model and the action reasoning model through only one set of state transition pairs, or may adjust the model parameters of the value reasoning model and the action reasoning model through multiple sets of state transition pairs. This embodiment is not limited here.
[0107] For example, in the current training cycle, when the state transition pair set includes only one set of state transition pairs, the computer device can obtain the target value corresponding to the set of state transition pairs, multiple preceding prediction values, predicted actions, and actions corresponding to the preceding state transition pairs, and adjust the model parameters of each value reasoning model through multiple preceding prediction values and target values. For example, for each value reasoning model in the value reasoning model set, the computer device obtains the first loss function of the current value reasoning model, and inputs the preceding prediction value and target value output by the current value reasoning model into the first loss function to obtain the first loss, and adjusts the model parameters of the current value model in the direction of minimizing the first loss. Correspondingly, the computer device adjusts the model parameters of the action reasoning model through multiple preceding prediction values, predicted actions, and actions corresponding to the preceding state transition pairs. For example, the computer device obtains the second loss function used to train the action reasoning model, and inputs multiple preceding prediction values, predicted actions, and actions corresponding to the preceding state transition pairs into the second loss function to obtain the second loss, and adjusts the model parameters in the direction of minimizing the second loss.
[0108] In the current training cycle, when the state transition pair set includes multiple groups of state transition pairs, the computer device can generate the target value corresponding to each group of state transition pairs, multiple previous predicted values, predicted actions, and actions corresponding to the previous state transition pairs. For each value reasoning model in the value reasoning model set, the computer device obtains the first loss function of the current value reasoning model, and generates a first loss for each group of state transition pairs through the first loss function. The computer device calculates the expectations of multiple first losses and adjusts the model parameters of the current value reasoning model in the direction of minimizing the expectation of the first loss. Correspondingly, the computer device obtains the second loss function of the action reasoning model, and generates a second loss for each group of state transition pairs through the second loss function. The computer device calculates the expectations of multiple second losses and adjusts the model parameters of the action reasoning model in the direction of minimizing the expectation of the second loss.
[0109] When entering the next training cycle, the computer device can re-sample from the offline data set to obtain a new set of state transition pairs, and continue to train each value reasoning model and action reasoning model based on the new set of state transition pairs until the training stop condition is reached, such as until the training cycle reaches a preset number of cycles, or until the training converges, and a trained action reasoning model and value reasoning model are obtained. Since the value reasoning model is used to assist the training of the action reasoning model, when the trained action reasoning model and value reasoning model are obtained, the computer device determines the information recommendation model based on the trained action reasoning model, and outputs the recommended information through the information recommendation model, while discarding the value reasoning model.
[0110] In the above embodiment, since this embodiment can be trained through offline data sets, compared with the traditional method that needs to obtain a large amount of online interaction data by continuously and repeatedly collecting target object data, and training the model through a large amount of online interaction data, this embodiment does not need to repeatedly collect target object data, thereby improving the training efficiency of the model.
[0111] In addition, since the value representing the long-term benefits of the recommended action can be output through the value reasoning model, the recommendation model obtained by training based on the value reasoning model can be biased towards generating recommended actions with high long-term value, thereby improving the quality of recommended information based on recommended actions with high long-term value, thereby improving the long-term satisfaction of the target object. Since the present invention can accurately output the value of the subsequent state of the target object after taking the corresponding action through the established value reasoning model, the value reasoning model in this application can identify content with huge single-step value but low long-term value (exaggerated titles, false information, unhealthy content, etc.), thereby enabling the recommendation model obtained by training based on the value reasoning model to reduce the probability of generating unhealthy recommendation information. Since the present application only needs to train a single action reasoning model, and the present application has lower requirements for the representation ability of a single action reasoning model, the model parameter amount of the action reasoning model can be controlled. By having fewer action reasoning models and fewer model parameters, the energy consumption during training can be effectively reduced.
[0112] In one embodiment, the model parameters of each value reasoning model are adjusted by the preceding prediction value and the succeeding prediction value, including: determining the target value according to the succeeding prediction value output by each value reasoning model; and adjusting the model parameters of each value reasoning model by the difference between the preceding prediction value and the target value.
[0113] Specifically, in order to solve the over-estimation problem caused by inconsistent distribution of training and test data, the computer device generates a model measurement value for measuring the reliability of the value reasoning model through the subsequent prediction values output by multiple value reasoning models, and obtains the target value for characterizing the value standard value through the model measurement value and each subsequent prediction value. Furthermore, for each value reasoning model in the multiple value reasoning models, the computer device determines the difference between the previous prediction value and the target value output by the current value reasoning model, and adjusts the model parameters of the current value reasoning model according to the difference between the previous prediction value and the target value. For example, the model parameters of the current value reasoning model are adjusted in the direction of reducing the difference between the previous prediction value and the target value.
[0114] In the embodiment of the present application, since multiple value reasoning models are set up, multiple subsequent prediction values output by the multiple value reasoning models can be obtained, so that a target value for representing the value standard value can be generated according to the multiple subsequent prediction values, and the value reasoning model can be trained based on the target value.
[0115] In one embodiment, the target value is determined based on the subsequent prediction values output by each value reasoning model, including: determining the degree of discreteness between the subsequent prediction values output by each value reasoning model, and determining the model measurement value based on the degree of discreteness; performing mean operation on the subsequent prediction values output by each value reasoning model to obtain the mean of the subsequent values, and determining the target value based on the model measurement value and the mean of the subsequent values.
[0116] Specifically, when multiple subsequent prediction values are obtained, the computer device determines the degree of discreteness between the subsequent prediction values, and obtains the model measurement value based on the discreteness, thereby measuring the reliability of the value reasoning model based on the model measurement value. For example, the computer device can perform variance calculation on multiple subsequent prediction values to obtain the variance of the subsequent prediction values, and obtain the model measurement value based on the variance of the subsequent prediction values. Further, the computer device performs mean calculation on the subsequent prediction values output by each value reasoning model to obtain the mean of the subsequent values, and applies the model measurement value to the mean of the subsequent values to obtain the target value. Among them, during the training process, by setting the model measurement value, the penalty intensity can be weakened when the pre-order prediction values output by multiple value prediction models are consistent, and when the variance corresponding to the pre-order prediction values output by multiple value prediction models is greater than the preset variance threshold, the penalty intensity is increased. Thus, the model parameters of the value model are continuously adjusted by the penalty intensity, so that the multiple value reasoning models after the parameter adjustment can output consistent results.
[0117] In one embodiment, the computer device can obtain the target value by the following formula:
[0118]
[0119] Where J is the target value, r is the reward function, γ is the preset discount factor, W(s′, a′) is the model metric value, is the mean value of the successor, N is the number of models of the value reasoning model, Q(s′, a′) is the successor predicted value, s′ is the state corresponding to the successor state transfer pair, a′ is the action in the successor state transfer pair, θ T is the model parameter of the value model, and arg max refers to the input or parameter value when the function outputs the maximum value.
[0120] In the above embodiment, by generating a model metric value, the reliability of the value reasoning model can be measured by the model metric value, so that the model parameters can be adjusted in the direction of obtaining a value reasoning model with high reliability, so that the results output by the value reasoning model after the model parameters are adjusted have high reliability, thereby solving the over-estimation problem caused by inconsistent distribution of training and test data.
[0121] In one embodiment, the degree of discreteness includes the successor variance. Determining the degree of discreteness between the successor prediction values output by each value reasoning model includes: performing a variance operation on the successor prediction values output by each value reasoning model to obtain the successor variance; determining the model measurement value according to the degree of discreteness includes: obtaining a preset training penalty coefficient, and obtaining the corresponding model measurement value according to the training penalty coefficient and the successor variance.
[0122] Specifically, when the subsequent prediction values output by the multiple value reasoning models are obtained, the computer device performs variance calculation on the multiple subsequent prediction values to obtain the subsequent variance. Further, the computer device obtains a preset training penalty coefficient, and obtains the model measurement value according to the training penalty coefficient and the subsequent variance. The training penalty coefficient can be freely set according to needs.
[0123] This application is more concerned with how to use model metrics to efficiently train the model. In the case of a huge action space, it can be considered that the traditional method of training the model by subtracting the variance from the mean is inefficient. Experiments show that the training method proposed in this application using model metrics has faster training efficiency and training effect than the traditional method. Figure 4 As shown in the figure, when training with a random training penalty coefficient that conforms to the normal distribution, the training method proposed in this article can take a larger training penalty coefficient (λ=100) while keeping the predicted value output by the value model in compliance with the prerequisite of being greater than zero, but the traditional method often pushes the predicted value to an illegal range. Figure 4 The correspondence between the prediction value and the training penalty coefficient in one implementation is shown.
[0124] In one embodiment, the computer device may determine the model metric value by the following formula:
[0125]
[0126]
[0127] Where W(s′, a′) is the model metric; Unc(s′, a′) is the variance between the subsequent predicted values, that is, the degree of dispersion; s′ represents the state corresponding to the subsequent state transition pair; a′ represents the action in the subsequent state transition pair; N is the number of value reasoning models; Q i (s′, a′) represents the successor value, and λ is the preset training penalty coefficient.
[0128] In the above embodiment, by generating model metric values, the reliability of the value model can be measured based on the model metric values, thereby constraining the value model to output more conservative and reliable results.
[0129] In one embodiment, model parameters of each value reasoning model are adjusted by the difference between the previous predicted value and the target value, including: determining the learning rate corresponding to each value reasoning model in the multiple value reasoning models; for each value reasoning model in the multiple value reasoning models, obtaining a first loss function corresponding to the current value reasoning model according to the difference between the previous predicted value output by the current value reasoning model and the target value; for each value reasoning model in the multiple value reasoning models, adjusting the model parameters of the current value model by the first loss function and learning rate corresponding to the current value model.
[0130] Specifically, in order to maintain the diversity of value reasoning models and thus increase the credibility of model metrics, a different learning rate can be used for each value reasoning model. The learning rate represents the speed at which information accumulates over time in a neural network. For example, a computer device can randomly determine the learning rate corresponding to each value reasoning model. For another example, a computer device can randomly sample a set of random numbers from a specific distribution and use this set of random numbers as the learning rate of multiple value reasoning models.
[0131] Furthermore, for each value reasoning model in the value reasoning model set, a first loss function corresponding to the current value reasoning model is obtained based on the difference between the previous predicted value and the target value output by the current value reasoning model, and the current value reasoning model is trained through the first loss function and the learning rate of the current value reasoning model.
[0132] In one embodiment, the computer device may determine the first loss function by the following formula:
[0133]
[0134] Among them, E (s,a,r,s′)~D represents the set of state transition pairs sampled from the offline dataset, i represents the i-th value reasoning model in the value reasoning model set, θ represents the model parameters in the value reasoning model, and θ T represents the model parameters in the value reasoning model at time t, Represents the target value, Represents the previous prediction value.
[0135] In the above embodiment, each value reasoning model in the value reasoning model set is trained through consistent target values and inconsistent learning rates, so that each value reasoning model can fit the same regression target at different learning rates, so that for the state transition pairs within the offline data set distribution, the value reasoning model will output the same result, while for the state transition pairs outside the offline data set distribution, the output of the value reasoning model has greater uncertainty. In addition, since the model metric value is used as a weight item to generate the target value, and the target value is used to train each value reasoning model in the value reasoning model set, the model metric value can also be used to achieve the purpose of increasing the penalty intensity when the results output by each value reasoning model are greatly different, so that the trained value reasoning model can output more conservative and accurate results even when facing state transition pairs outside the offline data set distribution.
[0136] In one embodiment, the predicted action is a predicted action corresponding to the preceding state transition pair; the model parameters of the action reasoning model are adjusted through the preceding prediction value, the predicted action and the standard action corresponding to the preceding state transition pair, including: determining the difference between the action in the preceding state transition pair and the predicted action to obtain the action difference; obtaining a second loss function based on the action difference and the preceding prediction value; and adjusting the model parameters of the action reasoning model through the second loss function.
[0137] Specifically, since the action in the previous state transition pair is the action actually taken by the target object in the state corresponding to the previous state transition pair, the action in the previous state transition pair can be used as the standard action, and the difference between the standard action and the predicted action is determined to obtain the action difference, and the second loss function is obtained through the action difference and the previous prediction value. For example, the computer device can determine the mean between multiple previous prediction values, and obtain the second loss function based on the action difference and the mean between multiple previous prediction values. For another example, the computer device can determine the middle value between multiple previous prediction values, and obtain the second loss function based on the middle value and the action difference. For another example, the computer device can randomly determine a previous prediction value from multiple previous prediction values, and obtain the second loss function through the action difference and the randomly determined previous prediction value. Further, the computer device adjusts the model parameters of the action reasoning model through the second loss function.
[0138] In this embodiment, the action reasoning model is trained by using the previous prediction value, so that the trained action reasoning model can pay more attention to high-value actions.
[0139] In one embodiment, a second loss function is obtained based on the action difference and the previous prediction value, including: taking the mean of the previous prediction values output by each of the value inference models to obtain the mean of the previous values, and obtaining the second loss function based on the action difference and the mean of the previous values.
[0140] Specifically, since there are multiple value reasoning models in the value reasoning model set, multiple value reasoning models can be used to perform value reasoning on the same previous state transition pair to obtain multiple previous prediction values. Further, the computer device determines the mean value between the multiple previous prediction values to obtain the previous value mean, and integrates the previous value mean as a weight term into the action difference to obtain the second loss function.
[0141] In one embodiment, the computer device may determine the second loss function by the following formula:
[0142]
[0143] Among them, E (s,a)~D Represents the state transition pairs sampled from the offline dataset. represents the action difference, and Q(s, a) represents the mean of the previous values.
[0144] In this embodiment, since the mean value of the previous order is used as the weight item, the weight item can be made more accurate.
[0145] In one embodiment, after obtaining the information recommendation model, the above method also includes: obtaining the information flow data to be recommended, and determining the state corresponding to the information flow data to be recommended; determining the recommended action adopted in the state corresponding to the information flow data to be recommended through the action reasoning model in the information recommendation model; and generating corresponding recommendation information according to the recommended action.
[0146] Specifically, when it is necessary to output recommended information to the target object through the information recommendation model, the computer device can obtain the target object's to-be-recommended information flow data, where the corresponding state s t The information flow data to be recommended may be the content browsed, clicked, or collected by the target object at multiple historical times before the current time. Further, the computer device splices the content browsed, clicked, or collected at multiple historical times to obtain a state corresponding to the information flow data to be recommended. Figure 5 , after obtaining the state S corresponding to the information flow data to be recommended tt When the computer device is in the state S corresponding to the information flow data to be recommended tt Perform word embedding and feature extraction to obtain the hidden state S to be processed tt, and the hidden state S to be processed tt The input is input into the action reasoning model to obtain the probability value of each action in the preset action set adopted by the target object in the state corresponding to the information flow data to be recommended, and the action with the maximum probability value is used as the recommended action. Figure 5 A schematic diagram of an information recommendation model in an embodiment is shown.
[0147] For example, when the state corresponding to the information flow data to be recommended represents the trajectory of the target object, that is, the target object browsed product A and then switched to browsing product B, and then collected product C after browsing product B, the recommended action output by the action reasoning model can be an action in the preset action set, for example, the target object can click the purchase control of product C, or the target object can browse product B, or the target object can click the purchase control of product A, etc., wherein the recommended action can be specifically determined according to the probability value. Thus, the recommendation model can generate corresponding recommendation information according to the recommended action output by the action reasoning model, for example, when the recommended action is that the target object clicks the purchase control of product C, the corresponding recommendation information generated can be discount information of product C.
[0148] In one of the embodiments, before obtaining the information flow data to be recommended of the target object, the computer device may obtain authorization from the target object, thereby obtaining the historical information to be processed when the authorization is obtained.
[0149] In the above embodiment, by determining the recommendation action to be adopted in the state corresponding to the information flow data to be recommended, generating recommendation information through the recommendation action, and outputting the recommendation information to the target object, the user experience can be improved through the recommendation information.
[0150] In one embodiment, generating corresponding recommendation information according to the recommended action includes: determining the action type to which the recommended action belongs; and generating corresponding recommendation information according to the recommended action when the action type to which the recommended action belongs is the target action type.
[0151] Specifically, the computer device may determine the action type of the generated recommended action, and when it is determined that the action type belongs to the target action type, generate corresponding recommendation information according to the recommended action. The target action type may be freely set according to demand, for example, the target action type may be determined to be a click type.
[0152] In one embodiment, when it is determined that the target action type is a click type, and the recommended action represents that the target object clicks on the purchase control of the product, the computer device can generate recommendation information corresponding to the product. For example, the computer device can filter out the target discount information corresponding to the product from a large number of discount information, and display the target discount information as recommended information to the target object. Since the recommended information can be directly displayed to the target object, the target object can be exempted from filtering out the target information that it wants to view from a large number of information, thereby improving the efficiency of the target object in determining the target information.
[0153] In this embodiment, by outputting recommendation information to the target object, the user experience can be improved through the recommendation information.
[0154] The present application also provides an application scenario, which applies the above-mentioned information recommendation model training method. Specifically, the application of the recommendation model training method in this application scenario is as follows:
[0155] refer to Figure 6 When the online service platform is an online shopping platform, the computer device can obtain information about multiple target objects browsing and purchasing goods through the online shopping platform, and generate an offline data set based on the obtained information. The computer device generates a state s corresponding to the target time through offline data at multiple historical times. t 、Action a t , and determine the state s at the next time adjacent to the target time t+1 、Action a t+1 , and the state s t 、Action a t , status t+1 and action a t+1 The input is sent to the word embedding module, which converts the input information into a vector and then sends the vector to the feature extraction module, which outputs the hidden state s. t and hidden state s t+1 . Further, the computing device passes the hidden state s t and hidden state s t+1 The hidden state s is input into multiple value reasoning models respectively, and the value reasoning model outputs the previous prediction value and the subsequent prediction value, and the target value is obtained through the subsequent prediction value, and the mean of the previous value is obtained through the previous prediction value. t The input is sent to the action reasoning model, and the action reasoning model outputs the predicted action. The computer device trains the value reasoning model through the previous predicted value and the target value. At the same time, the predicted action and action a t And the mean of the previous values are used to train the action reasoning model. Figure 6A schematic diagram of a training architecture of an information recommendation model in one embodiment is shown.
[0156] The present application also provides an application scenario, which applies the above-mentioned information recommendation model method. Specifically, the application of the recommendation model method in this application scenario is as follows:
[0157] When the online service platform is an online shopping platform, the terminal can obtain information about the target object browsing and purchasing goods through the online shopping platform, and generate information flow data to be recommended based on the obtained information. The terminal determines the state corresponding to the information flow data to be recommended, and determines the recommended action to be taken in the state corresponding to the information flow data to be recommended through the action reasoning model in the information recommendation model. When it is determined that the recommended action is to purchase goods, the terminal can generate discount information corresponding to the goods and display the discount information.
[0158] The above application scenarios are merely illustrative illustrations. It is to be understood that the application of the service-related data reporting methods provided in the embodiments of the present application is not limited to the above scenarios.
[0159] In a specific embodiment, referring to Figure 7 ,The information recommendation model training method includes the following steps:
[0160] S702, the computer device determines a target time and a next time adjacent to the target time; obtains a preceding state transition pair corresponding to the target time from an offline data set, and obtains a subsequent state transition pair corresponding to the next time adjacent to the target time from an offline data set.
[0161] S704, the computer device comprehensively obtains the preceding state transition pair and the subsequent state transition pair to obtain a state transition pair set corresponding to the target time.
[0162] S706, the computer device obtains a value reasoning model set; the value reasoning model set includes multiple value reasoning models; through each value reasoning model in the value reasoning model set, the value of the preceding state transfer pair in the state transfer pair set is predicted, and the preceding prediction value output by each value reasoning model is obtained.
[0163] S708, the computer device performs value prediction on the subsequent state transfer pairs in the state transfer pair set through each value reasoning model in the value reasoning model set, and obtains the subsequent prediction value output by each value reasoning model.
[0164] S710, the computer device determines the degree of discreteness between the subsequent predicted values output by each value reasoning model, and determines the model measurement value based on the degree of discreteness; performs mean operation on the subsequent predicted values output by each value reasoning model to obtain the mean of the subsequent values, and determines the target value based on the model measurement value and the mean of the subsequent values.
[0165] S712, the computer device determines the learning rate corresponding to each value reasoning model in the value reasoning model set; for each value reasoning model in the value reasoning model set, a first loss function corresponding to the current value reasoning model is obtained according to the difference between the previous predicted value and the target value output by the current value reasoning model; for each value reasoning model in the value reasoning model set, the model parameters of the current value model are adjusted by the first loss function and learning rate corresponding to the current value model.
[0166] S714, the computer device obtains an action reasoning model and a preset action set; through the action reasoning model, the probability value of adopting each action in the preset action set in the state corresponding to the preceding state transition pair in the state transition pair set is determined; based on the probability value of adopting each action in the preset action set, the predicted action adopted in the state corresponding to the preceding state transition pair is obtained.
[0167] S716, the computer device takes the action in the previous state transfer pair as the standard action, and obtains the action difference based on the difference between the standard action and the predicted action; calculates the mean of the previous predicted values output by each value reasoning model to obtain the previous value mean, and obtains the second loss function based on the action difference and the previous value mean; adjusts the model parameters of the action reasoning model through the second loss function.
[0168] S718, the computer device continues to perform value reasoning and adjust model parameters based on the adjusted value reasoning model, and continues to perform action reasoning and adjust model parameters based on the adjusted action reasoning model, until the training stop condition is reached, and determines the information recommendation model based on the action reasoning model obtained by stopping the training.
[0169] S720, the terminal obtains the information flow data to be recommended, and determines the state corresponding to the information flow data to be recommended; determines the recommended action to be adopted in the state corresponding to the information flow data to be recommended through the action reasoning model in the information recommendation model; generates corresponding recommendation information according to the recommended action.
[0170] In the above-mentioned information recommendation model training method, by obtaining a state transition pair set from an offline data set, value reasoning can be performed on the state transition pair set based on multiple value reasoning models to obtain multiple preceding prediction values and multiple subsequent prediction values, and action reasoning can be performed on the state transition pair set based on an action reasoning model to obtain a predicted action. By obtaining multiple preceding prediction values and multiple subsequent prediction values, the model parameters of each value reasoning model can be adjusted based on the preceding prediction value and the subsequent prediction value. By obtaining the predicted action, the model parameters of the action reasoning model can be adjusted based on the preceding prediction value, the predicted action and the corresponding standard action. By adjusting the model parameters of the value reasoning model and the action reasoning model, value reasoning and model parameter adjustment can be continued based on the adjusted value reasoning model, and action reasoning and model parameter adjustment can be continued based on the adjusted action reasoning model until the training stops when the training stops, and an action reasoning model for stopping training is obtained. By obtaining the action reasoning model for stopping training, a recommendation model for information recommendation can be obtained based on the action reasoning model. Since the present application can be trained through offline data sets, compared to the traditional method of obtaining a large amount of online interaction data by continuously and repeatedly collecting target object data, and training the model through a large amount of online interaction data, the present application does not need to repeatedly collect target object data, thereby improving the training efficiency of the model.
[0171] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0172] In one embodiment, Figure 8 As shown, an information recommendation model training device 800 is provided. The device can adopt a software module or a hardware module, or a combination of the two to become a part of a computer device. The device specifically includes: an acquisition module 802, a reasoning module 804 and an adjustment module 806, wherein:
[0173] The acquisition module 802 is used to acquire the state transition pair corresponding to the action record for the information flow data from the offline data set; the state transition pair is the state change information corresponding to the action record.
[0174] The reasoning module 804 is used to perform value prediction on the state transition pair through multiple value reasoning models to obtain the predicted value output by each value reasoning model; and to perform action prediction on the state transition pair through the action reasoning model to obtain the predicted action corresponding to the state transition pair.
[0175] The adjustment module 806 is used to adjust the value reasoning model and the action reasoning model based on the predicted value, predicted action and the corresponding action of the state transition until the training stop condition is reached, and determine the information recommendation model based on the action reasoning model obtained by stopping the training; input the information flow data to be recommended into the information recommendation model for information recommendation.
[0176] In one embodiment, reference Fig. 9 , the state transition pair includes a preceding state transition pair and a succeeding state transition pair; the acquisition module 802 also includes a data determination module 8021, which is used to determine a target time and a next time adjacent to the target time; obtain a preceding state transition pair corresponding to the target time from the offline data set, and obtain a succeeding state transition pair corresponding to the next time adjacent to the target time from the offline data set.
[0177] In one embodiment, the acquisition module also includes a data determination module 8021, which is also used to determine multiple historical times and multiple subsequent times corresponding to the target time; determine the offline data corresponding to each historical time from the offline data set, and obtain the target state corresponding to the target time based on the offline data corresponding to each historical time; determine the target action adopted in the target state from the offline data set; determine the offline data corresponding to each subsequent time from the offline data set, and obtain the successor state corresponding to the target time based on the offline data corresponding to each subsequent time period; and obtain a state transition pair corresponding to the target time by combining the target action, target state and successor state.
[0178] In one embodiment, the state transition pair includes a predecessor state transfer pair and a successor state transfer pair; the prediction value includes a predecessor prediction value corresponding to the predecessor state transfer pair and a successor prediction value corresponding to the successor state transfer pair; the adjustment module 806 is used to adjust the model parameters of each value reasoning model through the predecessor prediction value and the successor prediction value, and adjust the model parameters of the action reasoning model through the predecessor prediction value, the predicted action and the action corresponding to the predecessor state transfer pair; continue to perform value prediction and model parameter adjustment based on the adjusted value reasoning model, and continue to perform action prediction and model parameter adjustment based on the adjusted action reasoning model until the training stop condition is reached, and determine the information recommendation model based on the action reasoning model obtained by stopping the training.
[0179] In one embodiment, the reasoning module 804 also includes a parameter adjustment module 8042, which is used to determine the target value based on the subsequent predicted value output by each value reasoning model; and adjust the model parameters of each value reasoning model according to the difference between the previous predicted value and the target value.
[0180] In one embodiment, the parameter adjustment module 8042 is also used to determine the degree of discreteness between the subsequent prediction values output by each value reasoning model, and determine the model measurement value based on the degree of discreteness; perform mean operation on the subsequent prediction values output by each value reasoning model to obtain the mean of the subsequent values, and determine the target value based on the model measurement value and the mean of the subsequent values.
[0181] In one embodiment, the parameter adjustment module 8042 is also used to perform variance calculation on the subsequent prediction value output by each value inference model to obtain the subsequent variance; obtain the preset training penalty coefficient, and obtain the corresponding model measurement value based on the training penalty coefficient and the subsequent variance.
[0182] In one embodiment, the parameter adjustment module 8042 is also used to determine the learning rate corresponding to each value reasoning model in the multiple value reasoning models; for each value reasoning model in the multiple value reasoning models, a first loss function corresponding to the current value reasoning model is obtained according to the difference between the previous predicted value and the target value output by the current value reasoning model; for each value reasoning model in the multiple value reasoning models, the model parameters of the current value model are adjusted by the first loss function and the learning rate corresponding to the current value model.
[0183] In one embodiment, the reasoning module 804 also includes an action reasoning module 8043, which is used to obtain an action reasoning model and a preset action set; through the action reasoning model, the probability value of adopting each action in the preset action set in the state corresponding to the state transition pair is determined; based on the probability value of adopting each action in the preset action set, the predicted action adopted in the state corresponding to the state transition pair is obtained.
[0184] In one embodiment, the predicted action is a predicted action corresponding to a preceding state transfer pair; the reasoning module 804 also includes a parameter adjustment module 8042, which is used to take the action in the preceding state transfer pair as a standard action, and obtain an action difference based on the difference between the standard action and the predicted action; obtain a second loss function based on the action difference and the preceding prediction value; and adjust the model parameters of the action reasoning model through the second loss function.
[0185] In one embodiment, the parameter adjustment module 8042 is also used to calculate the mean of the previous prediction values output by each value reasoning model to obtain the mean of the previous values, and obtain the second loss function based on the action difference and the mean of the previous values.
[0186] In one embodiment, the information recommendation model training device 800 is also used to obtain information flow data to be recommended, and determine the state corresponding to the information flow data to be recommended; determine the recommended action adopted in the state corresponding to the information flow data to be recommended through the action reasoning model in the information recommendation model; and generate corresponding recommendation information according to the recommended action.
[0187] In one embodiment, the information recommendation model training device 800 is further used to determine the action type to which the recommended action belongs; when the action type to which the recommended action belongs is the target action type, corresponding recommendation information is generated according to the recommended action.
[0188] For the specific limitations of the information recommendation model training device, please refer to the limitations of the recommendation model training method above, which will not be repeated here. Each module in the above-mentioned information recommendation model training device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0189] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.10 shown.
[0190] The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store information recommendation model training data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an information recommendation model training method is implemented.
[0191] Those skilled in the art will understand that Fig.10The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0192] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.
[0193] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.
[0194] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in the above-mentioned method embodiments.
[0195] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).
[0196] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0197] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.
Claims
1. A method for training an information recommendation model, characterized in that: The method comprises: Acquire a state transition pair corresponding to an action record for information flow data from an offline data set; the state transition pair is state change information corresponding to the action record; the state transition pair includes a preceding state transition pair and a succeeding state transition pair; Through multiple value reasoning models, respectively perform value prediction on the state transfer pair to obtain the prediction value output by each value reasoning model; the prediction value includes the preceding prediction value corresponding to the preceding state transfer pair and the succeeding prediction value corresponding to the succeeding state transfer pair; Performing action prediction on the state transition pair through an action reasoning model to obtain a predicted action corresponding to the state transition pair; Determining a target value according to the subsequent predicted value output by each of the value reasoning models; Adjusting the model parameters of each value reasoning model according to the difference between the previous predicted value and the target value; Determine the difference between the action in the previous state transition pair and the predicted action corresponding to the previous state transition pair to obtain the action difference; Obtaining a second loss function according to the action difference and the previous prediction value, and adjusting model parameters of the action reasoning model by using the second loss function; Continue to perform value prediction and adjust model parameters based on the adjusted value reasoning model, continue to perform action prediction and adjust model parameters based on the adjusted action reasoning model, until the training stop condition is reached, and determine the information recommendation model based on the action reasoning model obtained by stopping the training; The information flow data to be recommended is input into the information recommendation model for information recommendation.
2. The method according to claim 1, characterized in that: The state transition pair includes a preceding state transition pair and a succeeding state transition pair; The step of acquiring a state transition pair corresponding to an action record for information flow data from an offline data set includes: determining a target time, and determining a next time adjacent to the target time; A preceding state transition pair corresponding to the target time is obtained from the offline data set, and a subsequent state transition pair corresponding to a next time adjacent to the target time is obtained from the offline data set.
3. The method according to claim 1, characterized in that: The step of acquiring a state transition pair corresponding to an action record for information flow data from an offline data set includes: Determine a plurality of historical times and a plurality of subsequent times corresponding to a target time; Determine the offline data corresponding to each of the historical times from the offline data set, and obtain the target state corresponding to the target time according to the offline data corresponding to each of the historical times; determining a target action to be taken in the target state from an offline dataset; Determine the offline data corresponding to each of the subsequent times from the offline data set, and obtain the subsequent state corresponding to the target time according to the offline data corresponding to each of the subsequent time periods; The target action, the target state and the successor state are integrated to obtain a state transition pair corresponding to the target time.
4. The method according to claim 1, characterized in that: The step of performing action prediction on the state transition pair by using an action reasoning model to obtain a predicted action corresponding to the state transition pair includes: Obtaining an action reasoning model and a preset action set; Determining, by means of the action reasoning model, a probability value of adopting each action in a preset action set in a state corresponding to the state transition pair; According to the probability value of adopting each action in the preset action set, a predicted action adopted in the state corresponding to the state transition pair is obtained.
5. The method according to claim 1, characterized in that: Determining the target value according to the subsequent predicted value output by each of the value reasoning models includes: Determine the degree of dispersion between the subsequent predicted values output by each of the value reasoning models, and determine the model metric value according to the degree of dispersion; The subsequent predicted values output by each of the value inference models are averaged to obtain the subsequent value mean, and the target value is determined based on the model measurement value and the subsequent value mean.
6. The method according to claim 5, characterized in that The degree of dispersion includes the subsequent variance, and the determination of the degree of dispersion between the subsequent predicted values output by each of the value inference models includes: Performing variance calculation on the subsequent predicted value output by each of the value inference models to obtain the subsequent variance; Determining the model metric value according to the discrete degree includes: A preset training penalty coefficient is obtained, and a corresponding model metric value is obtained according to the training penalty coefficient and the successor variance.
7. The method according to claim 1, characterized in that The adjusting the model parameters of each value reasoning model according to the difference between the previous predicted value and the target value includes: Determine a learning rate corresponding to each value reasoning model in the plurality of value reasoning models; For each value reasoning model among the multiple value reasoning models, a first loss function corresponding to the current value reasoning model is obtained according to the difference between the previous predicted value output by the current value reasoning model and the target value; For each value reasoning model among the multiple value reasoning models, the model parameters of the current value model are adjusted through the first loss function and learning rate corresponding to the current value model.
8. The method according to claim 1, characterized in that: The second loss function is obtained according to the action difference and the previous prediction value, including: The preceding predicted values output by each of the value inference models are averaged to obtain the preceding value mean, and a second loss function is obtained based on the action difference and the preceding value mean.
9. The method according to any one of claims 1 to 8, characterized in that: The step of inputting the information flow data to be recommended into the information recommendation model for information recommendation includes: Acquire information flow data to be recommended, and determine a state corresponding to the information flow data to be recommended; Determining, by means of an action reasoning model in the information recommendation model, a recommended action to be adopted in a state corresponding to the information flow data to be recommended; According to the recommended action, corresponding recommendation information is generated.
10. The method according to claim 9, characterized in that Generating corresponding recommendation information according to the recommended action includes: Determine the action type to which the recommended action belongs; When the action type to which the recommended action belongs is the target action type, corresponding recommendation information is generated according to the recommended action.
11. An information recommendation model training device, characterized in that: The device comprises: An acquisition module, used to acquire a state transition pair corresponding to an action record for information flow data from an offline data set; the state transition pair is state change information corresponding to the action record; the state transition pair includes a preceding state transition pair and a succeeding state transition pair; A reasoning module, configured to respectively perform value prediction on the state transfer pair through multiple value reasoning models to obtain the prediction value output by each value reasoning model; perform action prediction on the state transfer pair through an action reasoning model to obtain the prediction action corresponding to the state transfer pair; the prediction value includes a preceding prediction value corresponding to the preceding state transfer pair and a succeeding prediction value corresponding to the succeeding state transfer pair; An adjustment module is used to determine a target value based on a subsequent predicted value output by each of the value reasoning models; adjust the model parameters of each of the value reasoning models by the difference between the previous predicted value and the target value; determine the difference between the action in the previous state transfer pair and the predicted action corresponding to the previous state transfer pair to obtain an action difference; obtain a second loss function based on the action difference and the previous predicted value, and adjust the model parameters of the action reasoning model by the second loss function; continue to perform value prediction and model parameter adjustment based on the adjusted value reasoning model, and continue to perform action prediction and model parameter adjustment based on the adjusted action reasoning model until the training stop condition is reached, and determine an information recommendation model based on the action reasoning model obtained by stopping the training; input the information flow data to be recommended into the information recommendation model for information recommendation.
12. The device according to claim 11, characterized in that The state transition pair includes a preceding state transition pair and a succeeding state transition pair; the acquisition module also includes a data determination module, which is used to determine a target time and a next time adjacent to the target time; A preceding state transition pair corresponding to the target time is obtained from the offline data set, and a subsequent state transition pair corresponding to a next time adjacent to the target time is obtained from the offline data set.
13. The device according to claim 11, characterized in that The acquisition module also includes a data determination module, which is used to determine a plurality of historical times and a plurality of subsequent times corresponding to the target time; determine the offline data corresponding to each of the historical times from the offline data set, and obtain the target state corresponding to the target time according to the offline data corresponding to each of the historical times; determine the target action adopted in the target state from the offline data set; Determine the offline data corresponding to each of the successor times from the offline data set, and obtain the successor state corresponding to the target time based on the offline data corresponding to each of the successor time periods; and obtain the state transition pair corresponding to the target time by combining the target action, the target state and the successor state.
14. The device according to claim 11, characterized in that The reasoning module also includes an action reasoning module, which is used to obtain an action reasoning model and a preset action set; through the action reasoning model, the probability value of adopting each action in the preset action set in the state corresponding to the state transition pair is determined; based on the probability value of adopting each action in the preset action set, the predicted action adopted in the state corresponding to the state transition pair is obtained.
15. The device according to claim 11, characterized in that The reasoning module also includes a parameter adjustment module, which is used to determine the degree of discreteness between the subsequent prediction values output by each of the value reasoning models, and determine the model measurement value based on the degree of discreteness; perform mean operation on the subsequent prediction values output by each of the value reasoning models to obtain the mean of the subsequent values, and determine the target value based on the model measurement value and the mean of the subsequent values.
16. The device according to claim 15, characterized in that The degree of discreteness includes the successor variance, and the reasoning module also includes a parameter adjustment module, which is used to perform a variance operation on the successor prediction value output by each of the value reasoning models to obtain the successor variance; obtain a preset training penalty coefficient, and obtain a corresponding model measurement value based on the training penalty coefficient and the successor variance.
17. The device according to claim 11, characterized in that The reasoning module also includes a parameter adjustment module, which is used to determine the learning rate corresponding to each value reasoning model in the multiple value reasoning models; for each value reasoning model in the multiple value reasoning models, a first loss function corresponding to the current value reasoning model is obtained according to the difference between the previous predicted value output by the current value reasoning model and the target value; for each value reasoning model in the multiple value reasoning models, the model parameters of the current value model are adjusted by the first loss function and learning rate corresponding to the current value model.
18. The device according to claim 11, characterized in that The reasoning module also includes a parameter adjustment module, which is used to calculate the mean of the previous prediction values output by each of the value reasoning models to obtain the mean of the previous values, and to obtain a second loss function based on the action difference and the mean of the previous values.
19. The device according to any one of claims 11 to 18, characterized in that The information recommendation model training device is also used to obtain information flow data to be recommended and determine the state corresponding to the information flow data to be recommended; determine the recommended action adopted in the state corresponding to the information flow data to be recommended through the action reasoning model in the information recommendation model; and generate corresponding recommendation information according to the recommended action.
20. The device according to claim 19, characterized in that The information recommendation model training device is also used to determine the action type to which the recommended action belongs; when the action type to which the recommended action belongs is the target action type, corresponding recommendation information is generated according to the recommended action.
21. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 are implemented.
22. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.
23. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.
