Training method of reinforcement learning model, information recommendation method and device

Abstract

The disclosure provides a training method of a reinforcement learning model, an information recommendation method and device, and relates to the technical field of computers, in particular to the technical field of artificial intelligence and information flow recommendation. The specific implementation scheme of the training method is: obtaining a historical session behavior sequence of a target user in an information flow recommendation system; dividing the historical behavior sequence into an instant behavior sample and a cross-session behavior sample according to a session boundary state; training a first evaluation network in the reinforcement learning model using the instant behavior sample to optimize instant feedback revenue; training a second evaluation network in the reinforcement learning model using the cross-session behavior sample to optimize cross-session retention revenue; and jointly updating the reinforcement learning model based on the trained first evaluation network and second evaluation network to obtain a trained reinforcement learning model. The embodiments of the disclosure can effectively improve the data fusion accuracy of the computer system when processing multi-period feedback signals and improve the result quality.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to the fields of artificial intelligence and information flow recommendation technology. Background Technology

[0002] Existing recommendation systems often focus on short-term goals such as clicks and viewing time when characterizing user value. Although some reinforcement learning solutions introduce session-level long-term benefit modeling, their modeling granularity is relatively coarse, making it difficult to capture real-time changes in user interest after each refresh. Furthermore, existing solutions lack explicit representation of long-term lifecycle value such as user return visits and retention across sessions. When mixing short- and long-term feedback, the significant differences in their time spans and optimization directions easily lead to interference and gradient conflicts between different reward objectives, making it difficult for the model to converge stably. Therefore, they fail to achieve an effective balance between immediate interactive experience and long-term activity. Summary of the Invention

[0003] This disclosure provides a method for training a reinforcement learning model, an information recommendation method, an apparatus, a device, and a storage medium.

[0004] According to one aspect of this disclosure, a method for training a reinforcement learning model is provided, comprising: Obtain the historical session behavior sequence of the target user in the information flow recommendation system; Based on the session boundary state, the historical behavior sequence is divided into immediate behavior samples and cross-session behavior samples. The first evaluation network in the reinforcement learning model is trained using this immediate behavior sample to optimize the immediate feedback gain; The second evaluation network in the reinforcement learning model is trained using this cross-session behavior sample to optimize cross-session retention gains; The reinforcement learning model is jointly updated based on the first and second evaluation networks after training, resulting in a fully trained reinforcement learning model.

[0005] According to another aspect of this disclosure, an information recommendation method is provided, comprising: When a recommendation request is triggered, the state representation of the target user is obtained; wherein, the state representation includes a fusion feature of the current refresh context features and historical consumption features; The resource weighting action is determined based on the state representation using the policy network in a pre-trained reinforcement learning model; wherein the reinforcement learning model is obtained through the training method of the reinforcement learning model provided in any embodiment of this disclosure; The recommended results are output based on the resource allocation actions described.

[0006] According to another aspect of this disclosure, a training apparatus for a reinforcement learning model is provided, comprising: The sequence acquisition module is used to acquire the historical session behavior sequence of the target user in the information flow recommendation system; The sample segmentation module is used to divide the historical behavior sequence into immediate behavior samples and cross-session behavior samples based on the session boundary state. The first training module is used to train the first evaluation network in the reinforcement learning model using the immediate behavior sample to optimize the immediate feedback gain. The second training module is used to train the second evaluation network in the reinforcement learning model using the cross-session behavior sample to optimize cross-session retention benefits. The joint update module is used to jointly update the reinforcement learning model based on the first and second evaluation networks after training, so as to obtain the trained reinforcement learning model.

[0007] According to another aspect of this disclosure, an information recommendation device is provided, comprising: The representation acquisition module is used to acquire the state representation of the target user when a recommendation request is triggered; wherein, the state representation includes a fusion feature of the current refresh context feature and the historical consumption feature; The representation determination module is used to determine resource weighting actions based on the state representation using the policy network in a pre-trained reinforcement learning model; wherein the reinforcement learning model is obtained through the training method of the reinforcement learning model provided in any embodiment of this disclosure; The output module is used to output recommended results based on the resource allocation action.

[0008] According to another aspect of this disclosure, an electronic device is provided, comprising: At least one processor; and The memory is communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform any of the methods described in the present disclosure.

[0009] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform any of the methods according to embodiments of this disclosure.

[0010] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the methods according to embodiments of this disclosure.

[0011] This disclosure provides a reinforcement learning data processing scheme with stronger generalization ability and higher stability, which can effectively improve the data fusion accuracy of computer systems when processing multi-cycle feedback signals, thereby improving the overall computing power allocation efficiency and result output quality of the system.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is a schematic diagram based on the first embodiment of the present disclosure; Figure 2 This is a schematic diagram according to the second embodiment of the present disclosure; Figure 3 This is a schematic diagram of the structure of a training device for a reinforcement learning model provided according to an embodiment of this disclosure; Figure 4 This is a schematic diagram of the structure of the information recommendation device provided according to the embodiments of this disclosure; Figure 5 This is a block diagram of an electronic device used to implement any of the methods in the embodiments of this disclosure. Detailed Implementation

[0014] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0015] In related technologies, long-term value optimization schemes for recommendation systems often employ static weight fusion based on user historical behavior or conventional reinforcement learning methods based on the overall cumulative reward of a session. While these methods perform reasonably well when handling single optimization objectives or simple short-term interactive tasks (such as clicks, viewing time, etc.), they are ill-suited for complex recommendation scenarios involving the joint optimization of long- and short-term rewards across multiple industrial objectives. This is due to the coarse-grained modeling and lack of detailed characterization of real-time interest changes after each refresh, easily leading to misjudgments of instantaneous feedback and long-term retention value, resulting in a severe imbalance between short-term gains and long-term ecosystem. Furthermore, when handling mixed training of long- and short-term multi-objectives, traditional methods often employ a simple direct reward addition mechanism: relying entirely on fixed-weight reward fusion easily leads to severe gradient conflicts when the time spans of long- and short-term feedback differ greatly; while forcibly introducing complex multi-task network structures may result in mutual constraints on parameter updates between long- and short-term evaluation networks due to the lack of effective sample isolation mechanisms, making it difficult for the reinforcement learning model to converge stably. Furthermore, traditional recommendation architectures lack the ability to adaptively intervene in user groups with different activity levels and the dynamic modulation mechanism of underlying features at the output end, which can easily lead to a one-size-fits-all, crude weighting or a localized degradation of the recommendation experience.

[0016] To at least partially address one or more of the aforementioned problems and other potential issues, this disclosure proposes a training method for reinforcement learning models and an information recommendation method. By constructing a refined sample partitioning mechanism based on session boundary states and introducing an independent training architecture with dual evaluation networks possessing mutually exclusive mask labels, and through joint modeling that integrates cross-session state and refresh-level features, it achieves physical isolation optimization and smooth joint updates for immediate feedback gains and cross-session retention gains. This method significantly improves the model convergence speed and stability during multi-period objective mixed training, solves the pain point of mutual interference between long-term and short-term feedback in traditional multi-objective reinforcement learning, and enhances the dynamic adaptability of the unified model when facing different user behavior patterns (such as high, medium, and low activity levels) by introducing an independent modulation network based on user activity features for pre-weighted intervention of state representation. This achieves a unified improvement in single-interaction click conversion, cross-session lifecycle retention management, and personalized re-ranking intervention for each user.

[0017] This disclosure provides a training method for a reinforcement learning model and an information recommendation method, which can be applied to a training device for a reinforcement learning model or an information recommendation device. The device is located in an electronic device. This electronic device includes, but is not limited to, fixed devices and / or mobile devices. For example, fixed devices include, but are not limited to, servers, which can be cloud servers, ordinary servers, or distributed data processing nodes. Mobile devices include, but are not limited to, smartphones, tablets, and operation management terminals. In some possible implementations, the training method for the reinforcement learning model and the information recommendation method can also be implemented by a processor calling computer-readable instructions stored in memory.

[0018] The following describes in detail the implementation methods of this disclosure in conjunction with application scenarios and examples.

[0019] Figure 1 This is a flowchart illustrating the training method of the reinforcement learning model provided in this embodiment. Figure 1 As shown, the method includes the following steps: S110. Obtain the historical session behavior sequence of the target user in the information flow recommendation system.

[0020] In this embodiment, the historical session behavior sequence can be understood as an ordered set of interaction events between the user and the system over a past period. In specific implementations, this sequence feature can be extracted and constructed using a temporal neural network (such as LSTM or GRU). Alternatively, the self-attention mechanism of the Transformer architecture can be used to capture global dependencies in historical events. By acquiring long-term, high-dimensional behavioral data, rich environmental state representations are provided for the reinforcement learning model, enabling the model to go beyond predicting single-point behaviors and characterize user features from the perspective of the entire lifecycle.

[0021] For example, in short video distribution scenarios, this historical session behavior sequence can be a timestamp log sequence of user actions such as swiping, liking, and commenting over the past 7 days. As a specific implementation example, if applied to an e-commerce product waterfall layout, this sequence can be represented as an ordered combination of the category IDs of different product cards clicked by the user and the duration of their dwell time.

[0022] S120. Based on the session boundary state, divide the historical behavior sequence into immediate behavior samples and cross-session behavior samples.

[0023] In this embodiment, the session boundary state is used to identify the start and end points of the time window in which a user continuously uses the system. In specific implementations, sessions can be segmented by setting a fixed time interval threshold (e.g., inactivity exceeding 30 minutes). Alternatively, session boundaries can be hard-trimmed based on application foreground / background switching events (e.g., app launch and exit signals). By splitting sessions into immediate and cross-session samples, the mixing of short-term and long-term feedback is removed at the data level, avoiding mutual masking between features.

[0024] Taking a specific application scenario as an example, when a user opens the app and browses for 10 minutes during their morning commute before exiting, the data from those 10 minutes generates an immediate behavioral sample. In another example, the interval between the user exiting the app in the morning and reopening it in the evening will be tagged and extracted as a cross-session behavioral sample.

[0025] S130. Use the immediate behavior sample to train the first evaluation network in the reinforcement learning model to optimize the immediate feedback gain.

[0026] In this embodiment, the first evaluation network (typically a Critic network structure) is designed to focus on fitting the probability of positive user feedback within a short period. Specifically, a multilayer perceptron (MLP) can be used as the underlying structure. In alternative embodiments, a deep residual network (ResNet) structure can be employed to mitigate the vanishing gradient problem, thereby uncovering the nonlinear patterns of immediate feedback at a deeper level. By absorbing only immediate behavioral samples, the first evaluation network eliminates the interference of long-term noise, enabling it to make extremely high-precision value judgments (Q-value estimations) of the instantaneous reward of a single action.

[0027] Specifically, the first evaluation network outputs a scalar value representing the user's current satisfaction. Extending this further, in music recommendation software, the immediate feedback benefit of this network optimization could be the complete play rate of a single song.

[0028] S140. Use the cross-session behavior sample to train the second evaluation network in the reinforcement learning model to optimize cross-session retention benefits.

[0029] In this embodiment, the topology of the second evaluation network may be the same as or different from that of the first evaluation network, but its optimization objective is isolated to fit the retention value over a long period. In an alternative embodiment, to better handle long time spans, the second evaluation network may incorporate a graph neural network with a time decay mechanism to process cross-session topology graph data. This decoupled dual-network design overcomes the bottleneck of a single network easily getting trapped in local optima when dealing with multi-objective conflicts.

[0030] In one example, the optimization goal of the second evaluation network is to minimize the waiting time for users to return to the application. For instance, optimizing cross-session retention benefits means that users change from "opening the app once a week" to "opening it twice a day".

[0031] S150. The reinforcement learning model is jointly updated based on the first and second evaluation networks after training to obtain the trained reinforcement learning model.

[0032] In this embodiment, joint update refers to using the results of multiple value assessments to guide the evolution of the policy backbone (Actor). In implementation, a deterministic policy gradient algorithm can be used. In another embodiment, the pruning mechanism in a proximal policy optimization algorithm can also be used to achieve a smoother joint update. This step constructs a complete feedback loop, ensuring that the final model possesses both excellent short-term interaction induction capabilities and the restraint to protect the long-term active user ecosystem.

[0033] As a concrete implementation example, during the joint update process, the system will weight and fuse the 10 points output by the first evaluation network and the 8 points output by the second evaluation network to generate a comprehensive gradient for adjusting the recommendation action strategy.

[0034] According to the scheme of this disclosure embodiment, by introducing session boundary division and decoupling training of dual evaluation networks, the problem of gradient conflict between long and short reward signals in the same network is solved. The objective effect is that the model significantly reduces the cross-session churn rate while maintaining the click-through rate without degradation.

[0035] In one possible implementation, step S120 above includes the following steps: S121. Extract the data corresponding to a single refresh action in the historical session behavior sequence as an instant behavior sample.

[0036] In this embodiment, a single refresh action is an active or passive triggering event of the user acquiring a new batch of information. In specific implementation, a user's finger swiping down can be identified as a refresh. In another embodiment, for automatic continuous playback scenarios, the action of automatically and seamlessly switching streams after video playback is completed can also be recognized by the system as a refresh mechanism. This fine-grained action extraction makes the reinforcement learning interaction environment closer to the user's actual sensory experience cycle.

[0037] For example, if a user scrolls through 5 articles on a page, the interaction data of these 5 articles is packaged to form a real-time behavior sample.

[0038] S122. Extract the cross-session refresh action combination between two adjacent sessions as a cross-session behavior sample.

[0039] In this embodiment, the cross-session refresh action connects two time-disconnected usage periods. At the data processing level, global timestamp sorting and extraction can be performed based on the user ID. In an alternative embodiment, hash association extraction can also be performed based on the session unique identifier (Session ID) in a distributed database. This step establishes a data bridge between isolated session islands, enabling the model to predict "offline time."

[0040] Taking a specific application scenario as an example, the user's last refresh action before going to sleep last night and the first refresh action after waking up this morning are combined to form a cross-session behavior sample used to characterize the retention pattern of the sleep cycle.

[0041] According to the solution of this disclosure embodiment, by precisely defining the extraction rules of single refresh and cross-session action combination, the problem of local interest loss caused by coarse-grained modeling is solved, and accurate slicing of complex behavioral features in continuous spatiotemporal space is achieved.

[0042] In one possible implementation, step S121 above includes the following steps: S1211. Obtain the cumulative consumption behavior features of each recommended resource exposed within this single refresh action, and use them as the refresh-level features of this instantaneous behavior sample. These cumulative consumption behavior features are obtained by summing the interaction values ​​of multiple resources generated within the same refresh action.

[0043] In this embodiment, the refresh-level feature is an aggregation of the overall feedback from all resources issued in the same request. In practical implementation, besides performing a simple linear summation of the values, an alternative embodiment can use an exponentially weighted summation method to assign greater weight to resource interactions ranked higher, or a multimodal vector concatenation followed by dimensionality reduction can be used to construct the cumulative feature. This operation elevates the discrete feedback from individual resources to a "screen-level" macroscopic feedback, effectively filtering out environmental reward noise caused by fluctuations in the quality of individual content.

[0044] In one example, if three videos are released in a single refresh, and the user watches them for 10 seconds, 0 seconds, and 20 seconds respectively, the summed cumulative feature value is 30 seconds. Specifically, if a like occurs, the discrete like boolean values ​​(1, 0, 1) can be summed to a value of 2, collectively forming the refresh-level feature.

[0045] According to the solution of this disclosure embodiment, by summing the interactive values ​​of single-exposure resources to construct cumulative features, the problem of excessive variance in feedback of single resources is solved, and the smoothness and high confidence expression of state features are achieved.

[0046] In one possible implementation, step S122 above includes the following steps: S1221. Determine the last refresh action of the previous session in the historical session behavior sequence, and the first refresh action of the current session.

[0047] In this embodiment, the last and first refresh actions constitute the "anchor points" for user retention. This can be implemented by detecting the session end flag using a sliding window. Alternatively, state machine transition logic can be used to automatically capture these two anchor point actions. Clearly defining the anchor points can significantly reduce the computational load of invalid data.

[0048] For example, the system log analysis engine scans the timeline and locates the last network request command before exiting the app, determining it as the last refresh action.

[0049] S1222. The data corresponding to the last refresh action and the first refresh action are concatenated to form the cross-session state feature of the cross-session behavior sample.

[0050] In this embodiment, feature concatenation is a method for fusing data from different spatiotemporal regions. In a basic embodiment, concatenation can be performed directly at the end dimension of the tensor. In a more complex alternative, a cross-attention mechanism can be used to interactively weight the feature vectors of two actions, thereby uncovering the implicit logical connection between the two refreshes. This provides a powerful input directly pointing to the regression intent for the subsequent evaluation network.

[0051] As a concrete implementation example, the system directly combines the interest vector [0.1, 0.5] left from the last refresh with the environment vector [0.8, 0.2] brought in from the first refresh to form [0.1, 0.5, 0.8, 0.2], which is then input into the subsequent network.

[0052] According to the solution of this disclosure embodiment, by accurately locating and splicing the anchor point action data at the beginning and end of the session, the problem of feature distortion over a long period of time is solved, and a compact and efficient representation of cross-session states is achieved.

[0053] In one possible implementation, step S130 above includes the following steps: S131. Extract the click data and / or viewing duration data from the instant behavior sample.

[0054] In this embodiment, extracting specific interaction metrics is for quantifying instantaneous satisfaction. This can be achieved by parsing log tracking fields. In another embodiment, a streaming computing framework (such as Flink) can be used for real-time parsing and data cleaning.

[0055] S132. Calculate the refresh level instant reward based on the click data and / or the viewing duration data.

[0056] In this embodiment, reward calculation is a core step in transforming business metrics into reinforcement learning optimization directions. This can be implemented using a normalized linear weighted formula. Alternatively, a logarithmic or sigmoid function can be used to truncate and penalize extreme viewing durations to prevent overfitting.

[0057] From an external perspective, if the click data weight is set to 0.3 and the viewing time (mapped to 0 to 1) is set to 0.7, the system will use a weighted formula to calculate the final refresh level instant reward value as 0.85.

[0058] S133. Maximize the instantaneous reward of the refresh level using the first evaluation network to update the network parameters of the first evaluation network.

[0059] In this embodiment, maximizing the reward is achieved through gradient ascent or by minimizing the loss function. Specifically, the Adam optimizer can be used for parameter updates. Alternatively, the RMSProp optimizer can be used to adapt to changes in the learning rate for different feature dimensions.

[0060] In one example, the network calculates the mean squared error between the current predicted value and the target reward of 0.85, and then fine-tunes its own weighted nodes through backpropagation so that the next prediction is closer to 0.85.

[0061] According to the solution of this disclosure embodiment, by quantifying clicks and duration and maximizing immediate rewards, the problem of lack of short-term feedback guidance in the model is solved, and the experience of a single interaction is effectively enhanced.

[0062] In one possible implementation, step S140 above includes the following steps: S141. Obtain the actual cross-session time interval corresponding to the cross-session behavior sample.

[0063] In this embodiment of the disclosure, the actual cross-session time interval represents the true physical lapse time. This can be obtained by calculating the difference between two timestamps. Alternatively, the absolute physical time can be mapped to a relative time interval level that has been logarithmically smoothed (e.g., level 1 represents half an hour, level 2 represents half a day).

[0064] For example, the system subtracted the timestamps to find that the user had been offline for 14,400 seconds (4 hours).

[0065] S142. Predict the expected return probability corresponding to the cross-session behavior sample using a preset regression classification model.

[0066] In this embodiment, a regression classification model is introduced to mitigate the noise caused by the randomness of single real offline times. In implementation, tree models such as XGBoost can be used for probability prediction. In alternative embodiments, a deep survival analysis model can also be used to predict the probability density function returned at a specific time point.

[0067] As a concrete example of implementation, the regression model, based on the user's historical habits, predicts an 85% probability that the user will reopen the application within 4 hours.

[0068] S143. Calculate the long-term cross-session reward based on the actual cross-session time interval and the expected return probability.

[0069] In this embodiment of the disclosure, combining the actual value with the expected probability can generate a long-term supervision signal with higher confidence. Specifically, this can be achieved by multiplying the inverse of the time interval by the probability value. Alternatively, a Bayesian smoothing formula can be introduced, using the probability as a prior and the time interval as the posterior of the observation to calculate the final reward.

[0070] Specifically, the shorter the actual interval and the higher the expected probability, the smaller the long-term reward value (or the less negative penalty) will be calculated through the set inverse proportional function.

[0071] S144. Minimize the long-term reward across sessions using the second evaluation network to update the network parameters of the second evaluation network.

[0072] In this embodiment of the disclosure, for the time span attribute, the minimization operation is a guiding strategy to encourage users to return as quickly as possible. The implementation also relies on backpropagation based on gradient descent. In an alternative embodiment, delayed updates to the target network can be incorporated to maintain numerical stability during the minimization process.

[0073] In one example, the second evaluation network tries to predict and approximate the reward value that is expected to be minimized by adjusting the weights of its neuron connections.

[0074] According to the solution of this disclosure embodiment, by integrating the real time interval and the model prediction probability to construct and minimize the long-term reward, the technical defects of low confidence and high noise in long-term retention rewards are solved, and the objective effect of guiding users to return at high frequency is achieved.

[0075] In one possible implementation, the training process of this reinforcement learning model further includes the following steps: S160. Assign a first mask label to the instantaneous behavior sample in the experience replay pool, and assign a second mask label that is mutually exclusive with the first mask label to the cross-session behavior sample.

[0076] In this embodiment, a mask mechanism is used to achieve sample routing and isolation during training with mixed batch data. At the underlying implementation level, the first mask label can be set to [1, 0] in the matrix, and the second mask label can be set to [0, 1]. In an alternative embodiment, a Boolean index array can also be used to directly slice and label the tensor at the memory level.

[0077] From an external perspective, among the 128 mixed samples extracted from the experience replay pool, 80 ordinary refresh data were labeled with the first mask, and 48 data that were connected end to end were labeled with the second mask.

[0078] S170. During the backpropagation calculation of the reinforcement learning model, the effective backpropagation sample corresponding to the first evaluation network is determined according to the first mask label, so that the first evaluation network updates its gradient only based on the immediate behavior sample.

[0079] In this embodiment, the flow of invalid gradients can be physically blocked through mask multiplication. Specifically, the calculated full gradient matrix is ​​multiplied element-wise with the first mask tensor, causing the gradients of cross-session samples in the first evaluation network to be cleared to zero. In an alternative embodiment, graph computation of invalid samples can also be skipped through conditional branch control flow (such as tf.cond in deep learning frameworks).

[0080] For example, when the loss function returns the error, the mask labels [1, 0] ensure that only the errors of those 80 immediate samples are involved in the modification of the weights in the first evaluation network.

[0081] S180. Determine the valid return samples corresponding to the second evaluation network based on the second mask label, so that the second evaluation network performs gradient updates only based on the cross-session behavior samples.

[0082] In this embodiment, this is a mirror operation of the masking mechanism on another evaluation branch. The implementation and alternative embodiments correspond to step S170.

[0083] As a concrete implementation example, due to the mutual exclusivity of labels, the gradients of the remaining 48 cross-session samples are perfectly imported into the second evaluation network, achieving dedicated network training without interference.

[0084] According to the scheme of this disclosure, by introducing mutually exclusive mask labels to accurately isolate gradient backpropagation paths during backpropagation, the problem of mutual interference during multi-cycle target mixed training is completely solved, resulting in a significant improvement in model convergence speed and training stability.

[0085] In one possible implementation, the reinforcement learning model includes an online policy network and a target policy network; step S150 above includes the following steps: S151. Combine the immediate action value of the first evaluation network output with the long-term action value of the second evaluation network output to construct a target loss function.

[0086] In this embodiment, integrating immediate and long-term value is key to a unified strategy exit. In implementation, this can be achieved by linearly summing the values ​​of both with fixed hyperparameter weights. In an alternative embodiment, an adaptive attention mechanism can be introduced, allowing the model to automatically and dynamically allocate the proportion of immediate and long-term value based on the depth of the current session, thereby constructing a more flexible target loss function.

[0087] Taking a specific application scenario as an example, the loss function The action value Q is the score given to the policy network's output action in the evaluation network.

[0088] S152. Use the objective loss function to update the parameters of the online policy network in the reinforcement learning model.

[0089] In this embodiment, the Online Actor (OA) is the main entity responsible for real-time interaction and exploration tasks. Gradient ascent is performed by calculating the partial derivatives of the target loss function with respect to the OA parameters. Alternatively, a policy entropy regularization term can be added to encourage the OA to maintain the randomness and diversity of action outputs during parameter updates.

[0090] Specifically, based on the updated parameters, the online strategy network will output a combination of weighting actions that are more inclined to take into account both high click-through rates and fast return rates when faced with the same user status.

[0091] S153. Using a preset smooth update rate, the parameters of the updated online policy network are softly updated to the target policy network in the reinforcement learning model.

[0092] In this embodiment of the disclosure, a soft update mechanism is used to prevent oscillating divergence in value estimation. In specific implementations, a very small update rate parameter (e.g., ...) is typically used. An exponential moving average is assigned. In an alternative embodiment, periodic hard updates can also be used, where the parameters are fully replaced every fixed number of training steps.

[0093] For example, after each training session, the target policy network retains 99.5% of the old parameters and only absorbs 0.5% of the new parameters from the online policy network, thus maintaining the target's slow and stable movement.

[0094] According to the scheme of this disclosure embodiment, by combining an online / target dual network architecture with a smooth soft update mechanism to jointly optimize short-term and long-term values, the cold start collapse problem caused by the violent oscillation of a single-policy network is solved, and the high smoothness of policy evolution is achieved.

[0095] Figure 2 This is a flowchart illustrating the information recommendation method provided in this embodiment of the disclosure. Figure 2 As shown, the method includes at least the following steps: S210. Upon triggering a recommendation request, obtain the target user's state representation. This state representation includes a fusion of current refresh context features and historical consumption features.

[0096] In this embodiment, the information recommendation method is used in the actual application stage of the inference deployment. The trigger request can be a user swiping down the screen to request new content. State representation acquisition is the process of fusing real-time context with historical baselines. In a specific implementation, the two sets of features can be dimensionally aligned and then added together using a fully connected layer. Alternatively, bilinear pooling can be used to extract higher-order multiplicative interaction information between the context and historical features.

[0097] As a concrete example of implementation, when a user pulls down to refresh on a rainy weekend evening, the weather and time are used as contextual features, which are deeply integrated with the user's historical preference for science fiction movies.

[0098] S220. Using the policy network in a pre-trained reinforcement learning model, determine the resource weighting action based on the state representation. The reinforcement learning model is obtained through the training method described in any of the above embodiments.

[0099] In this embodiment, the trained policy network can directly map the optimal response action based on the high-dimensional state. Since the model has already incorporated the gradient essence of the long short-term evaluation network during training, only a single forward propagation is needed during inference. In an alternative embodiment, the output action can be scaled to a specified continuous spatial range using an activation function.

[0100] Furthermore, after receiving the fused features, the policy network outputs a vector containing multiple floating-point numbers, such as [1.2, 0.9, 1.05], for subsequent business intervention.

[0101] S230. Output the recommended results based on the resource allocation action.

[0102] In this embodiment, the final action needs to be applied to the actual resource list. This can be done through a simple multiplication strategy applied to the existing score. Alternatively, the action can be treated as an independent scoring channel and integrated with the existing base score using a multi-objective ranking mechanism.

[0103] In one example, this action boosted the weight of comedy-related resources by 1.2 times, ultimately resulting in comedy cards appearing significantly earlier in the recommendation waterfall returned to the user.

[0104] According to the solution of this disclosure, by introducing a pre-trained reinforcement learning model with a fusion of long and short-term perspectives at the inference end to output weighting actions, the problem of rigid and inflexible traditional recommendation rules is solved, and dynamic personalized distribution with more timely response and a longer-term perspective is achieved.

[0105] In one possible implementation, an information recommendation method further includes the following steps: S211. Obtain the activity characteristics of the target user.

[0106] In this embodiment, the activity feature is used to characterize the user's lifecycle stage and stickiness within the platform. Specifically, a continuous value can be obtained by calculating the number of active days (DAU / MAU ratio) of the user in the past 30 days. In an alternative embodiment, users can also be directly labeled with discrete "high activity", "medium activity", and "low activity" group features based on clustering algorithms.

[0107] For example, the system queries the user profile database and extracts the characteristic attributes of the target user as a "low-activity user who has not logged in for 3 consecutive days".

[0108] S212. Using the modulation network in the reinforcement learning model, the state representation is weighted and modulated according to the activity feature to obtain the modulated state representation.

[0109] In this embodiment of the disclosure, the modulation network achieves adaptive differentiation of strategies for different user groups. In implementation, a feature-level linear modulation layer can be used to generate scaling and translation coefficients that act on the state representation. In an alternative embodiment, activity features can be converted into a gating weight matrix to apply attention weights to the state representation.

[0110] Taking a specific application scenario as an example, the modulation network discovers that the user is a low-activity group, so it automatically amplifies the weight of the feature dimension "sensitive to high click-through rates" in the state representation.

[0111] Accordingly, step S220 includes the following steps: S221. The resource weighting action is determined by the policy network based on the state representation after modulation processing.

[0112] In this embodiment of the disclosure, the policy network receives the modulated and distorted state space and maps it to actions. This allows a single reinforcement learning model to automatically adjust the distribution of action preferences based on the population.

[0113] As a concrete implementation example, based on the adjusted state, the policy network outputs a more aggressive action, giving a very large weighting range of [1.5, 0.8], attempting to win back low-activity users through strong stimulation.

[0114] According to the scheme of this disclosure embodiment, by introducing a dedicated modulation network and using activity features for pre-weighted intervention, the technical bottleneck of the unified model being unable to be compatible with the user behavior patterns of different active circles is solved, and the adaptive dynamic strategy adjustment effect of "personalized for each person" is achieved at the underlying state level.

[0115] In one possible implementation, step S230 above includes the following steps: S231. Obtain the initial ranking score for the candidate recommended resources.

[0116] In this embodiment of the disclosure, the initial ranking score is typically provided by the preceding industrial-grade ranking funnel of the recommendation system. In implementation, it can be based on the click-through rate (pCTR) predicted by a deep neural network (such as the DIN or DIEN model in the fine-ranking stage). Alternatively, it can be a similarity score calculated quickly using a dual-tower model in the coarse-ranking stage.

[0117] In one example, after complex feature prediction, candidate article A received an initial ranking score of 0.64, and candidate article B received an initial score of 0.61.

[0118] S232. Use the resource weight adjustment action to perform numerical intervention on the initial ranking score to obtain the target ranking score.

[0119] In this embodiment, numerical intervention is the interface protocol for reinforcement learning to enter the traditional recommendation chain. Specifically, multiplicative intervention can be achieved by directly multiplying the initial score by a weighting action (such as vector component 1.1). In an alternative embodiment, to ensure the safety of the upper and lower bounds of the numerical value, an additive intervention method can also be adopted, which maps the action output to a small offset before performing the additive intervention.

[0120] Specifically, in article A, 0.64 multiplied by the action factor 0.9 becomes 0.576, and in article B, 0.61 multiplied by the action factor 1.2 becomes 0.732.

[0121] S233. Select at least one target resource from the candidate recommended resources as the recommendation result based on the target ranking score.

[0122] In this embodiment, the output is truncated based on the recalibrated score. This can be implemented using a quicksort algorithm or heapsort to retrieve the Top-K values ​​and combine them into a data packet of a specified format. Alternatively, in another embodiment, fallback rules such as shuffling or deduplication can be added at the business layer before truncating.

[0123] For example, after the system re-ranks the articles based on the target scores, article B, which was originally in second place, rises to first place and is displayed to end users as the Top 1 result.

[0124] According to the solution of this disclosure, by smoothly embedding the weighting action into the candidate resource rearrangement stage in the form of post-numerical intervention, the problem that reinforcement learning models are difficult to seamlessly integrate into the existing massive candidate set recommendation architecture is solved, and the presentation effect of the terminal resource list is improved in a low-cost and non-intrusive manner.

[0125] Figure 3 This is a schematic diagram of the structure of a training device for a reinforcement learning model provided according to an embodiment of this disclosure. Figure 3 The training device 300 for the reinforcement learning model shown includes: The sequence acquisition module 301 is used to acquire the historical session behavior sequence of the target user in the information flow recommendation system; The sample segmentation module 302 is used to segment the historical behavior sequence into immediate behavior samples and cross-session behavior samples according to the session boundary state. The first training module 303 is used to train the first evaluation network in the reinforcement learning model using the immediate behavior sample to optimize the immediate feedback benefit. The second training module 304 is used to train the second evaluation network in the reinforcement learning model using the cross-session behavior sample to optimize cross-session retention benefits. The joint update module 305 is used to jointly update the reinforcement learning model based on the trained first evaluation network and second evaluation network to obtain the trained reinforcement learning model.

[0126] In one possible implementation, the sample partitioning module 302 is specifically used for: Extract the data corresponding to a single refresh action in the historical session behavior sequence as an instant behavior sample; The cross-session refresh action combination between two adjacent sessions is extracted as a cross-session behavior sample.

[0127] In one possible implementation, when the sample segmentation module 302 extracts the data corresponding to a single refresh action in the historical session behavior sequence as an immediate behavior sample, it is specifically used for: The cumulative consumption behavior features of each recommended resource exposed within the single refresh action are obtained as the refresh-level features of the instant behavior sample. The cumulative consumption behavior feature is obtained by summing the interaction values ​​of multiple resources generated within the same refresh action.

[0128] In one possible implementation, when the sample segmentation module 302 extracts the cross-session refresh action combination between two adjacent sessions into a cross-session behavior sample, it is specifically used for: Determine the last refresh action of the previous session and the first refresh action of the current session in the historical session behavior sequence; The data corresponding to the last refresh action and the first refresh action are concatenated to form the cross-session state feature of the cross-session behavior sample.

[0129] In one possible implementation, the first training module 303 is specifically used for: Extract click data and / or viewing duration data from the real-time behavior samples; Calculate the refresh level instant reward based on the click data and / or the viewing duration data; The network parameters of the first evaluation network are updated by maximizing the instantaneous reward at the refresh level using the first evaluation network.

[0130] In one possible implementation, the second training module 305 is specifically used for: Obtain the actual cross-session time interval corresponding to the cross-session behavior sample; The expected return probability corresponding to the cross-session behavior sample is predicted using a preset regression classification model; Calculate the long-term cross-session reward based on the actual cross-session time interval and the expected return probability; The network parameters of the second evaluation network are updated by minimizing the long-term cross-session reward using the second evaluation network.

[0131] In one possible implementation, the training device 300 for the reinforcement learning model further includes a masking module for: Assign a first mask label to the instantaneous behavior sample in the experience replay pool, and assign a second mask label that is mutually exclusive with the first mask label to the cross-session behavior sample; Accordingly, the first training module 303 is also used for: During the backpropagation calculation of the reinforcement learning model, the effective backpropagation samples corresponding to the first evaluation network are determined according to the first mask label, so that the first evaluation network performs gradient updates only based on the immediate behavior samples. The second training module 305 is further configured to: determine the valid backhaul samples corresponding to the second evaluation network based on the second mask label, so that the second evaluation network performs gradient updates only based on the cross-session behavior samples.

[0132] In one possible implementation, the joint update module 304 is specifically used for: By combining the immediate action value of the first evaluation network output and the long-term action value of the second evaluation network output, a target loss function is constructed. The parameters of the online policy network in the reinforcement learning model are updated using the target loss function. Using a preset smooth update rate, the parameters of the updated online policy network are softly updated to the target policy network in the reinforcement learning model.

[0133] Figure 4 This is a schematic diagram of the structure of an information recommendation device provided according to embodiments of this disclosure. For example... Figure 4 The information recommendation device 400 shown includes: The representation acquisition module 401 is used to acquire the state representation of the target user when a recommendation request is triggered; wherein, the state representation includes a fusion feature of the current refresh context feature and the historical consumption feature; The representation determination module 402 is used to determine resource weighting actions based on the state representation using the policy network in the pre-trained reinforcement learning model; wherein the reinforcement learning model is obtained through the training method of the reinforcement learning model provided in any embodiment of this disclosure; Output module 403 is used to output recommendation results based on the resource weight adjustment action.

[0134] In one possible implementation, the characterization acquisition module 401 is further configured to: Obtain the activity characteristics of the target user; Accordingly, the characterization determination module 402 is specifically used for: The state representation is weighted and modulated based on the activity features using the modulation network in the reinforcement learning model to obtain the modulated state representation. The resource weighting action is determined using the policy network based on the state representation after modulation processing.

[0135] In one possible implementation, the output module 403 is specifically used for: Obtain the initial ranking score for the candidate recommended resources; The initial ranking score is numerically intervened using the resource allocation action to obtain the target ranking score. At least one target resource is selected from the candidate recommended resources as the recommendation result based on the target ranking score.

[0136] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0137] The acquisition, storage, and application of any type of information, such as user personal information, involved in the technical solutions disclosed herein comply with relevant laws and regulations and do not violate public order and good morals.

[0138] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0139] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0140] like Figure 5 As shown, device 500 includes a computing unit 501, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 502 or a computer program loaded from storage unit 508 into random access memory (RAM) 503. RAM 503 may also store various programs and data required for the operation of device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. Input / output (I / O) interface 505 is also connected to bus 504.

[0141] Multiple components in device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0142] The computing unit 501 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as reinforcement learning model training methods and information recommendation methods. For example, in some embodiments, the reinforcement learning model training methods and information recommendation methods can be implemented as computer software programs, which are tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program can be loaded and / or installed on device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the reinforcement learning model training methods and information recommendation methods described above can be performed. Alternatively, in other embodiments, computing unit 501 may be configured in any other suitable manner (e.g., by means of firmware) to perform training methods for reinforcement learning models or information recommendation methods.

[0143] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0144] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0145] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0146] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0147] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0148] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0149] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0150] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for training a reinforcement learning model, comprising: Obtain the historical session behavior sequence of the target user in the information flow recommendation system; Based on the session boundary state, the historical behavior sequence is divided into immediate behavior samples and cross-session behavior samples. The first evaluation network in the reinforcement learning model is trained using the aforementioned instantaneous behavior samples to optimize instantaneous feedback gains; The second evaluation network in the reinforcement learning model is trained using the cross-session behavior samples to optimize cross-session retention gains; The reinforcement learning model is jointly updated based on the first and second evaluation networks after training to obtain the trained reinforcement learning model.

2. The method according to claim 1, wherein, The step of dividing the historical session behavior sequence into immediate behavior samples and cross-session behavior samples based on session boundary states includes: Extract the data corresponding to a single refresh action in the historical session behavior sequence as an instant behavior sample; The cross-session refresh action combination between two adjacent sessions is extracted as a cross-session behavior sample.

3. The method according to claim 2, wherein, The step of extracting the data corresponding to a single refresh action in the historical session behavior sequence as an instant behavior sample includes: The cumulative consumption behavior features of each recommended resource exposed within the single refresh action are obtained as the refresh-level features of the instant behavior sample; wherein, the cumulative consumption behavior features are obtained by summing the interaction values ​​of multiple resources generated within the same refresh action.

4. The method according to claim 2, wherein, The step of extracting cross-session refresh action combinations between two adjacent sessions into cross-session behavior samples includes: Determine the last refresh action of the previous session and the first refresh action of the current session in the historical session behavior sequence; The data corresponding to the last refresh action and the first refresh action are concatenated to form the cross-session state feature of the cross-session behavior sample.

5. The method according to claim 1, wherein, The step of training the first evaluation network in the reinforcement learning model using the instantaneous behavior samples to optimize instantaneous feedback gains includes: Extract click data and / or viewing duration data from the real-time behavior samples; Calculate the refresh level instant reward based on the click data and / or the viewing duration data; The network parameters of the first evaluation network are updated by maximizing the instantaneous reward at the refresh level using the first evaluation network.

6. The method according to claim 1, wherein, The step of training the second evaluation network in the reinforcement learning model using the cross-session behavior samples to optimize cross-session retention gains includes: Obtain the actual cross-session time interval corresponding to the cross-session behavior sample; The expected return probability corresponding to the cross-session behavior sample is predicted using a preset regression classification model; Calculate the long-term cross-session reward based on the actual cross-session time interval and the expected return probability; The network parameters of the second evaluation network are updated by minimizing the long-term cross-session reward using the second evaluation network.

7. The method according to claim 1, wherein, The training process of the reinforcement learning model also includes: Assign a first mask label to the instantaneous behavior sample in the experience replay pool, and assign a second mask label that is mutually exclusive with the first mask label to the cross-session behavior sample; During the backpropagation calculation of the reinforcement learning model, the effective backpropagation samples corresponding to the first evaluation network are determined according to the first mask label, so that the first evaluation network performs gradient updates only based on the immediate behavior samples. The valid return samples corresponding to the second evaluation network are determined based on the second mask label, so that the second evaluation network performs gradient updates only based on the cross-session behavior samples.

8. The method according to claim 1, wherein, The reinforcement learning model includes an online policy network and a target policy network; the joint update of the reinforcement learning model based on the trained first evaluation network and second evaluation network includes: By combining the immediate action value of the first evaluation network output and the long-term action value of the second evaluation network output, a target loss function is constructed. The parameters of the online policy network in the reinforcement learning model are updated using the target loss function. Using a preset smooth update rate, the parameters of the updated online policy network are softly updated to the target policy network in the reinforcement learning model.

9. An information recommendation method, comprising: When a recommendation request is triggered, the state representation of the target user is obtained; wherein, the state representation includes a fusion feature of the current refresh context features and historical consumption features; Using a policy network in a pre-trained reinforcement learning model, resource weighting actions are determined based on the state representation; wherein the reinforcement learning model is obtained by the training method as described in any one of claims 1 to 8; The recommended results are output based on the resource allocation actions described.

10. The method of claim 9, further comprising: Obtain the activity characteristics of the target user; The state representation is weighted and modulated based on the activity features using the modulation network in the reinforcement learning model to obtain the modulated state representation. The method of utilizing the policy network in a pre-trained reinforcement learning model to determine resource weighting actions based on the state representation includes: The resource weighting action is determined using the policy network based on the state representation after modulation processing.

11. The method according to claim 9, wherein, The step of outputting recommendation results based on the resource allocation action includes: Obtain the initial ranking score for the candidate recommended resources; The initial ranking score is numerically intervened using the resource allocation action to obtain the target ranking score. At least one target resource is selected from the candidate recommended resources as the recommendation result based on the target ranking score.

12. A training device for a reinforcement learning model, comprising: The sequence acquisition module is used to acquire the historical session behavior sequence of the target user in the information flow recommendation system; The sample segmentation module is used to divide the historical behavior sequence into immediate behavior samples and cross-session behavior samples based on the session boundary state. The first training module is used to train the first evaluation network in the reinforcement learning model using the instantaneous behavior samples to optimize the instantaneous feedback gains. The second training module is used to train the second evaluation network in the reinforcement learning model using the cross-session behavior samples to optimize cross-session retention benefits. The joint update module is used to jointly update the reinforcement learning model based on the trained first evaluation network and second evaluation network to obtain the trained reinforcement learning model.

13. An information recommendation device, comprising: The representation acquisition module is used to acquire the state representation of the target user when a recommendation request is triggered; wherein, the state representation includes a fusion feature of the current refresh context feature and the historical consumption feature; The representation determination module is used to determine resource weighting actions based on the state representation using the policy network in a pre-trained reinforcement learning model; wherein the reinforcement learning model is obtained through the training method of the reinforcement learning model provided in any embodiment of this disclosure; The output module is used to output recommended results based on the resource allocation action.

14. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-11.

15. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-11.

16. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-11.

Images