A message pushing strategy optimization method and system based on deep reinforcement learning

Abstract

This invention relates to the field of data reasoning technology, and discloses a message push strategy optimization method and system based on deep reinforcement learning. The method includes: jointly encoding the static attributes, observation sequence, and selection identifier of the agent to generate a decision state vector and an action feature set; obtaining a preliminary value scalar by performing a dot product operation on the decision state vector and the action embedding vector; normalizing the scalar to obtain a policy probability distribution, and determining the action identifier to be executed accordingly; retrieving environmental interaction data based on the identifier to obtain a multi-dimensional feedback vector; updating the state representation weights based on the feedback, and constructing a prediction vector by combining the action features; finally, iterating the policy parameters based on the prediction vector to obtain an optimization index. This invention can improve the efficiency of message push strategy optimization based on deep reinforcement learning.

Description

Technical Field

[0001] This invention relates to the field of data reasoning technology, and in particular to a message push strategy optimization method and system based on deep reinforcement learning. Background Technology

[0002] In existing technologies, deep reinforcement learning methods applied to push notifications suffer from deficiencies in agent state construction. Existing methods typically fail to effectively and structurally co-encode long-term static user profile features with dynamic, time-series interaction records, resulting in generated decision state vectors that do not fully and accurately reflect the true state of the environment. This incomplete or imprecise state representation directly affects the agent's judgment of the current situation, becoming a source of policy decision-making errors and ultimately reducing the overall efficiency and effectiveness of the policy optimization process.

[0003] Meanwhile, existing technologies are inefficient in action selection and policy update mechanisms. On the one hand, the evaluation process for action value is computationally complex, failing to quickly produce reliable value scalars through efficient feature interaction and integration. On the other hand, policy network updates are often based on delayed or sparse global feedback, lacking an effective mechanism to utilize the instantaneous, multi-dimensional feedback vectors generated by a single interaction to adjust model parameters in real time and accurately. This results in slow policy iteration convergence, making it difficult to achieve rapid adaptive policy optimization in dynamically changing environments. Summary of the Invention

[0004] This invention provides a message push strategy optimization method and system based on deep reinforcement learning, the main purpose of which is to solve the problem of low efficiency in message push strategy optimization based on deep reinforcement learning.

[0005] To achieve the above objectives, this invention provides a message push strategy optimization method based on deep reinforcement learning, comprising:

[0006] The static attributes, observation sequences, and selection identifiers of the agent are jointly encoded to obtain the decision state vector and action feature set of the agent.

[0007] The decision state vector is multiplied by the embedding vector of the action feature library to obtain the preliminary value scalar of the agent.

[0008] The initial value scalar is normalized to obtain the policy probability distribution of the agent, and the action identifier to be executed by the agent is determined based on the policy probability distribution.

[0009] Based on the action identifier to be executed, retrieve the environmental interaction data of the agent to obtain the multidimensional feedback vector of the agent;

[0010] The state representation weights of the agent are updated based on the multidimensional feedback vector, and the prediction vector of the agent is constructed by combining the features of the action identifier to be executed.

[0011] The optimization index of the agent is obtained by iterating the policy parameters in the agent based on the prediction vector.

[0012] In a preferred embodiment, the joint encoding of the agent's static attributes, observation sequence, and selection identifier to obtain the agent's decision state vector and action feature set includes:

[0013] The user profile features of the intelligent agent are used as the static attributes, the message interaction records of the intelligent agent are used as the observation sequence, and the set of operable message types of the intelligent agent are used as the selection identifier.

[0014] The static attributes are mapped to the fully connected layer of the agent to obtain the first feature vector of the agent;

[0015] The observed sequence is temporally encoded to obtain the sequence dependency features of the agent;

[0016] The first feature vector is concatenated with the sequence dependency feature to obtain the decision state vector of the agent;

[0017] An embedding lookup is performed on the selected identifier to obtain the action feature set of the agent.

[0018] In a preferred embodiment, the step of performing a dot product operation between the decision state vector and the embedding vector of the action feature library set to obtain the preliminary value scalar of the agent includes:

[0019] Align the decision state vector with the embedding vector of the action feature library according to the feature dimension to obtain the projected state vector and projected action vector of the agent;

[0020] The interaction vector of the agent is obtained by multiplying the projection state vector and the projection action vector element by element according to their corresponding positions.

[0021] The elements in the interaction vector are summed to obtain the preliminary value scalar of the agent.

[0022] In a preferred embodiment, the step of normalizing the preliminary value scalar to obtain the agent's policy probability distribution, and determining the agent's action identifier based on the policy probability distribution, includes:

[0023] The initial value scalar is exponentially scaled to obtain the agent's preference intensity value;

[0024] The adjustment factor of the agent is obtained by aggregating the preference intensity values;

[0025] Projecting the value scalar onto the adjustment factor yields the probability distribution of the agent;

[0026] Based on the probability distribution, the selection identifier is randomly sampled to obtain the action identifier to be executed by the agent.

[0027] In a preferred embodiment, the step of retrieving the agent's environmental interaction data based on the action identifier to be executed, and obtaining the agent's multidimensional feedback vector, includes:

[0028] Push messages to the agent according to the message type corresponding to the action to be executed;

[0029] Collect the behavioral characteristics of the intelligent agent in response to the push message within a preset time window;

[0030] Numerical reward signals are assigned to the agent based on the category corresponding to the behavioral characteristics;

[0031] The reward signals are concatenated according to the feedback order to obtain the multidimensional feedback vector of the agent.

[0032] In a preferred embodiment, assigning a numerical reward signal to the agent based on the category corresponding to the behavioral feature includes:

[0033] Analyze the duration parameter and operation depth parameter in the behavioral characteristics;

[0034] Based on the numerical range of the duration parameter and the operation depth parameter, the refinement level of the behavioral feature in the corresponding category is determined;

[0035] Based on the refinement level, the reward parameters in the agent are forward-propagated to obtain the numerical reward signal of the agent.

[0036] In a preferred embodiment, updating the agent's state representation weights based on the multidimensional feedback vector and constructing the agent's prediction vector by combining the features of the action identifier to be executed includes:

[0037] The agent's long-term reward estimate is calculated by difference based on the immediate reward in the multidimensional feedback vector to obtain the agent's policy evaluation error.

[0038] Based on the strategy, the error is evaluated and the weights of the agent's state representation network are adjusted using a gradient to obtain the agent's instantaneous representation weights.

[0039] The instantaneous representation weights are fused with the action feature vectors corresponding to the action identifiers to be executed to obtain a comprehensive prediction vector.

[0040] In a preferred embodiment, the formula for calculating the strategy evaluation error includes:

[0041]

[0042] in, The error is evaluated for the strategy. For the aforementioned instant reward, As a discount factor, This is the estimated value at the next time step. For the original value estimate, This is the index for discrete time steps.

[0043] In a preferred embodiment, the step of iterating the policy parameters in the agent based on the prediction vector to obtain the optimization index of the agent includes:

[0044] Decouple the prediction vector to obtain the expected reward value of the agent;

[0045] Starting from the expected return value, the policy network of the agent is traversed in reverse to obtain the parameter gradient set of the agent;

[0046] Based on the feature gradient components in the parameter gradient set, the policy parameter values ​​in the agent are adjusted proportionally to obtain the optimization index of the agent.

[0047] To address the above problems, the present invention also provides a message push strategy optimization system based on deep reinforcement learning, the system comprising:

[0048] The feature extraction module jointly encodes the static attributes, observation sequence, and selection identifier of the agent to obtain the decision state vector and action feature set of the agent.

[0049] The preliminary value scalar module performs a dot product operation between the decision state vector and the embedding vector of the action feature library set to obtain the preliminary value scalar of the agent;

[0050] The action to be executed module normalizes the preliminary value scalar to obtain the policy probability distribution of the agent, and determines the action to be executed of the agent based on the policy probability distribution;

[0051] The multidimensional feedback data module retrieves the environmental interaction data of the agent based on the action identifier to be executed, and obtains the multidimensional feedback vector of the agent.

[0052] The prediction vector module updates the state representation weights of the agent based on the multidimensional feedback vector, and constructs the prediction vector of the agent by combining the features of the action identifier to be executed.

[0053] The optimization index module iterates through the policy parameters in the agent based on the prediction vector to obtain the optimization index of the agent.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] 1. This invention constructs a decision state vector that simultaneously encompasses long-term static attributes and short-term dynamic behaviors by jointly encoding user profiles, interaction sequences, and actionable message sets. This method utilizes fully connected layers and temporal coding networks to extract high-order features and ensures complete information fusion through vector concatenation, thereby generating a state representation that accurately reflects the true state of the environment. This technique enhances the depth and accuracy of the agent's understanding of complex user states, laying a reliable foundation for subsequent policy decisions.

[0056] 2. This invention further achieves efficient action evaluation through feature dimension alignment and vector interaction calculation, and utilizes real-time feedback data to adjust the model's gradient in real time. This method integrates multi-dimensional feedback into the update process of state representation weights, and generates a future-oriented comprehensive prediction vector based on the adjusted weights, ultimately driving the targeted optimization of policy network parameters. These technical measures enhance the real-time performance and accuracy of policy iteration, directly improving the learning efficiency and final performance of the policy model. Attached Figure Description

[0057] Figure 1 This is a flowchart illustrating a message push strategy optimization method based on deep reinforcement learning, as provided in an embodiment of the present invention.

[0058] Figure 2 This is a functional block diagram of a message push strategy optimization system based on deep reinforcement learning provided in an embodiment of the present invention;

[0059] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0060] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0061] This application provides a message push strategy optimization method based on deep reinforcement learning. The execution subject of this method includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this application: a server, a terminal, etc. In other words, the message push strategy optimization method based on deep reinforcement learning can be executed by software or hardware installed on a terminal device or a server device. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster. The server can be an independent server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0062] Reference Figure 1 The diagram shown is a flowchart illustrating a message push strategy optimization method based on deep reinforcement learning according to an embodiment of the present invention. In this embodiment, the message push strategy optimization method based on deep reinforcement learning includes:

[0063] In this embodiment of the invention, when jointly encoding the static attributes, observation sequence, and selection identifier of the agent to obtain the decision state vector and action feature set of the agent, it is specifically used for:

[0064] The user profile features of the intelligent agent are used as the static attributes, the message interaction records of the intelligent agent are used as the observation sequence, and the set of operable message types of the intelligent agent are used as the selection identifier.

[0065] The static attributes are mapped to the fully connected layer of the agent to obtain the first feature vector of the agent;

[0066] The observed sequence is temporally encoded to obtain the sequence dependency features of the agent;

[0067] The first feature vector is concatenated with the sequence dependency feature to obtain the decision state vector of the agent;

[0068] An embedding lookup is performed on the selected identifier to obtain the action feature set of the agent.

[0069] Specifically, static attributes are concretized into user profile features of the intelligent agent, which are structured data tables constructed by integrating users' basic attribute data, historical behavioral preference tags, and long-term interest indicators.

[0070] Specifically, mapping static attributes to the agent's fully connected layer is achieved through a feedforward neural network layer. This fully connected layer has fixed input and output dimensions, and its weight matrix and bias vector are determined during model initialization.

[0071] Specifically, temporal encoding of the observed sequence is accomplished using a recurrent neural network. The unit structure of this recurrent neural network is defined before training. The message interaction record sequence is input into the network step by step. At each time step, the network unit calculates the hidden state of the current step based on the input of the current step and the hidden state of the previous time step.

[0072] Specifically, the first feature vector is concatenated with the sequence dependency feature through a vector concatenation operation. The first feature vector and the sequence dependency feature are arranged side by side to form a longer vector.

[0073] Specifically, the embedding lookup for the selection identifier is performed using an embedding table. This embedding table is created during model initialization, and its number of rows equals the total number of types in the operable message type set. Each row is a fixed-length vector representing the embedding representation of a message type.

[0074] Furthermore, the observation sequence is materialized into a message interaction record of the agent, which is a timestamp-sorted list where each record contains an identifier for a push message, the type of user action triggered by that message, and the time when the action occurred. The selected identifier is materialized into a set of actionable message types for the agent, a fixed set containing the codes for all possible push message types.

[0075] Furthermore, the user profile features are fed into this layer as input vectors, where they are multiplied by the layer's weight matrix, and then a bias vector is added. Finally, the vectors are transformed by a preset non-linear activation function, and the output is the agent's first feature vector.

[0076] Furthermore, after the entire sequence input is complete, the hidden state generated at the last time step, or the result obtained by aggregating the hidden states of all time steps, is extracted as the sequence dependency feature of the agent.

[0077] Furthermore, physically, the elements of the two vectors are arranged sequentially to form a unified representation that includes static attributes and temporal history information. This newly formed vector is the agent's decision state vector.

[0078] Furthermore, based on the encoding of each message type in the selection identifier, the corresponding embedding vector is retrieved from the corresponding row of the embedding table. All these retrieved embedding vectors are collected together to form the action feature set of the agent.

[0079] In summary, by explicitly defining user profile features, message interaction records, and actionable message type sets as static attributes, observation sequences, and selection identifiers, respectively, a structured, multi-source, and heterogeneous input space is constructed for the agent. This clear division ensures that the model can simultaneously utilize the user's long-term stable characteristics, short-term dynamic behaviors, and all possible decision options, thus laying the foundation for generating a comprehensive and information-rich decision state and solving the problem of incomplete state representation caused by single or ambiguous input information.

[0080] In summary, by processing static attributes through fully connected layers, the original user profile features are mapped to a high-dimensional, non-linear continuous vector space, resulting in the first feature vector. This process extracts complex, deep-seated latent patterns from the static attributes, enabling subsequent computations to utilize these refined high-level features. Simultaneously, the observed sequences are temporally encoded, and a recurrent neural network captures the temporal dependencies and sequence patterns in message interaction records, generating sequence-dependent features. This allows the agent to understand the dynamic evolution of user behavior and contextual information.

[0081] In summary, concatenating the first feature vector with the sequence-dependent features directly merges all dimensions of the two vectors to generate the decision state vector. This method fully preserves all high-order feature information from static attributes and dynamic sequences, avoiding information loss or compression during the fusion process. It provides the agent with a unified, high-dimensional state representation that simultaneously embodies long-term preferences and short-term intentions, directly improving the accuracy and depth of the policy network's perception of the environmental state.

[0082] In summary, by embedding the selection identifiers, discrete, symbolic actionable message types are transformed into a low-dimensional, dense set of action feature vectors, forming an action feature set. This step embeds the action space into a continuous mathematical space, enabling the model to measure the correlation between actions and states through vector operations (such as dot products) rather than performing discrete, symbolic logical judgments. This significantly improves the efficiency and continuity of policy gradient calculation and action value evaluation, facilitating end-to-end optimization of the neural network.

[0083] In this embodiment of the invention, the step of performing a dot product operation between the decision state vector and the embedding vector of the action feature library set to obtain the preliminary value scalar of the agent is specifically used for:

[0084] Align the decision state vector with the embedding vector of the action feature library according to the feature dimension to obtain the projected state vector and projected action vector of the agent;

[0085] The interaction vector of the agent is obtained by multiplying the projection state vector and the projection action vector element by element according to their corresponding positions.

[0086] The elements in the interaction vector are summed to obtain the preliminary value scalar of the agent.

[0087] Specifically, feature dimension alignment is achieved through two independent linear transformation layers: one dedicated to processing the decision state vector, and the other dedicated to processing the embedding vector retrieved from the action feature library. The decision state vector is input into its dedicated linear transformation layer, which computes a fixed weight matrix and bias term to map the vector to a new vector with a specific dimension; this new vector is the projected state vector.

[0088] Specifically, the projected state vector and the projected action vector are ensured to have exactly the same length. Starting from the first element position, the value of the first element of the projected state vector is multiplied by the value of the first element of the projected action vector, and the result is used as the first element of the new vector.

[0089] Specifically, the interaction vector is treated as a list containing multiple numerical elements. Starting with the first element of the list, it is added to the second element to obtain an intermediate sum. Then, this intermediate sum is added to the third element of the list to obtain a new intermediate sum.

[0090] Furthermore, the embedding vectors of the action feature library are input into another dedicated linear transformation layer, which calculates the embedding vectors using its own fixed weight matrix and bias terms, mapping them to another new vector with the exact same specific dimension. This new vector is the projected action vector.

[0091] Furthermore, the same multiplication operation is performed on the second element position, and so on, until all elements in both vectors have been processed. All the results of multiplying at corresponding positions are collected sequentially to form a new vector, the cross vector.

[0092] Furthermore, this process is repeated, adding each remaining element in the list to the currently accumulated sum. When the last element in the list is also added, the final sum is a single value, i.e., the initial value scalar.

[0093] In summary, by mapping the decision state vector and action embedding vector to the same feature dimension through independent linear transformation layers, projected state vectors and projected action vectors are obtained. Mathematically, this operation creates a shared and comparable semantic space for states and actions, resolving the problem that state representations and action representations cannot be directly and effectively matched due to their different sources and structures. This enables subsequent computations to be based on aligned features with the same dimension, a crucial prerequisite for the model to achieve accurate value estimation.

[0094] In summary, the projected state vector and the projected action vector are multiplied element-wise to generate an interaction vector. This calculation simulates fine-grained, pairwise interactions between state and action features. The result of each element-wise multiplication represents the degree of matching or synergistic effect between the state and action on the corresponding feature dimension, thus transforming the abstract "state-action pair" relationship into a concrete, further-processable numerical vector, providing rich intermediate representations for evaluating the applicability of the action in a specific state.

[0095] In summary, summing all elements in the interaction vector yields a single, preliminary value scalar. This operation essentially integrates high-dimensional, fine-grained interaction features into a global evaluation score. By aggregating interaction information across all feature dimensions, it comprehensively judges the overall merit of the current action in the current state, thus outputting a scalar signal that can guide policy selection. This step completes the dimensionality reduction and induction from high-dimensional interaction features to a low-dimensional decision signal, and is the core link connecting feature computation and policy generation.

[0096] In this embodiment of the invention, the step of normalizing the preliminary value scalar to obtain the agent's policy probability distribution, and determining the agent's action identifier based on the policy probability distribution, is specifically used for:

[0097] The initial value scalar is exponentially scaled to obtain the agent's preference intensity value;

[0098] The adjustment factor of the agent is obtained by aggregating the preference intensity values;

[0099] Projecting the value scalar onto the adjustment factor yields the probability distribution of the agent;

[0100] Based on the probability distribution, the selection identifier is randomly sampled to obtain the action identifier to be executed by the agent.

[0101] Specifically, the mathematical constant e is used as the base, and the value of the initial input value scalar is used as the exponent to calculate the power of e.

[0102] Specifically, all the preference intensity values ​​obtained in the previous step, that is, the value corresponding to each action, are added together. This sum is the agent's adjustment factor.

[0103] Specifically, each preference intensity value is divided by the adjustment factor calculated in the previous step, which is the sum. Each preference intensity value divided by the adjustment factor yields a new value, which is between 0 and 1.

[0104] Specifically, first, a random number uniformly distributed between 0 and 1 is generated. Then, following the order of the actions, each probability value in the probability distribution is accumulated sequentially to form a cumulative probability interval.

[0105] Furthermore, this calculation process is performed once for each independent preliminary value scalar, and the resulting value is the preference intensity value for the corresponding action.

[0106] Furthermore, the set of all these new values ​​constitutes a probability distribution, where each value represents the probability of the corresponding action being selected, and the sum of all probabilities is 1.

[0107] Furthermore, accumulation stops when the accumulated probability value first exceeds or equals the generated random number. At this point, the action corresponding to the currently accumulated probability value becomes its selection flag. This selected flag is the action flag to be executed by the agent.

[0108] In summary, an exponential scaling of the initial value scalars to a base of the natural constant yields preference intensity values. This nonlinear transformation maps all value scalars to positive numbers, ensuring the mathematical feasibility of subsequent probability calculations. It also amplifies the relative differences in value ratings between different actions, enhancing the numerical significance of dominant actions. This resolves the potential issues of unstable numerical ranges or insignificant differences in the original value scalars, making the probability distribution more clearly reflect the preference order among actions.

[0109] In summary, the adjustment factor is obtained by aggregating all preference intensity values. This operation calculates the sum of preference intensity values, and its core function is to serve as a global normalization constant. By calculating this sum, it prepares the way for transforming each individual preference intensity value into a probability value. This is a crucial step in implementing a probability distribution, ensuring that the sum of all generated probabilities is 1, satisfying the basic axioms of probability distributions.

[0110] In summary, dividing each preference intensity value by an adjustment factor yields the agent's probability distribution. This division operation is a standard normalization process, transforming the preference intensity value for each action into a probability value between 0 and 1. This step directly converts the abstract concept of the model's preference intensity for actions into a concrete probability distribution that can be used for random sampling, thus transforming the policy from a deterministic choice into an explorable stochastic policy.

[0111] In summary, based on the obtained probability distribution, a roulette wheel selection method is used to randomly sample the selection identifiers to obtain the identifiers of the actions to be executed. This mechanism introduces randomness while adhering to the action probability weights. It solves the problems of insufficient exploration and getting trapped in local optima that may be caused by purely greedy strategies, realizes the trade-off between exploration and exploitation in reinforcement learning, and is the core operation that drives the agent to optimize its long-term strategy through trial and error learning.

[0112] In this embodiment of the invention, the step of retrieving the agent's environmental interaction data based on the action identifier to be executed, and obtaining the agent's multidimensional feedback vector, is specifically used for:

[0113] Push messages to the agent according to the message type corresponding to the action to be executed;

[0114] Collect the behavioral characteristics of the intelligent agent in response to the push message within a preset time window;

[0115] Numerical reward signals are assigned to the agent based on the category corresponding to the behavioral characteristics;

[0116] The reward signals are concatenated according to the feedback order to obtain the multidimensional feedback vector of the agent.

[0117] Specifically, the action identifier to be executed is an internal code. The system maintains a mapping table that uniquely associates each action identifier with a predefined message content template. Once the identifier is determined, the system retrieves the corresponding message template from the mapping table and generates a data block containing specific text or code.

[0118] Specifically, the preset time window is a fixed-length interval calculated from the moment the push is completed. The system monitors the simulated behavior output of the agent within this time window, and these behaviors are predefined into a finite number of types, such as "ignore", "click", "deep reading", and "negative feedback".

[0119] Specifically, assigning numerical reward signals to the agent based on the category corresponding to the behavioral characteristics is achieved through a preset, static category-reward mapping dictionary. This dictionary defines a fixed reward value corresponding to each possible behavioral category.

[0120] Specifically, the system retrieves the corresponding reward values ​​for each behavioral event in the original order of occurrence from the behavioral feature list. Then, these reward values ​​are treated as scalar elements and placed sequentially into the corresponding positions of a one-dimensional vector in the same order.

[0121] Furthermore, this data block is not sent through an external communication channel, but is directly placed into the agent's simulated interactive environment or internal input buffer as part of the environmental state to complete this push.

[0122] Furthermore, for each action that occurs, its type code and timestamp are recorded. At the end of the time window, all the action events recorded within the window are arranged in chronological order of their occurrence, forming an ordered list. This list represents the behavioral characteristics generated by the agent.

[0123] Furthermore, the system iterates through the list of behavioral features obtained in the previous step, and for each behavioral category in the list, it looks up its corresponding fixed reward value from the mapping dictionary. This lookup and assignment process does not involve dynamic calculation or comparison with external targets; it is merely a static table lookup based on the category.

[0124] Furthermore, this vector, which contains multiple reward scalars and whose order reflects the timing of behavioral feedback, is the agent's multidimensional feedback vector.

[0125] In summary, the simulated "push" is completed by mapping the action identifier to the model internally and inserting the corresponding message content. This step transforms the agent's abstract decision output into a concrete, evaluable simulated interaction event, providing a clear instance for subsequent evaluation of the policy's effectiveness. It establishes a bridge from the policy network output to the simulated environmental input and is a crucial step in initiating a complete "state-action-reward" interaction cycle.

[0126] In summary, the behavioral characteristics of the agent in response to simulated messages are collected within a preset time window and listed in chronological order. This operation defines a structured observation period, enabling the model to focus on the immediate and short-term subsequent effects of actions. Recording behavioral events in chronological order not only captures the "presence" of feedback but also retains its crucial "time sequence" dimension, providing structured data for analyzing dynamic patterns of user behavior and solving the problems of fragmented and unstructured feedback information.

[0127] In summary, based on a pre-defined static mapping dictionary, behavioral feature categories are directly converted into numerical reward signals. This process achieves a mapping from abstract, symbolic behavioral categories to concrete, mathematically computable scalar rewards. It provides a clear and consistent evaluation criterion, enabling the model to quantify the immediate effect of each action based on a uniform reward scale, which is the foundation for achieving reinforcement learning objectives.

[0128] In summary, multiple reward signals are concatenated into a multidimensional feedback vector in the original order in which the feedback occurs. This construction operation integrates the rewards obtained at discrete time points into an ordered vector. This vector not only contains the numerical information of the rewards but also implies the temporal correlation of the rewards through its order, forming a complete trajectory data segment that can be used for subsequent processing by algorithms such as temporal difference learning. This provides the necessary data structure for accurately calculating long-term value estimation and policy gradients.

[0129] In this embodiment of the invention, the step of allocating a numerical reward signal to the agent based on the category corresponding to the behavioral feature is specifically used for:

[0130] Analyze the duration parameter and operation depth parameter in the behavioral characteristics;

[0131] Based on the numerical range of the duration parameter and the operation depth parameter, the refinement level of the behavioral feature in the corresponding category is determined;

[0132] Based on the refinement level, the reward parameters in the agent are forward-propagated to obtain the numerical reward signal of the agent.

[0133] Specifically, each event recorded in the list of behavioral characteristics, in addition to the behavioral category code and timestamp, is also associated with a data packet, which explicitly stores two numerical fields: one representing the number of simulated time units experienced by the behavior from start to finish, i.e., the duration parameter.

[0134] Specifically, the system predefines a set of independent interval rules for each behavior category. For example, for the "deep reading" category, it may be defined that when the duration is a certain unit and the operation depth is a certain value, the refinement level is "shallow", and so on to "medium" and "deep".

[0135] Specifically, the agent contains a dedicated reward parameter network, whose input layer is designed to receive one-hot encodings of the refinement levels. First, the refinement levels obtained in the previous step are converted into a binary vector, where only the position representing that level is 1, and the rest are 0.

[0136] Furthermore, another predefined operation level is used to indicate the level triggered by the action. For example, for the "reading" action, level one represents browsing the title, level two represents reading the summary, and level three represents reading the full text. This level value is the operation depth parameter.

[0137] Furthermore, the system compares the duration parameter and operation depth parameter of the current behavior with the set of rules corresponding to its behavior category, finds the interval in which the parameter values ​​simultaneously meet the conditions, and the label marked on the interval is the refinement level of the behavior feature in the corresponding category.

[0138] Furthermore, this vector is input into the first layer of the reward parameter network, multiplied by the weight matrix of that layer, added with the bias vector, and passed through an activation function. This calculation result is then passed as input to the next layer of the network, repeating the same weighted summation and activation process until it reaches the output layer of the network. The final scalar value produced by the output layer is the numerical reward signal of the agent obtained in this calculation.

[0139] In summary, extracting two specific parameters—duration and operational depth—from behavioral characteristics allows for a shift in user behavior evaluation from discrete categorical judgments to continuous, multi-dimensional quantitative analysis. This process extracts finer-grained indicators that better reflect user engagement and intent intensity, providing a rich data foundation for generating more accurate and differentiated training signals. It also addresses the problems of traditional methods that neglect internal behavioral differences, resulting in coarse reward signals and weak training guidance.

[0140] In summary, based on pre-defined numerical range rules bound to behavior categories, duration and operation depth parameters are mapped to specific levels of refinement. This step establishes a standardized mapping from a continuous parameter space to a discrete semantic space. It enables the model to distinguish and characterize behaviors of different qualities within the same category based on clear and consistent rules, thereby refining and structuring the reward evaluation criteria and laying the logical foundation for the subsequent generation of non-uniform reward signals.

[0141] In summary, the refinement level is taken as input, and forward propagation is performed through a dedicated reward parameter network to ultimately output a numerical reward signal. The key to this mechanism is that reward generation no longer relies on a static lookup table, but is dynamically calculated through a learnable neural network. This allows the reward signal to be non-linearly and adaptively adjusted according to the high-level semantic concept of "refinement level," allowing the model to learn and internalize more complex reward allocation strategies during training, thereby generating higher-quality reinforcement learning signals that drive the policy towards the desired behavior.

[0142] In this embodiment of the invention, when updating the state representation weights of the agent based on the multidimensional feedback vector and constructing the prediction vector of the agent by combining the features of the action identifier to be executed, the specific steps are as follows:

[0143] The agent's long-term reward estimate is calculated by difference based on the immediate reward in the multidimensional feedback vector to obtain the agent's policy evaluation error.

[0144] Based on the strategy, the error is evaluated and the weights of the agent's state representation network are adjusted using a gradient to obtain the agent's instantaneous representation weights.

[0145] The instantaneous representation weights are fused with the action feature vectors corresponding to the action identifiers to be executed to obtain a comprehensive prediction vector.

[0146] Specifically, the first element is taken from the multidimensional feedback vector; this element represents the immediate reward the agent received in its most recent interaction. The agent's long-term reward estimate is a scalar value calculated by the agent's internal value network based on the current decision state vector.

[0147] Specifically, the gradient of the state characterization network output with respect to all its weights is calculated. This gradient represents the degree to which a change in each weight affects the policy evaluation error. The policy evaluation error is multiplied by a preset small adjustment factor, and then multiplied by the calculated gradient to obtain the adjustment amount for each weight.

[0148] Specifically, the action feature vector is an embedding vector retrieved from the action feature library using the action identifier to be executed. Fusion is achieved through a linear transformation, using the corresponding portion of the instantaneous representation weights as the transformation matrix.

[0149] Furthermore, the difference between the immediate reward value and the long-term return estimate is the strategy evaluation error.

[0150] Furthermore, by subtracting the corresponding adjustment amount from the original weights of the state representation network, all the updated weight values ​​constitute the agent's instantaneous representation weights.

[0151] Furthermore, the action feature vector is multiplied by the portion of the instantaneous representation weights designated as the transformation matrix. The product is then added to the portion of the instantaneous representation weights designated as the bias vector. The final output vector is the comprehensive prediction vector.

[0152] In summary, by calculating the difference between the immediate reward and the long-term return estimate, the policy evaluation error is obtained. This operation enables real-time quantitative evaluation of the agent's value prediction accuracy. This error signal directly measures the prediction bias of the current value network, providing clear direction and magnitude guidance for subsequent parameter updates. It solves the problem of slow or unstable convergence caused by the lack of immediate and directional feedback during model training and is the core driving force for the value function to approximate the true return.

[0153] In summary, based on the policy evaluation error, the weights of the state representation network are updated through gradient calculation and coefficient adjustment to obtain real-time representation weights. This process enables online, incremental optimization of the model's ability to understand environmental states. The error generated in each interaction is used to fine-tune the state representation network, allowing the network to dynamically adjust its focus in extracting features from the original data. This gradually teaches it to generate state representations that more accurately predict long-term returns, improving the model's adaptability and representation ability to complex environmental states.

[0154] In summary, the updated real-time representation weights are fused with the action feature vectors of the actions to be executed to generate a comprehensive prediction vector. This step combines optimized state understanding with specific action selection to produce a more accurate, future-oriented comprehensive prediction. This vector not only contains state evaluations based on the latest knowledge but also associates specific actions, providing higher-quality and more relevant target signals for subsequent policy parameter iterations, thereby directly improving the efficiency of the policy optimization loop and the performance of the final policy.

[0155] In this embodiment of the invention, the formula for calculating the strategy evaluation error is specifically the same as:

[0156]

[0157] in, The error is evaluated for the strategy. For the aforementioned instant reward, As a discount factor, This is the estimated value at the next time step. For the original value estimate, This is the index for discrete time steps.

[0158] Specifically, in the calculation of policy evaluation error, the immediate reward comes from the first scalar value directly extracted by the agent from the environmental feedback data after executing an action. The discount factor is a fixed constant pre-set during system initialization, used to adjust the weight of future rewards on the current estimate. The estimate for the next time step comes from the long-term reward prediction scalar calculated by the agent's internal value network based on the state vector of the next time step. The original value estimate comes from the long-term reward prediction scalar calculated by the same value network based on the decision state vector at the current time step. The discrete-time step index comes from the counting sequence of sequential interactions between the agent and the environment, identifying the position of the current interaction step.

[0159] Furthermore, this formula measures the difference between the agent's initial estimate of the value of the current state and the combined objective based on the actual reward and future predictions. The calculation first adds the next-time estimate obtained by multiplying the immediate reward by a discount factor, resulting in a new target value scalar. Then, the initial value estimate is subtracted from this target value scalar; the difference is the policy evaluation error. A positive error indicates that the initial estimate is lower than the actual reward, while a negative error indicates that the initial estimate is higher than the actual reward. Its absolute value directly reflects the degree of inaccuracy in the current value prediction.

[0160] In summary, during the agent's continuous learning process, the overall trend of the error value obtained by this calculation formula gradually approaches zero. With repeated adjustments to the weights of the value network, the agent's prediction of the state value becomes increasingly closer to the true target, which is composed of immediate rewards and future predictions, causing the fluctuation range of the error to continuously decrease. If the learning process is effective, the long-term mean of the error will stabilize near zero, signifying the convergence of value estimation and optimization of the policy; if the error continues to deviate from zero and shows no significant decreasing trend, it indicates that the learning process may require adjustment of preset parameters or that other stability issues exist.

[0161] In this embodiment of the invention, the step of iterating the policy parameters in the agent based on the prediction vector to obtain the optimization index of the agent is specifically used for:

[0162] Decouple the prediction vector to obtain the expected reward value of the agent;

[0163] Starting from the expected return value, the policy network of the agent is traversed in reverse to obtain the parameter gradient set of the agent;

[0164] Based on the feature gradient components in the parameter gradient set, the policy parameter values ​​in the agent are adjusted proportionally to obtain the optimization index of the agent.

[0165] Specifically, the composite prediction vector is a multi-dimensional vector in which the element at a specific index position is predefined as representing an estimate of the future cumulative return.

[0166] Specifically, the calculation process begins at the output layer of the policy network, where the output is the expected reward. First, the gradient of the activation value of the output layer neuron with respect to its input is calculated. Then, this gradient is passed down layer by layer along the network connections, following the chain rule.

[0167] Specifically, the system reads the gradient component corresponding to each weight from the parameter gradient set, multiplies the value of this gradient component by a preset, fixed, and positive number less than one, and the result of this product is the adjustment amount of the weight.

[0168] Furthermore, the decoupling operation involves directly accessing the vector, reading and outputting the scalar value at the specified index position, which is the agent's expected reward value.

[0169] Furthermore, at each layer, based on the input value, weight value, and gradient passed from the next layer, the partial derivative of each weight in the current layer with respect to the expected reward value is calculated. These partial derivatives are the gradient components of the weights in that layer. When the algorithm backtracks from the output layer to the input layer, it collects and organizes the weight gradient components calculated from all layers. The set of these gradient components constitutes the parameter gradient set of the agent.

[0170] Furthermore, the calculated adjustment amount is subtracted from the original value of the weight in the agent's policy network, and the resulting value is the updated policy parameter value. After repeating this process of "reading gradients, calculating adjustment amounts, and performing subtraction updates" for all weights in the policy network, all weight values ​​of the entire policy network are adjusted. This set of newly adjusted weight values ​​is defined as the agent's optimization metric.

[0171] In summary, extracting the element at a specified position from the comprehensive prediction vector as the expected reward value is a decoupling operation that transforms the vector output containing complex information into a single, explicit optimization objective scalar. This provides a clear and unique optimization direction for subsequent policy network parameter updates, allowing the entire backpropagation process to focus on directly improving this expected reward value. This solves the signal dispersion problem in multi-objective optimization, ensuring the stability of the learning process and the consistency of the objective.

[0172] In summary, starting with the expected return value, the gradient components corresponding to each weight parameter in the network are calculated backwards along the policy network structure using the backpropagation algorithm, forming a set of parameter gradients. This process precisely quantifies the degree and direction of the influence of each policy parameter on the final expected return. It efficiently and automatically distributes the error of the high-level objective to the low-level parameters, providing a complete mathematical basis for subsequent precise and targeted parameter adjustments, and is the core algorithmic support for training complex deep policy networks.

[0173] In summary, based on the calculated set of parameter gradients, all weights in the policy network are adjusted proportionally according to the gradient descent rule, ultimately yielding a new set of weight values ​​as optimization metrics. This step directly transforms gradient information into physical updates of model parameters, enabling iterative improvement of the policy. By continuously fine-tuning parameters along the gradient direction, the agent's decision-making policy is systematically guided to evolve in a direction that generates higher expected returns, thereby directly improving the performance of the policy model itself. This is the fundamental way for reinforcement learning agents to achieve capability growth.

[0174] Compared with the prior art, the present invention has the following beneficial effects:

[0175] 1. This invention constructs a decision state vector that simultaneously encompasses long-term static attributes and short-term dynamic behaviors by jointly encoding user profiles, interaction sequences, and actionable message sets. This method utilizes fully connected layers and temporal coding networks to extract high-order features and ensures complete information fusion through vector concatenation, thereby generating a state representation that accurately reflects the true state of the environment. This technique enhances the depth and accuracy of the agent's understanding of complex user states, laying a reliable foundation for subsequent policy decisions.

[0176] 2. This invention further achieves efficient action evaluation through feature dimension alignment and vector interaction calculation, and utilizes real-time feedback data to adjust the model's gradient in real time. This method integrates multi-dimensional feedback into the update process of state representation weights, and generates a future-oriented comprehensive prediction vector based on the adjusted weights, ultimately driving the targeted optimization of policy network parameters. These technical measures enhance the real-time performance and accuracy of policy iteration, directly improving the learning efficiency and final performance of the policy model.

[0177] like Figure 2 The diagram shown is a functional block diagram of a message push strategy optimization system based on deep reinforcement learning provided in an embodiment of the present invention.

[0178] The message push strategy optimization system 100 based on deep reinforcement learning described in this invention can be installed in an electronic device. Depending on the functions implemented, the message push strategy optimization system 100 based on deep reinforcement learning may include a feature extraction module 101, a preliminary value scalar module 102, an action to be executed module 103, a multi-dimensional feedback data module 104, a prediction vector module 105, and an optimization index module 106. The module described in this invention can also be called a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and which are stored in the memory of the electronic device.

[0179] In this embodiment, the functions of each module / unit are as follows:

[0180] The feature extraction module jointly encodes the static attributes, observation sequence, and selection identifier of the agent to obtain the decision state vector and action feature set of the agent.

[0181] The preliminary value scalar module performs a dot product operation between the decision state vector and the embedding vector of the action feature library set to obtain the preliminary value scalar of the agent;

[0182] The action to be executed module normalizes the preliminary value scalar to obtain the policy probability distribution of the agent, and determines the action to be executed of the agent based on the policy probability distribution;

[0183] The multidimensional feedback data module retrieves the environmental interaction data of the agent based on the action identifier to be executed, and obtains the multidimensional feedback vector of the agent.

[0184] The prediction vector module updates the state representation weights of the agent based on the multidimensional feedback vector, and constructs the prediction vector of the agent by combining the features of the action identifier to be executed.

[0185] The optimization index module iterates through the policy parameters in the agent based on the prediction vector to obtain the optimization index of the agent.

[0186] In the several embodiments provided by this invention, it should be understood that the disclosed methods and systems can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0187] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0188] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0189] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0190] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0191] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A message push strategy optimization method based on deep reinforcement learning, characterized in that, The method includes: The static attributes, observation sequences, and selection identifiers of the agent are jointly encoded to obtain the agent's decision state vector and action feature set, including: The user profile features of the intelligent agent are used as the static attributes, the message interaction records of the intelligent agent are used as the observation sequence, and the set of operable message types of the intelligent agent are used as the selection identifier. The static attributes are mapped to the fully connected layer of the agent to obtain the first feature vector of the agent; The observed sequence is temporally encoded to obtain the sequence dependency features of the agent; The first feature vector is concatenated with the sequence dependency feature to obtain the decision state vector of the agent; An embedding lookup is performed on the selected identifier to obtain the action feature set of the agent; The decision state vector is multiplied by the embedding vector of the action feature library to obtain the preliminary value scalar of the agent. The initial value scalar is normalized to obtain the policy probability distribution of the agent, and the action identifier to be executed by the agent is determined based on the policy probability distribution. Based on the action identifier to be executed, retrieve the environmental interaction data of the agent to obtain the multidimensional feedback vector of the agent; The state representation weights of the agent are updated based on the multidimensional feedback vector, and a prediction vector for the agent is constructed by combining the features of the action identifier to be executed, including: The agent's long-term reward estimate is calculated by difference based on the immediate reward in the multidimensional feedback vector to obtain the agent's policy evaluation error. The agent's long-term reward estimate is a scalar value calculated by the agent's internal value network based on the current decision state vector. Based on the strategy, the error is evaluated and the weights of the agent's state representation network are adjusted using a gradient to obtain the agent's instantaneous representation weights. The instantaneous representation weights are fused with the action feature vector corresponding to the action identifier to be executed to obtain the prediction vector; The optimization index of the agent is obtained by iterating the policy parameters in the agent based on the prediction vector.

2. The message push strategy optimization method based on deep reinforcement learning as described in claim 1, characterized in that, The step of performing a dot product operation between the decision state vector and the embedding vector of the action feature library set to obtain the preliminary value scalar of the agent includes: Align the decision state vector with the embedding vector of the action feature library according to the feature dimension to obtain the projected state vector and projected action vector of the agent; The interaction vector of the agent is obtained by multiplying the projection state vector and the projection action vector element by element according to their corresponding positions. The elements in the interaction vector are summed to obtain the preliminary value scalar of the agent.

3. The message push strategy optimization method based on deep reinforcement learning as described in claim 1, characterized in that, The process of normalizing the initial value scalar to obtain the agent's policy probability distribution, and determining the agent's action identifier based on the policy probability distribution, includes: The initial value scalar is exponentially scaled to obtain the agent's preference intensity value; The adjustment factor of the agent is obtained by aggregating the preference intensity values; Projecting the initial value scalar onto the adjustment factor yields the probability distribution of the agent; Based on the probability distribution, the selection identifier is randomly sampled to obtain the action identifier to be executed by the agent.

4. The message push strategy optimization method based on deep reinforcement learning as described in claim 1, characterized in that, The step of retrieving the agent's environmental interaction data based on the action identifier to obtain the agent's multidimensional feedback vector includes: Push messages to the agent according to the message type corresponding to the action to be executed; Collect the behavioral characteristics of the intelligent agent in response to the push message within a preset time window; Numerical reward signals are assigned to the agent based on the category corresponding to the behavioral characteristics; The reward signals are concatenated according to the feedback order to obtain the multidimensional feedback vector of the agent.

5. The message push strategy optimization method based on deep reinforcement learning as described in claim 4, characterized in that, Assigning numerical reward signals to the agent based on the category corresponding to the behavioral characteristics includes: Analyze the duration parameter and operation depth parameter in the behavioral characteristics; Based on the numerical range of the duration parameter and the operation depth parameter, the refinement level of the behavioral feature in the corresponding category is determined; Based on the refinement level, the reward parameters in the agent are forward-propagated to obtain the numerical reward signal of the agent.

6. The message push strategy optimization method based on deep reinforcement learning as described in claim 1, characterized in that, The formula for calculating the strategy evaluation error includes: in, The error is evaluated for the strategy. For the aforementioned instant reward, As a discount factor, for The estimated value at the next moment. For the original value estimate, This is the index for discrete time steps.

7. The message push strategy optimization method based on deep reinforcement learning as described in claim 1, characterized in that, The step of iterating the policy parameters in the agent based on the prediction vector to obtain the optimization index of the agent includes: Decouple the prediction vector to obtain the expected reward value of the agent; Starting from the expected return value, the policy network of the agent is traversed in reverse to obtain the parameter gradient set of the agent; Based on the feature gradient components in the parameter gradient set, the policy parameter values ​​in the agent are adjusted proportionally to obtain the optimization index of the agent.

8. A message push strategy optimization system based on deep reinforcement learning, used to implement the message push strategy optimization method based on deep reinforcement learning as described in any one of claims 1-7, characterized in that, The system includes: The feature extraction module jointly encodes the agent's static attributes, observation sequences, and selection identifiers to obtain the agent's decision state vector and action feature set, including: The user profile features of the intelligent agent are used as the static attributes, the message interaction records of the intelligent agent are used as the observation sequence, and the set of operable message types of the intelligent agent are used as the selection identifier. The static attributes are mapped to the fully connected layer of the agent to obtain the first feature vector of the agent; The observed sequence is temporally encoded to obtain the sequence dependency features of the agent; The first feature vector is concatenated with the sequence dependency feature to obtain the decision state vector of the agent; An embedding lookup is performed on the selected identifier to obtain the action feature set of the agent; The preliminary value scalar module performs a dot product operation between the decision state vector and the embedding vector of the action feature library set to obtain the preliminary value scalar of the agent; The action to be executed module normalizes the preliminary value scalar to obtain the policy probability distribution of the agent, and determines the action to be executed of the agent based on the policy probability distribution; The multidimensional feedback data module retrieves the environmental interaction data of the agent based on the action identifier to be executed, and obtains the multidimensional feedback vector of the agent. The prediction vector module updates the state representation weights of the agent based on the multidimensional feedback vector, and constructs the prediction vector of the agent by combining the features of the action identifier to be executed, including: The agent's long-term reward estimate is calculated by difference based on the immediate reward in the multidimensional feedback vector to obtain the agent's policy evaluation error. The agent's long-term reward estimate is a scalar value calculated by the agent's internal value network based on the current decision state vector. Based on the strategy, the error is evaluated and the weights of the agent's state representation network are adjusted using a gradient to obtain the agent's instantaneous representation weights. The instantaneous representation weights are fused with the action feature vector corresponding to the action identifier to be executed to obtain the prediction vector; The optimization index module iterates through the policy parameters in the agent based on the prediction vector to obtain the optimization index of the agent.

Images