A bulletin accurate pushing optimization method based on reinforcement learning

Abstract

The application relates to the technical field of artificial intelligence, and discloses a bulletin precise pushing optimization method based on reinforcement learning, which specifically comprises the following steps: collecting user logs and bulletin data, and splicing the data into a state vector through attenuation weighting and coding; taking the state vector as a condition, a diffusion generator trained by offline data generates K candidate actions through noise adding and noise removing, each of which contains a bulletin, a delay and a channel; inputting the candidate actions into a Q integrated body trained by an isotropy loss and a TD error, correcting the Q value by a conservative penalty positively correlated with the Q variance, and executing the maximum one; collecting feedback rewards, and storing a four-tuple composed of the front and back states and actions into a pool; calculating TD errors, isotropy losses and regular terms after batch sampling, and updating the two modules by weighting, and performing cyclic iteration. The application adopts diffusion generation and isotropy CQL, realizes multi-modal action generation and adaptive conservative evaluation, and takes into account exploration efficiency and long-term retention.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an optimization method for accurate announcement push based on reinforcement learning. Background Technology

[0002] Announcement push systems are crucial channels connecting users and information on internet platforms, and the quality of their push strategies directly impacts user experience and platform operational efficiency. With the expansion of the user base and dynamic changes in interests, static rule-based push methods struggle to cope with real-time shifts in user preferences. Deep reinforcement learning-based recommendation systems, with their sequential decision-making and long-term revenue modeling capabilities, have gained attention.

[0003] Offline reinforcement learning aims to learn policies from pre-collected static datasets without real-time interaction with the environment. Batch-Constrained Q-learning (BCQ) constrains the action space by learning the distribution of historical data, mitigating the overestimation problem caused by distribution bias; Conservative Q-learning (CQL) adds a regularization penalty term to the Bellman objective to suppress the Q-value estimation of out-of-distribution actions. However, these methods still have significant limitations in push notification scenarios: BCQ relies on conditional variational autoencoders (VAEs) to generate candidate actions, and VAEs tend to generate mean regression results from historical data, making it difficult to cover the diverse needs of long-tail users; CQL applies a uniformly conservative penalty to all out-of-distribution actions, failing to distinguish between high-uncertainty and relatively certain regions, resulting in overly conservative policies even in regions where accurate estimation is possible. Furthermore, existing methods rarely consider the randomness of user feedback in push notification scenarios and the data sparsity problem during the cold start phase for new users. Summary of the Invention

[0004] One objective of this invention is to propose an optimization method for accurate announcement push based on reinforcement learning. This invention introduces a conditional diffusion probability model to replace the variational autoencoder, integrates equivariant priors and conservative Q-learning frameworks to construct a Q-network ensemble, performs symmetric transformation and equivariant loss constraints on the state space, performs corrected Q-value evaluation and selection on candidate actions, and constructs an experience quadruple stored in the experience replay pool. It has the advantages of high announcement push accuracy, strong cold start adaptability, and good synergy between long-term and short-term benefits.

[0005] An optimization method for accurate announcement push based on reinforcement learning according to an embodiment of the present invention includes the following steps: Collect user logs and announcement data, obtain interest features by weighting log time decay, encode announcement categories and time expiration marks, and concatenate them to obtain a state vector; Based on the BCQ learning framework, an offline state-action pair is used to train a conditional diffusion generator instead of a standard variational autoencoder. The state vector is used as the conditional input to the diffusion generator. Forward noise is added to perturb the historical actions to pure Gaussian noise. Inverse noise is removed step by step with the current state as the condition to recover the complete action. After sampling and denoising from several random noise seeds, K candidate actions are output. The candidate actions include announcement identifier, push delay duration, and channel code. Based on the CQL learning framework, N Q sub-networks are integrated. A state translation and reflection symmetry group is defined. An isovariant loss is constructed to measure the mean square error of the Q value after inverse mapping of the original state and the transformed state. The isovariant loss and the TD error are weighted as the sub-network training loss. Conservative regularization is introduced to superimpose a penalty value on the Bellman objective. The penalty value is positively correlated with the variance of the N Q outputs. The candidate action inputs are integrated to obtain the base Q-mean, the penalty is subtracted to obtain the corrected Q, and the action with the largest corrected Q value is selected to be pushed. Collect user clicks, reads, and closes feedback to calculate rewards; Use a quadruple of reward, previous state, action, and next state to store the experience pool; Randomly sample batches of quadruples from the experience pool; Calculate the TD error, isovariance loss, and regularization term, and then perform a weighted sum to jointly update the diffusion generator and Q-integral. Use the updated parameters to re-execute candidate action generation and push decision.

[0006] Optionally, the interest features weighted by log time decay specifically include: User behavior logs are pre-weighted for behavior types, which include browsing, clicking, reading in full, and closing. For each log entry, a decay coefficient is determined based on the time difference between the time the behavior occurred and the current time. Adjust the behavior type weight of this log entry using the decay coefficient to obtain the contribution value of this log entry; sum the contribution values ​​of all log entries of the same user according to the announcement category dimension to obtain the interest score vector of this user in each announcement category; The interest score vector is normalized, and the normalized interest score vector is used as the interest feature.

[0007] Optionally, the training conditional diffusion generator, instead of the standard variational autoencoder, includes: The conditional diffusion generator adopts a denoised diffusion probability model architecture, which includes a forward denoising process and a reverse denoising process; The forward noise addition process sets a total number of steps T, with T ranging from 100 to 1000. A linear noise scheduling strategy is adopted, and Gaussian noise with zero mean and variance increasing with the number of steps is added to the current sample at each step, forming a Markov chain from the initial action sample to the standard Gaussian distribution. When the sample distribution reaches the Tth step, it is approximately a standard Gaussian distribution. The reverse denoising process learns from the standard Gaussian distribution and gradually restores the conditional distribution of the complete action along T steps. Each step uses the current state vector as the guiding condition and uses a trainable noise prediction network to receive three inputs: the noise sample of the current step, the current step index, and the state vector, and outputs the predicted noise added in this step. The noise prediction network adopts a U-shaped network structure, which includes a downsampling encoding path and an upsampling decoding path. Features are passed between each layer through skip connections, and the state vector is injected into the network at each decoding layer through a cross-attention mechanism. During the training phase, historical state-action pairs are randomly sampled from the offline dataset. For each action sample, a step index is randomly extracted. The noise sample and the real noise are generated according to the forward process. The noise sample, step index, and conditional state vector are input into the noise prediction network. The mean square error between the predicted noise and the real noise is calculated as the loss for this step. The losses of all steps are weighted and summed according to the preset weight table to obtain the total training loss. After training, the reverse denoising process can generate complete actions from random Gaussian noise step by step, consistent with the offline data distribution and constrained by the current state, given any current state vector.

[0008] Optionally, the step of sampling and denoising from several random noise seeds and then outputting K candidate actions includes: At each online decision-making moment, K random noise seeds are independently sampled from a standard Gaussian distribution. The value of K is preset to a positive integer between 5 and 20 according to business needs, and each noise seed is different from the others. K noise seeds are input into the trained conditional diffusion generator in parallel. Each noise seed independently executes the complete reverse denoising process. Each step is guided by the same state vector at the current time. After T iterations, the K noise seeds converge into K distinct complete actions. Each action corresponds to a different combination of announcement identifier, push delay duration and channel code. Apply the batch constraint mechanism of the BCQ framework to K complete actions: train an independent conditional variational autoencoder or Gaussian mixture model to learn the joint distribution of state-action in the offline dataset, use this model as an action validity discriminator, and calculate the conditional probability density estimate of each generated action in the given current state. When the probability density value is lower than a preset threshold, the action is rejected, and actions with a probability density value higher than the threshold are retained to form a candidate action set. If the number of candidate actions after constraint screening is insufficient, select several actions with the highest probability density values ​​from the remaining unscreened actions to make up the preset number K; The announcement identifier uses a one-hot encoded vector to represent the unique identity of the announcement to be pushed. The push delay duration is quantified in seconds to specify the push waiting time. The channel code uses discrete numerical values ​​to mark the type of push channel.

[0009] Optionally, the construction of the N Q-sub-network ensemble based on the CQL learning framework includes: Construct N independently initialized Q subnetworks. Each Q subnetwork uses the same neural network structure but has different parameter initialization values. N is a positive integer between 3 and 10.

[0010] The input to each Q-subnetwork is a state-action pair formed by concatenating the state vector and the action, and the output is a scalar Q-value; Define a set of symmetric transformation groups for the state space where the state vector is located. Each transformation in the group maps the state vector to the transformed state vector. The transformations include translation and reflection transformations in each dimension of the state space, and all transformations are invertible. Each transformation corresponds to a unique inverse transformation. For each Q-subnetwork, construct an isovariant loss term: sample the original state vector from the state space, apply any transformation from the symmetric group to the original state vector to obtain the transformed state vector, form a first state-action pair with the candidate action, input the first Q-value into the Q-subnetwork, form a second state-action pair with the transformed state vector and the same candidate action, input the second Q-value into the same Q-subnetwork, apply the inverse transformation corresponding to the transformation to the second Q-value to obtain the mapped Q-value, and calculate the mean square error between the first Q-value and the mapped Q-value as the isovariant loss under the transformation; For each Q subnetwork, the mean of the isovariant loss is calculated under all transformations and at each sampling state as the isovariant loss term of the subnetwork; The individual training loss of each Q subnetwork is composed of a weighted sum of an isovariant loss term and a standard temporal difference error term. The weight coefficient of the isovariant loss term is preset to a positive value, and the weight coefficient of the standard temporal difference error term is preset to 1. Each Q subnetwork updates its network parameters independently based on its own individual training loss.

[0011] Optionally, the introduction of conservative regularization in the Bellman objective superposition penalty value includes: Within the CQL framework, a conservative regularization penalty value is superimposed on the Bellman update objective of each Q subnetwork; The penalty value is calculated as follows: the current state vector and the candidate action are combined to form a state-action pair, which are then input into N Q sub-networks to obtain N Q-value outputs. The variance of the N Q-value outputs is calculated, and the variance is multiplied by a preset positive coefficient to obtain the penalty value. The conservative regularization loss of each Q subnetwork is the expected penalty value over all state-action pairs, which is estimated by averaging the batch of state-action pairs sampled from the experience replay pool. The complete training objective function of each Q subnetwork contains three terms: the standard temporal difference error term is used to ensure the temporal consistency of Q value estimation, the isovariance loss term is used to constrain the output stability of the Q function under state symmetric transformation, and the conservative regularization loss term is used to suppress the Q value estimation of out-of-distribution state-action pairs. The three losses are weighted and summed according to preset weight coefficients to obtain the total training loss of each Q sub-network. The weight coefficient of the conservative regularization loss term is positively correlated with the global variance of the output values ​​of the N Q sub-networks. The penalty value is included as a subtraction term in the Bellman objective during Q-value updates.

[0012] Optionally, the action of selecting the maximum corrected Q value to perform the push specifically includes: Each candidate action in the candidate action set is paired with the current state vector to form a state-action pair, which are then input into N Q-subnetworks. The arithmetic mean of the N Q values ​​for the same state-action pair is taken to obtain the basic Q-mean. The penalty value corresponding to the state-action pair is obtained according to the penalty value calculation method. The penalty value is subtracted from the basic Q-mean to obtain the corrected Q value. The K candidate actions are traversed, and their corrected Q values ​​are calculated. The candidate action with the largest value is selected as the optimal push decision.

[0013] Extract the announcement identifier from the optimal push decision to determine the announcement to be pushed, extract the push delay duration to determine the waiting time before push, and extract the channel code to determine the push channel; after the waiting time expires, send the announcement to the target user through the channel; record the actual push time after the push is completed.

[0014] Optionally, the process of storing the experience pool using a quadruple consisting of reward, previous state, action, and subsequent state specifically includes: The last data collection before the push is executed is used to generate a state vector as the previous state. The optimal push decision is used as the action. The scalar reward value calculated based on user interaction feedback is used as the reward. The user logs and announcement data are collected again after the push is executed, and a new state vector is generated by concatenating them as the next state. The four elements of the previous state, action, reward and next state are combined in order to form an experience quadruple.

[0015] The experience quadruples are added as independent records to the experience replay pool. The experience replay pool adopts a first-in-first-out queue structure. When the total number of experience quadruples stored in the pool reaches the preset capacity limit, the experience quadruples with the earliest storage time are deleted to free up storage space. The newly stored experience quadruples are marked with a timestamp.

[0016] Optionally, the step of randomly sampling a batch of quadruplets from the experience pool specifically includes: When the number of experience quadruples stored in the experience replay pool reaches the preset minimum sampling threshold, a batch sampling operation is triggered. B experience quadruples are randomly and uniformly drawn from the experience replay pool. Each quadruple has an equal probability of being drawn and each drawing is independent of the others. After the drawing, the original quadruples are still retained in the pool. The B drawn quadruples are organized into a training batch. In this batch, the previous state, action, reward, and post-state of each quadruple are aligned into four tensors.

[0017] Optionally, the calculation of TD error, isovariance loss, and regularization term, followed by a weighted summation and joint update of the diffusion generator and Q-integral, specifically includes: Extract B quadruplets from the training batch. Each quadruplet contains a previous state, an action, a reward, and a subsequent state. Input the previous state and action into the Q-integral to obtain the current Q-value. Input the subsequent state into the target Q-network to obtain the TD target value. Calculate the mean square error of both as the TD error. Substitute the isovariance loss term and the regularization term into the current batch and recalculate. Sum the TD error, isovariance loss, and regularization term according to preset weights to obtain the total loss. Use the total loss to update the gradients of the diffusion generator and the Q-integral. Replace the old parameters with the new parameters and re-execute the candidate action generation and push decision until training converges.

[0018] The beneficial effects of this invention are: (1) The present invention uses a conditional diffusion probability model to replace the traditional variational autoencoder. It generates a set of candidate actions from multiple random noise seeds through forward noise addition and reverse noise reduction processes. This can cover the multimodal distribution in historical data, avoid action generation from falling into mean regression, and significantly improve the diversity of push strategies and the coverage of long-tail users.

[0019] (2) Based on the CQL framework, this invention introduces the translation and reflection of the state space as a priori constraint, and uses the equivariance loss to guide the Q network to learn the output consistency under the state transformation. It can mine the symmetric structure of the data from small samples, significantly improve the Q value estimation accuracy in the data sparse scenario, and shorten the cold start time of new users.

[0020] (3) The present invention positively correlates the conservative regularization penalty value with the output variance of the Q-integrated network, automatically applies stronger penalties in the external region of the data distribution and weakens penalties in the dense data region, thereby achieving differentiated control of the conservative strength, solving the problem of the strategy being too conservative due to the uniform penalty of traditional CQL, and improving the overall benefit of the push system.

[0021] (4) This invention forms a closed-loop optimization mechanism of “generation-evaluation-execution-feedback-learning” by combining the conditional diffusion generator and the Q network ensemble through gradient update, so that the system can maintain long-term user retention while ensuring short-term click-through rate, and realize the continuous autonomous evolution of push strategy. Attached Figure Description

[0022] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a method for optimizing the accurate push of announcements based on reinforcement learning, as proposed in this invention. Figure 2 This is a schematic diagram of the candidate action generation process of the conditional diffusion generator based on the BCQ framework in the proposed method for optimizing accurate announcement push based on reinforcement learning. Figure 3 This is a schematic diagram of the equivariance conservative Q-value evaluation process based on the CQL framework for an optimization method for accurate announcement push based on reinforcement learning proposed in this invention. Detailed Implementation

[0023] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0024] refer to Figures 1-3 A reinforcement learning-based method for optimizing precise announcement push notifications includes the following steps: Collect user logs and announcement data, obtain interest features by weighting log time decay, encode announcement categories and time expiration marks, and concatenate them to obtain a state vector; Based on the BCQ learning framework, an offline state-action pair is used to train a conditional diffusion generator instead of a standard variational autoencoder. The state vector is used as the conditional input to the diffusion generator. Forward noise is added to perturb the historical actions to pure Gaussian noise. Inverse noise is removed step by step with the current state as the condition to recover the complete action. After sampling and denoising from several random noise seeds, K candidate actions are output. The candidate actions include announcement identifier, push delay duration, and channel code. Based on the CQL learning framework, N Q sub-networks are integrated. A state translation and reflection symmetry group is defined. An isovariant loss is constructed to measure the mean square error of the Q value after inverse mapping of the original state and the transformed state. The isovariant loss and the TD error are weighted as the sub-network training loss. Conservative regularization is introduced to superimpose a penalty value on the Bellman objective. The penalty value is positively correlated with the variance of the N Q outputs. The candidate action inputs are integrated to obtain the base Q-mean, the penalty is subtracted to obtain the corrected Q, and the action with the largest corrected Q value is selected to be pushed. Collect user clicks, reads, and closes feedback to calculate rewards; Use a quadruple of reward, previous state, action, and next state to store the experience pool; Randomly sample batches of quadruples from the experience pool; Calculate the TD error, isovariance loss, and regularization term, and then perform a weighted sum to jointly update the diffusion generator and Q-integral. Use the updated parameters to re-execute candidate action generation and push decision.

[0025] In this embodiment, the interest features obtained by weighting log time decay specifically include: User behavior logs are pre-weighted for behavior types, which include browsing, clicking, reading in full, and closing. For each log entry, a decay coefficient is determined based on the time difference between the time the behavior occurred and the current time. Adjust the behavior type weight of this log entry using the decay coefficient to obtain the contribution value of this log entry; sum the contribution values ​​of all log entries of the same user according to the announcement category dimension to obtain the interest score vector of this user in each announcement category; The interest score vector is normalized, and the normalized interest score vector is used as the interest feature.

[0026] Specifically, the preset behavior type weights for user behavior logs include: User behavior logs are pre-defined with four behavior types: browsing (initially weighted at 0.2), clicking (initially weighted at 0.5), reading completely (initially weighted at 0.8), and closing (initially weighted at -0.3). Browsing refers to an announcement being displayed on a user's device screen for more than 1 second without being clicked; clicking refers to a user actively clicking on the announcement title or content area; fully reading refers to a user staying on the announcement content page for more than 15 seconds; and closing refers to a user actively clicking the close button in the upper right corner of the announcement or swiping left to close the announcement card. The absolute value of the weight range for each behavior type is set to an upper limit of 1.0 and a lower limit of -1.0. Positive weights represent positive interest contributions, and negative weights represent negative interest contributions.

[0027] Specifically, for each log entry, the attenuation coefficient is determined based on the time difference between the time the action occurred and the current time. When constructing a state vector, the system uses the current system time as the base time and calculates the time difference between the occurrence time of each user behavior log record and the base time. The time difference is measured in seconds, and the system presets a decay half-life parameter with a value of 604,800 seconds, corresponding to 7 calendar days. The formula for calculating the decay coefficient is: the decay coefficient is equal to the negative power of 2, where the power is equal to the time difference divided by the decay half-life parameter.

[0028] When calculating the contribution value, the system multiplies the behavior type weight by the decay coefficient to obtain the contribution value of this log entry. The decay half-life parameter is set to 604,800 seconds during system initialization and can be dynamically adjusted according to business scenarios. For scenarios with high announcement push frequency, the decay half-life can be shortened to 3 days to accelerate interest updates; for announcement types with slow interest changes, the decay half-life can be extended to 14 days to retain the cumulative impact of historical behavior. The system administrator can adjust the parameters through the configuration file or management interface. After the adjustment, the system recalculates the decay coefficient and updates the interest features in the subsequent state vector. For newly registered users, due to the lack of historical logs, the system uniformly fills each dimension of the interest score vector with the initial values ​​of the interest categories filled in by the user during registration. The initial values ​​are weighted equally according to the interest categories selected by the user in the registration questionnaire. At the same time, the system starts to accumulate and calculate from the interaction logs generated after the push. In addition, the system assigns a unique identifier to each push event. After the push is completed, the actual push time of the push event is recorded and stored as a timestamp in the experience replay pool when the quadruple is updated. This is used to calculate the time weight of each experience during sampling, so that recent experiences account for a higher proportion in the overall optimization objective.

[0029] In this embodiment, the training conditional diffusion generator to replace the standard variational autoencoder includes: The conditional diffusion generator adopts a denoised diffusion probability model architecture, which includes a forward denoising process and a reverse denoising process; The forward noise addition process sets a total number of steps T, with T ranging from 100 to 1000. A linear noise scheduling strategy is adopted, and Gaussian noise with zero mean and variance increasing with the number of steps is added to the current sample at each step, forming a Markov chain from the initial action sample to the standard Gaussian distribution. When the sample distribution reaches the Tth step, it is approximately a standard Gaussian distribution. The reverse denoising process learns from the standard Gaussian distribution and gradually restores the conditional distribution of the complete action along T steps. Each step uses the current state vector as the guiding condition and uses a trainable noise prediction network to receive three inputs: the noise sample of the current step, the current step index, and the state vector, and outputs the predicted noise added in this step. The noise prediction network adopts a U-shaped network structure, which includes a downsampling encoding path and an upsampling decoding path. Features are passed between each layer through skip connections, and the state vector is injected into the network at each decoding layer through a cross-attention mechanism. During the training phase, historical state-action pairs are randomly sampled from the offline dataset. For each action sample, a step index is randomly extracted. The noise sample and the real noise are generated according to the forward process. The noise sample, step index, and conditional state vector are input into the noise prediction network. The mean square error between the predicted noise and the real noise is calculated as the loss for this step. The losses of all steps are weighted and summed according to the preset weight table to obtain the total training loss. After training, the reverse denoising process can generate complete actions from random Gaussian noise step by step, consistent with the offline data distribution and constrained by the current state, given any current state vector.

[0030] In the forward noise addition process, the total number of steps T is set, with T ranging from 100 to 1000. The linear noise scheduling strategy specifically includes: Set the total number of steps T, which is preset to an integer value between 200 and 800 based on the dimension and complexity of the action vectors in the offline dataset. Define a set of noise variance sequences from step 1 to step T, where the noise variance of step 1 is 0.0001, the noise variance of step T is 0.02, and the noise variance of each intermediate step is uniformly distributed in a linearly increasing manner between 0.0001 and 0.02. The linearly increasing means that the noise variance of this step is determined by the proportion of the current step in the total number of steps. Step 1 corresponds to the initial variance of 0.0001, step T corresponds to the final variance of 0.02, and the intermediate steps are uniformly distributed proportionally.

[0031] Specifically, the noise prediction network adopts a U-shaped network structure as follows: A symmetrical codec network architecture, named for its resemblance to the English letter "U". This structure consists of a downsampling encoding path on the left and an upsampling decoding path on the right, with the two paths connected laterally via jump links; The downsampling encoding path gradually reduces the spatial resolution of the feature map through convolutional and pooling layers while increasing the number of channels, extracting abstract semantic features of the input data layer by layer. The upsampling decoding path gradually restores the spatial resolution of the feature map through deconvolution or interpolation operations while reducing the number of channels, gradually restoring the abstract features to an output with the same size as the original input. Skip connections directly pass the feature maps of each level of the encoding path to the corresponding level of the decoding path, enabling the decoding layer to simultaneously utilize the high-level semantic features and low-level detail features extracted by the encoding layer, making up for the spatial detail information lost during downsampling and improving the accuracy of the output. In this invention, the U-shaped network serves as the backbone structure of the noise prediction network. It is responsible for receiving noisy samples, predicting the added noise, extracting multi-scale features of the noise samples through the encoding path, restoring them to their original size through the decoding path, and outputting a predicted noise tensor with the same size as the input sample.

[0032] In this embodiment, the output of K candidate actions after sampling and denoising from several random noise seeds includes: At each online decision-making moment, K random noise seeds are independently sampled from a standard Gaussian distribution. The value of K is preset to a positive integer between 5 and 20 according to business needs, and each noise seed is different from the others. K noise seeds are input into the trained conditional diffusion generator in parallel. Each noise seed independently executes the complete reverse denoising process. Each step is guided by the same state vector at the current time. After T iterations, the K noise seeds converge into K distinct complete actions. Each action corresponds to a different combination of announcement identifier, push delay duration and channel code. Apply the batch constraint mechanism of the BCQ framework to K complete actions: train an independent conditional variational autoencoder or Gaussian mixture model to learn the joint distribution of state-action in the offline dataset, use this model as an action validity discriminator, and calculate the conditional probability density estimate of each generated action in the given current state. When the probability density value is lower than a preset threshold, the action is rejected, and actions with a probability density value higher than the threshold are retained to form a candidate action set. If the number of candidate actions after constraint screening is insufficient, select several actions with the highest probability density values ​​from the remaining unscreened actions to make up the preset number K; The announcement identifier uses a one-hot encoded vector to represent the unique identity of the announcement to be pushed. The push delay duration is quantified in seconds to specify the push waiting time. The channel code uses discrete numerical values ​​to mark the type of push channel.

[0033] In this embodiment, constructing an ensemble of N Q-subnetworks based on the CQL learning framework includes: Construct N independently initialized Q subnetworks. Each Q subnetwork uses the same neural network structure but has different parameter initialization values. N is a positive integer between 3 and 10.

[0034] The input to each Q-subnetwork is a state-action pair formed by concatenating the state vector and the action, and the output is a scalar Q-value; Define a set of symmetric transformation groups for the state space where the state vector is located. Each transformation in the symmetric transformation group maps the state vector to the transformed state vector. The transformations include translation transformations and reflection transformations in each dimension of the state space. All transformations are invertible and each transformation corresponds to a unique inverse transformation. For each Q-subnetwork, construct an isovariant loss term: sample the original state vector from the state space, apply any transformation from the symmetric group to the original state vector to obtain the transformed state vector, form a first state-action pair with the candidate action, input the first Q-value into the Q-subnetwork, form a second state-action pair with the transformed state vector and the same candidate action, input the second Q-value into the same Q-subnetwork, apply the inverse transformation corresponding to the transformation to the second Q-value to obtain the mapped Q-value, and calculate the mean square error between the first Q-value and the mapped Q-value as the isovariant loss under the transformation; For each Q subnetwork, the mean of the isovariant loss is calculated under all transformations and at each sampling state as the isovariant loss term of the subnetwork; The individual training loss of each Q subnetwork is composed of a weighted sum of an isovariant loss term and a standard temporal difference error term. The weight coefficient of the isovariant loss term is preset to a positive value, and the weight coefficient of the standard temporal difference error term is preset to 1. Each Q subnetwork updates its network parameters independently based on its own individual training loss.

[0035] The transformations include translation and reflection transformations in each dimension of the state space, specifically: The translation transformation operates on any dimension of the state vector, adding or subtracting a predetermined offset from the original value of that dimension. For the user interest feature dimension, the offset is 0.05; for the timeliness marker and channel load rate dimension, the offset is 0.1; for the interval since the last push dimension, the offset is 60 seconds. Each transformation operates on only one dimension, while the other dimensions remain unchanged. If the transformation result exceeds the valid value range of this dimension, it is truncated to the interval boundary value. The reflection transformation mirrors any dimension of the state vector, inverting the value of that dimension around a preset center point. For the user interest feature dimension, timeliness marker dimension, and channel load rate dimension, the original value is mapped to 1 minus the original value, with 0.5 as the center point. For the interval since the last push dimension, the original duration is mapped to the center point multiplied by 2 minus the original duration, with half of the maximum interval duration as the center point. For the one-hot encoding dimension of the announcement category, the reflection transformation is defined as its own identity mapping to maintain the legal structure of the one-hot encoding. All transformations are reversible. The inverse of a translation transformation is a translation in the opposite direction, and the inverse of a reflection transformation is itself. By traversing and combining all dimensions and all transformation types, a complete group of symmetric transformations is constructed.

[0036] Specifically, the mean of the isovariant loss calculated under all transformations and at each sampling state is used as the isovariant loss term of the subnetwork, including: Extract a set of sampled states from the current training batch. The set of sampled states is the B state vector samples in the training batch, where B is the preset batch size and is a positive integer between 32 and 256. For each state vector in the sampled state set, apply all transformation operations in the symmetric transformation group in sequence, including translation and reflection transformations for each dimension of the state vector, to obtain the set of all transformed state vectors corresponding to this state vector. For each sampled state vector and its corresponding transformed state vector, a state-action pair is formed with the same action, input into the Q sub-network, and the mean square error between the Q value in the original state and the Q value after inverse transformation in the transformed state is calculated as the single isovariability loss of this sampled state under this transformation. For all state vectors in the sampled state set and all transformations in the symmetric transformation group, calculate the above-mentioned single equivariance loss respectively, sum all the single equivariance loss values, and then divide by the product of the number of sampled states and the total number of transformations to obtain the mean of the equivariance loss term of the subnetwork. This mean of the equivariance loss term reflects the average ability of the Q subnetwork to maintain equivariance under all representative states and all symmetric transformations. The smaller the mean, the stronger the output stability of the subnetwork for symmetric transformations in the state space. During training, each Q-subnetwork independently maintains its own mean calculation of the equivariance loss term, without interfering with each other. The mean of the equivariance loss term and the standard temporal difference error term are weighted and summed according to a preset weight coefficient, and then used as the individual training loss of the subnetwork to participate in backpropagation parameter updates. The preset weight coefficient is 0.1. In the early stage of training, the equivariance loss weight can be increased to 0.3 to strengthen the guiding role of the equivariance constraint, and then adjusted back to 0.1 in the later stage of training to focus on the convergence of the temporal difference error. The traversal calculation of each sampling state and all transformations is repeated in each training batch, and the equivariance loss value is dynamically updated as the training batch is updated.

[0037] The weight coefficient of the standard temporal difference error term is preset to 1. Each Q-subnetwork independently updates its network parameters based on its own individual training loss, specifically including: Each Q subnetwork has an independent target Q network. The parameters of each target Q network are periodically obtained from the current parameters of the corresponding Q subnetwork in a soft update manner. The soft update coefficient is preset to 0.005, that is, each time the target network parameter is equal to 0.995 multiplied by the previous target network parameter plus 0.005 multiplied by the current Q subnetwork parameter. Each Q-subnetwork and its respective target network's optimizer are maintained independently and do not interfere with each other. In the first 5000 steps of training, the isovariance loss weight coefficient is set to 0.3 to guide the network to prioritize learning isovariance constraints; thereafter, it is restored to 0.1. After each Q subnetwork is updated, the latest parameters are stored in the parameter server for online evaluation of the next round of candidate actions.

[0038] Specifically, the definition of the state translational reflection symmetry group includes: A symmetric transformation group is formed by all translation and reflection transformations in all dimensions of the state space. The set of elements in the group contains two types of transformation operations: translation transformations apply a positive or negative offset to any dimension of the state vector, and reflection transformations perform a numerical mirror flip of any dimension of the state vector around a preset center point. The combination operation of the group is defined as the sequential application of transformations, and the combination result of any two transformations still belongs to the group. For any dimension of the state vector, the translation transformation includes two directions: positive translation and negative translation. Assuming the absolute value of the offset is a preset constant, positive translation increases the offset in the current value of the dimension, negative translation decreases the offset in the current value of the dimension, and reflection transformation symmetrically flips the current value of the dimension around the center point. For multiple dimensions of the state vector, the group allows transformations to be applied to multiple dimensions simultaneously, but only one transformation operation is applied to the same dimension at the same time. A group satisfies the closure condition: applying any combination of two group transformations to the state vector is equivalent to applying any one of the group transformations. A group satisfies the identity element condition: a translation transformation with zero offset or an identity mapping at the reflection center constitutes the identity element, preserving the state vector unchanged. The group satisfies the inverse condition: the inverse of each translation transformation is the translation transformation in the opposite direction, and the inverse of each reflection transformation is itself, so that the group has a complete mathematical structure of a symmetric group. By traversing and combining all transformation elements and all dimensions in the state space, a complete symmetric transformation group is formed.

[0039] In this embodiment, the conservative regularization introduced in the Bellman objective superposition penalty value includes: Within the CQL framework, a conservative regularization penalty value is superimposed on the Bellman update objective of each Q subnetwork; The conservative regularization loss of each Q subnetwork is the expected penalty value over all state-action pairs, which is estimated by averaging the batch of state-action pairs sampled from the experience replay pool. The complete training objective function of each Q subnetwork contains three terms: the standard temporal difference error term is used to ensure the temporal consistency of Q value estimation, the isovariance loss term is used to constrain the output stability of the Q function under state symmetric transformation, and the conservative regularization loss term is used to suppress the Q value estimation of out-of-distribution state-action pairs. The three losses are weighted and summed according to preset weight coefficients to obtain the total training loss of each Q sub-network. The weight coefficient of the conservative regularization loss term is positively correlated with the global variance of the output values ​​of the N Q sub-networks. The penalty value is included as a subtraction term in the Bellman objective during Q-value updates.

[0040] The regularization penalty value is calculated as follows: The current state vector and candidate actions are combined to form a state-action pair, which are then input into N Q-subnetworks to obtain N Q-value outputs. The variance of the N Q-value outputs is calculated, and the variance is multiplied by a preset positive coefficient to obtain the penalty value.

[0041] In this embodiment, the action of selecting the maximum corrected Q value to perform the push specifically includes: Each candidate action in the candidate action set is paired with the current state vector to form a state-action pair, which are then input into N Q-subnetworks. The arithmetic mean of the N Q values ​​for the same state-action pair is taken to obtain the basic Q-mean. The penalty value corresponding to the state-action pair is obtained according to the penalty value calculation method. The penalty value is subtracted from the basic Q-mean to obtain the corrected Q value. The K candidate actions are traversed, and their corrected Q values ​​are calculated. The candidate action with the largest value is selected as the optimal push decision. Extract the announcement identifier from the optimal push decision to determine the announcement to be pushed, extract the push delay duration to determine the waiting time before push, and extract the channel code to determine the push channel; after the waiting time expires, send the announcement to the target user through the channel; record the actual push time after the push is completed.

[0042] In this embodiment, the experience pool is specifically composed of a quadruple consisting of reward, previous state, action, and subsequent state: The last data collection before the push is executed is used to generate a state vector as the previous state. The optimal push decision is used as the action. The scalar reward value calculated based on user interaction feedback is used as the reward. The user logs and announcement data are collected again after the push is executed, and a new state vector is generated by concatenating them as the next state. The four elements of the previous state, action, reward and next state are combined in order to form an experience quadruple.

[0043] The experience quadruples are added as independent records to the experience replay pool. The experience replay pool adopts a first-in-first-out queue structure. When the total number of experience quadruples stored in the pool reaches the preset capacity limit, the experience quadruples with the earliest storage time are deleted to free up storage space. The newly stored experience quadruples are marked with a timestamp.

[0044] In this embodiment, randomly sampling batch quadruples from the experience pool specifically includes: When the number of experience quadruples stored in the experience replay pool reaches the preset minimum sampling threshold, a batch sampling operation is triggered. B experience quadruples are randomly and uniformly drawn from the experience replay pool. Each quadruple has an equal probability of being drawn and each drawing is independent of the others. After the drawing, the original quadruples are still retained in the pool. The B drawn quadruples are organized into a training batch. In this batch, the previous state, action, reward, and post-state of each quadruple are aligned into four tensors.

[0045] In this embodiment, the calculation of TD error, isovariance loss, and regularization term, followed by a weighted summation and joint update of the diffusion generator and Q-integral, specifically includes: Extract B quadruplets from the training batch. Each quadruplet contains a previous state, an action, a reward, and a subsequent state. Input the previous state and action into the Q-integral to obtain the current Q-value. Input the subsequent state into the target Q-network to obtain the TD target value. Calculate the mean square error of both as the TD error. Substitute the isovariance loss term and the regularization term into the current batch and recalculate. Sum the TD error, isovariance loss, and regularization term according to preset weights to obtain the total loss. Use the total loss to update the gradients of the diffusion generator and the Q-integral. Replace the old parameters with the new parameters and re-execute the candidate action generation and push decision until training converges.

[0046] Example 1: An internet content platform operates an announcement push system serving clients across multiple industries, pushing various announcements to tens of millions of users daily, covering product updates, event notifications, system maintenance alerts, and more. The platform has accumulated approximately 280 million push logs over the past year. Each log entry includes the user ID, push content identifier, push time, push channel, and subsequent user behavior (clicks, reading time, closing, complaints, etc.). In actual operation, operators discovered three prominent problems with the traditional rule-based push strategy: First, the static priority strategy cannot adapt to the dynamic changes in user interests, resulting in low matching between pushed content and users' current needs; second, the lack of historical behavior data for newly registered users leads to blind trial-and-error, resulting in a poor push experience; third, increasing push frequency to improve click-through rates has led to a surge in user complaints, with some users choosing to disable notification permissions, damaging the long-term relationship between the platform and its users.

[0047] To verify the practical effectiveness of this invention, offline simulation experiments were conducted on the platform's historical push log dataset. The experiments compared this invention with three mainstream methods: the first is a rule-based priority strategy, pushing notifications according to their importance level and a preset time window, representing a commonly used solution in traditional industry; the second is the standard DQN algorithm, using click-through rate maximization as the reward objective, representing a push solution based on deep reinforcement learning; and the third is a solution combining CQL and a standard VAE generator, representing a more advanced approach to offline reinforcement learning in the push field.

[0048] The experiment selected five user groups with varying activity levels and interests on the platform, each containing approximately 50,000 users. The experiment lasted for 30 consecutive days. Evaluation metrics included push notification click-through rate, read completion rate (percentage of users reading for more than 15 seconds), user complaint rate, and user retention rate within 7 days after the push notification. Retention rate was calculated as the percentage of users who continued to log in or use the platform daily on the 7th day after the push notification. The five user groups were divided based on historical behavioral characteristics clustering results: Group 1 had a preference for technology news; Group 2 had a preference for lifestyle services; Group 3 had a preference for financial management; Group 4 had a preference for education and learning; and Group 5 consisted of users with no significant single preference.

[0049] Table 1: Performance metrics comparison of different push methods across different user groups

[0050] Table 1 shows that this invention achieved optimal performance across all user groups and evaluation metrics. In terms of click-through rate (CTR), this invention achieved 16.2% for users with a preference for technology and information, a 9.4 percentage point improvement compared to the rule-based strategy's 6.8%, representing a relative increase of approximately 138%; a 3.8 percentage point improvement compared to the standard DQN's 12.4%; and a 4.5 percentage point improvement compared to the CQL+VAE's 11.7%. For users with a preference for lifestyle services, this invention achieved a CTR of 17.1%, 4.3 percentage points higher than the CQL+VAE's 12.8%, representing the largest single-point improvement across all comparisons. Analysis suggests that lifestyle service users encompass multiple subcategories of interest, such as dining, travel, and shopping. The candidate actions generated by the standard VAE tend to be homogeneous, failing to capture the multi-peaked distribution characteristics within the group. In contrast, the conditional diffusion generator, by sampling and denoising from multiple random noise seeds in parallel, produces a significantly differentiated set of candidate actions, greatly improving the coverage probability of long-tail interest points. The benchmark click-through rate for users with a preference for financial management is generally low, with the rule-based strategy achieving only 4.2%. This is because such announcements have a narrow audience. However, this invention still achieved 15.8% in this group, proving that the model can accurately identify highly interested users and avoid users who are not interested, thus preventing broadcast-style push notifications from disturbing irrelevant users.

[0051] Regarding complaint rates, the standard DQN has an average complaint rate of 3.9%, indicating that algorithms solely focused on maximizing click-through rates frequently push highly attractive entertainment-related announcements. While short-term click performance may be acceptable, users experience aversion due to information overload. This invention achieves an average complaint rate of approximately 1.5% across all user groups, only 0.4 percentage points higher than the rule-based strategy's 1.1%, but achieves approximately a 3-fold improvement in click-through rate. This result is attributed to the positive correlation between the conservative regularization penalty value and the variance of the Q-integrated network output: when a state-action pair is outside the data distribution, the output differences of the N Q-subnetworks increase, the variance rises, and the penalty value increases accordingly. This automatically lowers the corrected Q-value of the action, causing it to fail in the decision-making competition, thus learning a reasonable push rhythm without relying on manually set frequency limits.

[0052] 7-day retention rate is a core metric for measuring the long-term effectiveness of a push notification system. This invention achieved an average retention rate of 73.9% across all four user groups, while the standard DQN only achieved 63.6%, a difference of 10.3 percentage points. This reveals that strategies that prioritize short-term click-through rates as the sole optimization objective fundamentally damage the long-term relationship between users and the platform. The rule-based strategy achieved an average retention rate of 70.8%, close to this invention, but at the cost of extremely low push volume and substandard reach of platform announcements. The CQL+VAE strategy achieved an average retention rate of 68.0%, indicating that a conservative strategy provides some protection for user retention. Compared to CQL+VAE, this invention achieves performance improvements of 5.6, 6.7, 6.2, and 5.3 percentage points across different groups, with an average improvement of 5.9 percentage points. This improvement is attributed to the joint update closed loop of the diffusion generator and the Q-integral: after each push notification, user feedback is converted into reward values ​​and stored in the experience pool. After batch sampling, the temporal difference error, isovariance loss, and regularization term are calculated. The weighted sum of these three factors yields the total loss, and gradient updates are performed on both modules simultaneously. This allows the generator to gradually produce actions that better match the current user state distribution, and the evaluator to converge to more accurate value judgments. Ultimately, this maximizes click-through efficiency while maintaining a low complaint rate, achieving synergistic optimization of short-term clicks and long-term retention. The above experimental results fully verify the feasibility and superiority of this invention in practical announcement push scenarios.

[0053] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for optimizing precise announcement push based on reinforcement learning, characterized in that, include: Collect user logs and announcement data, obtain interest features by weighting log time decay, encode announcement categories and time expiration marks, and concatenate them to obtain a state vector; Based on the BCQ learning framework, an offline state-action pair is used to train a conditional diffusion generator instead of a standard variational autoencoder. The state vector is used as the conditional input to the diffusion generator. Forward noise is added to perturb the historical actions to pure Gaussian noise. Inverse noise is removed step by step with the current state as the condition to recover the complete action. After sampling and denoising from several random noise seeds, K candidate actions are output. The candidate actions include announcement identifier, push delay duration, and channel code. Based on the CQL learning framework, N Q sub-networks are integrated. A state translation and reflection symmetry group is defined. An isovariant loss is constructed to measure the mean square error of the Q value after inverse mapping of the original state and the transformed state. The isovariant loss and the TD error are weighted as the sub-network training loss. Conservative regularization is introduced to superimpose a penalty value on the Bellman objective. The penalty value is positively correlated with the variance of the N Q outputs. The candidate action inputs are integrated to obtain the base Q-mean, the penalty is subtracted to obtain the corrected Q, and the action with the largest corrected Q value is selected to be pushed. Collect user clicks, reads, and closes feedback to calculate rewards; Use a quadruple of reward, previous state, action, and next state to store the experience pool; Randomly sample batches of quadruples from the experience pool; Calculate the TD error, isovariance loss, and regularization term, and then perform a weighted sum to jointly update the diffusion generator and Q-integral. Use the updated parameters to re-execute candidate action generation and push decision.

2. The method for optimizing precise announcement push based on reinforcement learning according to claim 1, characterized in that, The interest features obtained by weighting log time decay specifically include: User behavior logs are pre-weighted for behavior types, which include browsing, clicking, reading in full, and closing. For each log entry, a decay coefficient is determined based on the time difference between the time the behavior occurred and the current time. Adjust the behavior type weight of this log entry using the decay coefficient to obtain the contribution value of this log entry; sum the contribution values ​​of all log entries of the same user according to the announcement category dimension to obtain the interest score vector of this user in each announcement category; The interest score vector is normalized, and the normalized interest score vector is used as the interest feature.

3. The method for optimizing precise announcement push based on reinforcement learning according to claim 1, characterized in that, The training conditional diffusion generator, which replaces the standard variational autoencoder, includes: The conditional diffusion generator adopts a denoised diffusion probability model architecture, which includes a forward denoising process and a reverse denoising process; The forward noise addition process sets a total number of steps T, with T ranging from 100 to 1000. A linear noise scheduling strategy is adopted, and Gaussian noise with zero mean and variance increasing with the number of steps is added to the current sample at each step, forming a Markov chain from the initial action sample to the standard Gaussian distribution. When the sample distribution reaches the Tth step, it is approximately a standard Gaussian distribution. The reverse denoising process learns from the standard Gaussian distribution and gradually restores the conditional distribution of the complete action along T steps. Each step uses the current state vector as the guiding condition and uses a trainable noise prediction network to receive three inputs: the noise sample of the current step, the current step index, and the state vector, and outputs the predicted noise added in this step. The noise prediction network adopts a U-shaped network structure, which includes a downsampling encoding path and an upsampling decoding path. Features are passed between each layer through skip connections, and the state vector is injected into the network at each decoding layer through a cross-attention mechanism. During the training phase, historical state-action pairs are randomly sampled from the offline dataset. For each action sample, a step index is randomly extracted. The noise sample and the real noise are generated according to the forward process. The noise sample, step index, and conditional state vector are input into the noise prediction network. The mean square error between the predicted noise and the real noise is calculated as the loss for this step. The losses of all steps are weighted and summed according to the preset weight table to obtain the total training loss. After training, the reverse denoising process can generate complete actions from random Gaussian noise step by step, consistent with the offline data distribution and constrained by the current state, given any current state vector.

4. The method for optimizing precise announcement push based on reinforcement learning according to claim 1, characterized in that, The step of sampling and denoising from several random noise seeds to output K candidate actions includes: At each online decision-making moment, K random noise seeds are independently sampled from a standard Gaussian distribution. The value of K is preset to a positive integer between 5 and 20 according to business needs, and each noise seed is different from the others. K noise seeds are input into the trained conditional diffusion generator in parallel. Each noise seed independently executes the complete reverse denoising process. Each step is guided by the same state vector at the current time. After T iterations, the K noise seeds converge into K distinct complete actions. Each action corresponds to a different combination of announcement identifier, push delay duration and channel code. Apply the batch constraint mechanism of the BCQ framework to K complete actions: train an independent conditional variational autoencoder or Gaussian mixture model to learn the joint distribution of state-action in the offline dataset, use this model as an action validity discriminator, and calculate the conditional probability density estimate of each generated action given the current state. When the probability density value is lower than a preset threshold, the action is rejected, and actions with a probability density value higher than the threshold are retained to form a candidate action set. If the number of candidate actions after constraint screening is insufficient, select several actions with the highest probability density values ​​from the remaining unscreened actions to make up the preset number K; The announcement identifier uses a one-hot encoded vector to represent the unique identity of the announcement to be pushed. The push delay duration is quantified in seconds to specify the push waiting time. The channel code uses discrete numerical values ​​to mark the type of push channel.

5. The method for optimizing precise announcement push based on reinforcement learning according to claim 1, characterized in that, The ensemble of N Q-subnetworks built based on the CQL learning framework includes: Construct a Q-subnetwork containing N independently initialized subnetworks. Each Q-subnetwork uses the same neural network structure but the parameter initialization values ​​are different for each other. N is a positive integer between 3 and 10. The input to each Q-subnetwork is a state-action pair formed by concatenating the state vector and the action, and the output is a scalar Q-value; Define a set of symmetric transformation groups for the state space where the state vector is located. Each transformation in the group maps the state vector to the transformed state vector. The transformations include translation and reflection transformations in each dimension of the state space, and all transformations are invertible. Each transformation corresponds to a unique inverse transformation. For each Q-subnetwork, construct an isovariant loss term: sample the original state vector from the state space, apply any transformation from the symmetric group to the original state vector to obtain the transformed state vector, form a first state-action pair with the candidate action, input the first Q-value into the Q-subnetwork, form a second state-action pair with the transformed state vector and the same candidate action, input the second Q-value into the same Q-subnetwork, apply the inverse transformation corresponding to the transformation to the second Q-value to obtain the mapped Q-value, and calculate the mean square error between the first Q-value and the mapped Q-value as the isovariant loss under the transformation; For each Q subnetwork, the mean of the isovariant loss is calculated under all transformations and at each sampling state as the isovariant loss term of the subnetwork; The individual training loss of each Q subnetwork is composed of a weighted sum of an isovariant loss term and a standard temporal difference error term. The weight coefficient of the isovariant loss term is preset to a positive value, and the weight coefficient of the standard temporal difference error term is preset to 1. Each Q subnetwork updates its network parameters independently based on its own individual training loss.

6. The method for optimizing precise announcement push based on reinforcement learning according to claim 1, characterized in that, The introduced conservative regularization in the Bellman objective superposition penalty value includes: Within the CQL framework, a conservative regularization penalty value is superimposed on the Bellman update objective of each Q subnetwork; The conservative regularization loss of each Q subnetwork is the expected penalty value over all state-action pairs, which is estimated by averaging the batch of state-action pairs sampled from the experience replay pool. The complete training objective function of each Q subnetwork consists of three terms: standard temporal difference error, isovariance loss term, and conservative regularization loss term; The three losses are weighted and summed according to preset weight coefficients to obtain the total training loss of each Q sub-network. The weight coefficient of the conservative regularization loss term is positively correlated with the global variance of the output values ​​of the N Q sub-networks. The penalty value is included as a subtraction term in the Bellman objective during Q-value updates.

7. The method for optimizing precise announcement push based on reinforcement learning according to claim 1, characterized in that, The action of selecting the maximum corrected Q value to perform the push specifically includes: Each candidate action in the candidate action set is paired with the current state vector to form a state-action pair, which are then input into N Q-subnetworks. The base Q-mean is calculated for the N Q-values ​​of the same state-action pair. The penalty value corresponding to the state-action pair is obtained according to the penalty value calculation method. The corrected Q-value is obtained by calculating the base Q-mean. The K candidate actions are traversed, and their corrected Q-values ​​are calculated. The candidate action with the largest value is selected as the optimal push decision. Extract the announcement identifier from the optimal push decision to determine the announcement to be pushed, extract the push delay duration to determine the waiting time before push, and extract the channel code to determine the push channel; after the waiting time expires, send the announcement to the target user through the channel; record the actual push time after the push is completed.

8. The method for optimizing accurate announcement push based on reinforcement learning according to claim 1, characterized in that, The method of using a quadruple consisting of reward, previous state, action, and subsequent state to store the experience pool specifically includes: The last data collection before the push is executed is used to generate a state vector as the previous state, the optimal push decision is used as the action, the scalar reward value calculated based on user interaction feedback is used as the reward, and the user logs and announcement data are collected again after the push is executed, and a new state vector is generated by splicing them as the next state. The four elements of the previous state, action, reward and next state are combined in order to form an experience quadruple. The experience quadruples are appended as independent records to the experience replay pool. The experience replay pool adopts a first-in-first-out queue structure. When the total number of experience quadruples stored in the pool reaches the preset capacity limit, the experience quadruples with the earliest storage time are deleted to free up storage space. The newly stored experience quadruples are marked with a timestamp.

9. The method for optimizing precise announcement push based on reinforcement learning according to claim 1, characterized in that, Randomly sampling batch quadruples from the experience pool specifically includes: When the number of experience quadruples stored in the experience replay pool reaches the preset minimum sampling threshold, a batch sampling operation is triggered. B experience quadruples are randomly and uniformly drawn from the experience replay pool. Each quadruple has an equal probability of being drawn and each drawing is independent of the others. After the drawing, the original quadruples are still retained in the pool. The B drawn quadruples are organized into a training batch. In this batch, the previous state, action, reward, and post-state of each quadruple are aligned into four tensors.

10. The method for optimizing accurate announcement push based on reinforcement learning according to claim 1, characterized in that, The calculation of TD error, isovariance loss, and regularization term, followed by a weighted summation and joint update of the diffusion generator and Q-integral, specifically includes: Extract B quadruplets from the training batch. Each quadruplet contains a previous state, an action, a reward, and a subsequent state. Input the previous state and action into the Q-integral to obtain the current Q-value. Input the subsequent state into the target Q-network to obtain the TD target value. Calculate the mean square error of both as the TD error. Substitute the isovariance loss term and the regularization term into the current batch and recalculate. Sum the TD error, isovariance loss, and regularization term according to preset weights to obtain the total loss. Use the total loss to update the gradients of the diffusion generator and the Q-integral. Replace the old parameters with the new parameters and re-execute the candidate action generation and push decision until training converges.

Images