Cross-channel advertisement placement strategy optimization method fusing user behavior prediction and deep reinforcement learning

Abstract

This invention relates to the field of business forecasting data processing, and discloses a method for optimizing cross-channel advertising delivery strategies by integrating user behavior prediction and deep reinforcement learning. The method includes: acquiring user interaction sequences from multiple sources and generating feature tensors; extracting global conversion features reflecting conversion probabilities through an attention mechanism network; calculating the time change rate of the global conversion features to determine the intent evolution gradient; concatenating the global conversion features, real-time interaction feature vectors, and intent evolution gradient into a decision state space; inputting the decision state space into a strategy model, using the intent evolution gradient to compensate for trends in the reward function, and outputting resource allocation coefficients mapped to different channels; updating the strategy model based on feedback data. This invention establishes a coupling relationship between resource allocation actions and conversion windows, reduces data feedback latency, eliminates decision phase lag, and improves the robustness of business resource allocation.

Description

Technical Field

[0001] This invention belongs to the field of commercial forecasting data processing technology, and in particular relates to a method for optimizing cross-channel advertising delivery strategies that integrates user behavior prediction and deep reinforcement learning. Background Technology

[0002] In the current digital business system, omnichannel marketing collects and processes multi-source behavioral data generated by users across heterogeneous touchpoints to allocate business resources. Current technical solutions typically use temporal neural networks to build user conversion prediction models, determining the conversion probability of users on different channels based on historical interaction sequence characteristics, thereby providing a basis for budget allocation decisions. This approach has predictive accuracy in environments with stable data distribution and can support automated marketing management processes. However, in omnichannel interaction scenarios, user intent is a dynamic evolution process. Due to the logical damping characteristics of intent migration between different touchpoints such as search, browsing, and social interaction, the guiding effect of previous channel behavior on subsequent conversions decays non-linearly over time, and there is an asymmetric time offset in the data feedback from each channel. Existing technologies ignore the phase deviation in the intent evolution process, resulting in a lag in the system's perception of the conversion window. Resource allocation actions often deviate from the peak point of the true intent, causing a mismatch of business resources in the spatiotemporal dimensions.

[0003] In the deployment of digital business systems, not only are the physical nodes and fluid-like transmission architectures of the underlying hardware, which carry out data distribution, limited to causing asymmetric latency when connecting across channels, but the upper-layer software control methods also have shortcomings. For example, Chinese invention patent application CN121169485A discloses a real-time optimization method and system for advertising delivery based on reinforcement learning. This solution is based on continuous monitoring of user behavior, extracting the advertising delivery time period, and using it as a time parameter input into the optimization model based on the reinforcement learning mechanism. The order of push is adjusted according to the delivery simulation results. This existing technology implicitly relies on the idealized premise that the data of each channel is synchronized and the state is updated instantaneously. In the actual cross-channel high-frequency evolution, since there is an inevitable objective asynchronous delay in the physical link, this solution relies on discrete time periods as static parameters and fails to penetrate the surface time window to quantify the nonlinear decay momentum of user intent. The core premise is fundamentally mismatched with the actual boundary conditions, resulting in a phase misalignment between the control logic and the peak point of the actual intent when dealing with asynchronous delay.

[0004] Therefore, how to construct a resource allocation method that perceives the gradient of intent evolution and realizes decision phase hedging, and solve the problem of decision misalignment and resource allocation efficiency decline caused by the lag in intent evolution during the deployment process, has become the technical problem to be solved by this invention. Summary of the Invention

[0005] This invention aims to solve the problem of declining resource allocation efficiency caused by phase deviation in intent evolution and decision lag during cross-channel delivery.

[0006] In this technical solution, a method for optimizing cross-channel advertising delivery strategies that integrates user behavior prediction and deep reinforcement learning includes the following steps:

[0007] Step S1: Obtain user interaction sequences from multiple sources, normalize the user interaction sequences, and generate feature tensors representing user interest dimensions.

[0008] Step S2: Input the feature tensor into the attention mechanism network, extract the temporal features through the attention mechanism network, calculate the global conversion feature reflecting the user's conversion probability within a preset period and the real-time interaction feature vector at the current moment. The global conversion feature includes the expected conversion rate prediction value.

[0009] Step S3: Calculate the change in global transformation features within adjacent sampling periods and determine the intention evolution gradient, which characterizes the transformation intention evolution rate.

[0010] Step S4: Concatenate the global transformation features, real-time interaction feature vectors, and intent evolution gradients to construct the decision state space of the reinforcement learning agent.

[0011] Step S5: Input the decision state space into the strategy model, use the intention evolution gradient to perform phase hedging compensation calculation on the internal reward function, and output the resource allocation coefficients mapped to different delivery channels, wherein the sum of the resource allocation coefficients of each delivery channel is equal to 1.

[0012] Step S6: Monitor the business feedback data after the campaign, calculate the instant reward value based on the difference between the actual conversion rate and the predicted conversion rate, and combine it with the network parameters of the intent evolution gradient update strategy model.

[0013] Preferably, step S2 specifically includes: mapping the feature tensor to a high-dimensional linear space, calculating the association weights of different touchpoints in the user interaction sequence through a multi-head self-attention layer; using the association weights to perform weighted aggregation of touchpoint data features to generate global conversion features that reflect long-term behavioral dependencies, wherein when the length of the interaction sequence exceeds a preset threshold, the attention distribution of the multi-head self-attention layer is adjusted by a scaling factor to ensure that the output features can accurately characterize the user's interest migration trend among heterogeneous channels.

[0014] Preferably, the process of using the intention evolution gradient to perform phase hedging compensation calculation on the internal reward function in step S5 includes: identifying the algebraic sign of the intention evolution gradient; if the intention evolution gradient is negative and its absolute value is greater than a preset decay threshold, then the real-time reward weight of the delivery channel is adjusted using the decay factor reduction strategy model so that the resource allocation coefficient increases and monotonically decreases as the absolute value of the intention evolution gradient increases.

[0015] Preferably, before constructing the decision state space of the reinforcement learning agent in step S4, the method further includes: step S41, monitoring the data return latency of each delivery channel; step S42, when the maximum latency deviation between different delivery channels exceeds a preset deviation threshold, calling the time alignment unit to attach a timestamp index to the heterogeneous data stream, and eliminating the attribute offset of the feature vector in the decision state space in the time domain caused by asymmetric latency by performing delay caching alignment on the high-speed data stream.

[0016] Preferably, the strategy model is built on an asynchronous advantage evaluation architecture, and the decision step size of the reinforcement learning agent is limited by the convergence variance of the attention mechanism network. When the volatility of the convergence variance is detected to be greater than 15%, the decision step size is automatically reduced to 20% of the original step size to suppress numerical oscillations in the resource allocation coefficient calculation process.

[0017] Preferably, the feature tensor includes attribution features, which are obtained through the following sub-steps: constructing a virtual delivery environment, simulating the user behavior path after removing specific touchpoint data, calculating the difference in conversion probability before and after removal, and determining the difference in probability as the pure conversion gain of the specific touchpoint.

[0018] Preferably, the normalization process in step S1 involves privacy-preserving computation. By injecting random noise conforming to a Laplace distribution into the feature tensor, differential privacy desensitization is achieved, and it is ensured that the desensitized feature flow maintains the original temporal distribution characteristics of user behavior in the statistical dimension.

[0019] Preferably, when calculating the resource allocation coefficient, the real-time input-output ratio of all channels is extracted as a constraint variable. If the input-output ratio at the current moment is lower than the preset business safety threshold, the output value range boundary of the strategy model is forcibly contracted by restricting the output range of the activation function of the strategy model.

[0020] Preferably, during the campaign, user interest shifts caused by changes in the external market environment are captured, and the temporal evolution gradient of user click popularity is calculated. If the temporal evolution gradient is greater than 0.85, the learning rate weight of the fully connected layer of the strategy model is increased through the intent evolution gradient, so as to realize the adaptive iteration of the campaign strategy to market dynamics.

[0021] Compared to existing technologies, the cross-channel advertising delivery strategy optimization method of this invention, which integrates user behavior prediction and deep reinforcement learning, has the following advantages:

[0022] 1. In the optimization of cross-channel advertising placement strategies, by extracting the evolution gradient of user intent across heterogeneous channels, a temporal coupling relationship is established between the placement decision instruction and the intent migration trend. The system uses the prediction unit to capture the acceleration of intent probability on the time axis, so that the dynamic phase of resource allocation action and the actual conversion window are logically aligned. Due to the existence of this phase compensation mechanism based on intent inertia, the system effectively avoids decision overshoot caused by feedback signal lag when dealing with cross-channel flow behavior with logical damping characteristics, ensuring that advertising resources are accurately positioned before the peak of user intent.

[0023] 2. By dynamically weighting and splicing long-term intent tensors and real-time interaction features through intent fusion operators, a high-dimensional logical state space is constructed. This deep coupling at the feature level enables reinforcement learning agents to perceive the synergistic gain effect between channels in real time. Through the nonlinear mapping of cross-channel related features in the state space, the system eliminates the static assumption that each channel is independent in the traditional allocation model from the principle level, and realizes the global and adaptive configuration of heterogeneous channel business resources.

[0024] 3. An adaptive step size control mechanism based on predicted residual fluctuations is adopted to enhance the convergence stability of the system in a large-scale random data noise environment. When the intention evolution gradient oscillates violently, the strategy model hedges the decision risk by dynamically reducing the update step size. This risk control mechanism ensures the smooth evolution of the omnichannel budget allocation strategy under extreme conditions and solves the common strategy oscillation and allocation circuit breaker problems in high-frequency business decision-making. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating the implementation steps of the cross-channel advertising placement strategy optimization method of the present invention;

[0026] Figure 2 This is the logic diagram of phase offset compensation and parameter update for the gradient evolution intended by this invention. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0028] It should be noted that all directional and positional terms used in this invention, such as: up, down, left, right, front, back, vertical, horizontal, inner, outer, top, bottom, transverse, longitudinal, center, etc., are only used to explain the relative positional relationship and connection between components in a specific state (as shown in the accompanying drawings). They are only for the convenience of describing this invention and do not require that this invention be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention. In addition, the descriptions of "first," "second," etc., in this invention are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated.

[0029] In the description of this invention, unless otherwise explicitly specified and limited, the terms installation, connection, and linking should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections; they can refer to direct connections or indirect connections through an intermediate medium; they can refer to the internal connection of two components. For those skilled in the art, the specific meaning of the above terms in this invention can be understood according to the specific circumstances.

[0030] In the description of this specification, references to the terms "an embodiment," "some embodiments," "illustrative embodiments," "examples," "specific examples," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example, and the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0031] A method for optimizing cross-channel advertising delivery strategies that integrates user behavior prediction and deep reinforcement learning includes the following steps:

[0032] Step S1: Obtain user interaction sequences from multiple sources, normalize the user interaction sequences, and generate feature tensors representing user interest dimensions.

[0033] Step S2: Input the feature tensor into the attention mechanism network, extract the temporal features through the attention mechanism network, calculate the global conversion feature reflecting the user's conversion probability within a preset period and the real-time interaction feature vector at the current moment. The global conversion feature includes the expected conversion rate prediction value.

[0034] Step S3: Calculate the change in global transformation features within adjacent sampling periods and determine the intention evolution gradient, which characterizes the transformation intention evolution rate.

[0035] Step S4: Concatenate the global transformation features, real-time interaction feature vectors, and intent evolution gradients to construct the decision state space of the reinforcement learning agent.

[0036] Step S5: Input the decision state space into the strategy model, use the intention evolution gradient to perform phase hedging compensation calculation on the internal reward function, and output the resource allocation coefficients mapped to different delivery channels, wherein the sum of the resource allocation coefficients of each delivery channel is equal to 1.

[0037] Step S6: Monitor the business feedback data after the campaign, calculate the instant reward value based on the difference between the actual conversion rate and the predicted conversion rate, and combine it with the network parameters of the intent evolution gradient update strategy model.

[0038] Preferably, step S2 specifically includes: mapping the feature tensor to a high-dimensional linear space, calculating the association weights of different touchpoints in the user interaction sequence through a multi-head self-attention layer; using the association weights to perform weighted aggregation of touchpoint data features to generate global conversion features that reflect long-term behavioral dependencies, wherein when the length of the interaction sequence exceeds a preset threshold, the attention distribution of the multi-head self-attention layer is adjusted by a scaling factor to ensure that the output features can accurately characterize the user's interest migration trend among heterogeneous channels.

[0039] Preferably, the process of using the intention evolution gradient to perform phase hedging compensation calculation on the internal reward function in step S5 includes: identifying the algebraic sign of the intention evolution gradient; if the intention evolution gradient is negative and its absolute value is greater than a preset decay threshold, then the real-time reward weight of the delivery channel is adjusted using the decay factor reduction strategy model so that the resource allocation coefficient increases and monotonically decreases as the absolute value of the intention evolution gradient increases.

[0040] Preferably, before constructing the decision state space of the reinforcement learning agent in step S4, the method further includes: step S41, monitoring the data return latency of each delivery channel; step S42, when the maximum latency deviation between different delivery channels exceeds a preset deviation threshold, calling the time alignment unit to attach a timestamp index to the heterogeneous data stream, and eliminating the attribute offset of the feature vector in the decision state space in the time domain caused by asymmetric latency by performing delay caching alignment on the high-speed data stream.

[0041] Preferably, the strategy model is built on an asynchronous advantage evaluation architecture, and the decision step size of the reinforcement learning agent is limited by the convergence variance of the attention mechanism network. When the volatility of the convergence variance is detected to be greater than 15%, the decision step size is automatically reduced to 20% of the original step size to suppress numerical oscillations in the resource allocation coefficient calculation process.

[0042] Preferably, the feature tensor includes attribution features, which are obtained through the following sub-steps: constructing a virtual delivery environment, simulating the user behavior path after removing specific touchpoint data, calculating the difference in conversion probability before and after removal, and determining the difference in probability as the pure conversion gain of the specific touchpoint.

[0043] Preferably, the normalization process in step S1 involves privacy-preserving computation. By injecting random noise conforming to a Laplace distribution into the feature tensor, differential privacy desensitization is achieved, and it is ensured that the desensitized feature flow maintains the original temporal distribution characteristics of user behavior in the statistical dimension.

[0044] Preferably, when calculating the resource allocation coefficient, the real-time input-output ratio of all channels is extracted as a constraint variable. If the input-output ratio at the current moment is lower than the preset business safety threshold, the output value range boundary of the strategy model is forcibly contracted by restricting the output range of the activation function of the strategy model.

[0045] Preferably, during the campaign, user interest shifts caused by changes in the external market environment are captured, and the temporal evolution gradient of user click popularity is calculated. If the temporal evolution gradient is greater than 0.85, the learning rate weight of the fully connected layer of the strategy model is increased through the intent evolution gradient, so as to realize the adaptive iteration of the campaign strategy to market dynamics.

[0046] Example 1: In a large-scale omnichannel digital marketing system, when faced with the situation that user intent migrates frequently between heterogeneous channels and there is an asymmetric time offset in the data feedback from each channel, the system processes the original behavioral sequences generated by the same user on heterogeneous channels such as social media, search engines, mobile applications, and short video platforms to achieve real-time dynamic allocation of advertising resources and bid optimization. Since the migration of user intent between different touchpoints such as search, browsing, and social interaction has logical damping characteristics, the guiding effect of previous channel behaviors on subsequent conversions decays non-linearly over time. If the system generates decision instructions based solely on the collected conversion probability prediction values, there will be a decision phase lag, causing the placement action to deviate from the peak point of commercial intent, resulting in a mismatch of resources in the spatiotemporal dimension.

[0047] The system collects user interaction sequences from multiple sources. It processes the raw data such as click popularity, dwell time, and page jump depth in the user interaction sequences using a normalization operator to generate feature tensors representing the user's multi-dimensional interests. The feature tensors are then mapped to a high-dimensional linear space. The system uses a multi-head self-attention layer of an attention mechanism network to calculate the association weights of different touchpoints in the user interaction sequence. The touchpoint data features are then weighted and aggregated using the association weights to generate global conversion features that reflect long-term behavioral dependencies. At the same time, a real-time interaction feature vector reflecting the instantaneous intent at the current moment is generated. The global conversion features include the predicted conversion rate within a preset period.

[0048] The system calculates the change in global transformation features within adjacent sampling periods to determine the intention evolution gradient, which characterizes the rate of transformation intention evolution. This gradient characterizes the vector evolution trend of commercial intent over time. It captures the momentum change of user interests by obtaining the rate of change of intent probability over time. When the intent evolution gradient... When the value is positive and shows an increasing trend, it indicates that the user is in the period of rising intent; conversely, it indicates that the user is in the period of declining intent. In this way, the system transforms the static point prediction logic into a trend tracking logic with phase awareness, thereby solving the decision lag caused by asynchronous data feedback.

[0049] In constructing the decision state space of a reinforcement learning agent At that time, the system will combine global transformation features, real-time interactive feature vectors, and intent evolution gradients. Perform vector concatenation and combine the decision state space. Input policy model, policy model utilizes intention evolution gradient Phase hedging compensation is performed on the internal reward function, specifically to identify the intention evolution gradient. The algebraic sign and absolute value of the gradient, when intended to evolve. When the value is negative and its absolute value is greater than the preset decay threshold, the system determines that the user's conversion intention has entered the decay range. At this time, the decay factor adjustment strategy model is used to reduce the real-time reward weight of the specific advertising channel, thereby adjusting the output resource allocation coefficient. Random graph evolution gradient The absolute value of the resource allocation coefficient increases while it monotonically decreases. satisfy Under the constraints, for abnormal operating conditions where the real-time input-output ratio of all channels is lower than the preset business safety threshold, the control system extracts the resource allocation coefficient. An additional parameterized tensor clipping layer is connected in series before the input. This layer outputs activation values ​​by truncating the original network features corresponding to the restricted channels. And apply the boundary truncation mathematical formula Amplitude limiting is applied, among which, The system assigns a mandatory safety margin upper limit scalar parameter based on the current low input-output ratio. This cuts off the path for obtaining a large initial allocation weight for this risk channel at the bottom-level computing node, achieving a forced physical contraction of the model output value range boundary. The intention inertial phase compensation mechanism ensures that the delivery action is executed before the physical endpoint of the intention evolution. Under the asymmetric latency condition of 500ms backhaul from channel A and 10ms backhaul from channel B, the system predicts the decay trend of the intention and contracts the budget in advance due to the introduction of a gradient operator that reflects the acceleration of the intention, avoiding invalid delivery at the touchpoints entering the end of the intention. When the system detects that the convergence variance volatility of the attention mechanism network is greater than 15%, the reinforcement learning agent automatically adjusts the decision step size from the original 0.1 to 0.02, sacrificing local response speed for the stability of the overall channel budget allocation. After 5000 decision iterations, the system reaches a stable state, with the average return on investment increasing from 3.2 in the control group to 4.1, while the conversion volatility decreases from 14% to 6.5%.

[0050] Example 2: In a simulation test platform encompassing social media, mobile search, and short video interaction channels, the system verifies the real-time performance and accuracy of cross-channel behavioral data processing. The test platform is constructed from distributed computing nodes, generating user behavior feature sequences conforming to a Monte Carlo distribution. Asymmetric latency is introduced as an interference source, with the data return latency for social media channels set to 500ms and the latency for mobile search channels set to 10ms. Gaussian white noise with a signal-to-noise ratio of 20dB is superimposed on the original data stream. The system determines the sampling period during the initialization phase. The value of this parameter is limited by the feature extraction accuracy and computational resource load. When the user's interest migration frequency between heterogeneous channels is in the range of 0.5Hz to 2Hz, in order to meet the Nyquist sampling criterion and leave a 20% engineering margin, the system will adjust the sampling period. The sampling period is set to 50ms. If the value exceeds this threshold, the system will be unable to capture the instantaneous intent shift at short video touchpoints; if the sampling period... Below this value, a surge in processor interrupt frequency can cause memory bus bandwidth utilization to exceed 85%, leading to increased response latency.

[0051] The experiment included control group A, control group B, and the sample group of this invention. Control group A adopted a resource allocation method based on a static attribution model without introducing a gradient compensation mechanism; control group B constructed a decision state space. Time includes intention evolution gradient However, without invoking the phase offset compensation operator, observational data show that when the control group A processes 500ms high-latency data, the output resource allocation coefficient is... The phase deviation from the user conversion window reached 460ms, the average return on investment was 3.12, and under 20dB noise interference, the budget allocation showed a numerical oscillation of 14.5% in 5000 decision cycles. The average return on investment of control group B was 3.48, but when the intention entered the decay period, due to the lack of phase correction of the reward function, a budget redundancy of 12.3% was generated at the failure point.

[0052] The present invention utilizes a prediction unit to calculate the difference between the predicted values ​​of the expected conversion rate within adjacent sampling periods, and extracts the intended evolution gradient. Participate in constructing the decision state space When the intended evolution gradient When the value is negative and the absolute value reaches the preset attenuation threshold of 0.15, the system reduces the instantaneous reward weight of the strategy model through the attenuation factor. Actual test data shows that the average return on investment for the sample group of this invention under the same noise environment is 4.15, and the resource allocation coefficient... The response curve leads the intention peak by 45ms, thus locking the conversion window. When the convergence variance volatility of the attention mechanism network suddenly increases from 5% to 18%, the sample group of this invention triggers the adaptive step size control logic, adjusting the decision step size. The threshold was reduced from 0.1 to 0.02, lowering the volatility of resource allocation to 6.2%. Boundary stress testing was conducted on the selection of the attenuation threshold. When the attenuation threshold was set to 0.05, the system interpreted random data fluctuations as intentional attenuation, leading to an increase in the decision step size. With frequent contraction, the return on investment (ROI) drops to 3.35. When the decay threshold is set to 0.6, the system's perception of user interest decay becomes delayed, and resource allocation lags behind the conversion window closing time by 120ms. The ROI growth trend enters the saturation zone at 3.5. When the user intent migration intensity increases from 0.2 units / second to 1.5 units / second, the average ROI output of the sample group of this invention is positively correlated with the conversion intensity, confirming that the phase offset compensation mechanism maintains its correction capability under different intent offset intensities.

[0053] Example 3: In the data preprocessing scenario of a large-scale promotional event, the system handles situations with a sudden surge in user traffic and complex cross-channel interaction logic. The data preprocessing unit collects user interaction sequences from multiple sources to obtain raw feature vectors including click popularity and dwell time. Extract the original feature vector Maximum value within the preset history window Minimum value And calculate the normalized feature tensor. The specific calculation formula is as follows: ,in, These are the original feature values ​​collected. To preset the maximum value of this dimension feature within the historical window, This is the minimum value of this dimension feature within a preset historical window. To generate the normalized feature values, a differential privacy desensitization process is constructed based on Laplace distribution theory. This process generates a random noise matrix with the same dimension as the normalized feature tensor. This matrix is ​​then superimposed onto the feature tensor. The values ​​of each element in the noise matrix conform to the scale parameter. Laplace probability density distribution, scale parameter The calculation formula is as follows: ,in, The global sensitivity of the feature tensor is represented by the maximum change in the tensor's first norm caused by two adjacent interaction sequences. The privacy protection budget constant is always greater than zero. By setting scale parameter constraints, the numerical boundary of independent touch point trajectories is masked, ensuring that the deviation between the feature flow after adding noise and the original data is maintained within the preset tolerance range at the statistical expectation level, thus preserving the temporal transfer expectation value characteristics of the original group behavior.

[0054] Attention mechanism networks extract global intent features that reflect the long-term conversion probability of users. The sliding window extracts real-time interactive feature vectors that reflect instantaneous intentions. The linear mapping layer will incorporate global intent features. Converted into real-time interactive feature vectors Using the same 128-dimensional space, the intended fusion operator calculates the mutual information coefficients of both. To determine the feature alignment weights, if the mutual information coefficients... If the value is greater than 0.75, the system will increase the real-time interactive feature vector. The feature gain is used to capture instantaneous preferences, if the mutual information coefficient If the value is below 0.3, the system determines that the user's interests have shifted and increases the global intent feature. The splicing ratio is used to generate the decision state space of the current business system. Mutual information coefficient A dimensionless parameter to characterize the consistency between long-term intentions and short-term behavior. For a feature tensor containing the predicted value of the expected conversion rate, A vector reflecting the interaction strength at the current moment. To ensure the logical state of the input policy model is determined, the decision state space S of the input policy model must be defined. tThroughout the dynamic splicing ratio adjustment process, the tensor dimension remains absolutely fixed. The system is configured with mask alignment logic based on mutual information coefficients at the feature input end. When the splicing ratio of a certain feature vector is increased or decreased, the operator simultaneously calculates the number of feature elements to be retained and performs zero-padding and truncation operations on the extra redundant feature dimensions. At the physical memory level, the static conservation of the total number of dimensions of the entire state space is maintained, enabling the underlying fully connected neural network to seamlessly accept the reconstructed input tensor.

[0055] The calibration process for the attenuation threshold utilizes offline campaign log data streams to calculate the residual sequence between the system's omnichannel campaign actions and actual click-through rates over the past 30 calendar days. The statistical unit calculates the first derivative distribution of the residual sequence, and the upper bound of the 95% confidence interval of this distribution is determined as the calibration benchmark for the attenuation threshold. The calibration result under current conditions is 0.15. This is achieved when the real-time extracted intent evolution gradient... When this threshold is exceeded, the policy model identifies the algebraic sign of the intended evolution gradient. If the intended evolution gradient... If the value is negative, the instant reward weight of the corresponding delivery channel is reduced by using the decay factor to achieve contraction control of budget allocation, so that the identification accuracy of the final touchpoint of intent is maintained at 91%, and the dynamic phase of business resource allocation action and conversion window is logically aligned.

[0056] Example 4: In the case of processing an intent evolution sequence containing 12-dimensional user interaction features, the normalized feature tensor generated by the data preprocessing unit Mapped to a 128-dimensional embedding space; the attention mechanism network includes 8 parallel self-attention heads, each of which transforms the embedding vector into a query vector through a linear transformation matrix. Key vector AND value vector The system calculates the query vector. With key vector The attention score is determined by the dot product of the values. To suppress gradient vanishing caused by numerical bias, the score matrix is ​​divided by a scaling factor. ; The dimension of the key vector is 16. The multi-head self-attention layer adjusts the value vector based on the association weights. Weighted aggregation is employed to output a hidden state vector reflecting the nonlinear correlation between touchpoints. This hidden state vector, after layer normalization, serves as a component of the global conversion feature. The system extracts this hidden state vector and inputs it into a multilayer perceptron consisting of two fully connected layers for spatial dimensionality reduction. The first fully connected layer compresses the high-dimensional hidden state vector into a scalar feature. The second layer uses a sigmoid activation function to perform a nonlinear mapping on this scalar feature, forcibly binding its value within the probability range of 0 to 1. This posterior scalar value, calculated at the bottom layer, serves as the predicted conversion rate and is directly written into and fixed in the first data dimension of the global conversion feature tensor. Before feature tensor aggregation, a Markov decision simulation space is constructed using a test node array as a virtual delivery environment. The global conversion feature and the historical transition correlation matrix of all touchpoints are imported into the simulation space to maintain the inherent state transitions between existing heterogeneous touchpoints. While maintaining the unchanged transition probability distribution, the state transition probability and interaction activation weight of the node corresponding to the specific single-point touchpoint to be evaluated are forcibly overwritten to zero. After executing this forced overwrite instruction, the system processor calls the probability renormalization algorithm to mathematically repair the affected row vectors in the Markov transition matrix. Specifically, the original transition probability of the specific node is redistributed to all other non-zero active nodes pointed to by the row according to the relative numerical ratio. The normalized state transition probabilities are recalculated to ensure that the sum of all elements in the row after the update is still strictly equal to 1. This maintains the mathematical conservation of the closed nature of the underlying state transitions of the Markov system in the isolation calculation, triggering the reinforcement learning agent to perform forward reasoning and decomposition in the isolation matrix pool. The difference between the predicted value of the expected conversion rate before and after the overwrite is extracted and determined as the pure conversion gain of the corresponding touchpoint. The attribution feature dimension in the feature tensor is updated with this as an independent parameter.

[0057] Given the uneven feedback cycles of conversion returns across different channels, the strategy model employs a fully connected network architecture with two hidden layers, each containing 256 neurons, and uses a linear rectified function as the activation function; the system utilizes intent evolution gradients. Calculate the phase hedging reward correction amount ,when When it is positive, the phase hedging reward correction amount The value of is 0; when When the value is negative and its absolute value is greater than the decay threshold of 0.15, the phase hedging reward correction amount The reward function monotonically increases with the rate of change of the conversion probability; a first-order stimulated decay physicodynamic model is introduced to constrain the allocation of the reward function. Specifically, The calculation formula is as follows: ,in, This represents the highest cost penalty weighting coefficient corresponding to the target channel, with a value range limited to a closed interval between 0 and 1. This represents the target intent fading constant calculated based on historical feedback logs from the channel. Its value is always greater than zero. Based on an exponential nonlinear mapping rule, a unique, closed-loop quantitative compensation value is determined. To reward the correction value, The intended evolution gradient, and the phase hedging reward correction amount. The calculated reward value guides the reinforcement learning agent to reduce the resource allocation coefficient of the corresponding distribution channel in the calculation of the real-time reward value. The output layer of the policy model transforms the output tensor into one that satisfies the normalized exponential function. The numerical values ​​of the constraints maintain the stability of budget allocation in the current business system under heterogeneous channel environments. In the actual asynchronous reinforcement learning closed-loop execution mechanism, this contraction guidance does not forcibly intervene in the action output during a single forward propagation of forward reasoning, but rather through a phase-hedged reward correction. The feedback signal is passed to the Critic, which calculates and generates a time difference error (TD-error) with a decay penalty and backpropagates it to the policy network (Actor). This ensures that when the policy network, which has undergone this round of weight decay iterations, receives a homogeneous decision state space in the next sampling period, its naturally positive output resource allocation coefficients will be... It produces spontaneous physical contraction in algebraic absolute value.

[0058] Example 5: In the initial deployment scenario of a sub-channel on an e-commerce platform, the calibration unit extracts the anonymized interaction log streams from the last 72 consecutive running hours. By traversing the 64-dimensional, 128-dimensional, and 256-dimensional hidden layer dimensions in the simulation environment, the residual of the attention mechanism network for the reconstruction probability of user clicks within the next 300ms is calculated. ,in To characterize the numerical value of the prediction accuracy deviation, the system monitors the reconstructed residuals during this calibration process when the dimension increases from 64 to 128. The decrease was 42.5%, while the decrease rate was only 3.2% when the dimension increased from 128 to 256. Therefore, 128 was determined to be the dimension value of the high-dimensional embedding space. The system calculates the key vector... With query vector Use the element-wise covariance matrix to calibrate the scaling factor ,in and These are the key vector and the query vector, respectively. The dimension value of the key vector is set so that the output value range of each attention head is distributed within the response range of 0.15 to 0.85, thus establishing a benchmark feature space for the evolution characteristics of intent for a specific channel.

[0059] During system online operation, the timing alignment unit monitors the data return latency of heterogeneous channels in real time. By suspending the high-speed data stream through a delay buffer of 1024 data items in the memory bus, when the timestamp index deviation of different touch points exceeds a preset deviation threshold of 50ms, the system calls the timing alignment operator to align the temporal attributes of the heterogeneous feature tensor using a linear interpolation algorithm, eliminating the decision state space caused by hardware response asymmetry. Attribute offset, policy model calculates intention evolution gradient Based on the probability distribution before and after the intended peak point, the gradient critical value corresponding to the closing moment of the transition window is extracted to be -0.15, and the decay factor is determined to be 0.8 accordingly. The phase hedging reward correction amount is then calculated. The logic is when When the value is less than -0.15, the system uses a decay factor to reduce the expected return of the current delivery touchpoint, thus adjusting the strategy model's calculation of resource allocation coefficients. Reduce budget allocations to channels entering their decline phase to ensure logical alignment between business resource allocation actions and the dynamic phase of the conversion window.

[0060] In the offline pre-training scenario of reinforcement learning policy models, the system calls asynchronous advantage execution evaluation architecture to process the decision state space. The historical interaction trajectory is determined by defining a loss function. To guide the strategy network parameters Iterative updates, loss function The system consists of a weighted sum of policy gradient loss, value function loss, and entropy regularization term used to maintain policy diversity. Within each training step, the system calculates the advantage function value for the current action and adjusts the policy network accordingly to converge towards maximizing cumulative reward. This continues until the average reward volatility over 100 consecutive training cycles is less than a preset convergence threshold of 0.01. At this point, the system locks the current weight parameters of the policy model. It is recommended to add this paragraph after Example 5. In the heterogeneous channel resource sensitivity calibration scenario, the calibration unit extracts the mapping relationship between the average channel backhaul latency and the commercial intent decay rate to determine the attenuation factor value. The system simulates user interaction flows with different exponential decay characteristics in an offline simulation environment and calculates the resource allocation coefficient under different attenuation factor settings. Regarding the phase tracking error at the peak of the transition, the calibration process monitoring revealed that when the attenuation factor is in the range of 0.7 to 0.9, the phase tracking error varies with different intended evolution gradients. The response can be maintained within a stable range of less than 10ms. Therefore, based on the response sensitivity of a specific channel, 0.8 is selected as the attenuation factor calibration value in the current deployment environment through a linear interpolation algorithm within this range.

[0061] The embodiments of this application have been described above with reference to the accompanying drawings. Unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other. This application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit of this application and the scope of protection of this invention, and all of these forms are within the protection scope of this application.

Claims

1. A method for optimizing cross-channel advertising delivery strategies by integrating user behavior prediction and deep reinforcement learning, characterized in that, Includes the following steps: Step S1: Obtain user interaction sequences from multiple sources, normalize the user interaction sequences, and generate feature tensors representing user interest dimensions. Step S2: Input the feature tensor into the attention mechanism network, extract the temporal features through the attention mechanism network, calculate the global conversion feature reflecting the user's conversion probability within a preset period and the real-time interaction feature vector at the current moment. The global conversion feature includes the expected conversion rate prediction value. Step S3: Calculate the change in global transformation features within adjacent sampling periods and determine the intention evolution gradient, which characterizes the transformation intention evolution rate. Step S4: Concatenate the global transformation features, real-time interaction feature vectors, and intent evolution gradients to construct the decision state space of the reinforcement learning agent. Step S5: Input the decision state space into the strategy model, use the intention evolution gradient to perform phase hedging compensation calculation on the internal reward function, and output the resource allocation coefficients mapped to different delivery channels, wherein the sum of the resource allocation coefficients of each delivery channel is equal to 1. Step S6: Monitor the business feedback data after the campaign, calculate the instant reward value based on the difference between the actual conversion rate and the predicted conversion rate, and combine it with the network parameters of the intent evolution gradient update strategy model.

2. The method for optimizing cross-channel advertising delivery strategies by integrating user behavior prediction and deep reinforcement learning as described in claim 1, characterized in that, Step S2 specifically includes: mapping the feature tensor to a high-dimensional linear space, calculating the association weights of different touchpoints in the user interaction sequence through a multi-head self-attention layer; using the association weights to perform weighted aggregation of touchpoint data features to generate global conversion features that reflect long-term behavioral dependencies. When the length of the interaction sequence exceeds a preset threshold, the attention distribution of the multi-head self-attention layer is adjusted by a scaling factor to ensure that the output features can accurately represent the user's interest migration trend among heterogeneous channels.

3. The method for optimizing cross-channel advertising delivery strategies by integrating user behavior prediction and deep reinforcement learning as described in claim 1, characterized in that, The process of using the intention evolution gradient to perform phase hedging compensation calculation on the internal reward function in step S5 includes: identifying the algebraic sign of the intention evolution gradient; if the intention evolution gradient is negative and its absolute value is greater than the preset decay threshold, then the real-time reward weight of the delivery channel is adjusted using the decay factor reduction strategy model so that the resource allocation coefficient increases and monotonically decreases as the absolute value of the intention evolution gradient increases.

4. The method for optimizing cross-channel advertising delivery strategies by integrating user behavior prediction and deep reinforcement learning as described in claim 1, characterized in that, Before constructing the decision state space of the reinforcement learning agent in step S4, the method further includes: step S41, monitoring the data return latency of each delivery channel; step S42, when the maximum latency deviation between different delivery channels exceeds the preset deviation threshold, calling the time alignment unit to add a timestamp index to the heterogeneous data stream, and eliminating the attribute offset of the feature vector in the decision state space in the time domain caused by the asymmetric latency by performing delay caching alignment on the high-speed data stream.

5. The method for optimizing cross-channel advertising delivery strategies by integrating user behavior prediction and deep reinforcement learning as described in claim 1, characterized in that, The strategy model is built on an asynchronous advantage evaluation architecture, and the decision step size of the reinforcement learning agent is limited by the convergence variance of the attention mechanism network. When the volatility of the convergence variance is detected to be greater than 15%, the decision step size is automatically reduced to 20% of the original step size to suppress numerical oscillations in the resource allocation coefficient calculation process.

6. The method for optimizing cross-channel advertising delivery strategies by integrating user behavior prediction and deep reinforcement learning as described in claim 1, characterized in that, The feature tensor contains attribution features, which are obtained through the following sub-steps: construct a virtual delivery environment, simulate user behavior paths after removing specific touchpoint data, calculate the difference in conversion probability before and after removal, and determine the difference in probability as the pure conversion gain of the specific touchpoint.

7. The method for optimizing cross-channel advertising delivery strategies by integrating user behavior prediction and deep reinforcement learning as described in claim 1, characterized in that, The normalization process in step S1 involves privacy-preserving computation. By injecting random noise conforming to a Laplace distribution into the feature tensor, differential privacy desensitization is achieved, and it is ensured that the desensitized feature flow maintains the original temporal distribution characteristics of user behavior in the statistical dimension.

8. The method for optimizing cross-channel advertising delivery strategies by integrating user behavior prediction and deep reinforcement learning as described in claim 1, characterized in that, When calculating the resource allocation coefficient, the real-time input-output ratio of all channels is extracted as a constraint variable. If the input-output ratio at the current moment is lower than the preset business safety threshold, the output value range boundary of the strategy model is forcibly contracted by restricting the output range of the activation function of the strategy model.

9. The method for optimizing cross-channel advertising delivery strategies by integrating user behavior prediction and deep reinforcement learning as described in claim 1, characterized in that, During the campaign, the system captures user interest shifts caused by changes in the external market environment and calculates the time evolution gradient of user click popularity. If the time evolution gradient is greater than 0.85, the learning rate weight of the fully connected layer of the strategy model is increased through the intent evolution gradient to achieve adaptive iteration of the campaign strategy in response to market dynamics.

Images