A global marketing budget dynamic allocation method based on multi-agent reinforcement learning

Abstract

The application relates to the technical field of digital marketing, and discloses a global marketing budget dynamic allocation method based on multi-agent reinforcement learning, which comprises the following steps: S1, constructing an offline data set; S2, constructing a decoupling model; S3, supervised training and error suppression; S4, online deployment and budget allocation; and S5, feedback updating and retraining. The global marketing budget dynamic allocation method based on multi-agent reinforcement learning splits the originally jointly estimated state and action value problem into two independent supervised learning tasks by constructing an offline reinforcement learning model decoupled from the state and action, trains a state policy network and an action policy network respectively, fundamentally avoids the Q value overestimation problem caused by out-of-distribution actions in traditional offline reinforcement learning, enables the model to be stably trained on historical data without online exploration, greatly reduces the budget waste risk caused by trial and error in a real advertising system, and improves the reliability and safety of model training.

Description

Technical Field

[0001] This invention relates to the field of digital marketing technology, specifically to a dynamic allocation method for full-domain marketing budget based on multi-agent reinforcement learning. Background Technology

[0002] Traditional marketing budget allocation methods suffer from the following technical drawbacks: First, relying on online reinforcement learning requires extensive trial and error in real-world environments, leading to a high risk of wasted budget and difficulties in industrial implementation. Second, traditional offline reinforcement learning faces extrapolation error problems; when an agent encounters a combination of states and actions not present in historical data, the value estimation based on function approximation deviates significantly from the true value, causing the trained strategy to fail in actual deployment. Third, there is a lack of a comprehensive security mechanism from training to deployment; sudden changes in the model's output strategy may cause drastic fluctuations in campaign performance.

[0003] Therefore, there is an urgent need for a dynamic allocation method for the entire marketing budget that can be stably trained on offline data, effectively suppress extrapolation errors, and ensure the security of online deployment. Summary of the Invention

[0004] The purpose of this invention is to provide a dynamic allocation method for full-domain marketing budget based on multi-agent reinforcement learning, so as to solve the problems mentioned in the background art.

[0005] To solve the above technical problems, the present invention provides the following technical solution: a dynamic allocation method for full-domain marketing budget based on multi-agent reinforcement learning, comprising the following steps: S1, constructing an offline dataset: obtaining historical delivery data of each marketing channel and constructing an offline dataset, wherein the offline dataset contains the state characteristics of each channel at historical moments, budget allocation actions, and corresponding delivery effect rewards; S2. Construct a decoupled model: Construct an offline reinforcement learning model that decouples state and action. The model includes a state policy network and an action policy network. The state policy network is used to predict the target state based on the current global state, and the action policy network is used to generate budget allocation actions for each channel based on the target state. S3. Supervised training to suppress errors: Supervised learning is adopted, using state transition samples in the offline dataset to train the state policy network, and using target state and action mapping samples in the offline dataset to train the action policy network, so as to avoid extrapolation errors in offline reinforcement learning. S4. Online Deployment and Budget Allocation: Deploy the trained offline reinforcement learning model to the online marketing system and generate budget allocation strategies for each marketing channel based on real-time collected information on the overall marketing status. S5. Feedback Update Retraining: Update the offline dataset based on the actual deployment effect and periodically retrain the model.

[0006] Preferably, in the offline reinforcement learning model that decouples state and action, the training process of the state policy network is as follows: sample state pairs at adjacent time steps from the offline dataset, use the current state as input and the next state as the supervision label, and update the network parameters by minimizing the state prediction error; the training process of the action policy network is as follows: sample state and action pairs from the offline dataset, use the state as an approximation of the target state, use the state as input and the action as the supervision label, and update the network parameters by minimizing the action fitting error.

[0007] Preferably, the offline reinforcement learning model constructed in step S2 further includes a value evaluation network for evaluating the value of state-action pairs. The value evaluation network is trained using a Q-learning objective function with action constraints. This function only uses the state-action pairs that have appeared in the offline dataset to update the value. For out-of-distribution actions that do not appear in the dataset, no value extrapolation is performed, thereby suppressing the accumulation of extrapolation error.

[0008] Preferably, the status characteristics include at least one of the following information: historical spending amount, remaining budget, real-time competitive situation, user traffic characteristics, current conversion rate, and historical ROI data for each marketing channel; the budget allocation action includes the budget adjustment ratio or specific bid amount for each channel; and the campaign performance reward includes a comprehensive evaluation indicator of at least one of conversion number, ROI, and click-through rate.

[0009] Preferably, in step S3, when training the state policy network and the action policy network, an offline policy evaluation technique is used to verify the trained model: using an importance sampling weighting method, the expected return of the target policy is evaluated based on historical data generated by the behavior policy, and the model is deployed to the online marketing system only when the offline evaluation result meets the preset threshold condition.

[0010] Preferably, after generating the budget allocation strategy for each marketing channel in step S4, a strategy smoothing step is also included: the budget allocation action output by the model at the current moment is weighted and fused with the actual allocation action at the previous moment to obtain the final execution allocation scheme; the weighting coefficient is dynamically adjusted according to the real-time feedback of the online campaign effect to reduce the campaign risk caused by sudden changes in strategy.

[0011] Preferably, when constructing the offline dataset in step S1, a heterogeneous data fusion method is used to unify and organize marketing data from different data sources and different time granularities; for data-sparse channels, data augmentation technology is used to expand the sample size, including neighborhood sample synthesis based on state feature similarity and sliding window sampling based on time series.

[0012] Preferably, each marketing channel corresponds to an agent, and each agent shares action policy network parameters but maintains independent state observation; the state policy network is uniformly scheduled at the global level to achieve collaborative allocation of cross-channel budgets; agents share global information through a centralized value evaluation network, and after training, each agent makes independent decisions based on its local state.

[0013] Preferably, the model maintenance steps are also included: after online deployment, actual deployment data is continuously collected to build an incremental training set; when the size of the incremental training set reaches a preset threshold, offline retraining is triggered; during retraining, the priority sampling strategy in the experience replay buffer is used to prioritize the selection of samples with larger value assessment errors for learning, thereby accelerating the model's adaptation to abnormal deployment scenarios.

[0014] Compared with the prior art, the beneficial effects achieved by the present invention are: First, this invention constructs an offline reinforcement learning model that decouples state and action, splitting the problem of jointly estimating the value of state and action into two independent supervised learning tasks, training the state policy network and the action policy network separately. This fundamentally avoids the problem of Q-value overestimation caused by out-of-distribution actions in traditional offline reinforcement learning, enabling the model to be stably trained on historical data without online exploration. This significantly reduces the risk of wasting budget on trial and error in real advertising systems and improves the reliability and security of model training.

[0015] Secondly, this invention trains a value evaluation network using a Q-learning objective function with action constraints. It updates the value only using state-action pairs that have already appeared in the offline dataset, and does not extrapolate the value of out-of-distribution actions that do not appear in the dataset. At the same time, it combines offline policy evaluation technology to verify the trained model. It is only deployed to the online system when the evaluation result meets the preset threshold condition. Furthermore, after online deployment, the current output and historical actions are weighted and fused through a policy smoothing step to reduce the deployment risk caused by policy mutations. Thus, a full-chain security mechanism from offline training to online deployment is constructed. Attached Figure Description

[0016] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a schematic diagram of the data flow of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] This invention provides the following technical solutions: Example

[0019] Please see Figure 1 and Figure 2 A dynamic allocation method for full-domain marketing budget based on multi-agent reinforcement learning includes the following steps: S1, constructing an offline dataset: obtain historical delivery data of each marketing channel, construct an offline dataset, the offline dataset contains the state characteristics of each channel at historical moments, budget allocation actions, and corresponding delivery performance rewards; S2. Construct a decoupled model: Construct an offline reinforcement learning model that decouples state and action. The model includes a state policy network and an action policy network. The state policy network is used to predict the target state based on the current global state, and the action policy network is used to generate budget allocation actions for each channel based on the target state. S3. Supervised training to suppress errors: Supervised learning is adopted, using state transition samples in the offline dataset to train the state policy network, and using target state and action mapping samples in the offline dataset to train the action policy network, so as to avoid extrapolation errors in offline reinforcement learning. S4. Online Deployment and Budget Allocation: Deploy the trained offline reinforcement learning model to the online marketing system and generate budget allocation strategies for each marketing channel based on real-time collected information on the overall marketing status. S5. Feedback Update Retraining: Update the offline dataset based on the actual deployment effect and periodically retrain the model.

[0020] In the offline reinforcement learning model that decouples state and action, the training process of the state policy network is as follows: sample state pairs from adjacent time steps in the offline dataset, use the current state as input and the next state as the supervision label, and update the network parameters by minimizing the state prediction error; the training process of the action policy network is as follows: sample state and action pairs from the offline dataset, use the state as an approximation of the target state, use the state as input and the action as the supervision label, and update the network parameters by minimizing the action fitting error.

[0021] The state-policy network only needs to learn the mapping relationship between adjacent states, and the action-policy network only needs to learn the fitting relationship between states and historical actions. Neither of them involves value estimation of unseen state-action pairs, which significantly improves the training stability and policy reliability of the model on offline data. At the same time, the training of both networks is based on real samples in the offline dataset, which can be completed without online exploration, greatly reducing the risk of budget waste caused by trial and error in real advertising environments.

[0022] The offline reinforcement learning model constructed in step S2 also includes a value evaluation network, which is used to evaluate the value of state-action pairs. The value evaluation network is trained using a Q-learning objective function with action constraints. This function only uses the state-action pairs that have appeared in the offline dataset to update the value. For out-of-distribution actions that do not appear in the dataset, no value extrapolation is performed, thereby suppressing the accumulation of extrapolation error.

[0023] The Q-learning objective function is a core mathematical formula used in reinforcement learning to update the value evaluation network. Its basic form is: Target value = Immediate reward + Discount factor × Maximum action value in the next state. The network is trained by minimizing the error between the current Q-value and the target value. In this application, the objective function is improved to a form with action constraints, calculating the Q-value to update the target only on real-world state-action pairs in the offline dataset, and not performing value estimation or extrapolation for out-of-distribution actions.

[0024] By introducing a value evaluation network into the offline reinforcement learning model and training it with a Q-learning objective function with action constraints, the value updates are strictly limited to state-action pairs that have actually appeared in the offline dataset. No value extrapolation is performed for out-of-distribution actions that have not appeared in the dataset, thus fundamentally suppressing the generation and accumulation of extrapolation errors at the algorithmic level. Compared with traditional Q-learning methods, this mechanism effectively avoids the blind overestimation or underestimation of the value of unexplored actions caused by function approximation, making the output of the value evaluation network closer to the true distribution. This significantly improves the convergence stability and evaluation accuracy of the model during offline training, thereby ensuring the reliable theoretical performance guarantee of the budget allocation strategy when finally deployed to the online marketing system.

[0025] Status characteristics include at least one of the following: historical spending amount, remaining budget, real-time competitive situation, user traffic characteristics, current conversion rate, and historical ROI data for each marketing channel; budget allocation actions include budget adjustment ratios or specific bid amounts for each channel; and campaign performance rewards include a comprehensive evaluation indicator of at least one of the following: conversion volume, ROI, and click-through rate.

[0026] By clearly defining the specific connotations of state characteristics, budget allocation actions, and campaign performance rewards, a complete and quantifiable input-output space is provided for the offline reinforcement learning model. This allows the model to fully perceive the multi-dimensional real-time status of various marketing channels and output executable budget adjustment ratios or bid amounts accordingly. Ultimately, comprehensive evaluation indicators such as conversion volume, ROI, or click-through rate are used as optimization targets. This ensures that the state space accurately reflects the channel competitive landscape and resource constraints, the action space is practically operable, and the reward signals can comprehensively measure campaign performance. This effectively supports the supervised training of the state strategy network and action strategy network, making the budget allocation strategies learned by the model more closely aligned with the real needs of the omni-channel marketing scenario.

[0027] In step S3, when training the state policy network and action policy network, an offline policy evaluation technique is used to verify the trained model: using an importance sampling weighting method, the expected return of the target policy is evaluated based on historical data generated by the behavior policy. The model is only deployed to the online marketing system when the offline evaluation results meet the preset threshold conditions.

[0028] Offline strategy evaluation technology is a reinforcement learning method that evaluates the performance of a target strategy using only historical data collected from other behavioral strategies without actually executing the target strategy. Through statistical techniques such as importance sampling, it weights the action probability ratios of each sample in the historical data, thereby unbiasedly estimating the expected return of the target strategy in the current environment. This avoids the risks and costs associated with real online trial and error for strategy evaluation. In this application, an importance sampling weighting method is used to calculate the expected return estimate of the target strategy corresponding to the currently trained decoupled model based on the channel states and action samples generated by the behavioral strategies in the historical delivery data. This estimate is then compared with a preset safety threshold. Only when the offline evaluation result meets the threshold condition is the model allowed to be deployed to the online marketing system to execute real budget allocation; otherwise, it returns to the training phase for further optimization.

[0029] By introducing offline strategy evaluation technology before model deployment, and using the importance sampling weighting method to make an unbiased estimate of the expected return of the target strategy based on historical behavioral data, and deploying the model to the online marketing system only after the evaluation results meet the preset threshold conditions, a safe verification mechanism from offline training to online application is constructed. This effectively avoids the budget waste and effect fluctuation risk caused by directly putting underperforming strategies into the real deployment environment, and solves the pain point of traditional methods that can only rely on online trial and error verification due to the lack of reliable evaluation means.

[0030] After generating the budget allocation strategy for each marketing channel in step S4, a strategy smoothing step is also included: the budget allocation action output by the model at the current moment is weighted and merged with the actual allocation action at the previous moment to obtain the final execution allocation plan; the weighting coefficient is dynamically adjusted according to the real-time feedback of the online campaign performance to reduce the campaign risk caused by sudden changes in strategy.

[0031] By introducing a strategy smoothing step, the budget allocation action output by the model at the current moment is weighted and fused with the actual allocation action at the previous moment. The weighting coefficient is dynamically adjusted based on the real-time feedback of online campaign performance. This effectively suppresses drastic fluctuations in budget allocation caused by sudden changes in model output, and avoids campaign performance oscillations caused by instantaneous strategy changes in complex marketing environments. At the same time, the dynamic adjustment of the weighting coefficient enables the system to maintain high consistency when market changes are stable and to respond quickly when historical strategies are detected to be obviously ineffective.

[0032] In step S1, when constructing the offline dataset, a heterogeneous data fusion method is used to unify and organize marketing data from different data sources and different time granularities. For channels with sparse data, data augmentation techniques are used to expand the sample size. Data augmentation includes neighborhood sample synthesis based on state feature similarity and sliding window sampling based on time series.

[0033] Heterogeneous data fusion refers to the technical means of unifying, aligning and integrating marketing data from different data sources with different data structures, different time granularities or different sampling frequencies. It aims to eliminate problems such as inconsistent data formats, mismatched timestamps and differences in field semantics, and form a standardized offline dataset that can be directly used by the model. Data augmentation technology is a method to expand the dataset size by artificially constructing or transforming new training samples when the original data samples are insufficient. Common methods include synthesizing new samples based on the similarity between samples and sliding window sampling based on time series. In this application, firstly, a heterogeneous data fusion method is used to align and organize multi-source data such as consumption data, real-time bidding data, and user behavior logs from various marketing channels according to a unified time granularity and state feature dimension. Secondly, for low-traffic channels with sparse data, a neighborhood sample synthesis technique based on state feature similarity is adopted to find similar state samples in the feature space and interpolate to generate new samples. Additionally, a sliding window sampling technique based on time series is used to use state sequences within a continuous time window as multiple training samples, thereby effectively expanding the training sample size of sparse channels and improving the generalization ability and decision accuracy of the state policy network and action policy network on unpopular channels.

[0034] Each marketing channel corresponds to an agent. Each agent shares action policy network parameters but maintains independent state observation. The state policy network is uniformly scheduled at the global level to achieve collaborative allocation of budgets across channels. Agents share global information through a centralized value evaluation network. After training, each agent makes independent decisions based on its local state.

[0035] By constructing a multi-agent architecture with one agent for each marketing channel and designing shared action policy network parameters among agents to reduce training complexity, while maintaining independent state observations to preserve channel-specific characteristics, a flexible separation of the training and execution phases is achieved. The state policy network is uniformly scheduled at the global level, comprehensively considering the real-time state and remaining budget of each channel, enabling collaborative allocation of cross-channel budgets and resolving resource competition, avoiding budget inefficiencies or allocation conflicts caused by independent channel decisions. Agents share global information during training through a centralized value evaluation network, allowing each agent to perceive the policy influence of other agents during learning, thus converging to a collaboratively optimal joint policy. After training, each agent makes independent decisions based solely on its local state, reducing computational latency and communication overhead during online deployment and improving the system's real-time responsiveness.

[0036] It also includes model maintenance steps: after online deployment, continuously collect actual deployment data to build an incremental training set; when the size of the incremental training set reaches a preset threshold, trigger offline retraining; during retraining, use the priority sampling strategy in the experience replay buffer to prioritize the selection of samples with larger value assessment errors for learning, thereby accelerating the model's adaptation to abnormal deployment scenarios.

[0037] After online deployment, the model continuously collects actual deployment data to build an incremental training set, enabling it to absorb information about changes in the market environment and shifts in user behavior in real time. When the size of the incremental training set reaches a preset threshold, offline retraining is triggered, ensuring timely model updates while avoiding excessive training computation overhead. During retraining, a priority sampling strategy in the experience replay buffer is used to prioritize the selection of samples with larger value assessment errors for learning. This allows the model to focus on abnormal scenarios where the current strategy is not performing well or where the environment undergoes sudden changes, accelerating its adaptability to non-stationary factors such as market fluctuations and changes in the competitive landscape. This, in turn, enhances the model's continuous optimization capabilities in the task of allocating marketing budgets across the entire domain.

[0038] In practice, the system first acquires historical campaign data from various marketing channels to construct an offline dataset containing state features, budget allocation actions, and campaign performance rewards. Then, it builds an offline reinforcement learning model that decouples state and action, including a state-policy network for predicting the target state and an action-policy network for generating budget allocation actions. Next, a supervised learning approach is adopted, using state transition samples from the offline dataset to train the state-policy network and target state-action mapping samples to train the action-policy network, thus avoiding extrapolation errors in offline reinforcement learning. The trained model is then deployed to the online marketing system, generating budget allocation strategies for each channel based on real-time collected full-domain marketing state information. Finally, the offline dataset is updated based on actual campaign performance, and the model is periodically retrained to form a closed loop of continuous optimization.

[0039] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for dynamic allocation of full-domain marketing budget based on multi-agent reinforcement learning, characterized in that, Includes the following steps: S1. Construct an offline dataset: Obtain historical campaign data from each marketing channel and construct an offline dataset. The offline dataset includes the status characteristics of each channel at historical moments, budget allocation actions, and corresponding campaign performance rewards. S2. Construct a decoupled model: Construct an offline reinforcement learning model that decouples state and action. The model includes a state policy network and an action policy network. The state policy network is used to predict the target state based on the current global state, and the action policy network is used to generate budget allocation actions for each channel based on the target state. S3. Supervised training to suppress errors: Supervised learning is adopted, using state transition samples in the offline dataset to train the state policy network, and using target state and action mapping samples in the offline dataset to train the action policy network, so as to avoid extrapolation errors in offline reinforcement learning. S4. Online Deployment and Budget Allocation: Deploy the trained offline reinforcement learning model to the online marketing system and generate budget allocation strategies for each marketing channel based on real-time collected information on the overall marketing status. S5. Feedback Update Retraining: Update the offline dataset based on the actual deployment effect and periodically retrain the model.

2. The method for dynamic allocation of full-domain marketing budget based on multi-agent reinforcement learning according to claim 1, characterized in that: In the offline reinforcement learning model that decouples state and action, the training process of the state policy network is as follows: sample state pairs at adjacent time steps from the offline dataset, use the current state as input and the next state as the supervision label, and update the network parameters by minimizing the state prediction error; the training process of the action policy network is as follows: sample state and action pairs from the offline dataset, use the state as an approximation of the target state, use the state as input and the action as the supervision label, and update the network parameters by minimizing the action fitting error.

3. The method for dynamic allocation of full-domain marketing budget based on multi-agent reinforcement learning according to claim 1, characterized in that: The offline reinforcement learning model constructed in step S2 also includes a value evaluation network, which is used to evaluate the value of state-action pairs. The value evaluation network is trained using a Q-learning objective function with action constraints. This function only uses the state-action pairs that have appeared in the offline dataset to update the value. For out-of-distribution actions that do not appear in the dataset, no value extrapolation is performed, thereby suppressing the accumulation of extrapolation error.

4. The method for dynamic allocation of full-domain marketing budget based on multi-agent reinforcement learning according to claim 1, characterized in that: The status characteristics include at least one of the following: historical spending amount, remaining budget, real-time competitive situation, user traffic characteristics, current conversion rate, and historical ROI data for each marketing channel; the budget allocation action includes the budget adjustment ratio or specific bid amount for each channel; the campaign performance reward includes a comprehensive evaluation indicator of at least one of the following: conversion number, ROI, and click-through rate.

5. The method for dynamic allocation of full-domain marketing budget based on multi-agent reinforcement learning according to claim 1, characterized in that: In step S3, when training the state policy network and action policy network, an offline policy evaluation technique is used to verify the trained model: using an importance sampling weighting method, the expected return of the target policy is evaluated based on historical data generated by the behavior policy. The model is only deployed to the online marketing system when the offline evaluation results meet the preset threshold conditions.

6. The method for dynamic allocation of full-domain marketing budget based on multi-agent reinforcement learning according to claim 1, characterized in that: After generating the budget allocation strategy for each marketing channel in step S4, the method also includes a strategy smoothing step: weighting and fusing the budget allocation action output by the model at the current moment with the actual allocation action at the previous moment to obtain the final execution allocation scheme. The weighting coefficients are dynamically adjusted based on real-time feedback on online campaign performance to reduce the campaign risk caused by sudden changes in strategy.

7. The method for dynamic allocation of full-domain marketing budget based on multi-agent reinforcement learning according to claim 1, characterized in that: In step S1, when constructing the offline dataset, a heterogeneous data fusion method is used to unify and organize marketing data from different data sources and different time granularities. For data-sparse channels, data augmentation techniques are used to expand the sample size. The data augmentation includes neighborhood sample synthesis based on state feature similarity and sliding window sampling based on time series.

8. The method for dynamic allocation of full-domain marketing budget based on multi-agent reinforcement learning according to claim 1, characterized in that: Each marketing channel corresponds to an agent. Each agent shares action policy network parameters but maintains independent state observation. The state policy network is uniformly scheduled at the global level to achieve collaborative allocation of budgets across channels. Agents share global information through a centralized value evaluation network. After training, each agent makes independent decisions based on its local state.

9. The method for dynamic allocation of full-domain marketing budget based on multi-agent reinforcement learning according to claim 1, characterized in that: It also includes model maintenance steps: after online deployment, continuously collect actual deployment data to build an incremental training set; when the size of the incremental training set reaches a preset threshold, trigger offline retraining; during retraining, use the priority sampling strategy in the experience replay buffer to prioritize the selection of samples with larger value assessment errors for learning, thereby accelerating the model's adaptation to abnormal deployment scenarios.

Images