Content marketing effect evaluation and optimization method and system based on reinforcement learning

Abstract

This invention discloses a method and system for evaluating and optimizing the effectiveness of content marketing based on reinforcement learning, comprising the following steps: collecting data from content marketing operations and preprocessing it; constructing a sample dataset; constructing a constraint-aware implicit Q-learning model based on the sample dataset; inputting state data into a revenue state value network and a constraint state value network for training; obtaining revenue action value output and constraint action value output based on the revenue state value output and constraint state value output; fusing the revenue advantage data and constraint advantage data; performing weighted learning on the delivery action data based on a set of delivery action weights; making decisions based on a content delivery optimization strategy model to generate a content delivery plan, and periodically training the constraint-aware implicit Q-learning model. This invention uses a constraint-aware implicit Q-learning model to optimize the effectiveness of content marketing delivery.

Description

Technical Field

[0001] This invention relates to the field of intelligent content delivery and decision optimization technology, and in particular to a method and system for evaluating and optimizing the effectiveness of content marketing based on reinforcement learning. Background Technology

[0002] With the rapid development of digital marketing, content delivery is playing an increasingly important role in marketing campaigns. Traditional content marketing methods rely on manually formulated delivery strategies, often lacking targeted optimization and real-time feedback adjustments. This approach has certain limitations, especially when faced with a large amount of content and channel resources, making it difficult to efficiently integrate and optimize resources, resulting in greater uncertainty in marketing effectiveness.

[0003] Currently, data-driven optimization methods have seen some application, but existing technologies generally suffer from the following shortcomings. First, existing optimization models mostly rely on static rules or simple machine learning methods, which cannot effectively handle complex marketing constraints (such as budget, frequency control, and compliance requirements), resulting in a lack of flexibility and scalability in multi-objective optimization. Second, traditional reinforcement learning models often ignore multi-dimensional constraints in marketing, leading to strategies generated by the models that may not meet the complex constraints of real-world environments. Finally, most existing technologies rely on simple historical data learning, lacking dynamic adjustment and long-term optimization mechanisms, resulting in slow strategy updates and difficulty in responding to real-time changing market demands.

[0004] Therefore, how to provide methods and systems for evaluating and optimizing the effectiveness of content marketing based on reinforcement learning is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose a method and system for evaluating and optimizing the effectiveness of content marketing based on reinforcement learning. This invention fully utilizes reinforcement learning theory, multi-value function modeling, and constraint-aware optimization mechanisms to uniformly model user behavior, content characteristics, channel resources, and delivery constraints during the content marketing process. By constructing a constraint-aware implicit Q-learning model, it achieves the evaluation of content delivery effectiveness and strategy optimization. This invention generates delivery decision strategies that balance marketing revenue and business constraints through joint learning of revenue and constraint values. It can continuously update and optimize the strategy model in a dynamic environment, possessing advantages such as strong adaptability to complex constraints, high strategy stability, and sustainable improvement in content delivery optimization effectiveness.

[0006] The content marketing effectiveness evaluation and optimization method based on reinforcement learning according to embodiments of the present invention includes the following steps: Collect data from content marketing operations, preprocess it, and obtain a set of historical campaign log data; A sample dataset was constructed based on historical deployment log data. Based on the sample data set, a constraint-aware implicit Q-learning model is constructed, which includes a shared feature encoding network, a payoff state value network, a payoff action value network, a constraint state value network, a constraint action value network, and a policy network. The state data is input into the profit state value network and the constraint state value network for training, and the profit state value output and the constraint state value output are obtained. Based on the output of the benefit state value and the output of the constraint state value, the output of the benefit action value and the output of the constraint action value are obtained. Based on the value output of the benefit action and the value output of the benefit state, benefit advantage data is generated. Based on the value output of the constraint action and the value output of the constraint state, constraint advantage data is generated. These are then fused together to obtain the constraint perception advantage fusion result and the set of action weights for deployment. A content delivery optimization strategy model is obtained by weighted learning of delivery action data based on a set of delivery action weights. The decision-making process is based on the content delivery optimization strategy model, which generates a content delivery plan and controls its execution. The constraint-aware implicit Q-learning model is periodically trained to obtain an updated content delivery optimization strategy model.

[0007] Optionally, the preprocessing includes cleaning, deduplication, time alignment, and identification.

[0008] Optionally, constructing the sample dataset based on the historical deployment log dataset specifically includes: Each log entry is read from the historical deployment log data set, a sample key is determined based on the log entry, and the log entries are grouped and sorted by the sample key to obtain a candidate sample sequence set; Based on the candidate sample sequence set, a sample pair set is obtained. Based on the sample pair set, user feature vector, content feature vector, channel resource feature vector, constraint state feature vector, and context feature vector are generated to obtain state data. For each sample pair in the sample pair set, generate deployment action data, revenue feedback data, and constraint feedback data based on the current record; For each sample pair in the sample pair set, obtain the next state data based on the next record; The status data, deployment action data, revenue feedback data, constraint feedback data, and next status data are associated and encapsulated to form a single sample. The encapsulation process is then executed to obtain a sample data set.

[0009] Optionally, the construction of the constraint-aware implicit Q-learning model based on the sample data set, including a shared feature encoding network, a reward state value network, a reward action value network, a constraint state value network, a constraint action value network, and a policy network, specifically includes: Based on the sample data set, determine the model input of the constraint-aware implicit Q-learning model; Establish a shared feature coding network, configure the shared feature coding network to receive state data, perform feature coding processing on the state data, and output a shared feature representation vector; Establish a revenue state value network and a revenue action value network at the output of the shared feature encoding network; Establish a constrained state value network and a constrained action value network at the output of the shared feature encoding network; Establish a set of constraint multipliers and create a corresponding constraint multiplier update unit for each constraint multiplier; Establish an advantage calculation and fusion unit, a weight generation unit, and a policy network.

[0010] Optionally, the step of inputting state data into the revenue state value network and the constraint state value network for training, and obtaining the revenue state value output and the constraint state value output, specifically includes: Extract training sample groups from the sample data set, organize the training sample groups into training batches, and initialize the network parameters of the profit state value network and the constraint state value network. The state data from the training batch is input into the profit state value network to obtain the current profit state value output, and the next profit state value output is obtained based on the next state data. Generate a revenue status value monitoring target based on revenue feedback data, preset discount coefficients, and the next revenue status value output. Based on the current revenue state value output and revenue state value monitoring target, calculate the revenue state value loss and update the network parameters of the revenue state value network; The state data from the training batch is input into the constrained state value network to obtain the current constrained state value output, and the next constrained state value output is obtained based on the next state data. Generate constraint state value supervision target based on constraint feedback data, preset discount coefficient and next constraint state value output; Based on the current constraint state value output and the constraint state value supervision objective, calculate the constraint state value loss and update the network parameters of the constraint state value network. The reward state value network and the constraint state value network are updated until the preset stopping condition is met, and the trained reward state value network and constraint state value network are obtained. Input the state data from the sample dataset into the trained profit state value network and constraint state value network to obtain the profit state value output and constraint state value output.

[0011] Optionally, obtaining the profit action value output and constraint action value output based on the profit state value output and constraint state value output specifically includes: Extract action value training sample groups from the sample dataset and organize them into action value training batches; Read the output of the benefit state value and the output of the constraint state value, and align them with the state data in the action value training batch to obtain the training aligned data group; Initialize the network parameters of the reward action value network and the constraint action value network; For each sample in the action value training batch, a revenue action value supervision target is generated based on the revenue feedback data, the preset discount coefficient, and the revenue state value output corresponding to the next state data. Calculate the loss of revenue action value based on the revenue action value monitoring target, and update the network parameters of the revenue action value network; For the constraint action value network, a constraint action value supervision target is generated based on constraint feedback data, preset discount coefficients, and the constraint state value output corresponding to the next state data. The constraint action value loss is calculated based on the constraint action value supervision target, and the network parameters of the constraint action value network are updated. The reward action value network and the constraint action value network are updated until the preset termination condition is met, resulting in the trained reward action value network and constraint action value network. Based on the trained reward action value network and constraint action value network, the reward action value output and constraint action value output are obtained.

[0012] Optionally, the step of generating benefit advantage data based on the benefit action value output and benefit state value output, generating constraint advantage data based on the constraint action value output and constraint state value output, and performing fusion processing to obtain the constraint-aware advantage fusion result specifically includes: Extract advantage calculation sample groups from the sample dataset and organize them into advantage calculation batches; Calculate revenue advantage data based on revenue action value output and revenue status value output; Calculate constraint dominance data for constraint state value output and constraint action value output; Scale-homogenize the revenue advantage data and constraint advantage data to obtain scale-homogenized revenue advantage data and constraint advantage data. The constraint-aware advantage fusion result is obtained by fusing the scale-consistent benefit advantage data and constraint advantage data based on the constraint multiplier set, and a set of deployment action weights is generated.

[0013] Optionally, the step of weighting the delivery action data based on the set of delivery action weights to obtain the content delivery optimization strategy model specifically includes: Extract policy training sample groups from the sample dataset and organize them into policy training batches to initialize the network parameters of the policy network. For each sample in the policy training batch, the state data is input into the policy network to obtain the probability distribution of the action or the parameters of the action. The strategy learning objective is constructed based on the data of the delivery action and the probability distribution of the delivery action or the parameters of the delivery action, and the strategy learning objective is weighted based on the set of delivery action weights to obtain the weighted strategy learning objective. The policy learning loss is calculated based on the weighted policy learning objective, and the network parameters of the policy network are updated until the preset stopping condition for the policy network is met, thus obtaining the trained policy network. The trained policy network is output as the content delivery optimization policy model.

[0014] Optionally, the step of making decisions based on the content delivery optimization strategy model, generating a content delivery plan and controlling delivery execution, and periodically training the constraint-aware implicit Q-learning model to obtain an updated content delivery optimization strategy model specifically includes: Based on the content delivery optimization strategy model, a decision is made on the set of content to be delivered and the set of available channel resources to generate a content delivery plan; Based on the content delivery plan, execute content marketing campaigns, obtain campaign execution data, and generate campaign performance feedback data and constraint feedback data; The campaign performance feedback data and constraint feedback data are merged and processed to obtain a feedback data set; The feedback data set after the deployment is executed is merged and updated with the historical deployment log data set to obtain the updated historical deployment log data set; Based on the updated historical delivery log data set, the constraint-aware implicit Q-learning model is periodically trained to obtain the updated policy network, which is then used as the updated content delivery optimization policy model.

[0015] The content marketing effectiveness evaluation and optimization system based on reinforcement learning according to embodiments of the present invention includes the following modules: The data acquisition module is used to collect data from content marketing operations, perform preprocessing, and obtain a set of historical campaign log data. The sample construction module is used to construct a sample data set based on the historical deployment log data set; The model building module is used to construct a constraint-aware implicit Q-learning model based on a sample dataset, including a shared feature encoding network, a payoff state value network, a payoff action value network, a constraint state value network, a constraint action value network, and a policy network. The state value calculation module is used to input state data into the revenue state value network and the constraint state value network for training, and to obtain the revenue state value output and the constraint state value output. The action value calculation module is used to obtain the profit action value output and the constraint action value output based on the profit state value output and the constraint state value output. The advantage fusion module is used to generate benefit advantage data based on the benefit action value output and benefit state value output, and to generate constraint advantage data based on the constraint action value output and constraint state value output. The fusion process is then performed to obtain the constraint-aware advantage fusion result and the set of action weights for deployment. The strategy training module is used to perform weighted learning on the delivery action data based on the set of delivery action weights to obtain a content delivery optimization strategy model. The delivery execution module is used to make decisions based on the content delivery optimization strategy model, generate content delivery plans and control delivery execution, and periodically train the constraint-aware implicit Q-learning model to obtain the updated content delivery optimization strategy model.

[0016] The beneficial effects of this invention are: This invention constructs a constraint-aware implicit Q-learning model, integrating user behavior, content characteristics, channel resource characteristics, and placement constraints in the content marketing process into a reinforcement learning modeling framework. This enables systematic evaluation and optimization of content placement effectiveness. By establishing state value and action value models for both revenue and constraint values, the placement strategy can simultaneously characterize the impact of budget, frequency control, and compliance constraints on placement behavior while learning marketing revenue, thereby improving the feasibility and stability of content placement decisions in complex business scenarios.

[0017] This invention introduces a weighted learning mechanism based on an advantage function during the policy learning process. By fusing the payoff advantage and constraint advantage to generate deployment action weights, it effectively guides the policy network to prioritize deployment behaviors that balance payoff enhancement and constraint satisfaction. Compared to optimization methods that rely solely on a single payoff objective, this method reduces the volatility risk of the policy during training, enabling the generated deployment strategy to maintain relatively stable performance in long-term operation and avoiding policy failures caused by local optima or constraint neglect.

[0018] Furthermore, this invention achieves dynamic iterative optimization of content delivery strategies by continuously feeding back delivery execution results and periodically updating the historical delivery log data set. This mechanism enables the strategy model to be continuously updated as user behavior changes, content characteristics evolve, and channel resource status is adjusted, thereby enhancing the adaptability of the content marketing system to environmental changes and improving the accuracy of overall delivery performance evaluation and the continuous effectiveness of delivery optimization. Attached Figure Description

[0019] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of the method and system for evaluating and optimizing the effectiveness of content marketing based on reinforcement learning proposed in this invention; Figure 2 This is a schematic diagram of the structure of the constraint-aware implicit Q-learning model in the content marketing effectiveness evaluation and optimization method and system based on reinforcement learning proposed in this invention. Figure 3 This is a schematic diagram of the system structure of the content marketing effectiveness evaluation and optimization method and system based on reinforcement learning proposed in this invention. Detailed Implementation

[0020] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0021] refer to Figures 1-2 A reinforcement learning-based method for evaluating and optimizing the effectiveness of content marketing includes the following steps: Collect data from content marketing operations, preprocess it, and obtain a set of historical campaign log data; A sample dataset was constructed based on historical deployment log data. Based on the sample data set, a constraint-aware implicit Q-learning model is constructed, which includes a shared feature encoding network, a payoff state value network, a payoff action value network, a constraint state value network, a constraint action value network, and a policy network. The state data is input into the profit state value network and the constraint state value network for training, and the profit state value output and the constraint state value output are obtained. Based on the output of the benefit state value and the output of the constraint state value, the output of the benefit action value and the output of the constraint action value are obtained. Based on the value output of the benefit action and the value output of the benefit state, benefit advantage data is generated. Based on the value output of the constraint action and the value output of the constraint state, constraint advantage data is generated. These are then fused together to obtain the constraint perception advantage fusion result and the set of action weights for deployment. A content delivery optimization strategy model is obtained by weighted learning of delivery action data based on a set of delivery action weights. The decision-making process is based on the content delivery optimization strategy model, which generates a content delivery plan and controls its execution. The constraint-aware implicit Q-learning model is periodically trained to obtain an updated content delivery optimization strategy model.

[0022] In this embodiment, the collection of data from content marketing operations, after preprocessing, yields a historical campaign log data set, specifically including: Collect user behavior data, user profile data, content material data, channel and placement resource data, placement time and scenario context data, placement execution data, and performance feedback data from content marketing operations, and summarize them into a raw data set; The original collected data set is cleaned. The cleaning process includes formatting data fields, filling or removing missing fields, filtering outlier values, and validating the enumeration fields to obtain the cleaned data set. The cleaned dataset is deduplicated. Duplicate records are identified based on deduplication criteria consisting of user identifier, content identifier, channel identifier, placement identifier, and event timestamp. Data records that meet the preset priority rules are retained to obtain the deduplicated dataset. The deduplicated dataset is time-aligned by mapping the event timestamps of each data record to the time window corresponding to the preset time granularity, and generating a time alignment identifier for each record to obtain the time-aligned dataset. The time-aligned dataset is then tokenized by mapping user identifiers, content identifiers, channel identifiers, and placement identifiers to unique identifiers used within the system. Log keys are generated based on these identifiers to obtain the tokenized dataset. The identifiable data set is organized and stored according to the log key to form searchable log entries, and the historical log data set is output.

[0023] In this embodiment, constructing a sample data set based on a historical deployment log data set specifically includes: Each log entry is read from the historical delivery log data set. The sample key is determined based on the user identifier, content identifier, channel identifier, placement identifier, and time alignment identifier in the log entry. The log entries are then grouped and sorted using the sample key to obtain a candidate sample sequence set. The sample key is composed of the user identifier, content identifier, channel identifier, placement identifier, and time alignment identifier. For each candidate sample sequence in the candidate sample sequence set, determine the current record and the next record according to the order of the time alignment identifiers, and form a sample pair with the current record and the next record to obtain a sample pair set; For each sample pair in the sample pair set, extract the user profile field and user behavior field from the current record and generate a user feature vector; extract the content material field from the current record and generate a content feature vector; extract the channel and placement resource fields from the current record and generate a channel resource feature vector; extract the budget occupancy field, budget remaining field, frequency count field, frequency remaining field, and compliance mark or risk score field from the current record and generate a constraint status feature vector; extract the placement time period field and scenario context field from the current record and generate a context feature vector; concatenate the user feature vector, content feature vector, channel resource feature vector, constraint status feature vector, and context feature vector to obtain the status data corresponding to the sample pair; For each sample pair in the sample pair set, extract the content selection or sorting field, channel or placement selection field, placement timing field, placement frequency field, and bid or budget allocation field from the current record to generate placement action data corresponding to the current record; For each sample pair in the sample pair set, extract the effect feedback field and cost field from the current record, calculate the revenue feedback data according to the preset revenue calculation rules, and obtain the revenue feedback data corresponding to the current record. The preset revenue calculation rules include at least one of the following: conversion event count, transaction amount or order amount, and retention event count as positive items, and the campaign cost as a negative item for combined calculation. For each sample pair in the sample pair set, extract the budget consumption field, frequency control field, and compliance mark or risk score field from the current record, generate constraint feedback data according to the preset constraint calculation rules, and obtain the constraint feedback data corresponding to the current record. The preset constraint calculation rules include at least using the budget consumption amount as budget constraint feedback, using the excess amount exceeding the preset frequency threshold as frequency constraint feedback, and using the mark or risk score that triggers the compliance rule as compliance constraint feedback. For each sample pair in the sample pair set, extract the user profile field and user behavior field from the next record and generate the next user feature vector; extract the content material field from the next record and generate the next content feature vector; extract the channel and placement resource fields from the next record and generate the next channel resource feature vector; extract the budget occupancy field, budget remaining field, frequency count field, frequency remaining field, and compliance mark or risk score field from the next record and generate the next constraint state feature vector; extract the placement time period field and scenario context field from the next record and generate the next context feature vector; concatenate the next user feature vector, next content feature vector, next channel resource feature vector, next constraint state feature vector, and next context feature vector to obtain the next state data corresponding to the sample pair; The state data, deployment action data, revenue feedback data, constraint feedback data, and next state data are associated and encapsulated to form a single sample. The encapsulation process is repeated for each candidate sample sequence in the candidate sample sequence set to obtain a sample data set.

[0024] In this embodiment, based on the sample data set, the constraint-aware implicit Q-learning model, which includes a shared feature encoding network, a reward state value network, a reward action value network, a constraint state value network, a constraint action value network, and a policy network, specifically includes: Based on the sample data set, determine the model input of the constraint-aware implicit Q-learning model. The model input includes at least state data, delivery action data, benefit feedback data, constraint feedback data, and next state data. Establish a shared feature encoding network, configure the shared feature encoding network to receive state data, perform feature encoding processing on the state data, and output a shared feature representation vector; At the output of the shared feature encoding network, a revenue state value network and a revenue action value network are established. The revenue state value network is used to receive the shared feature representation vector and output the revenue state value output, while the revenue action value network is used to receive the shared feature representation vector and combine it with the action data to output the revenue action value output. At the output of the shared feature encoding network, at least one constraint state value network and at least one constraint action value network are established. Each constraint state value network is used to receive the shared feature representation vector and output the corresponding constraint state value output. Each constraint action value network is used to receive the shared feature representation vector and output the corresponding constraint action value output in combination with the deployment action data. The constraint state value output and the constraint action value output include at least one or more of the following: budget constraint value output, frequency constraint value output, and compliance constraint value output. Establish a set of constraint multipliers and create a corresponding constraint multiplier update unit for each constraint multiplier in the set. Configure the constraint multiplier update unit to: perform sliding statistics on constraint feedback of the same constraint dimension based on constraint feedback data according to a preset statistical window to generate constraint feedback statistical results, wherein the sliding statistics at least include summing the constraint feedback within the statistical window and dividing by the number of samples within the statistical window to obtain the window average; compare the constraint feedback statistical results with a preset constraint threshold to obtain the constraint deviation value; iteratively update the corresponding constraint multipliers based on the constraint deviation value and a preset update step size, and after each update, prune the updated constraint multipliers to non-negative values ​​not less than zero to obtain the updated set of constraint multipliers. An advantage calculation and fusion unit is established and configured to: generate benefit advantage data based on benefit action value output and benefit state value output; generate constraint advantage data based on constraint action value output and constraint state value output; perform scale-homogenization processing on the benefit advantage data and constraint advantage data respectively; the scale-homogenization processing includes calculating the batch mean and batch standard deviation of the advantage data within the same batch, subtracting the batch mean from the advantage data and then dividing by the batch standard deviation to obtain the scale-homogenized benefit advantage data and the scale-homogenized constraint advantage data; and perform fusion processing on the scale-homogenized benefit advantage data and the scale-homogenized constraint advantage data based on the constraint multiplier set to generate the constraint-aware advantage fusion result. Establish a weight generation unit and configure the weight generation unit to: multiply the constraint perception advantage fusion result by a preset temperature coefficient and then perform exponential mapping processing to obtain the initial weight result; perform upper and lower bound constraint processing on the initial weight result to obtain a set of delivery action weights that satisfy the preset lower bound and the preset upper bound; and perform normalization processing on the delivery action weight set so that the sum of each weight in the delivery action weight set is a preset normalization constant. Establish a policy network and configure the policy network to receive the shared feature representation vector and output the probability distribution of the delivery action or the delivery action parameters corresponding to the delivery action data; The policy network is trained by performing weighted learning on the action data in the sample dataset based on the action weight set. The shared feature encoding network, the payout state value network, the payout action value network, at least one constraint state value network, at least one constraint action value network, the constraint multiplier update unit, the advantage calculation and fusion unit, the weight generation unit, and the policy network are connected and assembled to obtain the constraint-aware implicit Q-learning model.

[0025] In this embodiment, the state data is input into the profit state value network and the constraint state value network for training, and the profit state value output and constraint state value output are obtained specifically including: A training sample set for training state values ​​is extracted from the sample data set. The training sample set includes at least the state data, benefit feedback data, constraint feedback data and next state data that are related to each other, and the training sample set is organized into a training batch. Initialize the network parameters of the profit-state value network and at least one constraint-state value network in the constraint-aware implicit Q-learning model, set the preset discount coefficient, preset learning rate, preset batch size and preset stopping condition, and obtain the initialized profit-state value network and at least one initialized constraint-state value network. Input the state data from the training batch into the profit state value network to obtain the current profit state value output, and input the next state data from the training batch into the profit state value network to obtain the next profit state value output. A revenue state value monitoring target is generated based on revenue feedback data, a preset discount coefficient, and the next revenue state value output. The revenue state value monitoring target is the target value obtained by discounting and summing the revenue feedback data and the preset discount coefficient on the next revenue state value output. The revenue state value loss between the current revenue state value output and the revenue state value monitoring target is calculated, and the network parameters of the revenue state value network are updated based on the revenue state value loss. For each constraint state value network in at least one constraint state value network, the state data in the training batch is input into the corresponding constraint state value network to obtain the current constraint state value output, and the next state data in the training batch is input into the corresponding constraint state value network to obtain the next constraint state value output. The constraint state value supervision target is generated based on the constraint feedback data of the constraint dimension consistent with the corresponding constraint state value network, the preset discount coefficient, and the next constraint state value output. The constraint state value supervision target is the target value obtained by discounting and summing the constraint feedback data of the corresponding constraint dimension with the preset discount coefficient for the next constraint state value output. The constraint state value loss between the current constraint state value output and the constraint state value supervision target is calculated, and the network parameters of the corresponding constraint state value network are updated according to the constraint state value loss. If the preset stopping condition is not met, the forward computation based on training batch, the generation of the supervision target of the profit state value network, the generation of the supervision target of the supervision target of the constraint state value network, the loss calculation and the network parameter update operation are continuously performed on the profit state value network and at least one constraint state value network until the preset stopping condition is met, and the trained profit state value network and at least one trained constraint state value network are obtained. The preset stopping conditions are used to determine whether the training of the profit state value network and the constraint state value network has ended. The preset stopping conditions may include at least one or more of the following: the number of training rounds reaches a preset upper limit, the change in state value loss is less than a preset convergence threshold, the performance of the verification batch is stable, the update of network parameters is less than a preset parameter change threshold, and the constraint feedback statistics and the change of constraint multipliers tend to be stable. They can be combined according to preset logical relationships. Input the state data from the sample dataset into the trained payoff-state-value network to obtain the payoff-state-value output for the state data. Input the state data from the sample dataset into at least one trained constraint-state-value network to obtain the constraint-state-value output for the state data.

[0026] In this embodiment, the specific steps for obtaining the profit action value output and the constraint action value output based on the profit state value output and the constraint state value output include: Extract action value training sample groups from the sample data set for training action value. Each action value training sample group includes at least interrelated state data, action delivery data, benefit feedback data, constraint feedback data, and next state data. Organize the action value training sample groups into action value training batches. Read the output of the benefit state value and the output of the constraint state value, and align them with the state data in the action value training batch according to the one-to-one correspondence of samples to obtain the training alignment data set. The training alignment data set includes at least state data, action data, output of the benefit state value, output of the constraint state value, benefit feedback data and constraint feedback data. Initialize the network parameters of the reward action value network and at least one constraint action value network in the constraint-aware implicit Q-learning model, set the preset discount coefficient, preset learning rate, preset batch size and preset stopping condition for the action value network, and obtain the initialized reward action value network and at least one initialized constraint action value network. For each sample in the action value training batch, a revenue action value supervision target is generated based on the revenue feedback data, the preset discount coefficient, and the revenue state value output corresponding to the next state data. The revenue action value supervision target is the target value obtained by discounting and summing the revenue feedback data and the revenue state value output corresponding to the next state data with the preset discount coefficient. The revenue action value loss is calculated based on the revenue action value supervision target, and the network parameters of the revenue action value network are updated according to the revenue action value loss. For each constraint action value network in at least one constraint action value network, a constraint action value supervision target is generated based on constraint feedback data of the constraint dimension consistent with the corresponding constraint action value network, a preset discount coefficient, and the constraint state value output corresponding to the next state data. The constraint action value supervision target is the target value obtained by discounting and summing the constraint feedback data of the corresponding constraint dimension and the constraint state value output corresponding to the next state data with the preset discount coefficient. The constraint action value loss is calculated based on the constraint action value supervision target, and the network parameters of the corresponding constraint action value network are updated according to the constraint action value loss. If the preset termination condition for the action value network is not met, continue to generate the profit action value supervision target and the constraint action value supervision target based on the action value training batch and perform loss calculation and network parameter update until the preset termination condition for the action value network is met, and obtain the trained profit action value network and at least one trained constraint action value network. Preset termination conditions may include at least one or more of the following: the number of training rounds reaches a preset upper limit, the action value loss converges, the performance of the validation sample set is stable, the network parameter update magnitude is lower than a threshold, the difference between the action value output and the corresponding state value output is stable, and the constraint feedback statistical results are stable. These conditions can be combined according to preset logical relationships. The trained reward action value network is applied to the state data and delivery action data in the sample dataset to obtain the reward action value output for the state data and delivery action data. At least one trained constraint action value network is applied to the state data and delivery action data in the sample dataset to obtain the constraint action value output for the state data and delivery action data.

[0027] In this embodiment, benefit advantage data is generated based on the benefit action value output and benefit state value output, and constraint advantage data is generated based on the constraint action value output and constraint state value output. These are then fused to obtain the constraint-aware advantage fusion result, which specifically includes: Extract advantage calculation sample groups from the sample data set to generate advantages and weights. The advantage calculation sample groups include at least the state data, deployment action data, revenue state value output, revenue action value output, constraint state value output, and constraint action value output that are related to each other. The advantage calculation sample groups are then organized into advantage calculation batches. For each sample in the advantage calculation batch, the profit advantage data is calculated based on the profit action value output and the profit status value output. The profit advantage data is calculated by subtracting the profit status value output from the profit action value output. For each sample in the advantage calculation batch, for each constraint dimension corresponding to the constraint state value output and constraint action value output, the constraint advantage data of the constraint dimension is calculated separately. The constraint advantage data is calculated by subtracting the constraint state value output of the constraint dimension from the constraint action value output of the constraint dimension. The yield advantage data and constraint advantage data are scaled to be consistent. The scale consistency process includes: calculating the batch mean and batch standard deviation of the yield advantage data and the batch mean and batch standard deviation of the constraint advantage data for each constraint dimension within the advantage calculation batch, and subtracting the corresponding batch mean from the corresponding advantage data and then dividing by the corresponding batch standard deviation to obtain the scaled yield advantage data and the scaled constraint advantage data. The constraint-aware advantage fusion result is obtained by fusing the scale-homogenized gain advantage data and the scale-homogenized constraint advantage data based on the constraint multiplier set. The fusion process satisfies the following relationship: The constraint-aware advantage fusion result is equal to the sum of the product of the constraint multiplier of each constraint dimension and the constraint advantage data after scale uniformity minus the scale uniformity of the benefit advantage data. Among them, the constraint multipliers of each constraint dimension are the corresponding constraint multipliers in the constraint multiplier set; The set of weights for deployment actions is generated based on the constraint-aware advantage fusion result. The generation method of the set of weights for deployment actions includes: multiplying the constraint-aware advantage fusion result by a preset temperature coefficient and then performing exponential mapping processing to obtain the initial weight result; performing upper and lower bound constraints on the initial weight result to obtain the truncated weight result; and performing normalization processing on the truncated weight result so that the sum of each weight in the set of weights for deployment actions is a preset normalization constant.

[0028] Output the set of action weights for the delivery action, and associate the set of action weights with the corresponding state data and action data in the advantage calculation sample group.

[0029] In this embodiment, the content delivery optimization strategy model is obtained by weighted learning of the delivery action data based on the weight set of delivery actions, specifically including: Extract policy training sample groups for training the policy network from the sample data set. The policy training sample groups include at least the state data, the action data, and the action weights that correspond one-to-one with the action data. The policy training sample groups are then organized into policy training batches. Initialize the network parameters of the policy network in the constraint-aware implicit Q-learning model, set the preset learning rate, preset batch size and preset stopping condition for the policy network, and obtain the initialized policy network. For each sample in the policy training batch, the state data is input into the policy network to obtain the probability distribution of the delivery action or the delivery action parameters corresponding to the state data. A strategy learning objective is constructed based on the delivery action data and the probability distribution or parameters of the delivery actions. The strategy learning objective is then weighted based on the set of delivery action weights to obtain a weighted strategy learning objective, which satisfies the following relationship: The learning objective of the weighted strategy is equal to the result of multiplying the corresponding weight in the set of action weights by the log probability of the action data under the action probability distribution and taking the negative value, or equal to the result of multiplying the corresponding weight in the set of action weights by the negative log likelihood of the action data and the action parameter and taking the negative value. The policy learning loss is calculated based on the weighted policy learning objective, and the network parameters of the policy network are updated according to the policy learning loss. If the preset stopping condition for the policy network is not met, the system continues to perform weighted policy learning target generation, policy learning loss calculation and network parameter update based on the policy training batch until the preset stopping condition for the policy network is met, and the trained policy network is obtained. The trained policy network is output as the content delivery optimization policy model.

[0030] In this embodiment, decision-making is based on the content delivery optimization strategy model, a content delivery plan is generated and its execution is controlled, and the constraint-aware implicit Q-learning model is periodically trained to obtain the updated content delivery optimization strategy model, specifically including: During the execution of content marketing campaigns, decisions are made based on the content campaign optimization strategy model to determine the set of content to be campaigned and the set of available channel resources, and a content campaign plan is generated. The content campaign plan includes campaign actions for each content item in the set of content to be campaigned and each channel resource in the set of available channel resources. The set of content to be delivered refers to the set of content that meets the current delivery conditions and is selected from historical delivery log data or content material data during the content marketing delivery execution phase. Each content item in the set of content to be delivered has a corresponding content feature vector. The available channel resource set refers to the set of channel resources that can be used for advertising during the content marketing campaign execution phase, which is selected from all channel resources based on channel and placement resource data, campaign time period, and resource availability status. Each channel resource in the available channel resource set has a corresponding channel resource feature vector. Transmit the delivery actions in the content delivery plan to the content delivery execution system, control the delivery execution system to execute the content marketing delivery according to the delivery actions, obtain the delivery execution data of the delivery execution system, and generate delivery performance feedback data and constraint feedback data based on the delivery execution data; The campaign performance feedback data and constraint feedback data are merged and processed to obtain a set of feedback data after the campaign execution. The campaign performance feedback data includes at least the click-through rate, conversion rate, user retention or revenue data corresponding to each campaign action, and the constraint feedback data includes at least budget consumption, frequency control or compliance score information. The feedback data set after the campaign execution is merged and updated with the historical campaign log data set to obtain the updated historical campaign log data set, which includes all historical campaign records and related feedback data. Based on the updated historical deployment log data set, the constraint-aware implicit Q-learning model is periodically trained. The periodic training includes: constructing a sample data set based on the updated historical deployment log data set, training the revenue state value network and at least one constraint state value network to obtain the updated revenue state value output and the updated constraint state value output. Based on the updated profit state value output and the updated constraint state value output, train a profit action value network and at least one constraint action value network to obtain the updated profit action value output and the updated constraint action value output. Further, a set of delivery action weights is generated based on the updated value output of the benefit action, the updated value output of the constraint action, the updated value output of the benefit state, and the updated value output of the constraint state. The delivery action weights are then used to train the policy network by weighted learning of the delivery action data in the sample data set, so as to update the policy network. The updated policy network will be used as the updated content delivery optimization policy model for the next cycle of content delivery optimization decisions.

[0031] refer to Figure 3 A content marketing effectiveness evaluation and optimization system based on reinforcement learning includes the following modules: The data acquisition module is used to collect data from content marketing operations, perform preprocessing, and obtain a set of historical campaign log data. The sample construction module is used to construct a sample data set based on the historical deployment log data set; The model building module is used to construct a constraint-aware implicit Q-learning model based on a sample dataset, including a shared feature encoding network, a payoff state value network, a payoff action value network, a constraint state value network, a constraint action value network, and a policy network. The state value calculation module is used to input state data into the revenue state value network and the constraint state value network for training, and to obtain the revenue state value output and the constraint state value output. The action value calculation module is used to obtain the profit action value output and the constraint action value output based on the profit state value output and the constraint state value output. The advantage fusion module is used to generate benefit advantage data based on the benefit action value output and benefit state value output, and to generate constraint advantage data based on the constraint action value output and constraint state value output. The fusion process is then performed to obtain the constraint-aware advantage fusion result and the set of action weights for deployment. The strategy training module is used to perform weighted learning on the delivery action data based on the set of delivery action weights to obtain a content delivery optimization strategy model. The delivery execution module is used to make decisions based on the content delivery optimization strategy model, generate content delivery plans and control delivery execution, and periodically train the constraint-aware implicit Q-learning model to obtain the updated content delivery optimization strategy model.

[0032] Example 1: To verify the feasibility of this invention in practice, it was applied to a content delivery optimization scenario on an online content marketing platform. This platform simultaneously delivers text, images, short videos, and news feed content to multiple user groups, covering various online channel resources, and is subject to multiple constraints such as budget control, user reach frequency limits, and content compliance requirements. In this scenario, the platform needs to make daily decisions on combining a large number of candidate content and channel resources. Traditional methods relying on manual rules or static models are difficult to adapt to changes in user behavior in a timely manner, easily leading to problems such as large fluctuations in delivery performance, low resource utilization, and exceeding constraints.

[0033] In practical applications, the system first continuously collects data generated during the content marketing process. This data includes user browsing, click, and conversion behavior data; user profile-related feature data; attribute data for different content materials; resource status data for various channels and placements; and performance feedback data and constraint feedback data generated after each campaign execution. After cleaning, deduplicating, and time-aligning this data, the system constructs a historical campaign log dataset. Based on this historical campaign log dataset, the system further constructs a sample dataset, assigning corresponding status data, campaign action data, revenue feedback data, constraint feedback data, and next status data for each sample, which are then used for subsequent reinforcement learning modeling.

[0034] During the model training phase, the system constructs a constraint-aware implicit Q-learning model based on the sample dataset. A shared feature encoding network is used to uniformly encode user features, content features, and channel resource features, training both a revenue state value network and a constraint state value network to characterize the expected revenue and constraint consumption levels under different delivery states. Building upon this, the system further trains a revenue action value network and a constraint action value network to obtain the revenue action value output and constraint action value output under different delivery action conditions. Subsequently, the system calculates revenue advantage data based on the revenue action value output and revenue state value output, and calculates constraint advantage data based on the constraint action value output and constraint state value output. An advantage fusion module generates a constraint-aware advantage fusion result, which in turn generates a set of delivery action weights. During training, the strategy network performs weighted learning on historical delivery actions based on the delivery action weight set, thus forming a content delivery optimization strategy model.

[0035] During the campaign execution phase, the system makes decisions based on the content delivery optimization strategy model, considering the set of content to be delivered and the set of available channel resources. It automatically generates a content delivery plan and controls the delivery execution. After delivery, the system continuously collects new performance feedback data and constraint feedback data, and feeds them back to update the historical delivery log data set. This allows for periodic training of the constraint-aware implicit Q-learning model, enabling the strategy network to continuously optimize in response to changes in user behavior and resource status.

[0036] Statistical analysis of the operational data during the implementation process revealed that, compared to a control scheme that did not employ the method of this invention, the overall conversion performance of content delivery was more stable, budget consumption and frequency control were more balanced, and constraint exceedance events were significantly reduced. After multiple rounds of iterative training, the system can gradually increase the delivery weight of high-value content while reducing the probability of selecting inefficient or high-constraint-risk delivery actions, thereby improving overall marketing effectiveness without exceeding business constraints.

[0037] To further verify the feasibility and beneficial effects of the present invention in practice, key indicators before and after adopting the method of the present invention were compared and statistically analyzed, and the results are shown in the table.

[0038] Table 1 Comparison of Content Marketing Effectiveness Evaluation and Optimization Implementation Results

[0039] As can be seen from the table above, this invention demonstrates significant improvements over traditional delivery strategies in several key performance indicators for content marketing effectiveness evaluation and delivery optimization. Firstly, regarding user behavior response metrics, the average click-through rate increased from 2.34% to 2.98%, and the average conversion rate increased from 0.87% to 1.26% after adopting the method of this invention. This indicates that the delivery strategy generated based on the constraint-aware implicit Q-learning model can more accurately match user interests and content characteristics, improve the matching degree between content and users, and thus effectively promote user interaction.

[0040] In terms of cost and resource utilization efficiency, this invention also demonstrates relatively stable improvement effects. The cost per conversion decreased from 38.6 to 31.2, indicating that under the same budget constraints, the system can reduce the frequency of inefficient campaigns and increase the probability of selecting high-value campaigns by weighted learning of campaign actions. Simultaneously, the budget utilization rate increased from 91.5% to 97.8%, reflecting an enhanced coordination capability of the system at the budget constraint perception and execution levels, making budget consumption closer to the expected goals and avoiding resource idleness or ineffective consumption.

[0041] In terms of constraint control and strategy stability, this invention demonstrates more significant advantages. The frequency exceeding the limit ratio decreased from 6.4% to 1.9%, and the compliance trigger ratio decreased from 3.1% to 0.8%. This indicates that by jointly modeling the value of constraint states and constraint actions, and introducing a constraint advantage modulation mechanism during the strategy training phase, the system can proactively avoid high-constraint-risk delivery behaviors during strategy generation. Furthermore, the fluctuation range of strategy stability has decreased from large to small, indicating that this invention, through periodic training and feedback mechanisms, allows the strategy model to gradually converge during multiple iterations, reducing significant fluctuations in delivery effectiveness. In summary, these results demonstrate that this invention effectively balances cost control and business constraint enforcement while improving marketing effectiveness, validating its application value in real-world content marketing scenarios.

[0042] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for evaluating and optimizing the effectiveness of content marketing based on reinforcement learning, characterized in that, Includes the following steps: Collect data from content marketing operations, preprocess it, and obtain a set of historical campaign log data; A sample dataset was constructed based on historical deployment log data. Based on the sample data set, a constraint-aware implicit Q-learning model is constructed, which includes a shared feature encoding network, a payoff state value network, a payoff action value network, a constraint state value network, a constraint action value network, and a policy network. The state data is input into the profit state value network and the constraint state value network for training, and the profit state value output and the constraint state value output are obtained. Based on the output of the benefit state value and the output of the constraint state value, the output of the benefit action value and the output of the constraint action value are obtained. Based on the value output of the benefit action and the value output of the benefit state, benefit advantage data is generated. Based on the value output of the constraint action and the value output of the constraint state, constraint advantage data is generated. These are then fused together to obtain the constraint perception advantage fusion result and the set of action weights for deployment. A content delivery optimization strategy model is obtained by weighted learning of delivery action data based on a set of delivery action weights. The decision-making process is based on the content delivery optimization strategy model, which generates a content delivery plan and controls its execution. The constraint-aware implicit Q-learning model is periodically trained to obtain an updated content delivery optimization strategy model.

2. The method for evaluating and optimizing the effectiveness of content marketing based on reinforcement learning according to claim 1, characterized in that, The preprocessing includes cleaning, deduplication, time alignment, and identification.

3. The method for evaluating and optimizing the effectiveness of content marketing based on reinforcement learning according to claim 1, characterized in that, The construction of the sample data set based on the historical deployment log data set specifically includes: Each log entry is read from the historical deployment log data set, a sample key is determined based on the log entry, and the log entries are grouped and sorted by the sample key to obtain a candidate sample sequence set; Based on the candidate sample sequence set, a sample pair set is obtained. Based on the sample pair set, user feature vector, content feature vector, channel resource feature vector, constraint state feature vector, and context feature vector are generated to obtain state data. For each sample pair in the sample pair set, generate deployment action data, revenue feedback data, and constraint feedback data based on the current record; For each sample pair in the sample pair set, obtain the next state data based on the next record; The status data, deployment action data, revenue feedback data, constraint feedback data, and next status data are associated and encapsulated to form a single sample. The encapsulation process is then executed to obtain a sample data set.

4. The method for evaluating and optimizing the effectiveness of content marketing based on reinforcement learning according to claim 1, characterized in that, The constraint-aware implicit Q-learning model, constructed based on a sample data set and including a shared feature encoding network, a reward state value network, a reward action value network, a constraint state value network, a constraint action value network, and a policy network, specifically includes: Based on the sample data set, determine the model input of the constraint-aware implicit Q-learning model; Establish a shared feature coding network, configure the shared feature coding network to receive state data, perform feature coding processing on the state data, and output a shared feature representation vector; Establish a revenue state value network and a revenue action value network at the output of the shared feature encoding network; Establish a constrained state value network and a constrained action value network at the output of the shared feature encoding network; Establish a set of constraint multipliers and create a corresponding constraint multiplier update unit for each constraint multiplier; Establish an advantage calculation and fusion unit, a weight generation unit, and a policy network.

5. The method for evaluating and optimizing the effectiveness of content marketing based on reinforcement learning according to claim 1, characterized in that, The step of inputting state data into the profit state value network and the constraint state value network for training, and obtaining the profit state value output and the constraint state value output specifically includes: Extract training sample groups from the sample data set, organize the training sample groups into training batches, and initialize the network parameters of the profit state value network and the constraint state value network. The state data from the training batch is input into the profit state value network to obtain the current profit state value output, and the next profit state value output is obtained based on the next state data. Generate a revenue status value monitoring target based on revenue feedback data, preset discount coefficients, and the next revenue status value output. Based on the current revenue state value output and revenue state value monitoring target, calculate the revenue state value loss and update the network parameters of the revenue state value network; The state data from the training batch is input into the constrained state value network to obtain the current constrained state value output, and the next constrained state value output is obtained based on the next state data. Generate constraint state value supervision target based on constraint feedback data, preset discount coefficient and next constraint state value output; Based on the current constraint state value output and the constraint state value supervision objective, calculate the constraint state value loss and update the network parameters of the constraint state value network. The reward state value network and the constraint state value network are updated until the preset stopping condition is met, and the trained reward state value network and constraint state value network are obtained. Input the state data from the sample dataset into the trained profit state value network and constraint state value network to obtain the profit state value output and constraint state value output.

6. The method for evaluating and optimizing the effectiveness of content marketing based on reinforcement learning according to claim 1, characterized in that, The specific steps for obtaining the profit action value output and constraint action value output based on the profit state value output and constraint state value output include: Extract action value training sample groups from the sample dataset and organize them into action value training batches; Read the output of the benefit state value and the output of the constraint state value, and align them with the state data in the action value training batch to obtain the training aligned data group; Initialize the network parameters of the reward action value network and the constraint action value network; For each sample in the action value training batch, a revenue action value supervision target is generated based on the revenue feedback data, the preset discount coefficient, and the revenue state value output corresponding to the next state data. Calculate the loss of revenue action value based on the revenue action value monitoring target, and update the network parameters of the revenue action value network; For the constraint action value network, a constraint action value supervision target is generated based on constraint feedback data, preset discount coefficients, and the constraint state value output corresponding to the next state data. The constraint action value loss is calculated based on the constraint action value supervision target, and the network parameters of the constraint action value network are updated. The reward action value network and the constraint action value network are updated until the preset termination condition is met, resulting in the trained reward action value network and constraint action value network. Based on the trained reward action value network and constraint action value network, the reward action value output and constraint action value output are obtained.

7. The method for evaluating and optimizing the effectiveness of content marketing based on reinforcement learning according to claim 1, characterized in that, The process of generating benefit advantage data based on benefit action value output and benefit state value output, and generating constraint advantage data based on constraint action value output and constraint state value output, and then fusing them to obtain the constraint-aware advantage fusion result specifically includes: Extract advantage calculation sample groups from the sample dataset and organize them into advantage calculation batches; Calculate revenue advantage data based on revenue action value output and revenue status value output; Calculate constraint dominance data for constraint state value output and constraint action value output; Scale-homogenize the revenue advantage data and constraint advantage data to obtain scale-homogenized revenue advantage data and constraint advantage data. The constraint-aware advantage fusion result is obtained by fusing the scale-consistent benefit advantage data and constraint advantage data based on the constraint multiplier set, and a set of deployment action weights is generated.

8. The method for evaluating and optimizing the effectiveness of content marketing based on reinforcement learning according to claim 1, characterized in that, The weighted learning of the delivery action data based on the weight set of delivery action actions to obtain the content delivery optimization strategy model specifically includes: Extract policy training sample groups from the sample dataset and organize them into policy training batches to initialize the network parameters of the policy network. For each sample in the policy training batch, the state data is input into the policy network to obtain the probability distribution of the action or the parameters of the action. The strategy learning objective is constructed based on the data of the delivery action and the probability distribution of the delivery action or the parameters of the delivery action, and the strategy learning objective is weighted based on the set of delivery action weights to obtain the weighted strategy learning objective. The policy learning loss is calculated based on the weighted policy learning objective, and the network parameters of the policy network are updated until the preset stopping condition for the policy network is met, thus obtaining the trained policy network. The trained policy network is output as the content delivery optimization policy model.

9. The method for evaluating and optimizing the effectiveness of content marketing based on reinforcement learning according to claim 1, characterized in that, The process of making decisions based on the content delivery optimization strategy model, generating a content delivery plan and controlling its execution, and periodically training the constraint-aware implicit Q-learning model to obtain the updated content delivery optimization strategy model specifically includes: Based on the content delivery optimization strategy model, a decision is made on the set of content to be delivered and the set of available channel resources to generate a content delivery plan; Based on the content delivery plan, execute content marketing campaigns, obtain campaign execution data, and generate campaign performance feedback data and constraint feedback data; The campaign performance feedback data and constraint feedback data are merged and processed to obtain a feedback data set; The feedback data set after the deployment is executed is merged and updated with the historical deployment log data set to obtain the updated historical deployment log data set; Based on the updated historical delivery log data set, the constraint-aware implicit Q-learning model is periodically trained to obtain the updated policy network, which is then used as the updated content delivery optimization policy model.

10. The content marketing effectiveness evaluation and optimization system based on reinforcement learning according to claim 1, wherein the content marketing effectiveness evaluation and optimization method based on reinforcement learning according to any one of claims 1 to 9 is characterized in that, Includes the following modules: The data acquisition module is used to collect data from content marketing operations, perform preprocessing, and obtain a set of historical campaign log data. The sample construction module is used to construct a sample data set based on the historical deployment log data set; The model building module is used to construct a constraint-aware implicit Q-learning model based on a sample dataset, including a shared feature encoding network, a payoff state value network, a payoff action value network, a constraint state value network, a constraint action value network, and a policy network. The state value calculation module is used to input state data into the revenue state value network and the constraint state value network for training, and to obtain the revenue state value output and the constraint state value output. The action value calculation module is used to obtain the profit action value output and the constraint action value output based on the profit state value output and the constraint state value output. The advantage fusion module is used to generate benefit advantage data based on the benefit action value output and benefit state value output, and to generate constraint advantage data based on the constraint action value output and constraint state value output. The fusion process is then performed to obtain the constraint-aware advantage fusion result and the set of action weights for deployment. The strategy training module is used to perform weighted learning on the delivery action data based on the set of delivery action weights to obtain a content delivery optimization strategy model. The delivery execution module is used to make decisions based on the content delivery optimization strategy model, generate content delivery plans and control delivery execution, and periodically train the constraint-aware implicit Q-learning model to obtain the updated content delivery optimization strategy model.

Images