Advertisement bidding method, system and device based on diffusion model and medium

Abstract

The invention discloses an advertisement bidding method, system and device based on a diffusion model and a medium, and the method comprises the steps: obtaining historical advertisement bidding data, and generating first track data according to the historical advertisement bidding data; inputting the first trajectory data into a training and evaluation model based on reinforcement learning to generate second trajectory data; inputting the second track data into an advertisement bidding model based on a diffusion model and an inverse dynamic learning model for training to obtain a trained advertisement bidding model; and according to the trained advertisement bidding model, carrying out real-time bidding on the advertisement. According to the method, a set of complete automatic bidding framework is constructed by combining the diffusion model and the inverse dynamics model, and a special track data processing and enhancing method is matched. The method enables the model to effectively learn the mapping from the historical track to the optimal bidding strategy, achieves the precise conversion from the state to the action, and can remarkably improve the click rate of advertisement putting compared with the related technology.

Description

Technical Field

[0001] The present application belongs to the field of advertising delivery technology, and in particular relates to an advertising bidding method, system, device and medium based on a diffusion model. Background Art

[0002] In the programmatic advertising industry, demand-side platforms (DSPs) serve as a core tool for advertisers to deliver ads. They use real-time bidding (RTB) technology to precisely display ads to target users and collect commissions from advertisers based on ad performance (such as click-through rate or conversion rate). As competition in the advertising market intensifies, advertisers are increasingly demanding more refined budget allocation. DSPs, on the other hand, need to optimize bidding algorithms to maximize ad effectiveness within budget constraints, thereby improving revenue structure and resource allocation efficiency.

[0003] In programmatic advertising transactions, real-time bidding (RTB) systems enable dynamic allocation of advertising resources through millisecond-level bidding decisions. However, the core technologies of existing RTB ad bidding algorithms still have significant flaws, resulting in low budget utilization for advertisers. Traditional algorithms primarily rely on static prediction models based on historical click-through rates (CTR) or conversion rates (CVR) during the bidding process, failing to fully integrate real-time user behavior trajectories (such as page dwell time, cross-device interactions, and immediate search intent). This results in insufficiently accurate matching of bidding decisions with users' immediate needs.

[0004] Due to the poor effectiveness of existing advertising bidding algorithms, a large number of advertisements cannot accurately hit the target user groups. The budgets invested by advertisers are used to reach non-potential customers, resulting in ineffective waste of funds, significantly reducing the conversion rate and return on investment of advertising, and hindering the healthy and sustainable development of the advertising industry.

[0005] Therefore, there is an urgent need to develop a more efficient and accurate RTB advertising bidding algorithm to improve the effectiveness of advertising delivery, reduce advertiser costs, and achieve optimal allocation of advertising resources. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide an advertising bidding method, system, device and medium based on a diffusion model to solve the problem of inaccurate bidding in the advertising bidding algorithm in the related art.

[0007] In order to solve the above technical problems, this application provides the following technical solutions:

[0008] In a first aspect, the present application provides an advertising bidding method based on a diffusion model, comprising:

[0009] Acquire historical advertising bidding data, and generate first trajectory data based on the historical advertising bidding data;

[0010] Inputting the first trajectory data into a preset reinforcement learning-based training and evaluation model to generate second trajectory data;

[0011] Inputting the second trajectory data into a preset advertising bidding model based on a diffusion model and an inverse dynamics learning model for training to obtain a trained advertising bidding model;

[0012] The advertisements are bid in real time according to the trained advertisement bidding model.

[0013] Furthermore, the first trajectory data includes: state data, action data, reward data, termination mark data and next state data.

[0014] Furthermore, the reinforcement learning-based training and evaluation model includes: a data loading and preprocessing submodule, a parameter initialization submodule, a model training submodule, and a model evaluation submodule.

[0015] Furthermore, the training and evaluation model based on reinforcement learning adopts the following reward function:

[0016]

[0017] Among them, Return t is the sum of the rewards of the current trajectory, T is the number of the last time step of the current trajectory; r i is the reward of the i-th time step; is the result after normalization.

[0018] Furthermore, the advertising bidding model based on the diffusion model and the inverse dynamics learning model includes: a diffusion model and an inverse dynamics model;

[0019] The diffusion model is used to predict the future state using the second trajectory and the expected return;

[0020] The inverse dynamics model is used to predict actions based on the states of the previous two time steps, the state of the current time step and the predicted state of the next time step.

[0021] Furthermore, the second trajectory data is input into a preset advertising bidding model based on a diffusion model and an inverse dynamics learning model for training to obtain a trained advertising bidding model. The training process uses the following loss function:

[0022]

[0023] in, is the loss function of the diffusion model, is the loss function of the inverse dynamics learning model, x0 is the original unnoised input data; ∈ is the real noise sampled from the standard normal distribution; t is the diffusion time step; s t is the state at time step t; s t+1 is the state at time step t+1; For predicted actions.

[0024] Furthermore, the method further includes: evaluating the advertising bidding model and optimizing the advertising bidding model according to the evaluation result.

[0025] In a second aspect, the present application further provides an advertising bidding system based on a diffusion model, comprising:

[0026] an acquisition module, configured to acquire historical advertisement bidding data and generate first trajectory data based on the historical advertisement bidding data;

[0027] a data processing module, configured to input the first trajectory data into a preset reinforcement learning-based training and evaluation model to generate second trajectory data;

[0028] a training module, configured to input the second trajectory data into a preset advertising bidding model based on a diffusion model and an inverse dynamics learning model for training, thereby obtaining a trained advertising bidding model;

[0029] The advertisement bidding module is used to conduct real-time bidding on advertisements based on the trained advertisement bidding model.

[0030] In a third aspect, the present application also provides a computer electronic device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the above-mentioned advertising bidding method based on the diffusion model are implemented.

[0031] In a fourth aspect, the present application further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any one of the above-mentioned diffusion model-based advertising bidding methods.

[0032] The present application provides an advertising bidding method, system, device, and medium based on a diffusion model, which have the following beneficial effects:

[0033] This application combines the diffusion model with the inverse dynamics model to build a complete automated bidding framework. The automated bidding framework of this application deeply integrates the probabilistic modeling capability of the diffusion model with the state-action mapping characteristics of the inverse dynamics model to build an end-to-end bidding strategy generation system. Compared with the traditional method that relies on a single model, this solution uses the diffusion model to accurately model the implicit distribution of historical trajectory data, combined with the inverse dynamics model to reversely deduce the state-action conversion relationship, effectively solving the problem of weak policy correlation in high-dimensional and sparse trajectory data. Through the supporting trajectory data processing and enhancement module, the model can fully explore the potential patterns in the trajectory data during the training phase, significantly improving the mapping accuracy from complex state space to optimal bidding actions. In actual tests, this technical solution has increased the click-through rate (CTR) of ads by 20%-40%, and through the causal constraint mechanism of the bidding strategy, it has simultaneously optimized the ad conversion rate (CVR) by 15%-30%, thereby creating a higher return on investment (ROI) for advertising platforms and advertisers. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0035] Figure 1 This is a flow chart of an advertising bidding method based on a diffusion model in an embodiment of the present application;

[0036] Figure 2 Schematic diagram of the training process of the reinforcement learning-based training and evaluation model in the embodiment of the present application;

[0037] Figure 3 Schematic diagram of the format of the first trajectory data in an embodiment of the present application;

[0038] Figure 4 Schematic diagram of the structure of the advertising bidding model based on the diffusion model and the inverse dynamics learning model in the embodiment of the present application;

[0039] Figure 5 This is a structural diagram of an advertising bidding system based on a diffusion model in an embodiment of the present application;

[0040] Figure 6 It is a structural diagram of a computer electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0042] It should be noted that when an element is referred to as being "fixed to" another element, it may be directly on the other element or there may be an intermediate element. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be an intermediate element. Conversely, when an element is referred to as being "directly on" another element, there is no intermediate element. The terms "vertical," "horizontal," "left," "right," and similar expressions used herein are for illustrative purposes only.

[0043] In this application, unless otherwise expressly specified or limited, terms such as "mounted," "connected," "connect," and "fixed" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integration; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; and internal connections between two components or interactions between two components. Those skilled in the art will understand the specific meanings of these terms in this application based on specific circumstances.

[0044] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. Throughout the description of this application, "plurality" means two or more, unless otherwise specifically defined.

[0045] The terms used in one or more embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present application. The singular forms "a", "the", and "the" used in one or more embodiments of the present application are also intended to include plural forms unless the context clearly indicates otherwise.

[0046] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. The terms used in the template description herein are for the purpose of describing specific embodiments only and are not intended to limit this application. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0047] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of the present application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of one or more embodiments of the present application, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "when..." or "when...".

[0048] Currently, existing advertising bidding methods have the following problems:

[0049] Traditional algorithm based on linear bidding:

[0050] Because linear bidding relies on a fixed scaling factor, α, it lacks flexibility when market prices fluctuate significantly. For example, when increased competition drives up costs, budgets may be exhausted prematurely. Conversely, when competition weakens, insufficient bidding may lead to missed opportunities for high-value traffic and low click-through rates.

[0051] Bidding method based on logistic regression:

[0052] Logistic regression relies heavily on feature engineering, requiring manual design of high-quality features. Furthermore, model parameters are trained based on historical data, making it difficult to adapt to changing market conditions. If market distribution shifts (e.g., due to changes in user behavior), the accuracy of the predicted pCTR decreases, and bidding performance deteriorates.

[0053] Bidding strategy based on greedy algorithm:

[0054] Greedy algorithms focus on short-term gains and ignore long-term budget allocation. For example, overspending on budget in the early stages of a bidding process can lead to missing out on high-quality traffic due to insufficient resources later on, resulting in lower-than-expected overall returns.

[0055] Reinforcement Learning Methods:

[0056] Traditional reinforcement learning relies on extensive trial-and-error to estimate state-action values ​​(such as Q-values), resulting in high computational costs and slow convergence. In the high-frequency, low-latency (millisecond-level) RTB scenario, frequent environmental interactions are difficult to implement, making the algorithm difficult to implement practically. Furthermore, traditional reinforcement learning models incur high costs when interacting with real-world bidding environments.

[0057] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes in certain embodiments will not be repeated. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0058] Please refer to Figure 1 The embodiment of the present application provides an advertisement bidding method based on a diffusion model, which includes at least the following steps:

[0059] S10: Acquire historical advertisement bidding data, and generate first trajectory data based on the historical advertisement bidding data.

[0060] Specifically, in this application, in this example, first, the bidding data of historical advertisements can be obtained through the advertisement bidding platform.

[0061] In one embodiment, after obtaining the historical advertising bidding data, the historical advertising bidding data may be preprocessed, wherein the preprocessing includes:

[0062] Noise removal: Various noises may exist in the data, such as sensor measurement errors, environmental interference, etc. These noises will affect the learning effect of the algorithm and need to be removed through filtering, smoothing and other methods.

[0063] Handling missing values: Missing values ​​in the data need to be handled according to the specific situation. Common methods include deleting samples with missing values, filling missing values ​​with statistics such as mean, median, or mode, or using more complex interpolation methods.

[0064] Remove outliers: Outliers may be caused by data collection errors or special circumstances, and they may have a significant impact on model training. You can identify and remove outliers by setting thresholds, using boxplots, and other methods.

[0065] Secondly, after pre-processing the historical advertising bidding data, it is necessary to convert the historical advertising bidding data into first track data.

[0066] In one embodiment, the first trajectory data includes: state data, action data, reward data, termination mark data, and next state data.

[0067] Specifically, the format of the generated first trajectory data is as follows:

[0068] Taking daily historical advertising data as input, a set of (S t ,A t ,R t ,S t+1 ,done) format for reinforcement learning trajectory, which is used for subsequent strategy training and evaluation. The following is the definition of trajectory data:

[0069] 1. State data definition:

[0070] Each state S tIt is a fixed-length feature vector that fully describes the environment, resource constraints, and historical behavior trends at the current time step. The state vector consists of the following 16 dimensions:

[0071]

[0072]

[0073] 2. The action data action is defined as follows:

[0074] Action A t Represents the overall bidding tendency of the bidding agent at the current time step, which is calculated based on the bids and estimated values ​​of all impression opportunities at the current time step:

[0075]

[0076] Among them, bidd i A unique identifier representing a single bid, pctr i Indicates the estimated ctr for this impression opportunity.

[0077] 3. The reward data Reward is defined as follows:

[0078] This study defines rewards as the total number of clicks on impressions won by the bidding agent:

[0079]

[0080] Click indicates whether a click occurred, 1 for click occurred, and 0 for click did not occur.

[0081] 4. Termination mark data done

[0082] At each time step, it is calculated whether the current trajectory has reached the terminal state. The calculation rules are as follows:

[0083]

[0084] Among them, step represents the time step of the bidding, and remainingBudget represents the remaining budget after the end of this step.

[0085] It should be noted that once the mark is done = 1, it indicates that the current trajectory is the end state and no S will be generated later. t+1 (next state data).

[0086] 5. Next state data S t+1 :

[0087] In the non-terminal state, the system will set the state S of the next time step to t+1is constructed as the successor state of the current trajectory; and for the terminal time step (done=1), its S t+1 Set to None to mark the end of the track.

[0088] S20: Input the first trajectory data into a preset training and evaluation model based on reinforcement learning to generate second trajectory data.

[0089] In one embodiment of the application, the preset reinforcement learning-based training and evaluation model includes: a data loading and preprocessing submodule, a parameter initialization submodule, a model training submodule, and a model evaluation submodule.

[0090] For the sake of subsequent description, the preset reinforcement learning-based training and evaluation model will be referred to as the DDTrainer model.

[0091] See also Figure 2 In this embodiment, first, it is necessary to initialize the parameters of the DDTrainer model using the parameter initialization submodule. During the DDTrainer initialization process, the main tasks are to set the path configuration (such as data directory, model storage directory), experimental parameters (advertiser ID, budget ratio), model computing device (CPU / GPU), etc.

[0092] Then, the first trajectory data is loaded and preprocessed through the data loading and preprocessing submodule, as follows:

[0093] This module is primarily used to read click log data and generate trajectory sequences required for reinforcement learning. Preprocessing includes: arranging each action sequence into a standard reinforcement learning triplet (state, action, return). Each sequence is padded with zeros according to the specified step_len to align the lengths, and a binary mask is generated to mark the valid bits.

[0094] For example, please refer to Figure 3 ,exist Figure 3 In , Epoch is the trajectory number, indicating that this is the 205th trajectory generated, and step represents the time step of the 205th trajectory.

[0095] The triples are filled with the corresponding target attributes extracted from these attributes; if the advertising budget is used up, the subsequent empty time steps will be filled with 0-tuples, that is, (0, 0, 0).

[0096] In one embodiment of the present application, the preset reinforcement learning-based training and evaluation model uses the following reward function:

[0097] Return is calculated by summing the rewards within a sequence and then applying a sigmoid function to it, limiting its range to (0, 1) to ensure stable returns during model training. All input data is Z-score normalized before being passed to the model to improve numerical stability during training.

[0098] Specifically, the return of each trajectory is first simply summed and then normalized using the sigmoid function:

[0099]

[0100] Among them, Return t is the sum of the rewards of the trajectory; T is the number r of the last time step of the trajectory i is the reward at the i-th time step.

[0101] After obtaining the summation result, normalize the result

[0102]

[0103] is the result after normalization, and its value is (0, 1).

[0104] S30: Input the second trajectory data into a preset advertising bidding model based on a diffusion model and an inverse dynamics learning model for training to obtain a trained advertising bidding model.

[0105] Specifically, in this embodiment, the second trajectory data is input into the advertising bidding model based on the diffusion model and the inverse dynamics learning model through the DDTrainer model for training.

[0106] It should be noted that, for the sake of subsequent description, in the subsequent description, the preset advertising bidding model based on the diffusion model and the inverse dynamics learning model will be referred to as the DFUSER model.

[0107] The specific training process is as follows: The model training phase is handled by the run() function in the DDTrainer model, which uses batch data training for parameter optimization. Training data is fed into the DFUSER model in batches through the DDTrainer model, where forward propagation, loss calculation, and backpropagation are performed sequentially. The overall loss value is monitored during training, and the model parameters are saved when the loss reaches the current optimal value to ensure optimal training results.

[0108] It should be noted that the run() function is a complete training process controller used to train the DFUSER model. Its core responsibilities include data loading, preprocessing, model training, model saving, and evaluation. Its general process flow is as follows:

[0109] 1. Environment and parameter configuration

[0110] Automatically use the GPU. If the GPU is not available, use the CPU.

[0111] batch_size = 256 (one batch trains 256 data)

[0112] 2. Model Initialization

[0113] Initialize a DFUSER instance and transfer it to the CPU / GPU

[0114] 3. Data preparation and normalization

[0115] Read data from the specified path and normalize the state vector:

[0116]

[0117] Among them, s is the original state vector,

[0118] μ is the mean of all state vectors, and the calculation formula is:

[0119]

[0120] N is the total sample size

[0121] σ is the standard deviation, defined as:

[0122]

[0123] s norm is the normalized state vector

[0124] 4. Training

[0125] Traverse all training data and train the model. If the total loss of the current epoch is better than the historical best, save the model parameters. If the total number of training steps exceeds step_num = 20000, terminate the training.

[0126] It should be noted that the DFUSER model (DFUSER model) is a reinforcement learning model based on the diffusion model and inverse dynamics learning model of temporal convolution, used to learn the mapping from state to action. During model training and inference, to improve numerical stability and enhance the expressiveness of conditional control, the original reward must be normalized: the reward value is mapped to the interval (0, 1) to construct a continuous scalar signal that can be used for conditional generation, denoted as returns. During inference, the model receives input as a trajectory x consisting of a historical state-action-time step combination, and the expected return of the future trajectory.

[0127] In a certain embodiment of the present application, the advertising bidding model based on the diffusion model and the inverse dynamics learning model includes: a diffusion model and an inverse dynamics model;

[0128] The diffusion model is used to predict the future state using the second trajectory and the expected return;

[0129] The inverse dynamics model is used to predict actions based on the states of the previous two time steps, the state of the current time step and the predicted state of the next time step.

[0130] Please note that Figure 4 The diffusion model in this application is a diffusion model based on time convolution, and its specific structure is as follows:

[0131] The backbone network for diffusion modeling uses an improved Temporal U-Net structure to enhance the model's expressiveness in time series tasks. Its overall architecture is designed as follows:

[0132] Input layer: The input of the diffusion model is the state sequence x perturbed by Gaussian noise t , with dimensions (B,H,D), where: B is the batch size; H is the time step range (for example, set one day to 96); D is the dimension of the state; and receives the time step t and the return returns.

[0133] Embedding layer: Through Sinusoidal Positional Encoding (Sinusoidal Positional Encoding), each scalar time step tt is mapped into a high-dimensional vector of dimension d (d = 128) to capture the periodic and continuous characteristics of time. Similarly, the returns are nonlinearly mapped through a fully connected network (MLP) and projected to the same dimension d as the time embedding. Subsequently, the time embedding and the return embedding are concatenated on the feature dimension to form a fused vector with a final dimension of [B, 2d]. This fused embedding serves as a global control signal to guide the model to produce target-consistent outputs at each diffusion step.

[0134] Downsampling stage (Encoder): The encoder adopts a multi-layered structure consisting of multiple ResidualTemporalBlocks, which gradually extract temporal features from the input. Each block consists of two convolutional blocks (Conv1dBlock), with the temporal return fusion information (temb) obtained by the embedding layer embedded in the middle. Residual connections are used to enhance feature preservation and gradient flow. The output of each layer is cached before downsampling, forming skip connections, providing the decoder with the information support needed to recover details.

[0135] Upsampling stage (Decoder): In the decoding stage, the decoder gradually reconstructs the multi-scale temporal features extracted by the encoder. Its architecture is symmetrical with the encoder and consists of several ResidualTemporalBlocks. In each layer, the corresponding encoder output is first extracted from the skip connection and concatenated with the current feature to enhance the fusion effect of local details and high-level semantics. Each block consists of two Conv1dBlocks, and time embedding modulation is introduced. In specific layers, the Upsample1d operation can be selectively applied to expand the time dimension and achieve gradual recovery of the temporal granularity. The stability of the model is maintained by residual connections, and the differences between high- and low-level features are bridged by skip connections to ensure that the final output matches the input distribution as much as possible in structure and content.

[0136] Output layer: The output layer consists of a series of sequentially arranged convolutional modules to ensure that the final output has a dimension of (B, H, D).

[0137] Please continue reading Figure 4 , the inverse dynamics model in this application is constructed as follows:

[0138] The inverse dynamics model inv_model in this application is a feedforward neural network used to calculate the inverse dynamics model based on the state S of the previous two time steps. t-2 、S t-1 , the state S of the current time step t , and the predicted next time step state St+1 The model input is the concatenated vector of these four states, which passes through a two-layer fully connected structure: the input is first mapped to 256 dimensions, then activated by ReLU and mapped to the action dimension action_dim, and the output is the prediction of action.

[0139] During the model training phase, the DFUSER model adopts a joint training strategy to optimize the diffusion model and the inverse dynamics model at the same time. The loss is defined as follows:

[0140] Diffusion loss: Use weighted mean square error (L2) loss to measure the difference between the noise predicted by the model from the noisy state and the actual noise, as follows:

[0141] The diffusion model is trained by gradually adding noise to the data and learning the denoising process. Its loss function is usually the weighted mean square error (L2 loss) between the predicted noise and the true noise:

[0142]

[0143] Where: x0 is the original unnoised input data (such as image, state, etc.); ∈ is the real noise sampled from the standard normal distribution; t is the diffusion time step, which is usually randomly sampled during training;

[0144] x t is the noise sample at time step t, calculated as:

[0145] α t It is the noise retention coefficient at each time tt in the diffusion process, which controls how much of the "original signal" is retained in the t-th step.

[0146] is the cumulative product of all α from time step 1 to t:

[0147] ∈ θ (x t ,t) is the model input x t and the noise predicted at time t

[0148] w(t) is a weighting coefficient for different time steps (for example, it can be set to w(t) = 1 or decayed as a function of αt), which is used for balanced training.

[0149] Inverse dynamics model loss: Using standard mean squared error loss, the supervised model predicts the correct action from the state sequence, as follows:

[0150] The goal of the inverse dynamics model is to predict the action from a sequence of adjacent states. It is trained using the standard mean squared error loss:

[0151]

[0152] Among them, s t is the state at time step t; s t+1 is the state at time step t+1; a t For real actions (data collected from the environment); The inverse dynamics model (parameter is φ) is used to calculate the state pair (s t ,s t+1 ) predicted action.

[0153] Finally, through the above training method, the trained advertising bidding model (DFUSER model) based on the diffusion model and the inverse dynamics learning model is obtained.

[0154] S40: Conduct real-time bidding on advertisements based on the trained advertisement bidding model.

[0155] Specifically, in this embodiment, real-time bidding is performed on advertisements using the trained advertisement bidding model.

[0156] Specifically, after obtaining the trained advertising bidding model, you can bid for ads in real time based on real-time data. The bidding formula is as follows:

[0157] Bid=pctr*action

[0158] Among them, bid is the bid, pctr is the estimated click-through rate (obtained by the upstream model, here we use it as the raw data of the display opportunity); action is the action (value) inferred by the DFUSER model, that is, the value output by the advertising bidding model.

[0159] In one embodiment of the present application, the method further includes:

[0160] S50: Evaluate the advertisement bidding model, and optimize the advertisement bidding model according to the evaluation result.

[0161] It is understandable that in order to make the bidding strategy of the advertising bidding model more accurate, this application also evaluates the trained advertising bidding model, and then optimizes the bidding strategy of the advertising bidding model based on the evaluation results. The evaluation process is as follows:

[0162] In this embodiment, the model evaluation mechanism is implemented by the DDTrainer model evaluate() function, and its purpose is to simulate the actual performance of the current model in a real advertising bidding scenario. The core steps of the evaluation process include: for daily advertising traffic data, the bidding agent DDAgent (whose function is only to process the output action obtained by DFUSER through input into a bid) loads the DFUSER model and executes a simulated bidding process. The simulated bidding process will be described in detail later. When the model budget is exhausted or the daily time limit is reached, the evaluation process will terminate, and all clicks obtained by the model will be counted as a benchmark for evaluating model performance.

[0163] It should be noted that the basis for simulated bidding is the display opportunities that have been won in the past. The winning prices of these display opportunities are used as their market prices and provided to the intelligent agent (DDAgent) for bidding. If the intelligent agent's bid is greater than the market price of the display opportunity, it is deemed that the intelligent agent has won the display opportunity.

[0164] DDAgent's bidding follows the following formula:

[0165] Bid=pctr*action

[0166] Where bid is the bid price, pctr is the estimated click-through rate (obtained from the upstream model, and here we use it as the raw data of the display opportunity); action is the action (value) inferred by the DFUSER model.

[0167] Ultimately, the bidding results for each time step will be saved in a data frame, and then the bidding strategy of the advertising bidding model will be optimized based on these data frames.

[0168] The present application provides an advertising bidding method based on a diffusion model, which has the following beneficial effects:

[0169] This application combines the diffusion model with the inverse dynamics model to build a complete automated bidding framework. The automated bidding framework of this application deeply integrates the probabilistic modeling capability of the diffusion model with the state-action mapping characteristics of the inverse dynamics model to build an end-to-end bidding strategy generation system. Compared with the traditional method that relies on a single model, this solution uses the diffusion model to accurately model the implicit distribution of historical trajectory data, combined with the inverse dynamics model to reversely deduce the state-action conversion relationship, effectively solving the problem of weak policy correlation in high-dimensional and sparse trajectory data. Through the supporting trajectory data processing and enhancement module, the model can fully explore the potential patterns in the trajectory data during the training phase, significantly improving the mapping accuracy from complex state space to optimal bidding actions. In actual tests, this technical solution has increased the click-through rate (CTR) of ads by 20%-40%, and through the causal constraint mechanism of the bidding strategy, it has simultaneously optimized the ad conversion rate (CVR) by 15%-30%, thereby creating a higher return on investment (ROI) for advertising platforms and advertisers.

[0170] See also Figure 5 The embodiment of the present application further provides an advertisement bidding system 200 based on a diffusion model, including:

[0171] An acquisition module 201 is configured to acquire historical advertisement bidding data and generate first trajectory data based on the historical advertisement bidding data;

[0172] A data processing module 202 is configured to input the first trajectory data into a preset reinforcement learning-based training and evaluation model to generate second trajectory data;

[0173] A training module 203 is configured to input the second trajectory data into a preset advertising bidding model based on a diffusion model and an inverse dynamics learning model for training, thereby obtaining a trained advertising bidding model;

[0174] The advertisement bidding module 204 is configured to conduct real-time bidding on advertisements based on the trained advertisement bidding model.

[0175] See also Figure 6 An embodiment of the present application also provides a computer electronic device 300, including a memory 303 and a processor 302, wherein the memory 303 stores a computer program, and when the processor executes the computer program, it implements the steps of any of the above-mentioned diffusion model-based advertising bidding methods.

[0176] Specifically, the electronic device 300 includes: a transceiver 301, a bus interface and a processor 302. The processor 302 is used to obtain historical advertising bidding data and generate first trajectory data based on the historical advertising bidding data; input the first trajectory data into a preset training and evaluation model based on reinforcement learning to generate second trajectory data; input the second trajectory data into a preset advertising bidding model based on a diffusion model and an inverse dynamics learning model for training to obtain a trained advertising bidding model; and perform real-time bidding on advertisements based on the trained advertising bidding model.

[0177] In the embodiment of the present application, the electronic device 300 further includes: a memory 303. Figure 6 In the embodiment, the bus architecture may include any number of interconnected buses and bridges, specifically linking various circuits such as one or more processors represented by processor 302 and memory represented by memory 303. The bus architecture may also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 301 may be a plurality of components, i.e., a transmitter and a receiver, providing a unit for communicating with various other devices over a transmission medium. The processor 302 is responsible for managing the bus architecture and general processing, and the memory 303 may store data used by the processor 302 when performing operations.

[0178] An embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any one of the above-mentioned diffusion model-based advertising bidding methods.

[0179] In this embodiment, the computer-readable storage medium may be a non-volatile storage medium or a volatile storage medium. For example, the computer storage medium may include, but is not limited to, a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, among other media capable of storing program code.

[0180] In all examples shown and described herein, any specific values ​​should be interpreted as merely exemplary and not limiting, and thus other examples of the exemplary embodiments may have different values.

[0181] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0182] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and structure diagrams in the accompanying drawings show the possible architectures, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in an alternative implementation, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the structure diagram and / or flowchart, and the combination of boxes in the structure diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.

[0183] In addition, the functional modules or units in the various embodiments of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0184] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a terminal device (which can be a smart phone, personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application.

[0185] The above is only a specific implementation method of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the scope of protection of the present application.

Claims

1. An advertising bidding method based on a diffusion model, characterized in that: include: Acquire historical advertising bidding data, and generate first trajectory data based on the historical advertising bidding data; Inputting the first trajectory data into a preset reinforcement learning-based training and evaluation model to generate second trajectory data; Inputting the second trajectory data into a preset advertising bidding model based on a diffusion model and an inverse dynamics learning model for training to obtain a trained advertising bidding model; The advertisements are bid in real time according to the trained advertisement bidding model.

2. The advertising bidding method according to claim 1, characterized in that: The first trajectory data includes: state data, action data, reward data, termination mark data and next state data.

3. The advertising bidding method according to claim 1, characterized in that: The training and evaluation model based on reinforcement learning includes: a data loading and preprocessing submodule, a parameter initialization submodule, a model training submodule, and a model evaluation submodule.

4. The advertising bidding method according to claim 1 or 2, characterized in that: The training and evaluation model based on reinforcement learning adopts the following reward function: Among them, Return t is the sum of the rewards of the current trajectory, T is the number of the last time step of the current trajectory; r i is the reward of the i-th time step; Return norm,t is the result after normalization.

5. The advertising bidding method according to claim 1, characterized in that: The advertising bidding model based on the diffusion model and the inverse dynamics learning model includes: a diffusion model and an inverse dynamics model; The diffusion model is used to predict the future state using the second trajectory and the expected return; The inverse dynamics model is used to predict actions based on the states of the previous two time steps, the state of the current time step and the predicted state of the next time step.

6. The advertising bidding method according to claim 1, characterized in that: The second trajectory data is input into a preset advertising bidding model based on a diffusion model and an inverse dynamics learning model for training to obtain a trained advertising bidding model. The training process uses the following loss function: in, is the loss function of the diffusion model, is the loss function of the inverse dynamics learning model, x0 is the original unnoised input data; ∈ is the real noise sampled from the standard normal distribution; t is the diffusion time step; s t is the state at time step t; s t+1 is the state at time step t+1; For predicted actions.

7. The advertising bidding method according to claim 1, characterized in that: The method further comprises: The advertising bidding model is evaluated, and the advertising bidding model is optimized according to the evaluation result.

8. An advertising bidding system based on a diffusion model, characterized in that: include: an acquisition module, configured to acquire historical advertisement bidding data and generate first trajectory data based on the historical advertisement bidding data; a data processing module, configured to input the first trajectory data into a preset reinforcement learning-based training and evaluation model to generate second trajectory data; a training module, configured to input the second trajectory data into a preset advertising bidding model based on a diffusion model and an inverse dynamics learning model for training, thereby obtaining a trained advertising bidding model; The advertisement bidding module is used to conduct real-time bidding on advertisements based on the trained advertisement bidding model.

9. A computer electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the diffusion model-based advertising bidding method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the advertising bidding method based on the diffusion model according to any one of claims 1 to 7 are implemented.

Images