Advertisement click rate dynamic prediction method and system based on deep reinforcement learning

Abstract

The application relates to the field of advertisement click rate dynamic prediction, and discloses an advertisement click rate dynamic prediction method and system based on deep reinforcement learning, which comprises the following steps: performing multidimensional analysis on historical advertisement clicks and user behavior data, dynamically identifying core user groups based on clustering, graph structure analysis and behavior sequence modeling, generating a user feature vector, simultaneously constructing a user behavior trend evolution vector, analyzing high click active distribution in different time periods to form a time period weight vector; combining a user's recent advertisement exposure sequence, extracting a guide deviation of the advertisement to the user's interest, forming an instant psychological preference state vector, and performing multi-layer fusion on advertisement multi-modal content to generate click probability change trend strategy information under different situations; and fusing the click probability change trend strategy information with real-time click, interaction, conversion data and user behavior sequences after advertisement delivery to generate a dynamically adjusted click rate prediction result. The application has the advantage of improving prediction accuracy.

Description

Technical Field

[0001] This invention relates to the field of dynamic prediction of ad click-through rate (CTR), specifically to a method and system for dynamic prediction of ad CTR based on deep reinforcement learning. Background Technology

[0002] In digital advertising, click-through rate (CTR) is a crucial metric for evaluating campaign effectiveness. Existing technologies typically predict CTR based on users' historical click behavior, interests, and ad content characteristics, and then formulate campaign strategies accordingly. However, these methods have shortcomings. First, the time window in which an ad first reaches users with specific ad preferences significantly impacts click-through rate, but existing methods often neglect the choice of initial launch time, leading to ads being delivered during periods of low activity for these users, thus significantly reducing CTR. Second, user interest modeling relies heavily on overall historical behavior, lacking precise user identification for specific ad types or content preferences, making it difficult to effectively target and optimize for core audiences. Furthermore, recently encountered ads may influence or change users' psychological preferences, affecting their interest in new ads, and existing technologies do not incorporate these dynamic changes in psychological preferences into their analysis, resulting in biased predictions. Finally, the dynamic matching between ad content and user psychological states is not adequately considered, and static analysis cannot guarantee that the initial ad will achieve optimal click-through rate within the critical time window. Therefore, designing a dynamic CTR prediction method and system based on deep reinforcement learning to improve prediction accuracy is essential. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention provides a method and system for dynamic prediction of ad click-through rates based on deep reinforcement learning, which has the advantage of improving prediction accuracy and solves the problems mentioned in the background technology.

[0004] To achieve the aforementioned goal of improving prediction accuracy, this invention provides the following technical solution: a dynamic prediction method for ad click-through rates based on deep reinforcement learning, comprising the following steps: Multi-dimensional analysis of historical ad clicks and user behavior data is performed to extract click frequency, dwell time, interaction type, and ad theme preference. Based on clustering, graph structure analysis, and behavioral sequence modeling, core user groups are dynamically identified, enhanced user feature vectors are generated, and user behavior trend evolution vectors are constructed. By combining enhanced user feature vectors and user behavior trend evolution vectors, we analyze the distribution of high click-through activity in different time periods, and dynamically consider user activity periodicity, holiday effects, multi-device cross-platform behavior and advertising environment factors to form time period weight vectors. By combining the time-time weight vector and the user's recent ad exposure sequence, the bias in how ads guide user interests is extracted, forming an instant psychological preference state vector; The enhanced user feature vector, time period weight vector, psychological preference state vector and advertising multimodal content are fused in multiple layers. The potential click results of different delivery schemes are simulated through deep generative networks to generate click probability change trend strategy information under different scenarios. By integrating click probability trend information under different scenarios with real-time click, interaction, conversion data and user behavior sequences after ad placement, and adaptively updating the data through reward mechanisms and strategy deduction, a dynamically adjusted click-through rate prediction result is generated.

[0005] The preferred process for dynamically identifying core user groups based on clustering, graph structure analysis, and behavioral sequence modeling is as follows: Perform multi-dimensional analysis of historical ad clicks and user behavior data to extract user behavior characteristics; Using clustering algorithms to analyze user group behavior patterns; By analyzing graph structures, we can identify the relationships between users and model behavioral sequences to dynamically identify core user groups. By integrating the behavioral characteristics of the core user group with the group's behavioral patterns, an enhanced user feature vector is generated. By combining user behavior sequence trends with historical preference changes, a user behavior trend evolution vector is constructed.

[0006] Preferably, the process of analyzing the distribution of high click-through rates and activity levels across different time periods is as follows: Based on enhanced user feature vectors and user behavior trend evolution vectors, the number of clicks and interaction frequency in different time periods are statistically analyzed. Identify high-activity periods by combining users' historical behavior patterns; Based on time series analysis, determine the user's click tendency and activity distribution in each time period, and output the user activity index and click tendency distribution for each time period.

[0007] Preferably, the process of forming the time-time weight vector is as follows: Based on user activity index and click tendency distribution for each time period, we collect information on daily user activity patterns, factors influencing holidays and special events, multi-device usage, and advertising competition environment. By combining the user activity index with various influencing factors, the overall activity weight of users in each time period is calculated. The activity weights for different time periods are weighted, and the calculated activity weights for each time period are standardized. The standardized active weights for each time period are organized into a vector form to form a complete time period weight vector.

[0008] Preferably, the process of forming an instantaneous psychological preference state vector is as follows: Based on the complete time-time weight vector, collect recent ad exposure sequences and interaction data of users; Using attention mechanisms and time-series analysis methods, we can assess the impact of each advertisement on user psychological preferences. The intensity of each advertisement's guidance on user interest is weighted and fused with the time period weight vector to calculate the instantaneous psychological preference bias value; The weighted deviation values ​​are mapped to an instantaneous psychological preference state vector.

[0009] Preferably, the process of multi-layer fusion of enhanced user feature vectors, time period weight vectors, psychological preference state vectors, and advertising multimodal content is as follows: The enhanced user feature vector, time period weight vector, and real-time psychological preference state vector are normalized. The association weights between user features and advertising multimodal features are calculated using an attention mechanism; Features are interactively fused through multi-layer neural networks, including feature weighting, nonlinear transformation, and multimodal feature fusion operations. Attention mapping across time periods and modalities is introduced during the fusion process; The fused multi-layer features are combined to generate the final integrated state representation vector.

[0010] Preferably, the process of generating click probability change trend strategy information under different scenarios is as follows: Based on the final comprehensive state representation vector, obtain the set of advertising placement schemes to be evaluated and related placement conditions; The comprehensive state representation vector is used as input, and combined with different delivery schemes, it is input into a deep generative network for simulation prediction. Calculate the potential click probability of each option under the current user characteristics, time period, and psychological preferences; By combining simulation results with user interest dynamics and time windows, click probability change trend information covering different scenarios and strategy combinations is generated.

[0011] Preferably, the process of integrating the click probability change trend information under different scenarios with real-time click, interaction, conversion data and user behavior sequences after ad placement is as follows: Based on the click probability change trend information covering different scenarios and strategy combinations, advertising data is collected in real time, including clicks, interactions, conversions and user behavior sequences; Align the collected real-time delivery data with the strategy information in terms of time and user dimensions; By integrating actual feedback with simulated strategy information, user status and ad response are updated. By using a time-series update method, the actual feedback is mapped into the policy information to form the updated policy input.

[0012] Preferably, the process of generating dynamically adjusted click-through rate prediction results is as follows: The reward function of reinforcement learning is used to evaluate the actual click-through rate based on the deviation between the actual click-through rate and the predicted click-through rate in the updated policy input; Combine strategy simulation to examine the potential effects of different advertising placement adjustment plans; Based on reward feedback and simulation results, the strategy generation model parameters are dynamically updated, and the final click-through rate prediction result is output, which is dynamically adjusted according to user behavior, psychological preferences, and time windows.

[0013] A deep reinforcement learning-based dynamic click-through rate prediction system for advertisements includes: User profiling module: Performs multi-dimensional analysis of historical ad clicks and user behavior data to identify core user groups and generate enhanced user feature vectors and behavioral trend evolution vectors; Time Analysis Module: Based on user characteristics and behavioral trends, analyzes the distribution of high click-through activity in different time periods and forms a time period weight vector; Psychological preference module: Combining time weights and recent ad exposure sequences, extracting the bias in how ads guide user interests, and generating an instant psychological preference state vector; Strategy generation module: It integrates user characteristics, time weight, psychological preferences and multimodal advertising content, and simulates the potential click results of different delivery plans through deep generative networks to generate strategy information on click probability change trends; The adaptive optimization module integrates strategy information with real-time click, interaction, conversion data, and user behavior sequences, and dynamically updates and generates the final click-through rate prediction result through reward mechanisms and strategy deduction.

[0014] Compared with existing technologies, this invention provides a method and system for dynamic prediction of ad click-through rates based on deep reinforcement learning, which has the following beneficial effects: This invention analyzes user historical ad clicks and behavior data from multiple dimensions, and dynamically identifies core user groups by combining clustering, graph structure, and behavioral sequence modeling. This generates enhanced user feature vectors and behavioral trend evolution vectors, enabling a more precise depiction of target users' interests and behavioral patterns. By analyzing user activity distribution across different time periods and forming time-period weight vectors, it achieves accurate matching between ad launch times and peak user click periods. Combining recent ad exposure sequences and psychological preference modeling, it captures the bias in how ads guide user interests, forming real-time psychological preference states and providing dynamic input for personalized targeting. Through multimodal feature fusion and deep generative networks simulating potential click results from different targeting schemes, it generates click probability trend information covering various scenarios, improving the predictability and refinement of targeting strategies. Finally, by combining real-time click, interaction, and conversion data with user behavior sequences, it adaptively updates click-through rate predictions through reward mechanisms and strategy deduction, thereby enhancing the targeting, timeliness, and accuracy of ad targeting. This improves ad effectiveness, optimizes resource utilization, and adapts to dynamic changes in user behavior and preferences. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the method of the present invention; Figure 2 This is a schematic diagram of the structure of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Example 1: Please refer to Figure 1 As shown in the embodiment of the present invention, the method for dynamic prediction of ad click-through rate based on deep reinforcement learning includes the following steps: S1: Perform multi-dimensional analysis on historical ad clicks and user behavior data to extract click frequency, dwell time, interaction type, and ad theme preference. Based on clustering, graph structure analysis, and behavioral sequence modeling, dynamically identify core user groups, generate enhanced user feature vectors, and construct user behavior trend evolution vectors. The process of dynamically identifying the core user group in S1 based on clustering, graph structure analysis, and behavioral sequence modeling is as follows: Perform multi-dimensional analysis of historical ad clicks and user behavior data to extract user behavior characteristics; Historical ad click records and user behavior data are retrieved from the advertising system database, including ad exposure time, click time, dwell time, interaction type, browsing order, and ad theme category. For each user, their behavior sequence is organized chronologically, and the frequency, average dwell time, and interaction distribution of various behaviors are statistically analyzed. The behavioral data is quantified, such as accumulating click frequency by hour, dwell time by seconds, and representing interaction type using encoding. Ad theme preferences are represented as vectors, and the click ratio and average interaction level of users across different themes are statistically analyzed. Finally, a behavioral feature matrix is ​​formed for each user.

[0018] Using clustering algorithms to analyze user group behavior patterns; The user behavior feature matrix is ​​input into the clustering algorithm to group users. The clustering method adopts a density-based iterative clustering strategy. First, the Euclidean distance matrix between user behavior feature vectors is calculated. Then, the cluster centers are found iteratively and users with high similarity are grouped into the same cluster. Each cluster represents a group of users with similar click behavior and interest preferences. During the clustering process, a distance threshold and a minimum cluster size are set to ensure high similarity within each cluster and significant differences between clusters. The clustering results are output as the cluster label and cluster center vector corresponding to each user.

[0019] By analyzing graph structures, we can identify the relationships between users and model behavioral sequences to dynamically identify core user groups. Based on user behavior sequences and social or interaction data, a user association graph is constructed, where each node represents a user, and the edge weights represent the behavioral similarity or interaction intensity between users. Behavioral similarity is calculated by the cosine similarity of historical click topics and time series, and interaction intensity is generated by accumulating the number of times users click on the same advertisement or are active in the same time period. The graph structure is analyzed using graph clustering or graph convolutional network methods to identify highly related subgraph regions between users. Combined with the node behavior characteristics within the subgraph, the core user group is dynamically filtered, and the core user node set and its association weight matrix are output.

[0020] By integrating the behavioral characteristics of the core user group with the group's behavioral patterns, an enhanced user feature vector is generated. The cluster center vectors and the features of core user nodes are fused together. A weighted average method is used to synthesize the feature vector of each user with the center vector of its cluster according to the weights. At the same time, the weights are adjusted by combining the association weights of the nodes in the graph. During the fusion process, the features of different dimensions are normalized to obtain the enhanced feature vector of each core user.

[0021] By combining user behavior sequence trends with historical preference changes, a user behavior trend evolution vector is constructed. Analyze the trends of each user's behavioral characteristics over time, including daily, weekly, and monthly click volume changes, fluctuations in topic interests, and evolution of interaction preferences. Use time series analysis methods to calculate the rate of behavioral change and trend vector, and map the behavioral change information into a continuous vector representation. Each dimension corresponds to the evolution value of a behavioral characteristic in different time windows, forming a user behavior trend evolution vector.

[0022] S2: Combining the enhanced user feature vector and the user behavior trend evolution vector, analyze the distribution of high click-through activity in different time periods, and dynamically consider user activity periodicity, holiday effect, multi-device cross-platform behavior and advertising environment factors to form a time period weight vector; The process of analyzing the distribution of high click-through activity in different time periods in S2 is as follows: Based on enhanced user feature vectors and user behavior trend evolution vectors, the number of clicks and interaction frequency in different time periods are statistically analyzed. Based on enhanced user feature vectors and user behavior trend evolution vectors, the system retrieves each user's historical ad exposure records, click records, and interaction behaviors from the advertising system database, including browsing time, likes, shares, and comments. The data is aggregated according to hourly or minute-level time windows. For each time window, the system calculates the total number of clicks, interactions, and interaction ratios of users within that window, and stores the statistical results as a time period matrix.

[0023] Identify high-activity periods by combining users' historical behavior patterns; Using a time-time matrix, the system analyzes user behavior patterns based on historical data. Daily and weekly click and interaction data are arranged in time series. For each user or user cluster, the system calculates the average click-through rate, interaction rate, and standard deviation for each time period. Time periods exceeding a certain threshold of the average value are identified as high-activity periods. The threshold can be set based on the distribution of historical data. The results are output in list format.

[0024] Based on time series analysis, determine the user's click tendency and activity distribution in each time period, and output the user activity index and click tendency distribution for each time period; Based on high-activity periods, click tendency is calculated for each time period, which is the proportion of clicks in that time period to the total number of clicks throughout the day. At the same time, interaction tendency is calculated, which represents the distribution of different interaction types in that time period. The click tendency and activity level of each time period are standardized to form the user activity index vector and click tendency distribution vector for each time period.

[0025] The process of forming the time-time weight vector in S2 is as follows: Based on user activity index and click tendency distribution for each time period, we collect information on daily user activity patterns, factors influencing holidays and special events, multi-device usage, and advertising competition environment. Based on user activity index and click tendency distribution for each time period, daily activity pattern data is obtained from the advertising system and user behavior database, including user login frequency and dwell time at different times of the day; information on holidays and special events is also collected, including national statutory holidays, industry promotion days, and dates of major events; multi-device usage is obtained, including user activity data and login duration on mobile phones, tablets, and desktops; information on the advertising competition environment is collected, including the volume of similar ads, display frequency, and click-through rate of competing ads. The above data is aligned according to time periods and stored uniformly as a time period feature matrix.

[0026] By combining the user activity index with various influencing factors, the overall activity weight of users in each time period is calculated. The collected data on various influencing factors are combined with user activity index and click tendency distribution. A weighted linear combination or multi-factor scoring method is used to calculate the comprehensive activity weight for each time period. In the weight calculation, the activity index percentage is set based on historical behavior data distribution, for example, 50% of the total weight; holidays and events account for 20%; multi-device usage accounts for 20%; and the advertising competition environment accounts for 10%. The comprehensive activity weight value is calculated for each user or user cluster, generating a list of time period weights.

[0027] The activity weights for different time periods are weighted, and the calculated activity weights for each time period are standardized. The active time period weights are weighted and adjusted to take into account long-term behavioral trends and recent behavioral changes. Then, the weighted time period weights are standardized so that the weight values ​​fall within the range of zero to one. The standardization method can be min-max normalization, which subtracts the minimum value from the weight of each user's time period and divides it by the difference between the maximum and minimum values. The processed data forms a standardized active time period weight matrix.

[0028] The standardized active weights for each time period are organized into a vector form to form a complete time period weight vector. The processed standardized time period weight matrix is ​​arranged in chronological order and organized into a vector form, generating a complete time period weight vector for each user.

[0029] S3: Combining the time period weight vector and the user's recent ad exposure sequence, extract the bias in the guidance of ad interests on users to form an instant psychological preference state vector; The process of forming the instantaneous psychological preference state vector in S3 is as follows: Based on the complete time-time weight vector, collect recent ad exposure sequences and interaction data of users; Based on a complete time-period weight vector, the system collects the user's ad exposure sequence over the past week from the advertising system, including ad identifier, exposure time, exposure channel, and ad type information. At the same time, it collects user interaction data, including click count, dwell time, swipe ratio, sharing behavior, and collection behavior. The collected data is organized in chronological order to form a sequence table of the user's recent advertising behavior.

[0030] Using attention mechanisms and time-series analysis methods, we can assess the impact of each advertisement on user psychological preferences. The ad exposure sequence and interaction data are input into the attention mechanism model to construct a temporal correlation matrix. The contribution weight of each ad content to user interests and psychological preferences is calculated. The attention calculation considers the user's dwell time, interaction depth, and the matching degree of ad type in the sequence. The user's response signal to different ad categories is mapped into a numerical psychological preference influence value. The output is the psychological preference weight vector of each ad, with a length equal to the ad feature dimension and a numerical range that can be normalized to between zero and one, representing the intensity of the ad's guidance on user interests.

[0031] The intensity of each advertisement's guidance on user interest is weighted and fused with the time period weight vector to calculate the instantaneous psychological preference bias value; The psychological preference weight vector of the advertisement is weighted and fused with the weight vector of the user's corresponding time period. The weighting method is to multiply the psychological preference influence value of each advertisement by the time period weight value of its corresponding exposure time period, and then sum the weighted values ​​of all advertisements in the same time period to obtain the time period weighted psychological preference distribution. The entire calculation process is processed independently for each user to form a daily or weekly real-time psychological preference deviation value matrix.

[0032] The weighted deviation values ​​are mapped to an instantaneous psychological preference state vector; The instant psychological preference deviation value matrix is ​​expanded according to the time period order and advertising feature dimension, and mapped to an instant psychological preference state vector of fixed length.

[0033] S4: The enhanced user feature vector, time period weight vector, psychological preference state vector and advertising multimodal content are fused in multiple layers. The potential click results of different delivery schemes are simulated through deep generative networks to generate click probability change trend strategy information under different scenarios. The process of multi-layer fusion in S4, which integrates enhanced user feature vectors, time-time weight vectors, psychological preference state vectors, and multimodal advertising content, is as follows: The enhanced user feature vector, time period weight vector, and real-time psychological preference state vector are normalized. The enhanced user feature vector, time period weight vector, and instant psychological preference state vector are normalized respectively, mapping the values ​​of each vector to between zero and one, ensuring that features of different dimensions and scales are comparable in subsequent calculations. The normalization method uses min-max normalization, mapping the minimum value of each feature dimension to zero and the maximum value to one, maintaining the proportional relationship of each dimension. At the same time, the normalization parameters are recorded for use in subsequent inverse normalization or feature reconstruction.

[0034] The association weights between user features and advertising multimodal features are calculated using an attention mechanism; The normalized user feature vector and the multimodal content vector of the advertisement are input into the attention mechanism model. The attention mechanism generates a weight matrix of each modality in the context of user behavior by calculating the similarity or relevance score between the user features and the features of each advertisement modality.

[0035] Features are interactively fused through multi-layer neural networks, including feature weighting, nonlinear transformation, and multimodal feature fusion operations. The weight matrix and feature vectors of each modality are used as inputs and then passed through a multi-layer neural network for feature interaction and fusion. Each layer of the network includes linear weighting, non-linear activation function transformation and feature recombination operation. By passing through the layers and combining the feature weights of different modalities, a complex non-linear mapping between user features and advertising content is achieved. The output dimension of each layer of the network is gradually adjusted. After dimensionality reduction and projection from the original feature dimension, a unified intermediate representation vector is finally formed.

[0036] Attention mapping across time periods and modalities is introduced during the fusion process; In the multi-layer fusion process, a cross-time period and cross-modal attention mapping mechanism is introduced to globally correlate user behavior information from different time periods with the modal features of advertisements. Through matrix multiplication and normalization operations, an attention distribution matrix of time periods and modalities is generated. Each element represents the attention weight of behavior in a specific time period to a specific advertisement modality, and is combined with the output of the multi-layer fusion network.

[0037] The fused multi-layer features are combined to generate the final integrated state representation vector; The feature vectors output by the fusion of multi-layer neural networks are integrated with the attention mapping results across time periods and modalities, and then combined and normalized to form a fixed-length comprehensive state representation vector.

[0038] The process of generating click probability change trend strategy information under different scenarios in S4 is as follows: Based on the final comprehensive state representation vector, obtain the set of advertising placement schemes to be evaluated and related placement conditions; Based on the final integrated state representation vector, the advertising campaign plans to be evaluated are collected, including the ad content number, display position, display frequency, target user group and campaign time period information. At the same time, the corresponding campaign conditions are collected, such as bidding price, budget limit, target region and device type.

[0039] The comprehensive state representation vector is used as input, and combined with different delivery schemes, it is input into a deep generative network for simulation prediction. The comprehensive state representation vector is combined with the vector of each advertising campaign in a one-to-one correspondence between users and campaigns. The result is then input into a deep generative network. The deep generative network adopts a multi-layer feedforward neural network structure, including linear transformation layers, non-linear activation function layers, and normalization layers. The network calculates the potential click probability layer by layer. The network performs feature fusion and non-linear mapping on the input vector and outputs the click probability distribution of each user under each campaign, with a value range of zero to one.

[0040] Calculate the potential click probability of each option under the current user characteristics, time period, and psychological preferences; For each advertising campaign, the probability distribution output by the network is used to perform statistical calculations by user dimension to obtain the average potential click probability, standard deviation, and maximum and minimum click probability values ​​of the campaign under the current user characteristics, time period, and psychological preferences.

[0041] By combining simulation results with user interest dynamics and time windows, click probability change trend information covering different scenarios and strategy combinations is generated; The calculated potential click probability is combined with the user behavior trend evolution vector, time period weight vector, and real-time psychological preference state vector to construct a click probability change matrix according to the time series and user group dimensions. Each row of the matrix represents a different user group, each column represents a different time period, and each element is the predicted click probability in the corresponding context. By sorting, weighting, and aggregating the matrix, click probability change trend information covering different contexts and strategy combinations is generated.

[0042] S5: Integrate the click probability change trend strategy information under different scenarios with real-time click, interaction, conversion data and user behavior sequences after ad placement, and adaptively update through reward mechanism and strategy deduction to generate dynamically adjusted click-through rate prediction results.

[0043] The process in S5 that integrates click probability change trend information under different scenarios with real-time click, interaction, conversion data and user behavior sequences after ad delivery is as follows: Based on the click probability change trend information covering different scenarios and strategy combinations, advertising data is collected in real time, including clicks, interactions, conversions and user behavior sequences; Based on the click probability trend information covering different scenarios and strategy combinations, real-time advertising data is collected, including the number of clicks, interactions, and conversions for each ad, as well as the corresponding user behavior sequence data. The user behavior sequence data includes the timestamp of each ad exposure, device type, geographical location, browsing duration, and interaction type with the ad. The collection frequency is set to once per minute, and the data is recorded in a time series database to ensure that the data is arranged in chronological order. At the same time, abnormal data is marked and preliminarily cleaned.

[0044] Align the collected real-time delivery data with the strategy information in terms of time and user dimensions; The collected real-time delivery data is aligned with the strategy information according to user identifiers and timestamps. Each user's click, interaction, and conversion data for each time period are matched one-to-one with the predicted click probability. Data across devices or platforms is uniformly mapped, and user identifiers on different devices and platforms are mapped to unique user IDs, ensuring that behavioral data within the time window is completely mapped to the strategy information matrix.

[0045] By integrating actual feedback with simulated strategy information, user status and ad response are updated. The aligned real-time delivery data is fused with the click probability obtained from simulation prediction. The difference between the actual click rate and the predicted click rate for each user in each time period is calculated. Combined with the user behavior sequence, the response to different ad types is weighted and updated to generate the current state vector for each user.

[0046] By using a time-series update method, the actual feedback is mapped into the policy information to form the updated policy input; The rolling time window method is adopted to accumulate and average real-time data over a continuous time period, and to map the latest user behavior feedback into the policy information. For each time window, the updated prediction probability, interest bias and weight distribution are calculated, and a new policy input matrix is ​​formed.

[0047] The process of generating dynamically adjusted click-through rate prediction results in S5 is as follows: The reward function of reinforcement learning is used to evaluate the actual click-through rate based on the deviation between the actual click-through rate and the predicted click-through rate in the updated policy input; Based on the updated policy input matrix, the predicted click-through rate (CTR) data for each user in each time period is extracted. At the same time, the actual number of clicks and exposures in the corresponding time period are collected, and the actual CTR is calculated. Through the reinforcement learning reward function, the deviation between each predicted data and the actual CTR is numerically calculated to obtain the reward or penalty value for each user in the current time period, which is recorded as a deviation matrix.

[0048] Combine strategy simulation to examine the potential effects of different advertising placement adjustment plans; Using the current strategy to generate a model, different advertising scenarios are simulated based on the deviation matrix. Each simulation scenario includes the ad display order, ad time period selection, and ad content combination. The model calculates the changes in potential click probability under the current user characteristics, time period weight, and psychological preference state. The simulation results are recorded according to user and time period, including potential click probability, interaction weight, and conversion rate estimation, providing a basis for subsequent reinforcement learning parameter updates.

[0049] Based on reward feedback and simulation results, the strategy generation model parameters are dynamically updated, and the final click-through rate prediction result is output, which is dynamically adjusted according to user behavior, psychological preferences and time windows. Based on the reward value and simulation results, the parameters of the strategy generation model are adjusted using gradients. By iteratively calculating the error between the forward propagation output and the reward feedback, backpropagation is performed to update the weights, including the connection weights in the multi-layer neural network, the attention mechanism parameters, and the time series prediction model parameters. The updated strategy generation model is then applied to the current user feature vector, time period weight vector, and psychological preference state vector to calculate the final predicted click-through rate for each user in each time period. The output results are stored in matrix form, with each record containing the user ID, time period number, dynamically adjusted click-through rate, corresponding ad ID, and campaign identifier. This matrix can be directly used as a reference input for the next round of ad campaigns, forming a continuously iterative dynamic prediction process.

[0050] A deep reinforcement learning-based dynamic click-through rate prediction system for advertisements includes: User profiling module: Performs multi-dimensional analysis of historical ad clicks and user behavior data to identify core user groups and generate enhanced user feature vectors and behavioral trend evolution vectors; Time Analysis Module: Based on user characteristics and behavioral trends, analyzes the distribution of high click-through activity in different time periods and forms a time period weight vector; Psychological preference module: Combining time weights and recent ad exposure sequences, extracting the bias in how ads guide user interests, and generating an instant psychological preference state vector; Strategy generation module: It integrates user characteristics, time weight, psychological preferences and multimodal advertising content, and simulates the potential click results of different delivery plans through deep generative networks to generate strategy information on click probability change trends; The adaptive optimization module integrates strategy information with real-time click, interaction, conversion data, and user behavior sequences, and dynamically updates and generates the final click-through rate prediction result through reward mechanisms and strategy deduction.

[0051] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0052] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for dynamic prediction of ad click-through rate based on deep reinforcement learning, characterized in that, Includes the following steps: Multi-dimensional analysis of historical ad clicks and user behavior data is performed to extract click frequency, dwell time, interaction type, and ad theme preference. Based on clustering, graph structure analysis, and behavioral sequence modeling, core user groups are dynamically identified, enhanced user feature vectors are generated, and user behavior trend evolution vectors are constructed. By combining enhanced user feature vectors and user behavior trend evolution vectors, we analyze the distribution of high click-through activity in different time periods, and dynamically consider user activity periodicity, holiday effects, multi-device cross-platform behavior and advertising environment factors to form time period weight vectors. By combining the time-time weight vector and the user's recent ad exposure sequence, the bias in how ads guide user interests is extracted, forming an instant psychological preference state vector; The enhanced user feature vector, time period weight vector, psychological preference state vector and advertising multimodal content are fused in multiple layers. The potential click results of different delivery schemes are simulated through deep generative networks to generate click probability change trend strategy information under different scenarios. By integrating click probability trend information under different scenarios with real-time click, interaction, conversion data and user behavior sequences after ad placement, and adaptively updating the data through reward mechanisms and strategy deduction, a dynamically adjusted click-through rate prediction result is generated.

2. The method for dynamic prediction of ad click-through rate based on deep reinforcement learning according to claim 1, characterized in that, The process of dynamically identifying core user groups based on clustering, graph structure analysis, and behavioral sequence modeling is as follows: Perform multi-dimensional analysis of historical ad clicks and user behavior data to extract user behavior characteristics; Using clustering algorithms to analyze user group behavior patterns; By analyzing graph structures, we can identify the relationships between users and model behavioral sequences to dynamically identify core user groups. By integrating the behavioral characteristics of the core user group with the group's behavioral patterns, an enhanced user feature vector is generated. By combining user behavior sequence trends with historical preference changes, a user behavior trend evolution vector is constructed.

3. The method for dynamic prediction of ad click-through rate based on deep reinforcement learning according to claim 2, characterized in that, The process of analyzing the distribution of high click-through rates in different time periods is as follows: Based on enhanced user feature vectors and user behavior trend evolution vectors, the number of clicks and interaction frequency in different time periods are statistically analyzed. Identify high-activity periods by combining users' historical behavior patterns; Based on time series analysis, determine the user's click tendency and activity distribution in each time period, and output the user activity index and click tendency distribution for each time period.

4. The method for dynamic prediction of ad click-through rate based on deep reinforcement learning according to claim 3, characterized in that, The process of forming the time-time weight vector is as follows: Based on user activity index and click tendency distribution for each time period, we collect information on daily user activity patterns, factors influencing holidays and special events, multi-device usage, and advertising competition environment. By combining the user activity index with various influencing factors, the overall activity weight of users in each time period is calculated. The activity weights for different time periods are weighted, and the calculated activity weights for each time period are standardized. The standardized active weights for each time period are organized into a vector form to form a complete time period weight vector.

5. The method for dynamic prediction of ad click-through rate based on deep reinforcement learning according to claim 4, characterized in that, The process of forming an instantaneous psychological preference state vector is as follows: Based on the complete time-time weight vector, collect recent ad exposure sequences and interaction data of users; Using attention mechanisms and time-series analysis methods, we can assess the impact of each advertisement on user psychological preferences. The intensity of each advertisement's guidance on user interest is weighted and fused with the time period weight vector to calculate the instantaneous psychological preference bias value; The weighted deviation values ​​are mapped to an instantaneous psychological preference state vector.

6. The method for dynamic prediction of ad click-through rate based on deep reinforcement learning according to claim 5, characterized in that, The process of multi-layer fusion of enhanced user feature vectors, time period weight vectors, psychological preference state vectors, and multimodal advertising content is as follows: The enhanced user feature vector, time period weight vector, and real-time psychological preference state vector are normalized. The association weights between user features and advertising multimodal features are calculated using an attention mechanism; Features are interactively fused through multi-layer neural networks, including feature weighting, nonlinear transformation, and multimodal feature fusion operations. Attention mapping across time periods and modalities is introduced during the fusion process; The fused multi-layer features are combined to generate the final integrated state representation vector.

7. The method for dynamic prediction of ad click-through rate based on deep reinforcement learning according to claim 6, characterized in that, The process of generating click probability change trend strategy information under different scenarios is as follows: Based on the final comprehensive state representation vector, obtain the set of advertising placement schemes to be evaluated and related placement conditions; The comprehensive state representation vector is used as input, and combined with different delivery schemes, it is input into a deep generative network for simulation prediction. Calculate the potential click probability of each option under the current user characteristics, time period, and psychological preferences; By combining simulation results with user interest dynamics and time windows, click probability change trend information covering different scenarios and strategy combinations is generated.

8. The method for dynamic prediction of ad click-through rate based on deep reinforcement learning according to claim 7, characterized in that, The process of integrating click probability change trend information under different scenarios with real-time click, interaction, conversion data and user behavior sequences after ad delivery is as follows: Based on the click probability change trend information covering different scenarios and strategy combinations, advertising data is collected in real time, including clicks, interactions, conversions and user behavior sequences; Align the collected real-time delivery data with the strategy information in terms of time and user dimensions; By integrating actual feedback with simulated strategy information, user status and ad response are updated. By using a time-series update method, the actual feedback is mapped into the policy information to form the updated policy input.

9. The method for dynamic prediction of ad click-through rate based on deep reinforcement learning according to claim 8, characterized in that, The process of generating dynamically adjusted click-through rate prediction results is as follows: The reward function of reinforcement learning is used to evaluate the actual click-through rate based on the deviation between the actual click-through rate and the predicted click-through rate in the updated policy input; Combine strategy simulation to examine the potential effects of different advertising placement adjustment plans; Based on reward feedback and simulation results, the strategy generation model parameters are dynamically updated, and the final click-through rate prediction result is output, which is dynamically adjusted according to user behavior, psychological preferences, and time windows.

10. A dynamic prediction system for ad click-through rate based on deep reinforcement learning, characterized in that, Applied to the method as described in claims 1-9, characterized in that it comprises: User profiling module: Performs multi-dimensional analysis of historical ad clicks and user behavior data to identify core user groups and generate enhanced user feature vectors and behavioral trend evolution vectors; Time Analysis Module: Based on user characteristics and behavioral trends, analyzes the distribution of high click-through activity in different time periods and forms a time period weight vector; Psychological preference module: Combining time weights and recent ad exposure sequences, extracting the bias in how ads guide user interests, and generating an instant psychological preference state vector; Strategy generation module: It integrates user characteristics, time weight, psychological preferences and multimodal advertising content, and simulates the potential click results of different delivery plans through deep generative networks to generate strategy information on click probability change trends; The adaptive optimization module integrates strategy information with real-time click, interaction, conversion data and user behavior sequences, and dynamically updates and generates the final click-through rate prediction result through reward mechanisms and strategy deduction.

Images