A reinforcement learning-based intelligent advertising delivery optimization system and method for e-commerce.

By using a reinforcement learning-based method to optimize e-commerce advertising, the method optimizes e-commerce advertising in different time zones, identifies turning points in exposure by using the interest retention intensity sequence, generates exploratory actions and corrects strategies, and solves the problems of lag and strategy adaptability in existing e-commerce advertising methods, achieving precise targeting and efficient utilization.

CN123118178APending Publication Date: 2026-11-17CHONGQING CATERPILLAR E-COMMERCE CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611599858.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-10-15
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing e-commerce advertising optimization technologies are lagging and one-sided, failing to accurately capture dynamic changes in user interests, resulting in wasted advertising traffic and inflated advertising costs. Furthermore, reinforcement learning-based advertising optimization models lack differentiated strategy iterations, leading to low model iteration accuracy and poor adaptability of advertising strategies.

Method used

By using a reinforcement learning-based approach, e-commerce advertising is optimized in segments. By segmenting exposure touchpoints and identifying turning exposure touchpoints based on interest retention intensity sequences, a set of exploratory actions is generated and applied to the advertising strategy. The advertising strategy is then adjusted based on the rate of decline in interest retention intensity, achieving dynamic optimization.

Benefits of technology

It has enabled precise optimization of e-commerce advertising, improved the accuracy and targeting of advertising strategies, reduced advertising traffic waste, and increased return on investment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123118178A_ABST
    Figure CN123118178A_ABST
Patent Text Reader

Abstract

This application provides a reinforcement learning-based intelligent advertising delivery optimization system and method for e-commerce, relating to the field of advertising delivery optimization technology. The method includes: identifying a segmented sequence of touchpoints consisting of consecutive non-converting exposure segments and conversion termination exposure segments within an advertising delivery cycle; determining the interest retention intensity sequence corresponding to each consecutive non-converting exposure segment; dividing the segmented sequence of touchpoints into an exploration interval and a convergence interval based on the turning exposure touchpoints identified by the interest retention intensity sequence; determining the delivery strategy after exploration within the exploration interval; and using the rate of decline of the interest retention intensity sequence after the turning exposure touchpoint as a reward signal for reinforcement learning within the convergence interval, determining the correction magnitude, correcting the exploration delivery strategy accordingly, and applying the corrected delivery strategy to the next advertising delivery cycle. This application can achieve precise delivery optimization of e-commerce advertising under reinforcement learning dynamic optimization.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of advertising optimization technology, and more specifically, to an intelligent advertising optimization system and method for e-commerce based on reinforcement learning. Background Technology

[0002] With the rapid development of the e-commerce industry and the widespread adoption of digital marketing, e-commerce advertising has become a core means for platforms and merchants to acquire traffic, promote conversions, and increase revenue. Currently, most mainstream e-commerce advertising optimization solutions are based on static user profiles, historical conversion data, and fixed placement strategy models. They rely heavily on statistical patterns from past placement data for strategy iteration, resulting in significant lag and bias. On the one hand, existing technologies generally lack a refined breakdown of user behavior across the entire advertising exposure chain. Most focus solely on optimizing strategies based on the final conversion result, ignoring the dynamic changes in user interest across continuous exposure touchpoints within the advertising campaign. This makes it difficult to accurately capture the complete behavioral evolution of users from browsing and attention to churn and conversion, and to identify the rising, peaking, and declining patterns of user interest retention. This easily leads to problems such as blindly continuing to target users with declining interest and prematurely terminating targeting of potential conversion users, resulting in wasted advertising traffic and inflated placement costs.

[0003] On the other hand, existing reinforcement learning-based ad optimization models have relatively simple state input and reward mechanisms, failing to differentiate ad delivery intervals and distinguish the characteristics of different stages of ad exploration and convergence. Traditional optimization methods often employ a uniform strategy iteration logic. In the exploration stage, before user interest decays, they cannot accurately mine the sampling value of potential conversion traffic, missing high-quality delivery opportunities. In the stage where user interest continues to decline, there is a lack of a dynamic correction mechanism based on the interest decay rate, making it difficult to precisely fine-tune the ad delivery strategy, resulting in low model iteration accuracy and poor ad delivery strategy adaptability. At the same time, the fixed ad delivery strategy iteration mode cannot adapt to the dynamic fluctuations of user interest within a single ad delivery cycle, resulting in lagging strategy updates and insufficient optimization targeting. Ultimately, this leads to unstable ad conversion rates and difficulty in continuously improving the return on investment, severely restricting the development of refined and intelligent e-commerce ad delivery.

[0004] Therefore, there is an urgent need for an intelligent e-commerce advertising delivery optimization method that can achieve dynamic optimization through reinforcement learning in different regions, in order to solve the industry problems of insufficient optimization accuracy and low traffic utilization of existing technologies, and improve the accuracy and cost-effectiveness of advertising delivery. Summary of the Invention

[0005] This application provides a reinforcement learning-based intelligent e-commerce advertising delivery optimization system and method, which can achieve precise delivery optimization of e-commerce advertisements under reinforcement learning dynamic optimization.

[0006] Firstly, this application provides a method for intelligent optimization of e-commerce advertising based on reinforcement learning, comprising the following steps:

[0007] During the advertising campaign period, user behavior sequences at the exposure level are obtained and segmented according to exposure touchpoints to obtain a touchpoint segment sequence consisting of continuous non-converting exposure segments and conversion termination exposure segments. The interest retention intensity sequence corresponding to each continuous non-converting exposure segment is then determined.

[0008] Based on the interest retention intensity sequence, the turning exposure touch point is identified, and the touch point segment sequence is divided into the deployment exploration interval and the deployment convergence interval, with the turning exposure touch point as the boundary.

[0009] Within the exploration range, the interest retention intensity sequence is used as the state input for reinforcement learning to generate a set of exploration actions that allow exposure touchpoints that have not yet decreased to be sampled again, and these actions are applied to the current delivery strategy to obtain the post-exploration delivery strategy.

[0010] Within the convergence interval of the campaign, the rate of decline of the interest retention intensity sequence after the turning exposure touchpoint is used as the reward signal for reinforcement learning. The correction magnitude is determined and the campaign strategy after the exploration is corrected accordingly. The corrected campaign strategy is then applied to the next advertising campaign cycle.

[0011] In some embodiments, acquiring user behavior sequences at the exposure level during the advertising campaign period and segmenting them according to exposure touchpoints to obtain a touchpoint segment sequence consisting of continuous non-converting exposure segments and conversion termination exposure segments specifically includes:

[0012] During the advertising campaign period, user exposure, clicks, and conversion behaviors of e-commerce ads are collected at each exposure touchpoint, forming a sequence of user behaviors at the exposure level sorted by timestamp.

[0013] Based on the user behavior sequence, exposure touchpoints with no conversion behavior between adjacent exposure touchpoints are grouped into a continuous unconverted exposure segment, and a conversion termination exposure segment is generated at the exposure touchpoint where the first conversion behavior occurs.

[0014] By concatenating the continuous unconverted exposure segment and the converted termination exposure segment according to their timestamps, a contact segment sequence consisting of the continuous unconverted exposure segment and the converted termination exposure segment is obtained.

[0015] In some embodiments, determining the interest retention intensity sequence corresponding to each consecutive unconverted exposure segment specifically includes:

[0016] For each continuous unconverted exposure segment, the click behavior, dwell time and page browsing depth corresponding to each exposure touchpoint are extracted to form a behavioral feature vector.

[0017] Based on the behavioral feature vector, the behavioral attenuation between adjacent exposure contacts is calculated in the order of the timestamps of the exposure contacts to obtain a behavioral attenuation sequence.

[0018] The behavior attenuation sequence is mapped to the interest retention intensity that varies with the exposure contact point, and arranged in the order of the exposure contact points to obtain the interest retention intensity sequence corresponding to each consecutive unconverted exposure segment.

[0019] In some embodiments, mapping the behavior attenuation sequence to interest retention intensity that varies with exposure touchpoint specifically includes:

[0020] The behavior decay sequence is normalized to obtain a normalized behavior decay sequence.

[0021] Based on the normalized behavior attenuation sequence, an exponential attenuation function is constructed with the exposure touch point number as the independent variable, and the value of the exponential attenuation function at the corresponding exposure touch point is used as the interest retention intensity of the exposure touch point.

[0022] The interest retention intensity of each exposure contact is arranged in order of exposure contact to obtain the interest retention intensity that varies with the exposure contact.

[0023] In some embodiments, identifying the transition exposure touchpoint based on the interest retention intensity sequence specifically includes:

[0024] The interest retention intensity sequence is subjected to first-order difference according to the exposure contact order to obtain the interest retention intensity difference sequence;

[0025] In the interest retention intensity difference sequence, locate the difference element that changes from a positive value to a non-positive value, and determine the exposure contact corresponding to the difference element as the turning point exposure contact where the interest retention intensity changes from rising to falling.

[0026] In some embodiments, dividing the sequence of touch points into a deployment exploration interval and a deployment convergence interval, with the inflection exposure touch point as the boundary, specifically includes:

[0027] Arrange all the obtained turning exposure points in the order of exposure points to obtain the set of turning exposure points;

[0028] In the contact point segmentation sequence, the exposure contact point before the first turning exposure contact point and located after the start position of the contact point segmentation sequence is included in the first deployment exploration interval;

[0029] Each turning exposure touch point itself, as well as the exposure touch points after that turning exposure touch point and before the next turning exposure touch point, are included in the delivery convergence interval corresponding to that turning exposure touch point.

[0030] The exposure touchpoints before each subsequent turning exposure touchpoint and after the previous turning exposure touchpoint are included in the next deployment exploration range, and the exposure touchpoints after the last turning exposure touchpoint are included in the last deployment convergence range.

[0031] In some embodiments, within the exploration range, using the interest retention intensity sequence as the state input for reinforcement learning to generate a set of exploration actions that allow exposure touchpoints that have not yet decreased to be sampled further specifically includes:

[0032] Within the aforementioned exploration range, the sequence of interest retention intensity falling within the exploration range is input into the state encoding layer of the reinforcement learning policy network in the order of exposure touchpoints to obtain state feature representations.

[0033] Based on the state characteristics, exposure touchpoints whose interest retention intensity is still rising are selected within the exploration range to form a candidate exposure touchpoint set.

[0034] Using the set of candidate exposure touchpoints as the action space, the reinforcement learning policy network samples and generates a set of exploratory actions to continue deploying to the candidate exposure touchpoints.

[0035] Secondly, this application provides a reinforcement learning-based intelligent e-commerce advertising delivery optimization system for executing a reinforcement learning-based intelligent e-commerce advertising delivery optimization method. The system includes:

[0036] The acquisition module is used to acquire user behavior sequences at the exposure level during the advertising campaign period, and segment them according to exposure touchpoints to obtain a touchpoint segment sequence consisting of continuous non-converting exposure segments and conversion termination exposure segments, and determine the interest retention intensity sequence corresponding to each continuous non-converting exposure segment.

[0037] The processing module is used to identify the turning exposure touchpoints based on the interest retention intensity sequence, and divide the touchpoint segment sequence into a deployment exploration interval and a deployment convergence interval with the turning exposure touchpoints as the boundary.

[0038] The processing module is further configured to, within the exploration range, use the interest retention intensity sequence as the state input for reinforcement learning, generate a set of exploration actions that allow exposure touchpoints that have not yet decreased to be sampled again, and apply them to the current delivery strategy to obtain the post-exploration delivery strategy;

[0039] The execution module is used to determine the correction magnitude and adjust the explored delivery strategy accordingly within the delivery convergence interval, using the rate of decline of the interest retention intensity sequence after the turning exposure touchpoint as a reward signal for reinforcement learning, and then applying the adjusted delivery strategy to the next advertising delivery cycle.

[0040] Thirdly, this application provides a computer device, the computer device including a memory and a processor, the memory storing code, and the processor being configured to acquire the code and execute the above-described reinforcement learning-based intelligent advertising delivery optimization method.

[0041] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described reinforcement learning-based intelligent advertising delivery optimization method.

[0042] The technical solutions provided by the embodiments disclosed in this application have the following beneficial effects:

[0043] The reinforcement learning-based intelligent e-commerce advertising delivery optimization system and method provided in this application can achieve precise delivery optimization of e-commerce advertisements under dynamic optimization through reinforcement learning. Firstly, using exposure touchpoints as the smallest collection and decision-making unit, discrete exposure logs are organized into structured segmented data with clear conversion termination boundaries. Furthermore, behavioral features with varying dimensions are uniformly mapped to a sequence of interest retention intensity that changes sequentially along the exposure touchpoints. This provides a stable, non-oscillating, and comparable input for identifying subsequent turning exposure touchpoints, avoiding estimation biases in interest retention intensity caused by mixed exposure touchpoints or varying dimensions of behavioral features. Secondly, by locating the change in interest retention intensity from increasing to decreasing... The declining inflection point of exposure touchpoints uniquely and definitively anchors the boundary between exploration and convergence at the inflection point where interest shifts from accumulation to decline. This avoids the arbitrariness of relying on empirical thresholds to divide intervals, strictly limiting the exploration interval to the range of exposure touchpoints where interest has not yet declined, and the convergence interval to the range of exposure touchpoints where interest has entered a decline phase. Then, using state feature representation as a filtering criterion, the sampling range is limited to exposure touchpoints where interest is still rising, avoiding invalid exploration of exposure touchpoints where interest has declined. The matching strategy parameters are then updated with weighted probabilities based on sampling probabilities, ensuring that the adjustment magnitude of the strategy parameters is proportional to the sampling probability and that adjustments during the exploration phase do not contaminate the convergence interval. Finally, the rate of decline in interest retention intensity is used as a reward signal to drive the value network output correction magnitude, directly linking the strategy correction magnitude to the rate of interest decline. This achieves interval-based processing of exploration and correction, and allows the corrected delivery strategy to iterate continuously with the advertising cycle, constantly updating the strategy network parameters, thus improving the overall accuracy, targeting, and convergence efficiency of the delivery strategy correction. In summary, the technical solution provided in this application can achieve precise delivery optimization of e-commerce advertising under reinforcement learning dynamic optimization. Attached Figure Description

[0044] Figure 1 This is an exemplary flowchart of a reinforcement learning-based intelligent advertising delivery optimization method for e-commerce, as shown in some embodiments of this application.

[0045] Figure 2This is a schematic diagram illustrating the principle of transition exposure contact recognition according to some embodiments of this application;

[0046] Figure 3 This is a schematic diagram of the structure of an e-commerce advertising intelligent delivery optimization system based on reinforcement learning, as shown in some embodiments of this application.

[0047] Figure 4 This is a schematic diagram of the structure of a computer device that implements a reinforcement learning-based intelligent advertising delivery optimization method according to some embodiments of this application. Detailed Implementation

[0048] To better understand the technical solution of this application, the technical solution of this application will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0049] refer to Figure 1 The figure is an exemplary flowchart of an e-commerce advertising intelligent delivery optimization method based on reinforcement learning, according to some embodiments of this application. The figure mainly includes the following steps:

[0050] In step S101, user behavior sequences at the exposure level are obtained within the advertising campaign period, and segmented according to exposure touchpoints to obtain a touchpoint segmentation sequence consisting of continuous non-converted exposure segments and conversion termination exposure segments, and the interest retention intensity sequence corresponding to each continuous non-converted exposure segment is determined.

[0051] It should be noted that the advertising campaign cycle in this application refers to the time window from the start of the advertising campaign to the completion of a strategy update and effect recovery for the advertising campaign. It takes the exposure touchpoint as the smallest collection and decision-making unit, and covers the complete collection process of exposure, click and conversion behavior on all exposure touchpoints within the time window. It also serves as a closed-loop iteration cycle of the reinforcement learning strategy from state input, action generation, reward calculation to strategy correction. After each advertising campaign cycle ends, the corrected campaign strategy becomes the initial campaign strategy for the next advertising campaign cycle, and so on, rolling and iterating in cycles.

[0052] In some embodiments, obtaining a sequence of user behavior at the exposure level during the advertising campaign period and segmenting it by exposure touchpoint to obtain a touchpoint segment sequence consisting of continuous non-converting exposure segments and conversion termination exposure segments is achieved through the following steps:

[0053] During the advertising campaign period, user exposure, clicks, and conversion behaviors of e-commerce ads are collected at each exposure touchpoint, forming a sequence of user behaviors at the exposure level sorted by timestamp.

[0054] Based on the user behavior sequence, exposure touchpoints with no conversion behavior between adjacent exposure touchpoints are grouped into a continuous unconverted exposure segment, and a conversion termination exposure segment is generated at the exposure touchpoint where the first conversion behavior occurs.

[0055] By concatenating the continuous unconverted exposure segment and the converted termination exposure segment according to their timestamps, a contact segment sequence consisting of the continuous unconverted exposure segment and the converted termination exposure segment is obtained.

[0056] In practice, firstly, within the advertising campaign period, user exposure, clicks, and conversions of e-commerce ads are collected in units of exposure touchpoints, forming a user behavior sequence at the exposure level ordered by timestamps. An exposure touchpoint is the smallest unit of ad delivery where a user's ad display request is responded to by the ad delivery engine and the ad is rendered. The user behavior sequence is a collection of exposure, click, and conversion events of the same user arranged chronologically over time. Specifically, the ad delivery engine writes an exposure log each time an ad is rendered. The log fields include user identifier, ad identifier, exposure timestamp, click flag, and conversion flag. The click flag is set by the click event listener interface when the user clicks the ad, and the conversion flag is set by the conversion feedback interface when the user completes an order or payment. The exposure logs, arranged in ascending order of exposure timestamps, constitute the user behavior sequence at the exposure level. For example, if user U1 generates three exposure logs at timestamps T1, T2, and T3, with only a click at T2 and a conversion at T3, then the user behavior sequence at the exposure level for that user is obtained after sorting by timestamps. Secondly... Based on the user behavior sequence, exposure touchpoints with no conversion behavior between adjacent exposure touchpoints are grouped into continuous unconverted exposure segments. A conversion termination exposure segment is generated at the exposure touchpoint where a conversion behavior first occurs. A continuous unconverted exposure segment refers to a segment in the user behavior sequence consisting of multiple consecutive exposure touchpoints where the conversion flag is not set. A conversion termination exposure segment refers to a segment in the user behavior sequence consisting of exposure touchpoints where the conversion flag is set for the first time. Specifically, this is achieved by traversing each exposure touchpoint in the user behavior sequence, ordered by its exposure timestamp in ascending order. For each exposure touchpoint, its conversion flag is read one by one. When an exposure touchpoint with an unset conversion flag is read, it is written into the current continuous unconverted exposure segment. When an exposure touchpoint with a set conversion flag is read, the writing of the current continuous unconverted exposure segment ends and the exposure touchpoint is truncated to generate a conversion termination exposure segment. For example, in the user behavior sequence, the conversion flag of exposure touchpoint T1 is unset, the conversion flag of exposure touchpoint T2 is unset, and the conversion flag of exposure touchpoint T3 is set. Then T1 and T2 are merged into a continuous unconverted exposure segment, and T3 is truncated to generate a conversion termination exposure segment.Finally, the continuous unconverted exposure segments and the conversion termination exposure segments are concatenated according to their timestamps to obtain a contact segment sequence composed of continuous unconverted exposure segments and conversion termination exposure segments. The contact segment sequence refers to a set of exposure contact segments ordered by exposure timestamps, consisting of at least one continuous unconverted exposure segment and at most one conversion termination exposure segment connected sequentially. Specifically, the smallest exposure timestamp within each continuous unconverted exposure segment and conversion termination exposure segment is used as the sorting key. Each continuous unconverted exposure segment and conversion termination exposure segment is then written into the same segment list in ascending order according to this sorting key, thus obtaining the contact segment sequence. For example... The consecutive unconverted exposure segments T1 and T2, and the conversion termination exposure segment T3, are concatenated in ascending order of minimum exposure timestamps. This results in a touchpoint segment sequence composed of the consecutive unconverted exposure segments and the conversion termination exposure segment. This touchpoint segment sequence serves as input data for subsequently determining the interest retention intensity sequence corresponding to each consecutive unconverted exposure segment. It should be noted that the conversion termination exposure segment, as the first exposure touchpoint where conversion behavior occurs, indicates that the user has completed the conversion. It does not participate in the calculation of the subsequent interest retention intensity decline rate, nor is it included in the delivery convergence interval. Instead, it is used as a positive sample for training the reinforcement learning strategy network.

[0057] This solution collects user behavior sequences by exposure touchpoints and divides them into continuous non-converted exposure segments and conversion termination exposure segments based on whether conversion behavior occurs. This organizes the originally discrete and disordered exposure logs into structured segmented data with clear conversion termination boundaries, thereby providing a unified and comparable input unit for the subsequent calculation of interest retention intensity sequences and avoiding the estimation bias of interest retention intensity caused by mixed exposure touchpoints.

[0058] In some embodiments, determining the interest retention intensity sequence corresponding to each consecutive unconverted exposure segment is achieved by the following steps:

[0059] For each continuous unconverted exposure segment, the click behavior, dwell time and page browsing depth corresponding to each exposure touchpoint are extracted to form a behavioral feature vector.

[0060] Based on the behavioral feature vector, the behavioral attenuation between adjacent exposure contacts is calculated in the order of the timestamps of the exposure contacts to obtain a behavioral attenuation sequence.

[0061] The behavior attenuation sequence is mapped to the interest retention intensity that varies with the exposure contact point, and arranged in the order of the exposure contact points to obtain the interest retention intensity sequence corresponding to each consecutive unconverted exposure segment.

[0062] In practice, firstly, for each consecutive unconverted exposure segment, the click behavior, dwell time, and page browsing depth corresponding to each exposure touchpoint are extracted to form a behavioral feature vector. This behavioral feature vector is a feature representation composed of three components: click behavior, dwell time, and page browsing depth, combined according to the exposure touchpoint. Specifically, for each exposure touchpoint within the consecutive unconverted exposure segment, the click flag is read from the ad delivery engine's exposure and click logs and encoded as a click behavior component (1 for clicks, 0 for no clicks). The data is also read from the landing page's event tracking data. The dwell time component is obtained by subtracting the page load completion timestamp and the page exit timestamp. The page browsing depth component is obtained by reading the page browsing level count from the landing page tracking data. The three components are concatenated in a fixed order of click behavior, dwell time, and page browsing depth to form the behavior feature vector of the exposure touchpoint. For example, if the click behavior of exposure touchpoint P1 in a certain continuous unconverted exposure segment is 1, the dwell time is 8 seconds, and the page browsing depth is 2, then its behavior feature vector is (1,8,2). Next, based on the behavior feature vector, the exposure touchpoints are sorted in order of their timestamps. The behavior attenuation between adjacent exposure touchpoints is calculated sequentially to obtain a behavior attenuation sequence. Behavior attenuation refers to the degree of difference between the behavioral feature vectors of two adjacent exposure touchpoints, calculated on a component-by-component basis. The behavior attenuation sequence is a sequence composed of the behavior attenuation amounts between all adjacent exposure touchpoints within a continuous unconverted exposure segment, arranged in the order of the exposure touchpoint timestamps. Specifically, the absolute values ​​of the differences between the behavioral feature vectors of adjacent exposure touchpoints are taken, and then a weighted sum is calculated. The weights can be pre-set based on the contribution of click behavior, dwell time, and page browsing depth to the interest retention intensity. For example, click behavior weight is 0.5, dwell time weight is 0.3, and page browsing depth weight is 0.2. This yields the behavior attenuation amount between adjacent exposure touchpoints. This behavior attenuation amount is written into the sequence in the order of the exposure touchpoint timestamps to obtain the behavior attenuation sequence. The behavior attenuation sequence is then mapped to the interest retention intensity that varies with the exposure touchpoints and arranged in the order of the exposure touchpoints to obtain the interest retention intensity sequence corresponding to each continuous unconverted exposure segment. Interest retention intensity refers to the numerical value representing the degree of user interest in the advertisement corresponding to the continuous unconverted exposure segment.

[0063] In some embodiments, mapping the behavior attenuation sequence to interest retention intensity that varies with exposure touchpoints is achieved using the following steps:

[0064] The behavior decay sequence is normalized to obtain a normalized behavior decay sequence.

[0065] Based on the normalized behavior attenuation sequence, an exponential attenuation function is constructed with the exposure touch point number as the independent variable, and the value of the exponential attenuation function at the corresponding exposure touch point is used as the interest retention intensity of the exposure touch point.

[0066] The interest retention intensity of each exposure contact is arranged in order of exposure contact to obtain the interest retention intensity that varies with the exposure contact.

[0067] In practice, firstly, the behavior attenuation sequence is normalized to the maximum and minimum values ​​to ensure that the values ​​fall within the [0,1] interval, resulting in a normalized behavior attenuation sequence. Secondly, an exponential decay function is constructed using the exposure touchpoint number as the independent variable and the normalized behavior attenuation as the attenuation coefficient. The value obtained by substituting each exposure touchpoint number into the exponential decay function is used as the interest retention intensity of that exposure touchpoint. For example, if the normalized behavior attenuation sequence is 0.4 and 0.6, and the exposure touchpoint numbers are 1, 2, and 3, then the value of the exponential decay function is calculated using the exposure touchpoint number as the independent variable and the corresponding normalized behavior attenuation as the attenuation coefficient. The resulting interest retention intensities of each exposure touchpoint are arranged in order of exposure touchpoints to obtain the interest retention intensity sequence corresponding to the continuous unconverted exposure segment.

[0068] It should be noted that the interest retention intensity sequence in this application refers to the numerical sequence formed by arranging the interest retention intensities of each exposure touchpoint in the continuous unconverted exposure segment according to the chronological order of the exposure touchpoint timestamps. By mapping the behavior attenuation sequence to a value that changes with the exposure touchpoints, the interest retention intensity has clear monotonicity and comparability. This unifies the original discrete and dimensionless behavioral characteristics into a scalar sequence that changes monotonically along the exposure touchpoint sequence, providing a stable input for subsequent first-order differential positioning of turning exposure touchpoints and providing a reliable basis for the division of the deployment exploration interval and deployment convergence interval.

[0069] In step S102, the turning exposure touchpoint is identified based on the interest retention intensity sequence, and the touchpoint segment sequence is divided into a deployment exploration interval and a deployment convergence interval, with the turning exposure touchpoint as the boundary.

[0070] In some embodiments, identifying the transition exposure touchpoint based on the interest retention intensity sequence is achieved through the following steps:

[0071] The interest retention intensity sequence is subjected to first-order difference according to the exposure contact order to obtain the interest retention intensity difference sequence;

[0072] In the interest retention intensity difference sequence, locate the difference element that changes from a positive value to a non-positive value, and determine the exposure contact corresponding to the difference element as the turning point exposure contact where the interest retention intensity changes from rising to falling.

[0073] In specific implementation, firstly, the interest retention intensity sequence is subjected to first-order difference according to the exposure contact order to obtain the interest retention intensity difference sequence. First-order difference refers to the operation of subtracting the preceding element from the following element in any two adjacent elements of the interest retention intensity sequence. The interest retention intensity difference sequence is a sequence formed by arranging the difference values ​​obtained after the first-order difference of the interest retention intensity sequence according to the exposure contact order. Specifically, let the interest retention intensity sequence be S=[s1,s2,…,sn], where n is the number of exposure contacts in the continuous unconverted exposure segment. Then, d1=s2-s1, d2=s3-s2,…, d(n-1)=sn-s(n- 1) Arrange d1 to d(n-1) in the order of exposure contacts to obtain the interest retention intensity difference sequence D=[d1,d2,…,d(n-1)]; then, locate the difference element that changes from positive to non-positive value in the interest retention intensity difference sequence, and determine the exposure contact corresponding to the difference element as the turning point exposure contact where the interest retention intensity changes from increasing to decreasing, and thus obtain all the turning points exposure contacts. The change from positive to non-positive value refers to the adjacent change relationship where a difference element in the difference sequence is positive and the next difference element is zero or negative. The turning point exposure contact refers to the exposure contact corresponding to the change from increasing to decreasing interest retention intensity.

[0074] refer to Figure 2 This figure is a schematic diagram of the principle of identifying turning exposure touchpoints according to some embodiments of this application. In this figure, the horizontal axis represents exposure touchpoints, which correspond to the advertising exposure nodes that reach the user in chronological order (P1, P2, P3, P4, P5, P6...Pn, where Pn represents the nth exposure touchpoint at the end of the sequence, and n is the sequence number of the last exposure touchpoint in the segmented sequence. n is a variable natural number that varies with the length of the segmented sequence for different users). It is used to quantify the user's interest level in the advertisement at each exposure touchpoint. The curve represents the trend of interest retention intensity as the exposure touchpoint progresses. P1 to P5 is the exploration interval for ad placement. In this interval, the intensity of user interest retention first increases, then stabilizes briefly, and then continues to increase. P5 is the turning point of exposure, which is the dividing peak where the intensity of interest retention changes from increasing to decreasing. P5 to Pn is the convergence interval for ad placement. In this interval, the intensity of interest retention continues to decrease with subsequent exposure touchpoints. This figure intuitively illustrates the principle of dividing the ad placement exploration interval and the ad placement convergence interval with the turning point of exposure as the boundary. It provides the principle support for subsequent reinforcement learning to generate ad placement exploration actions in the exploration interval and construct reward signals based on the rate of decrease of interest retention intensity in the convergence interval to correct the ad placement strategy.

[0075] It should be noted that, in this application, the turning point of exposure touchpoint is used to characterize the critical exposure position where user interest in an advertisement changes from accumulation to decline as exposure touchpoints increase. This solution locates the difference element that changes from positive to non-positive in the interest retention intensity difference sequence, so that the critical exposure position is uniquely and definitively anchored at the inflection point where interest retention intensity changes from rising to falling, avoiding the arbitrariness of interval division caused by relying on empirical thresholds or artificially set cut-off points. At the same time, this turning point of exposure touchpoint serves as the common boundary between the ad exploration interval and the ad convergence interval, so that the exploration interval is strictly limited to the range of exposure touchpoints where interest has not yet declined, and the convergence interval is strictly limited to the range of exposure touchpoints where interest has entered a decline.

[0076] In some embodiments, dividing the sequence of contact points into a deployment exploration interval and a deployment convergence interval, with the inflection exposure contact point as the boundary, is achieved through the following steps:

[0077] Arrange all the obtained turning exposure points in the order of exposure points to obtain the set of turning exposure points;

[0078] In the contact point segmentation sequence, the exposure contact point before the first turning exposure contact point and located after the start position of the contact point segmentation sequence is included in the first deployment exploration interval;

[0079] Each turning exposure touch point itself, as well as the exposure touch points after that turning exposure touch point and before the next turning exposure touch point, are included in the delivery convergence interval corresponding to that turning exposure touch point.

[0080] The exposure touchpoints before each subsequent turning exposure touchpoint and after the previous turning exposure touchpoint are included in the next deployment exploration range, and the exposure touchpoints after the last turning exposure touchpoint are included in the last deployment convergence range.

[0081] In specific implementation, all acquired turning exposure touchpoints are arranged in the order of exposure touchpoints to obtain a set of turning exposure touchpoints. This set refers to the set of all exposure touchpoints in the aforementioned interest retention intensity difference sequence that satisfy the condition that the preceding difference element is positive and the following difference element is non-positive, arranged according to their timestamp order in the touchpoint segment sequence. For example, if three turning exposure touchpoints with exposure touchpoint numbers 3, 7, and 11 are identified in the touchpoint segment sequence, then they are arranged in ascending order to obtain the set {3,7,11}. Then, in the touchpoint segment sequence, the touchpoints preceding the first turning exposure touchpoint and located at the beginning of the touchpoint segment sequence are... The exposure touchpoints after the initial placement are assigned to the first deployment exploration interval, i.e., exposure touchpoints numbered 1 to 2 are assigned to the first deployment exploration interval. Each turning exposure touchpoint itself, as well as the exposure touchpoints after it and before the next turning exposure touchpoint, are assigned to the deployment convergence interval corresponding to that turning exposure touchpoint, i.e., exposure touchpoints numbered 3 to 6 are assigned to the first deployment convergence interval, exposure touchpoints numbered 7 to 10 are assigned to the second deployment convergence interval, and exposure touchpoints numbered 11 and after it are assigned to the third deployment convergence interval. Each subsequent turning exposure touchpoint, and the exposure touchpoint after the previous turning exposure touchpoint, is assigned to the next deployment exploration interval, i.e., exposure touchpoints numbered 4 to 6... The light touchpoint is assigned to the second projection exploration interval, exposure touchpoints numbered 8 to 10 are assigned to the third projection exploration interval, and exposure touchpoints after the last turning exposure touchpoint are assigned to the last projection convergence interval, i.e., exposure touchpoints after number 11 are assigned to the last projection convergence interval. The projection exploration interval is a segment consisting of exposure touchpoints where the interest retention intensity is increasing between two adjacent turning exposure touchpoints, and this segment is a left-closed, right-open interval. The projection convergence interval is a segment consisting of the turning exposure touchpoint itself and the exposure touchpoints after it and before the next turning exposure touchpoint where the interest retention intensity is decreasing, and this segment is a left-closed, right-open interval. Each turning exposure touchpoint... Points belong only to the subsequent delivery convergence interval and are no longer considered as closing elements of the preceding delivery exploration interval. Each delivery exploration interval and each delivery convergence interval are arranged alternately according to the exposure touchpoint order, and any two adjacent intervals have no overlap or omission in the exposure touchpoint dimension. The resulting interval sequence serves as the input data for subsequently inputting the interest retention intensity sequence into the state encoding layer of the reinforcement learning policy network to generate the set of exploration actions within each delivery exploration interval, and inputting the interest retention intensity decline rate sequence into the reinforcement learning value network to generate reward signals and determine the correction magnitude within each delivery convergence interval. This interval sequence refers to the segmented set formed by the alternating arrangement of delivery exploration intervals and delivery convergence intervals according to the exposure touchpoint order.

[0082] It should be noted that in this application, the exploration interval is used to characterize the segment consisting of exposure touchpoints where interest retention intensity is in the rising phase and has not yet declined; that is, the range where user interest is still accumulating and continued sampling is expected to improve conversion probability. The convergence interval is used to characterize the segment consisting of exposure touchpoints where interest retention intensity is in the declining phase and has already declined; that is, the range where user interest has entered the decline phase, and continued sampling yields diminishing returns, requiring a shift to strategy adjustment. By dividing the touchpoint segment sequence into an alternating sequence of exploration and convergence intervals using all turning exposure touchpoints as boundaries, the generation of reinforcement learning exploration actions and the calculation of reward signals each fall within intervals that match their semantics. Exploration actions are only applied to exposure touchpoints where interest is still rising, and reward signals are only taken from exposure touchpoints where interest has declined, avoiding strategy oscillations caused by exploring and adjusting within the same interval.

[0083] In step S103, within the deployment exploration interval, the interest retention intensity sequence is used as the state input for reinforcement learning to generate a set of exploration actions that allow exposure touchpoints that have not yet decreased to continue to be sampled, and these actions are applied to the current deployment strategy to obtain the deployment strategy after exploration.

[0084] In some embodiments, within the deployment exploration interval, using the interest retention intensity sequence as the state input for reinforcement learning to generate a set of exploration actions that allow exposure touchpoints that have not yet experienced a decline to continue sampling is achieved through the following steps:

[0085] Within the aforementioned exploration range, the sequence of interest retention intensity falling within the exploration range is input into the state encoding layer of the reinforcement learning policy network in the order of exposure touchpoints to obtain state feature representations.

[0086] Based on the state characteristics, exposure touchpoints whose interest retention intensity is still rising are selected within the exploration range to form a candidate exposure touchpoint set.

[0087] Using the set of candidate exposure touchpoints as the action space, the reinforcement learning policy network samples and generates a set of exploratory actions to continue deploying to the candidate exposure touchpoints.

[0088] In specific implementation, firstly, within the deployment exploration range, the interest retention intensity sequence falling within this range is input into the state encoding layer of the reinforcement learning policy network in the order of exposure touchpoints to obtain a state feature representation. The reinforcement learning policy network is a parameterized neural network that takes states as input and action probability distributions as outputs. The state encoding layer is the network layer in the reinforcement learning policy network responsible for mapping the original states into fixed-length feature vectors. The state feature representation is the feature vector obtained after mapping the interest retention intensity sequence through the state encoding layer. Specifically, the interest retention intensity sequence falling within the deployment exploration range is concatenated into a fixed-length input vector in the order of exposure touchpoints (where zeros are used to pad if the sequence length is insufficient, and the length is truncated to the deployment exploration range if it exceeds the limit). The number of exposure touchpoints within a certain exploration interval is determined by inputting a fixed-length input vector into a state encoding layer consisting of fully connected layers and activation functions. After linear transformation of the fully connected layers and nonlinear mapping of the activation functions, the fixed-length feature vector at the output of the state encoding layer is taken as the state feature representation. For example, if the interest retention intensity sequence within a certain exploration interval is [0.3, 0.5, 0.7], it is concatenated into an input vector [0.3, 0.5, 0.7] and input into the state encoding layer. After mapping through two fully connected layers and a ReLU activation function, the state feature representation is obtained. Then, based on the state feature representation, exposure touchpoints with increasing interest retention intensity are selected within the exploration interval to form a candidate exposure touchpoint set. The term "set" refers to the collection of exposure touchpoints within the exploration interval whose interest retention intensity difference element is positive. Specifically, using state features as query conditions, the interest retention intensity of each exposure touchpoint within the exploration interval is subtracted from the previous element according to the exposure touchpoint order. Exposure touchpoints with positive difference values ​​are added to the candidate exposure touchpoint list in order, thus obtaining the candidate exposure touchpoint set. Finally, using the candidate exposure touchpoint set as the action space, the reinforcement learning policy network samples and generates a set of exploration actions for further exposure of the candidate exposure touchpoints. The action space refers to the set of actions that the reinforcement learning policy network can select, and the exploration action set refers to the set of actions generated by the reinforcement learning policy network for the candidate exposure touchpoints. The set consists of the exploration actions sampled from each exposure touchpoint in the set, used to indicate further deployment. The exploration action set is generated according to the order of each deployment exploration interval in the interval sequence. In specific implementation, the state feature representation is input into the policy output layer of the reinforcement learning policy network. After normalization by the Softmax function, the sampling probability corresponding to each exposure touchpoint in the candidate exposure touchpoint set is obtained. Bernoulli sampling is performed on each exposure touchpoint in the candidate exposure touchpoint set according to each sampling probability, or the top K are selected according to the probability. The further deployment action corresponding to the exposure touchpoint that hit the sampling is written into the exploration action set. For example, the candidate exposure touchpoint set is {1,2,3}, and the sampling probabilities given by the policy output layer are 0.6, 0.3, and 0.1. If exposure touchpoints 1 and 2 are sampled and hit according to probability, a set of exploratory actions for further deployment to exposure touchpoints 1 and 2 is generated. This set of exploratory actions serves as input data for subsequently obtaining the strategy parameters that match the corresponding exposure touchpoints in the current deployment strategy, forming the set of strategy parameters to be updated.

[0089] Specifically, the pre-training process of the reinforcement learning policy network is as follows: User behavior sequences at the exposure level are collected from the historical exposure logs of the advertising platform according to the aforementioned touchpoint segmentation method. The interest retention intensity sequence corresponding to each consecutive non-converting exposure segment, as well as the set of turning exposure touchpoints, the exploration interval, and the convergence interval are determined according to the aforementioned method. The interest retention intensity sequence within each exploration interval is used as a state sample, and the continued delivery action corresponding to the exposure touchpoint where a click or conversion actually occurs within that exploration interval is used as the action label for that state sample, forming a state-action sample pair. Each state-action sample pair is then divided into a training set and a validation set according to a set ratio. A reinforcement learning policy network is constructed, consisting of a state encoding layer and a policy output layer. The network is constructed by connecting the following layers: the state encoding layer uses a fully connected layer and a ReLU activation function to map the input interest retention intensity sequence into a state feature representation; the policy output layer uses a fully connected layer and a Softmax function to map the state feature representation into the sampling probability corresponding to each exposure touchpoint in the candidate exposure touchpoint set; taking state-action sample pairs in the training set as input, the cross-entropy loss function is used to measure the difference between the sampling probability output by the policy output layer and the action label; the policy network parameters are iteratively updated using stochastic gradient descent or Adam optimizer at a set learning rate until the cross-entropy loss function no longer decreases on the validation set or reaches the set number of iterations; training is then stopped and the policy network parameters are saved, thus obtaining the pre-trained reinforcement learning policy network. It should be noted that the reinforcement learning policy network and the reinforcement learning value network (described later) work together in each advertising campaign cycle: the policy network is responsible for generating a set of exploration actions within the campaign exploration interval, while the value network is responsible for outputting reward signals within the campaign convergence interval; the policy network is updated based on the sampling probability and cross-entropy loss corresponding to the set of exploration actions, while the value network is updated based on the interest retention intensity decline rate sequence and subsequent exposure conversion revenue. The two are trained independently but share user behavior sequence data within the same advertising campaign cycle.

[0090] This scheme uses the state feature representation obtained by the state coding layer through the interest retention intensity sequence within each ad placement exploration interval as the screening criterion. It converges exposure touchpoints with increasing interest retention intensity into a candidate exposure touchpoint set and uses this set as the action space. This limits the sampling range of the reinforcement learning strategy network to exposure touchpoints that have not yet declined, avoiding ineffective exploration of exposure touchpoints where interest has faded. At the same time, it uses the sampling probability given by the strategy output layer to sample the candidate exposure touchpoint set to generate an exploration action set. This ensures that the distribution of exploration actions is based on the interest evolution trend reflected by the state feature representation, while retaining the exploration capability of probabilistic random sampling. This allows for continued ad placement with lower trial and error costs when interest is still rising, and a natural reduction in the tendency to continue ad placement when interest is nearing a turning point through sampling probability. This provides a clear probabilistic basis for subsequent weighted updates of matching strategy parameters based on sampling probability, ultimately improving the targeting of ad placement strategy exploration and the convergence efficiency of strategy correction in the next ad placement cycle.

[0091] In some embodiments, the post-exploration deployment strategy is determined using the following steps:

[0092] Obtain the strategy parameters in the current delivery strategy that match the exposure touchpoints corresponding to the set of exploration actions in each delivery exploration interval, and form a set of strategy parameters to be updated;

[0093] The set of strategy parameters to be updated is weighted and updated according to the sampling probability corresponding to each exploration action in the set of exploration actions to obtain the updated strategy parameters.

[0094] The updated strategy parameters replace the set of strategy parameters to be updated in the current delivery strategy to obtain the explored delivery strategy. The explored delivery strategy is composed of segments of the updated strategy parameters corresponding to each delivery exploration interval, arranged in the order of the interval sequence.

[0095] In practice, firstly, the strategy parameters that match the exposure touchpoints corresponding to the set of exploration actions within each exploration interval in the current delivery strategy are obtained, forming a set of strategy parameters to be updated. The current delivery strategy refers to the set of parameters that the advertising engine is executing during the current advertising period to determine whether each exposure touchpoint is delivered and with what bid and targeting conditions. Strategy parameters refer to the adjustable parameters configured according to the exposure touchpoint dimension in the current delivery strategy (e.g., bid coefficient, targeting condition weight, frequency control threshold). The set of strategy parameters to be updated refers to the exposure touchpoints in the current delivery strategy that are matched with the set of exploration actions within each exploration interval. The set consists of a one-to-one correspondence of strategy parameters. In practice, using the exposure touchpoint identifier indicated by each exploration action in the exploration action set as an index, the bidding coefficient, targeting condition weight, and frequency control threshold corresponding to each exposure touchpoint identifier are retrieved one by one from the strategy parameter configuration of the delivery engine. The retrieved parameters are then written into the list of strategy parameters to be updated according to the exposure touchpoint identifier, thus forming the set of strategy parameters to be updated. For example, if the exploration action set indicates that exposure touchpoints 1 and 2 should continue to be delivered, then the bidding coefficients 0.8 and 0.6 and the targeting condition weights 0.5 and 0.4 corresponding to exposure touchpoints 1 and 2 are retrieved from the strategy parameter configuration table and written into the set of strategy parameters to be updated; then… The updated policy parameters are obtained by weighted updating the policy parameter set according to the sampling probability corresponding to each exploration action in the exploration action set. The sampling probability refers to the probability that each exposure touchpoint in the candidate exposure touchpoint set is sampled and hit, given by the Softmax function after normalization of the policy output layer of the reinforcement learning policy network. Weighted updating refers to the operation of adding the increment in the policy gradient direction to the baseline value using each policy parameter in the policy parameter set as the baseline value and the corresponding sampling probability as the weight, according to a set learning rate. Specifically, for each exploration action in the exploration action set, the policy parameters are updated using the following formula: New: Updated parameters = Current parameters + Learning rate × Sampling probability × Policy gradient, where the policy gradient is obtained through backpropagation of the reinforcement learning policy network. The learning rate is a preset value such as 0.01 or 0.001. For example, if the current bid coefficient of exposure touchpoint 1 is 0.8, the sampling probability is 0.6, the learning rate is 0.01, and the policy gradient is 0.5, then the updated bid coefficient is 0.8 + 0.01 × 0.6 × 0.5 = 0.803. If the current bid coefficient of exposure touchpoint 2 is 0.6, the sampling probability is 0.3, the learning rate is 0.01, and the policy gradient is 0.4, then the updated bid coefficient is 0.6 + 0.01 × 0.3 × 0.4 = 0.6012. The updated strategy parameters for each exposure touchpoint are written into the updated strategy parameter list according to the exposure touchpoint identifier to obtain the updated strategy parameters. Finally, the updated strategy parameters replace the set of strategy parameters to be updated in the current delivery strategy to obtain the explored delivery strategy. The explored delivery strategy is composed of segments of updated strategy parameters corresponding to each delivery exploration interval, arranged in the order of the interval sequence. The interval sequence refers to a set of segments formed by alternating exposure touchpoints of delivery exploration intervals and delivery convergence intervals. Segment splicing refers to the operation of writing the updated strategy parameters corresponding to each delivery exploration interval into the delivery strategy configuration table in the order of each delivery exploration interval in the interval sequence, and writing the original strategy parameters corresponding to each delivery convergence interval into the same delivery strategy configuration table while keeping them unchanged. In practice, the system iterates through each exploration interval in sequence, updating the strategy parameters for each exploration interval and overwriting the original strategy parameters in the corresponding positions of the delivery strategy configuration table according to the exposure touchpoint identifier. The strategy parameters for each convergence interval remain unchanged from their original values ​​in the current delivery strategy. The resulting delivery strategy configuration table after the iteration is the explored delivery strategy. For example, if the interval sequence is exploration interval 1, convergence interval 1, exploration interval 2, and convergence interval 2, then the updated strategy parameters for exploration interval 1, the original strategy parameters for convergence interval 1, the updated strategy parameters for exploration interval 2, and the original strategy parameters for convergence interval 2 are written into the same delivery strategy configuration in this order to obtain the explored delivery strategy.

[0096] It should be noted that the post-exploration delivery strategy is used to characterize an intermediate delivery strategy based on the current delivery strategy. This strategy updates the parameters of exposure touchpoints where interest retention intensity is still increasing by weighting the sampling probability, while keeping the original strategy parameters unchanged for the remaining exposure touchpoints. This provides the strategy basis for the next stage of correction based on reward signals. This scheme retrieves the set of strategy parameters to be updated by using the exposure touchpoints hit by the exploration action set as an index, and updates them by weighting the sampling probability corresponding to each exploration action. This makes the adjustment range of the strategy parameters proportional to the sampling probability given by the reinforcement learning policy network. The higher the sampling probability of the exposure touchpoint, the larger the parameter adjustment, and the lower the probability, the smaller the adjustment. This directly converts the probability information of the exploration action into a continuous adjustment of the strategy parameters, avoiding the parameter jumps caused by simply adjusting by the same amount based on whether the hit or not is successful.

[0097] In step S104, within the convergence interval of the advertising campaign, the rate of decline of the interest retention intensity sequence after the turning exposure touchpoint is used as the reward signal for reinforcement learning. The correction magnitude is determined and the explored advertising strategy is corrected accordingly. The corrected advertising strategy is then applied to the next advertising campaign cycle.

[0098] In some embodiments, within the delivery convergence interval, the rate of decline of the interest retention intensity sequence after the inflection exposure point is used as the reward signal for reinforcement learning. The correction magnitude is determined and implemented according to the delivery strategy after the correction exploration using the following steps:

[0099] Within the delivery convergence interval, the interest retention intensity sequence falling within the delivery convergence interval is obtained, and the interest retention intensity after the starting and turning exposure touch point of the delivery convergence interval is differentially divided according to the exposure touch point order to obtain the interest retention intensity decrease rate sequence.

[0100] The interest retention intensity decline rate sequence is input into the reinforcement learning value network to obtain the reward signal;

[0101] Based on the reward signal, determine the correction magnitude of the strategy parameters corresponding to the deployment convergence interval in the deployment strategy after exploration, and correct the deployment strategy after exploration according to the correction magnitude to obtain the corrected deployment strategy.

[0102] In specific implementation, firstly, within the delivery convergence interval, the sequence of interest retention intensities falling within the interval is obtained. Then, the interest retention intensities after the initial turning exposure point of the delivery convergence interval are differentially analyzed according to the exposure point order to obtain the interest retention intensity decline rate sequence. This sequence refers to the sequence formed by arranging the difference values ​​obtained by subtracting the preceding element from the subsequent element of the interest retention intensities after the initial turning exposure point of the delivery convergence interval according to the exposure point order. Specifically, starting from the next exposure point after the initial turning exposure point of the delivery convergence interval, the interest retention intensities of adjacent exposure points are differentially analyzed one by one according to the exposure point order. The interest retention intensity is subtracted, and the difference values ​​are written into a sequence according to the exposure touchpoint order to obtain the interest retention intensity decline rate sequence. For example, if the interest retention intensity after the initial turning exposure touchpoint in a certain convergence interval is 0.7, 0.5, and 0.3 respectively, then the differences are 0.5-0.7=-0.2 and 0.3-0.5=-0.2, and the interest retention intensity decline rate sequence is [-0.2,-0.2]. Then, the interest retention intensity decline rate sequence is input into a reinforcement learning value network to obtain a reward signal. The reinforcement learning value network is a parameterized neural network that takes state or state-action features as input and state value or action value as output. The reward signal is... This refers to the scalar value output by the reinforcement learning value network, which characterizes the strategy correction reward corresponding to the rate of decline in interest retention intensity within the convergence interval of the campaign. Specifically, the interest retention intensity decline rate sequence is concatenated into a fixed-length input vector according to the exposure touchpoint order. This input is fed into the value network, which consists of fully connected layers and activation functions connected sequentially. After linear transformation of the fully connected layers and nonlinear mapping of the activation functions, the output scalar value is taken as the reward signal. The value network parameters are obtained by training sample pairs consisting of the interest retention intensity decline rate sequence within the convergence interval of the campaign and its subsequent exposure conversion rewards from historical campaign data using a mean squared error loss function and an Adam optimizer. For example, the interest retention intensity decline... The rate sequence [-0.2, -0.2] is input into the value network, and the output reward signal is -0.15. Finally, based on the reward signal, the correction magnitude of the strategy parameters corresponding to the delivery convergence interval in the explored delivery strategy is determined, and the explored delivery strategy is corrected according to the correction magnitude to obtain the corrected delivery strategy. The correction magnitude refers to the amount of adjustment made to the strategy parameters corresponding to the delivery convergence interval according to the reward signal value. In specific implementation, the reward signal is input into a pre-set linear mapping relationship to obtain the correction magnitude of the strategy parameters corresponding to each exposure touchpoint in the delivery convergence interval. For example, if the reward signal is -0.15 and the mapping coefficient is 0.1, then the correction magnitude is -0.015. Based on the correction magnitude, the bid coefficients, targeting condition weights, and frequency control thresholds for each exposure touchpoint corresponding to the convergence interval in the explored delivery strategy are added together one by one. The correction magnitude for each convergence interval is determined and corrected segment by segment according to the interval sequence. The corrected strategy parameters for each convergence interval are then concatenated with the uncorrected strategy parameters for each exploration interval in the explored delivery strategy, segment by segment according to the interval sequence, to obtain the corrected delivery strategy. This corrected delivery strategy serves as the input data for subsequent deployment to the delivery engine in the next advertising cycle and for obtaining the initial delivery strategy for the next advertising cycle.

[0103] Specifically, the pre-training process of the reinforcement learning value network refers to the initialization and training of the value network parameters using historical advertising data before actual deployment, enabling it to map the interest retention intensity decline rate sequence into a reward signal. In practice, user behavior sequences at the exposure level are collected from the historical exposure logs of the advertising platform according to the aforementioned touchpoint segmentation method. The interest retention intensity sequence, the set of turning exposure touchpoints, the deployment exploration interval, and the deployment convergence interval corresponding to each consecutive non-converting exposure segment are determined according to the aforementioned method. For each deployment convergence interval, the interest retention intensity sequence falling within that interval is obtained. The interest retention intensity after the starting turning exposure touchpoint of the deployment convergence interval is then differentially analyzed according to the exposure touchpoint order to obtain the interest retention intensity decline rate sequence. This interest retention intensity decline rate sequence is used as the input sample, and each exposure touchpoint within the deployment convergence interval is then tested within a subsequently set time window. The exposure conversion revenue generated (e.g., exposure conversion rate or exposure transaction amount) is used as the reward label for the input sample, forming an input sample-reward label sample pair. Each sample pair is divided into a training set and a validation set according to a set ratio. A reinforcement learning value network is constructed, which consists of fully connected layers and activation functions connected sequentially. The number of nodes in the input layer is equal to the fixed length of the interest retention intensity decline rate sequence. The output layer is a single scalar node, and the hidden layers use the ReLU activation function. Using the input sample-reward label sample pairs in the training set as input, the mean squared error loss function is used to measure the difference between the reward signal output by the value network and the reward label. The parameters of the value network are iteratively updated using stochastic gradient descent or Adam optimizer at a set learning rate until the mean squared error loss function no longer decreases on the validation set or reaches the set number of iterations. Training is then stopped and the value network parameters are saved, thus obtaining the pre-trained reinforcement learning value network.

[0104] It should be noted that the pre-set linear mapping relationship refers to a linear correspondence established with the reward signal output by the reinforcement learning value network as the independent variable, the correction magnitude of the strategy parameters corresponding to the convergence interval of the deployment as the dependent variable, and a fixed proportional coefficient. Specifically, the correction magnitude is equal to the reward signal multiplied by the mapping coefficient, where the mapping coefficient is a pre-set constant used to scale the dimensions of the reward signal to the adjustable range of the strategy parameters. The mapping coefficient can be set according to the type of strategy parameters (for example, the mapping coefficient corresponding to the bid coefficient is 0.1, the mapping coefficient corresponding to the targeting condition weight is 0.05, and the mapping coefficient corresponding to the frequency control threshold is 1). The mapping coefficient is obtained by linear regression fitting between the reward signal and the adjustment amount of the strategy parameters that improves the subsequent exposure conversion revenue in historical deployment data, or it can be pre-set manually in the range of 0.01 to 1 based on experience. There is no limitation here.

[0105] In some embodiments, applying the revised delivery strategy to the next advertising cycle is achieved through the following steps:

[0106] The revised delivery strategy is deployed to the delivery engine of the next advertising cycle in the order of exposure touchpoints to obtain the initial delivery strategy for the next advertising cycle.

[0107] In the next advertising campaign cycle, ads are delivered to the exposure touchpoints using the initial delivery strategy, and the exposure, click and conversion behaviors of each exposure touchpoint are collected to obtain the user behavior sequence at the exposure level in the next advertising campaign cycle.

[0108] The user behavior sequence for the next advertising cycle is fed back to the reinforcement learning policy network to update the policy parameters of the reinforcement learning policy network.

[0109] In practice, firstly, the revised delivery strategy is deployed to the delivery engine for the next advertising cycle according to the order of exposure touchpoints, thus obtaining the initial delivery strategy for the next advertising cycle. The delivery engine refers to the advertising server program that receives advertising delivery requests, decides whether to deliver ads according to the delivery strategy, and determines the bidding and targeting conditions for delivery. The initial delivery strategy refers to the delivery strategy that the delivery engine actually loads and takes effect at the start of the next advertising cycle. Specifically, the bidding coefficient, targeting condition weight, and frequency control threshold corresponding to each exposure touchpoint in the revised delivery strategy are written into the delivery engine's strategy configuration table one by one according to the exposure touchpoint identifier. The delivery engine reads this strategy configuration table and loads it at the start of the next advertising cycle, thus obtaining the initial delivery strategy. The initial placement strategy for the next advertising campaign period is then determined. Within the next advertising campaign period, ads are placed on exposure touchpoints according to the initial placement strategy, and the exposure, click, and conversion behaviors of each touchpoint are collected to obtain the user behavior sequence at the exposure level for the next advertising campaign period. Specifically, the ad delivery engine writes an exposure log according to the initial placement strategy each time an ad is rendered within the next advertising campaign period. The log fields include user identifier, ad identifier, exposure timestamp, click flag, and conversion flag. The click flag is set by the click event listener interface when the user clicks the ad, and the conversion flag is set by the conversion feedback interface when the user completes an order or payment. The exposure logs are then sorted in ascending order of exposure timestamp to obtain the next... The user behavior sequence at the exposure level of the advertising campaign cycle is then processed. Finally, the user behavior sequence for the next advertising campaign cycle is fed back to the reinforcement learning strategy network to update the strategy parameters of the reinforcement learning strategy network. Feedback refers to writing the user behavior sequence at the exposure level of the next advertising campaign cycle into the training sample cache of the reinforcement learning strategy network according to the exposure touchpoint identifiers. Updating the strategy parameters of the reinforcement learning strategy network involves re-dividing the interval sequence according to the aforementioned touchpoint segmentation method based on the fed-back user behavior sequence, re-determining the set of turning exposure touchpoints, using the interest retention intensity sequence within each campaign exploration interval as state samples, and using the continued campaign actions corresponding to the exposure touchpoints where clicks or conversions actually occurred within each campaign exploration interval as action indicators. The process involves iteratively updating the strategy network parameters using the cross-entropy loss function and the Adam optimizer. For example, after backfeeding the user behavior sequence at the exposure level for the next advertising campaign, the interval sequence is re-divided and the set of turning exposure touchpoints is redefined. The interest retention intensity sequence [0.4, 0.6] within the advertising exploration interval is taken as the state sample, and the continued advertising action corresponding to the exposure touchpoint that actually occurred in the interval is taken as the action label. The strategy network parameters are iteratively updated using the cross-entropy loss function and the Adam optimizer with a learning rate of 0.01. The updated strategy network parameters are used as the network parameters for generating the set of exploration actions in the next advertising campaign, thus enabling the strategy network to iterate with each advertising campaign.

[0110] Furthermore, in another aspect of this application, in some embodiments, this application provides an intelligent e-commerce advertising delivery optimization system based on reinforcement learning, referencing... Figure 3 The figure is a schematic diagram of the structure of an e-commerce advertising intelligent delivery optimization system based on reinforcement learning, according to some embodiments of this application. The e-commerce advertising intelligent delivery optimization system based on reinforcement learning includes: an acquisition module 201, a processing module 202, and an execution module 203, which are described below:

[0111] The acquisition module 201 in this application is mainly used to collect user behavior sequences at the exposure level during the advertising campaign period, and segment them according to the exposure touchpoints to obtain a touchpoint segment sequence consisting of continuous non-converted exposure segments and conversion termination exposure segments, and determine the interest retention intensity sequence corresponding to each continuous non-converted exposure segment.

[0112] Processing module 202 in this application is mainly used to identify the turning exposure touch point based on the interest retention intensity sequence, and divide the touch point segment sequence into a delivery exploration interval and a delivery convergence interval with the turning exposure touch point as the boundary.

[0113] The processing module 202 is further configured to, within the deployment exploration interval, use the interest retention intensity sequence as the state input for reinforcement learning, generate a set of exploration actions that allow exposure touchpoints that have not yet decreased to be sampled again, and apply them to the current deployment strategy to obtain the deployment strategy after exploration;

[0114] The execution module 203 in this application is mainly used to, within the convergence interval of the advertising campaign, take the rate of decline of the interest retention intensity sequence after the turning exposure touchpoint as the reward signal for reinforcement learning, determine the correction magnitude and correct the explored advertising strategy accordingly, and apply the corrected advertising strategy to the next advertising campaign cycle.

[0115] In addition, this application also provides a computer device, which includes a memory and a processor. The memory stores code, and the processor is configured to acquire the code and execute the above-described reinforcement learning-based intelligent advertising delivery optimization method.

[0116] In some embodiments, reference Figure 4 The figure is a schematic diagram of the structure of a computer device implementing a reinforcement learning-based intelligent e-commerce advertising delivery optimization method according to some embodiments of this application. The reinforcement learning-based intelligent e-commerce advertising delivery optimization method in the above embodiments can... Figure 4 The computer device shown is used to implement this, and the computer device includes at least one processor 301, a communication bus 302, a memory 303, and at least one communication interface 304.

[0117] The processor 301 can be a general-purpose central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more devices used to control the execution of the reinforcement learning-based e-commerce advertising intelligent delivery optimization method in this application.

[0118] The communication bus 302 can be used to transmit information between the aforementioned components.

[0119] The memory 303 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disks or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 303 may exist independently and be connected to the processor 301 via the communication bus 302. The memory 303 may also be integrated with the processor 301.

[0120] The memory 303 stores program code for executing the scheme of this application, and its execution is controlled by the processor 301. The processor 301 executes the program code stored in the memory 303. The program code may include one or more software modules. In the above embodiments, the determination of the intelligent e-commerce advertising delivery optimization method based on reinforcement learning can be achieved by the processor 301 and one or more software modules in the program code in the memory 303.

[0121] Communication interface 304 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.

[0122] In a specific implementation, as one example, a computer device may include multiple processors, each of which may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. Here, a processor may refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0123] The aforementioned computer device can be a general-purpose computer device or a special-purpose computer device. In specific implementations, the computer device can be a desktop computer, a portable computer, a network server, a handheld digital assistant (PDA), a mobile phone, a tablet computer, a wireless terminal device, a communication device, or an embedded device. This application does not limit the type of computer device.

[0124] In addition, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described reinforcement learning-based intelligent advertising delivery optimization method.

[0125] Although preferred embodiments of this application have been described, those skilled in the art, once they have learned the basic inventive concept, can make other changes and modifications to these embodiments.

[0126] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application.

Claims

1. A method for optimizing intelligent delivery of e-commerce advertisements based on reinforcement learning, characterized in that, Includes the following steps: During the advertising campaign period, user behavior sequences at the exposure level are obtained and segmented according to exposure touchpoints to obtain a touchpoint segment sequence consisting of continuous non-converting exposure segments and conversion termination exposure segments. The interest retention intensity sequence corresponding to each continuous non-converting exposure segment is then determined. Based on the interest retention intensity sequence, the turning exposure touch point is identified, and the touch point segment sequence is divided into the deployment exploration interval and the deployment convergence interval, with the turning exposure touch point as the boundary. Within the exploration range, the interest retention intensity sequence is used as the state input for reinforcement learning to generate a set of exploration actions that allow exposure touchpoints that have not yet decreased to be sampled again, and these actions are applied to the current delivery strategy to obtain the post-exploration delivery strategy. Within the convergence interval of the campaign, the rate of decline of the interest retention intensity sequence after the turning exposure touchpoint is used as the reward signal for reinforcement learning. The correction magnitude is determined and the campaign strategy after the exploration is corrected accordingly. The corrected campaign strategy is then applied to the next advertising campaign cycle.

2. The method of claim 1, wherein, During the advertising campaign period, user behavior sequences at the exposure level are obtained and segmented according to exposure touchpoints. The resulting touchpoint segment sequence consists of continuous non-converting exposure segments and conversion termination exposure segments. Specifically, it includes: During the advertising campaign period, user exposure, clicks, and conversion behaviors of e-commerce ads are collected at each exposure touchpoint, forming a sequence of user behaviors at the exposure level sorted by timestamp. Based on the user behavior sequence, exposure touchpoints with no conversion behavior between adjacent exposure touchpoints are grouped into a continuous unconverted exposure segment, and a conversion termination exposure segment is generated at the exposure touchpoint where the first conversion behavior occurs. By concatenating the continuous unconverted exposure segment and the converted termination exposure segment according to their timestamps, a contact segment sequence consisting of the continuous unconverted exposure segment and the converted termination exposure segment is obtained.

3. The method of claim 1, wherein, Determining the interest retention intensity sequence corresponding to each consecutive unconverted exposure segment specifically includes: For each continuous unconverted exposure segment, the click behavior, dwell time and page browsing depth corresponding to each exposure touchpoint are extracted to form a behavioral feature vector. Based on the behavioral feature vector, the behavioral attenuation between adjacent exposure contacts is calculated in the order of the timestamps of the exposure contacts to obtain a behavioral attenuation sequence. The behavior attenuation sequence is mapped to the interest retention intensity that varies with the exposure contact point, and arranged in the order of the exposure contact points to obtain the interest retention intensity sequence corresponding to each consecutive unconverted exposure segment.

4. The method of claim 3, wherein, Mapping the behavior attenuation sequence to interest retention intensity that varies with exposure touchpoints specifically includes: The behavior decay sequence is normalized to obtain a normalized behavior decay sequence. Based on the normalized behavior attenuation sequence, an exponential attenuation function is constructed with the exposure touch point number as the independent variable, and the value of the exponential attenuation function at the corresponding exposure touch point is used as the interest retention intensity of the exposure touch point. The interest retention intensity of each exposure contact is arranged in order of exposure contact to obtain the interest retention intensity that varies with the exposure contact.

5. The method of claim 1, wherein, The specific features of identifying the turning point exposure touchpoint based on the interest retention intensity sequence include: The interest retention intensity sequence is subjected to first-order difference according to the exposure contact order to obtain the interest retention intensity difference sequence; In the interest retention intensity difference sequence, locate the difference element that changes from a positive value to a non-positive value, and determine the exposure contact corresponding to the difference element as the turning point exposure contact where the interest retention intensity changes from rising to falling.

6. The method of claim 1, wherein, Using the aforementioned turning exposure touchpoint as the boundary, the touchpoint segment sequence is divided into a deployment exploration interval and a deployment convergence interval, specifically including: Arrange all the obtained turning exposure points in the order of exposure points to obtain the set of turning exposure points; In the contact point segmentation sequence, the exposure contact point before the first turning exposure contact point and located after the start position of the contact point segmentation sequence is included in the first deployment exploration interval; Each turning exposure touch point itself, as well as the exposure touch points after that turning exposure touch point and before the next turning exposure touch point, are included in the delivery convergence interval corresponding to that turning exposure touch point. The exposure touchpoints before each subsequent turning exposure touchpoint and after the previous turning exposure touchpoint are included in the next deployment exploration range, and the exposure touchpoints after the last turning exposure touchpoint are included in the last deployment convergence range.

7. The method of claim 1, wherein, Within the aforementioned exploration range, the interest retention intensity sequence is used as the state input for reinforcement learning to generate a set of exploration actions that allow exposure touchpoints that have not yet decreased to be sampled. Specifically, this includes: Within the aforementioned exploration range, the sequence of interest retention intensity falling within the exploration range is input into the state encoding layer of the reinforcement learning policy network in the order of exposure touchpoints to obtain state feature representations. Based on the state characteristics, exposure touchpoints whose interest retention intensity is still rising are selected within the exploration range to form a candidate exposure touchpoint set. Using the set of candidate exposure touchpoints as the action space, the reinforcement learning policy network samples and generates a set of exploratory actions to continue deploying to the candidate exposure touchpoints.

8. A system for performing the method of any one of claims 1 to 7. The system includes: The acquisition module is used to acquire user behavior sequences at the exposure level during the advertising campaign period, and segment them according to exposure touchpoints to obtain a touchpoint segment sequence consisting of continuous non-converting exposure segments and conversion termination exposure segments, and determine the interest retention intensity sequence corresponding to each continuous non-converting exposure segment. The processing module is used to identify the turning exposure touchpoints based on the interest retention intensity sequence, and divide the touchpoint segment sequence into a deployment exploration interval and a deployment convergence interval with the turning exposure touchpoints as the boundary. The processing module is further configured to, within the exploration range, use the interest retention intensity sequence as the state input for reinforcement learning, generate a set of exploration actions that allow exposure touchpoints that have not yet decreased to be sampled again, and apply them to the current delivery strategy to obtain the post-exploration delivery strategy; The execution module is used to determine the correction magnitude and adjust the explored delivery strategy accordingly within the delivery convergence interval, using the rate of decline of the interest retention intensity sequence after the turning exposure touchpoint as a reward signal for reinforcement learning, and then applying the adjusted delivery strategy to the next advertising delivery cycle.

9. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing code, and the processor being configured to retrieve the code and execute the reinforcement learning-based intelligent advertising delivery optimization method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the reinforcement learning-based intelligent delivery optimization method for e-commerce advertising as described in any one of claims 1 to 7.