Media information delivery strategy optimization method and system based on reinforcement learning

Abstract

The application provides a media information delivery strategy optimization method and system based on reinforcement learning, relates to the technical field of information delivery, and comprises the following steps: analyzing a user behavior sequence, detecting a mutation point, and constructing a hidden Markov model to quantify a behavior mode drift; generating candidate actions for a current environment, using a model to simulate and predict a state transition path and a drift amplitude, and screening out to-be-verified actions meeting conditions; setting an execution observation window to monitor a real-time data distribution offset, deciding to execute or withdraw an action, and finally updating a model and related threshold parameters according to an execution reward. The application realizes dynamic optimization and risk control of a delivery strategy, and improves delivery effect and stability.

Description

Technical Field

[0001] This invention relates to the field of information delivery technology, and in particular to a method and system for optimizing media information delivery strategies based on reinforcement learning. Background Technology

[0002] In the fields of digital marketing and media delivery, optimizing delivery strategies to improve user conversion rates and advertising effectiveness is a core objective. Existing technologies typically rely on reinforcement learning frameworks, which learn optimal strategies through continuous interaction between the agent and the delivery environment. Based on the current environmental state and historical interaction data, candidate delivery actions are directly generated, and pre-trained models, such as deep Q-networks or policy gradient models, are used to evaluate the expected long-term rewards of these actions. The action with the highest reward estimate is selected for execution, and the model parameters are updated based on the actual reward feedback obtained after execution, thereby iteratively optimizing the strategy. This process emphasizes learning stable patterns from historical data and assumes that user behavior and environmental dynamics are relatively stable over a certain period of time.

[0003] However, these conventional reinforcement learning methods face significant challenges in practical applications. User interests, market trends, or the competitive environment can shift rapidly or abruptly, and models trained based on the assumption of historical stability struggle to capture this non-stationarity. This leads to the continuous execution of actions learned from outdated patterns, causing a mismatch between the delivery strategy and the current real-world environment. This not only wastes delivery resources but also damages user experience and brand image by pushing irrelevant information. Another common problem is the weak risk control mechanism when exploring new strategies. When trying new actions that may bring high returns but are uncertain, there is a lack of a careful verification and safety buffer mechanism based on real-time data before final execution. If the prediction on which the new action is based is biased, direct execution will immediately lead to negative business results. The learning mechanism can only slowly correct itself afterward through negative rewards. The entire process lacks agile intervention and rollback capabilities, making the strategy optimization process costly and risky. Summary of the Invention

[0004] This invention provides a method and system for optimizing media information delivery strategies based on reinforcement learning, which can solve the problems in the prior art.

[0005] A first aspect of this invention provides a method for optimizing media information delivery strategies based on reinforcement learning, comprising:

[0006] Obtain the current deployment environment status and historical deployment data;

[0007] Extract user behavior sequences from historical delivery data, detect mutation points in user behavior sequences and divide them into multiple behavior segments based on mutation points, construct hidden Markov models for each behavior segment to obtain segment state transition matrices, and compare adjacent segment state transition matrices to quantify the drift magnitude of behavior patterns.

[0008] Multiple candidate delivery actions are generated based on the current delivery environment. Each candidate delivery action is substituted into a hidden Markov model to simulate the state evolution, predict the state transition path and behavior pattern drift of each candidate delivery action, and select the candidate delivery actions whose behavior pattern drift exceeds the preset drift threshold and whose state transition path terminates at the transformation state as delivery actions to be verified.

[0009] Set up an execution observation window, collect real-time data streams of the current deployment environment status within the execution observation window, and calculate the data distribution offset between the data distribution and the path anchor point distribution. When the data distribution offset is greater than or equal to a preset safety threshold, record negative feedback and trigger action rollback. When the execution observation window ends and the data distribution offset is less than the safety threshold, execute the deployment action to be verified.

[0010] The execution results are collected to calculate the reward value, and the hidden Markov model parameters, drift threshold, and safety threshold are updated based on the reinforcement learning mechanism using the reward value.

[0011] In one optional embodiment, user behavior sequences are extracted from historical delivery data, abrupt changes in the user behavior sequences are detected, and the sequences are segmented into multiple behavior segments based on these abrupt changes. A hidden Markov model is constructed for each behavior segment to obtain a segment state transition matrix. The magnitude of behavior pattern drift is quantified by comparing the state transition matrices of adjacent segments, including:

[0012] Extract user behavior sequences arranged in chronological order from historical data, use a sliding window to scan the user behavior sequences to obtain multiple behavior windows, and calculate the behavior type distribution entropy and behavior time interval variance within each behavior window;

[0013] The length of the sliding window is dynamically adjusted based on the distribution entropy of the behavior type of the current behavior window.

[0014] Calculate the difference in behavior type distribution entropy and the difference in behavior temporal interval variance between adjacent behavior windows to determine the comprehensive difference between windows. Based on the comprehensive difference sequence between windows, identify local maxima. Mark the window boundaries corresponding to the local maxima as mutation points. Use the mutation points as the dividing boundaries to divide the user behavior sequence into multiple behavior segments.

[0015] For each behavior segment, a hidden Markov model is constructed to obtain the segment state transition matrix corresponding to the behavior segment;

[0016] Extract the state transition matrix of adjacent behavior segments, calculate the difference in state transition probability and the difference in transition path structure between the state transition matrices, and obtain the quantized value of the behavior pattern drift amplitude by weighted summation.

[0017] In one optional embodiment, constructing a hidden Markov model for each behavior segment to obtain the segment state transition matrix corresponding to the behavior segment includes:

[0018] The sequence of user interaction behavior types within a behavior segment is used as an observation sequence. The number of behavior types in the observation sequence is counted to determine the number of hidden states in the Hidden Markov Model.

[0019] Initialize the state transition probability matrix and observation probability matrix of the Hidden Markov Model;

[0020] The observation sequence is traversed to identify observation locations containing transformation behaviors, and behavioral subsequences of a preset length are extracted from each observation location as key behavioral subsequences.

[0021] The frequency of occurrence of the key behavior subsequence in the observation sequence is counted, and the ratio of the frequency of occurrence to the total length of the observation sequence is calculated to obtain the key behavior weight coefficient.

[0022] The state transition probability matrix and the observation probability matrix are iteratively updated using the expectation-maximization algorithm. During each update of the state transition probability matrix, the hidden state transition path corresponding to the key action subsequence in the hidden state space is calculated, and a key action weight coefficient is applied to the state transition probabilities in the hidden state transition path.

[0023] The iteration stops when the parameter changes of the state transition probability matrix and the observation probability matrix are less than a preset convergence threshold, and the current state transition probability matrix is ​​output as the segment state transition matrix corresponding to the behavior segment.

[0024] In one optional embodiment, multiple candidate delivery actions are generated based on the current delivery environment state. Each candidate delivery action is substituted into a Hidden Markov Model for state evolution simulation to predict the state transition path and behavior pattern drift amplitude of each candidate delivery action. Candidate delivery actions whose behavior pattern drift amplitude exceeds a preset drift threshold and whose state transition path terminates at a transformation state are selected as delivery actions to be verified, including:

[0025] Extract user characteristics, time characteristics, and content characteristics from the current delivery environment to generate multiple candidate delivery actions;

[0026] Select the hidden Markov model corresponding to the most recent behavior segment, take each candidate delivery action as the initial observation input, execute the forward algorithm to calculate the hidden state sequence corresponding to each candidate delivery action, and extract the transition between adjacent hidden states in the hidden state sequence as the state transition path.

[0027] Traverse the state transition paths of each candidate delivery action, determine whether the terminating hidden state of the state transition path corresponds to the transformation behavior observation, and filter out the candidate delivery actions whose state transition paths terminate at the transformation state.

[0028] The state transition path of each candidate action is matched with the fragment state transition matrix of the adjacent behavior fragments, and the magnitude of behavior pattern drift caused by each candidate action is predicted based on the degree of difference in path transition probabilities.

[0029] Candidate campaign actions that filter behavioral pattern drift amplitudes exceeding a preset drift threshold are selected as campaign actions to be verified. Candidate campaign actions that simultaneously satisfy the condition that the state transition path terminates at the conversion state and the behavioral pattern drift amplitude exceeds the preset drift threshold are selected as campaign actions to be verified.

[0030] In one optional embodiment, the state transition paths of each candidate delivery action are traversed, and it is determined whether the terminating hidden state of the state transition path corresponds to a transformation behavior observation. Candidate delivery actions whose state transition paths terminate at a transformation state are selected, including:

[0031] Extract user behavior sequences containing conversion behaviors from historical campaign data, and statistically analyze the last observed behavior type before the conversion occurred.

[0032] In the Hidden Markov Model, the observation probability of the last observed behavior before the transformation behavior is generated in each hidden state is calculated, and the hidden state whose observation probability exceeds the preset probability threshold is marked as the transformation state.

[0033] Traverse the state transition paths of each candidate action and extract the terminating hidden state of the state transition path.

[0034] Determine whether the terminating hidden state belongs to the set of marked transition states, and filter out candidate delivery actions whose state transition paths terminate at the transition state.

[0035] In one optional embodiment, an execution observation window is set up. Within this window, real-time data streams of the current deployment environment are collected, and the data distribution offset between the current data stream and the path anchor point distribution is calculated. When the data distribution offset is greater than or equal to a preset safety threshold, negative feedback is recorded, and action reversal is triggered. When the execution observation window ends and the data distribution offset is less than the safety threshold, the deployment action to be verified is executed, including:

[0036] Set the execution observation window duration according to the state transition path corresponding to the action to be verified. Collect real-time data stream of the current delivery environment state by time step within the execution observation window and extract user behavior features from the real-time data stream.

[0037] From historical deployment data, extract historical observation data corresponding to each hidden state node in the state transition path of the deployment action to be verified, and construct the conditional probability distribution of each hidden state node as the path anchor point distribution set.

[0038] The real-time data stream is mapped to the hidden state node sequence of the state transition path of the action to be verified according to time steps. The distribution deviation between the real-time observation data of each time step and the path anchor point distribution of the corresponding hidden state node is calculated. The distribution deviation of each time step is accumulated by time weighting to obtain the data distribution offset.

[0039] When the data distribution offset is greater than or equal to the preset safety threshold, negative feedback is recorded and action rollback is triggered. When the observation window ends and the data distribution offset is less than the safety threshold, the action to be verified is executed.

[0040] In one optional embodiment, mapping the real-time data stream to the hidden state node sequence of the state transition path of the action to be verified by time steps includes:

[0041] The real-time data stream is divided into ordered real-time observation data sequences according to the acquisition time step;

[0042] Extract the hidden state node sequence from the state transition path of the action to be verified, and obtain the length of the hidden state node sequence and the expected dwell time of each hidden state node.

[0043] Construct a time alignment matrix between the real-time observation data sequence and the hidden state node sequence, and calculate the observation probability matching degree between each time step in the real-time observation data sequence and each hidden state node in the hidden state node sequence.

[0044] Based on the observation probability matching degree and the expected dwell time of the hidden state node, dynamic time warping is performed to calculate the optimal time alignment path;

[0045] Based on the optimal time alignment path, each time step in the real-time observation data sequence is mapped to the corresponding hidden state node in the hidden state node sequence, thus establishing the mapping relationship between time steps and hidden state nodes.

[0046] A second aspect of this invention provides a media information delivery strategy optimization system based on reinforcement learning, comprising:

[0047] The status acquisition unit is used to acquire the current deployment environment status and historical deployment data;

[0048] The behavior analysis unit is used to extract user behavior sequences from historical delivery data, detect mutation points in user behavior sequences and divide them into multiple behavior segments based on mutation points, construct hidden Markov models for each behavior segment to obtain the segment state transition matrix, and compare the state transition matrices of adjacent segments to quantify the drift amplitude of behavior patterns.

[0049] The action generation unit is used to generate multiple candidate action for the current delivery environment state, substitute each candidate action into the Hidden Markov Model to simulate the state evolution, predict the state transition path and behavior pattern drift of each candidate action, and select the candidate action whose behavior pattern drift exceeds the preset drift threshold and whose state transition path terminates in the transformation state as the action to be verified.

[0050] The safety verification unit is used to set the execution observation window, collect the real-time data stream of the current deployment environment status within the execution observation window and calculate the data distribution offset between the data distribution and the path anchor point distribution. When the data distribution offset is greater than or equal to the preset safety threshold, negative feedback is recorded and the action is triggered to revert. When the execution observation window ends and the data distribution offset is less than the safety threshold, the deployment action to be verified is executed.

[0051] The model update unit is used to collect execution results and calculate reward values. Based on the reinforcement learning mechanism, the reward values ​​are used to update the parameters, drift threshold, and safety threshold of the Hidden Markov Model.

[0052] A third aspect of the present invention provides an electronic device, comprising:

[0053] processor;

[0054] Memory used to store processor-executable instructions;

[0055] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0056] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0057] In this embodiment of the invention, dynamic changes in user behavior patterns are effectively identified. By detecting and segmenting abrupt changes in user behavior sequences, continuous behavior data is divided into multiple relatively stable behavior segments, providing a quantitative basis for understanding the non-stationary evolution of user interests and behaviors, and enabling the perception of changes in macro-behavioral trends. After generating multiple candidate actions for the current environmental state, a forward state evolution simulation is performed using a pre-constructed Hidden Markov Model (HMM) to predict the state transition path triggered by each action and the resulting behavioral pattern drift magnitude. By screening actions that can both trigger significant behavioral pattern changes and ultimately lead to a transformation state, exploratory strategies that break with convention and effectively guide users towards the target behavior are identified. By introducing an execution observation window and a dynamic safety mechanism, the execution risks brought by exploratory actions are significantly reduced. The self-iteration and adaptive adjustment of strategy optimization parameters are realized. By collecting the results after action execution and calculating the reward value, a reinforcement learning mechanism is used to simultaneously update the parameters of the HMM, the judgment threshold for behavioral pattern drift, and the offset threshold for safety monitoring. Attached Figure Description

[0058] Figure 1 This is a flowchart illustrating the media information delivery strategy optimization method based on reinforcement learning, as described in an embodiment of the present invention.

[0059] Figure 2 This is for the deployment of action state transition and verification process. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0061] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0062] Figure 1 This is a flowchart illustrating the media information delivery strategy optimization method based on reinforcement learning according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0063] Obtain the current deployment environment status and historical deployment data;

[0064] Extract user behavior sequences from historical delivery data, detect mutation points in user behavior sequences and divide them into multiple behavior segments based on mutation points, construct hidden Markov models for each behavior segment to obtain segment state transition matrices, and compare adjacent segment state transition matrices to quantify the drift magnitude of behavior patterns.

[0065] Multiple candidate delivery actions are generated based on the current delivery environment. Each candidate delivery action is substituted into a hidden Markov model to simulate the state evolution, predict the state transition path and behavior pattern drift of each candidate delivery action, and select the candidate delivery actions whose behavior pattern drift exceeds the preset drift threshold and whose state transition path terminates at the transformation state as delivery actions to be verified.

[0066] Set up an execution observation window, collect real-time data streams of the current deployment environment status within the execution observation window, and calculate the data distribution offset between the data distribution and the path anchor point distribution. When the data distribution offset is greater than or equal to a preset safety threshold, record negative feedback and trigger action rollback. When the execution observation window ends and the data distribution offset is less than the safety threshold, execute the deployment action to be verified.

[0067] The execution results are collected to calculate the reward value, and the hidden Markov model parameters, drift threshold, and safety threshold are updated based on the reinforcement learning mechanism using the reward value.

[0068] In one optional embodiment, user behavior sequences are extracted from historical delivery data, abrupt changes in the user behavior sequences are detected, and the sequences are segmented into multiple behavior segments based on these abrupt changes. A hidden Markov model is constructed for each behavior segment to obtain a segment state transition matrix. The magnitude of behavior pattern drift is quantified by comparing the state transition matrices of adjacent segments, including:

[0069] Extract user behavior sequences arranged in chronological order from historical data, use a sliding window to scan the user behavior sequences to obtain multiple behavior windows, and calculate the behavior type distribution entropy and behavior time interval variance within each behavior window;

[0070] The length of the sliding window is dynamically adjusted based on the distribution entropy of the behavior type of the current behavior window.

[0071] Calculate the difference in behavior type distribution entropy and the difference in behavior temporal interval variance between adjacent behavior windows to determine the comprehensive difference between windows. Based on the comprehensive difference sequence between windows, identify local maxima. Mark the window boundaries corresponding to the local maxima as mutation points. Use the mutation points as the dividing boundaries to divide the user behavior sequence into multiple behavior segments.

[0072] For each behavior segment, a hidden Markov model is constructed to obtain the segment state transition matrix corresponding to the behavior segment;

[0073] Extract the state transition matrix of adjacent behavior segments, calculate the difference in state transition probability and the difference in transition path structure between the state transition matrices, and obtain the quantized value of the behavior pattern drift amplitude by weighted summation.

[0074] In one specific implementation, user behavior sequences are extracted from historical campaign data in ascending order of timestamps. Behavior types include discrete events such as exposure, clicks, dwell time, and conversions. An initial sliding window length of 50 behavior events is set, and the user behavior sequences are scanned. For each behavior window, the frequency of occurrence of each behavior type within the window is counted, and the behavior type distribution entropy is calculated. Specifically, based on the frequency of occurrence of each behavior type within the window, its product with the corresponding logarithm is summed and the negative value is taken to obtain the behavior type distribution entropy. Simultaneously, the time intervals between adjacent behavior events within the window are extracted, and the variance of all time intervals is calculated as the behavior time-series interval variance. This indicator reflects the stability of the user behavior rhythm.

[0075] The sliding window length is dynamically adjusted based on the behavior type distribution entropy of the current behavior window. When the behavior type distribution entropy is higher than 2.5, it indicates that the behavior patterns within the window are complex and diverse, and the window length is increased to 1.2 times to capture more complete pattern information; when the behavior type distribution entropy is lower than 1.2, it indicates that the behavior patterns are simple, and the window length is reduced to 0.8 times to improve the sensitivity of mutation detection. The window length is limited to between 30 and 100 events.

[0076] Calculate the absolute value of the difference in entropy between the behavior type distributions of two adjacent behavior windows, and simultaneously calculate the absolute value of the difference in variance between the behavior time intervals. Standardize the entropy difference and assign it a weight of 0.6, and standardize the variance difference and assign it a weight of 0.4. Summate these weighted values ​​to obtain the overall inter-window difference. Iterate through all adjacent window pairs to form an inter-window difference sequence. Find local maxima in this sequence, determined by the following criteria: the current difference is simultaneously greater than the difference values ​​of the three positions before and after it, and the difference exceeds 1.5 times the global mean. Mark the window boundary moments corresponding to the identified local maxima as mutation points, and use these mutation points as dividing boundaries to segment the user behavior sequence into multiple behavior segments.

[0077] For each behavioral segment, an observation sequence is constructed based on the behavioral type, sequence of occurrence, and time interval characteristics within the segment. Behaviors such as exposure, clicks, dwell time, and conversion are encoded into discrete observations in chronological order, and characteristics such as behavioral frequency and dwell time are used as observation inputs. Based on this, the hidden state set is defined into four categories: exploration state, interest state, decision state, and churn state. A Hidden Markov Model (HMM) is used to model the observation sequence. During model training, the algorithm automatically learns the probability distribution of the hidden state corresponding to each observation, thus obtaining the hidden state sequence corresponding to the observation sequence. The Baum-Welch algorithm is used to train the HMM, iteratively optimizing the state transition probability matrix, observation probability matrix, and initial state probability vector. The iteration terminates when the change in the log-likelihood function is less than 0.001 or after 200 iterations. The trained state transition probability matrix is ​​extracted as the segment state transition matrix corresponding to the behavioral segment. The matrix has a dimension of 4×4, and its elements represent the probability of transitioning from one hidden state to another.

[0078] Extract the state transition matrix for two time-adjacent behavioral segments. Calculate the difference in state transition probabilities, measured by the square root of the sum of squared differences between corresponding elements. Calculate the difference in transition path structure, considering transition edges with state transition probabilities greater than 0.1 as valid paths. Construct a directed graph, and use graph edit distance to measure the structural difference between the two transition graphs. Graph edit distance is obtained by calculating the minimum number of operations required to delete nodes, edges, and edges. Standardize the state transition probability difference and assign it a weight of 0.7, and standardize the transition path structure difference and assign it a weight of 0.3. Sum these weighted values ​​to obtain a quantified value of the behavioral pattern drift amplitude. The larger this value, the more significant the change in user behavior pattern between adjacent time periods.

[0079] In one optional embodiment, constructing a hidden Markov model for each behavior segment to obtain the segment state transition matrix corresponding to the behavior segment includes:

[0080] The sequence of user interaction behavior types within a behavior segment is used as an observation sequence. The number of behavior types in the observation sequence is counted to determine the number of hidden states in the Hidden Markov Model.

[0081] Initialize the state transition probability matrix and observation probability matrix of the Hidden Markov Model;

[0082] The observation sequence is traversed to identify observation locations containing transformation behaviors, and behavioral subsequences of a preset length are extracted from each observation location as key behavioral subsequences.

[0083] The frequency of occurrence of the key behavior subsequence in the observation sequence is counted, and the ratio of the frequency of occurrence to the total length of the observation sequence is calculated to obtain the key behavior weight coefficient.

[0084] The state transition probability matrix and the observation probability matrix are iteratively updated using the expectation-maximization algorithm. During each update of the state transition probability matrix, the hidden state transition path corresponding to the key action subsequence in the hidden state space is calculated, and a key action weight coefficient is applied to the state transition probabilities in the hidden state transition path.

[0085] The iteration stops when the parameter changes of the state transition probability matrix and the observation probability matrix are less than a preset convergence threshold, and the current state transition probability matrix is ​​output as the segment state transition matrix corresponding to the behavior segment.

[0086] In one specific implementation, the extracted sequence of user interaction behaviors within the behavioral fragment is used as the input observation sequence. The observation sequence may include multiple behavior types such as browsing, clicking, adding to cart, favoriting, and conversion. The number of distinct behavior types in the observation sequence is counted; for example, when the observation sequence includes four behavior types (browsing, clicking, adding to cart, and conversion), the number of behavior types is 4. Based on this number, the number of hidden states in the Hidden Markov Model is determined. Typically, the number of hidden states can be set to the number of behavior types plus 1 or 2 to capture implicit transitions between different user interest states.

[0087] During initialization, the elements of the state transition probability matrix are set to uniformly distributed values ​​or randomly initialized, and the observation probability matrix is ​​also randomly initialized. The entire observation sequence is traversed to identify all observation location indices containing transformation behaviors. Tracing back from each transformation behavior location, a fixed-length behavior subsequence is extracted as the key behavior subsequence. This preset length can be set to 3 to 7 behavior steps depending on the business scenario. For example, when the transformation behavior is at index 15, a subsequence of length 5 is extracted forward, i.e., the behavior sequence from index 10 to 14.

[0088] The total frequency of each key behavior subsequence in the complete observed sequence is counted. The ratio of this frequency to the total length of the observed sequence is calculated to obtain the key behavior weight coefficient. If a key behavior subsequence appears 10 times and the total length of the observed sequence is 200, the weight coefficient is 0.05. This coefficient quantifies the correlation strength between a specific behavior pattern and the transformation result.

[0089] The expectation-maximization (EM) algorithm is used for iterative parameter updates. In the E-step, the forward-backward algorithm is used to calculate the posterior probability distribution of the observed sequence given the current model parameters. In the M-step, when updating the state transition probability matrix, special processing is performed on key action subsequences: the key action subsequences are mapped to the hidden state space, and the most probable hidden state transition path is obtained through Viterbi decoding. The state transition probability values ​​involved in this path are multiplied by the corresponding key action weight coefficient to enhance them, giving the state transition path leading to transformation a higher probability weight. When updating the observation probability matrix, the standard EEM algorithm's update formula is used to calculate the probability of emitting each observed symbol in each hidden state.

[0090] After each iteration, the changes in all elements of the state transition probability matrix and the observation probability matrix are calculated. The L1 norm or maximum absolute difference of the matrix element differences is used as a measure of parameter change. When this change is less than a preset convergence threshold, such as 0.001, the model parameters are considered to have converged, and the iteration process is terminated. The state transition probability matrix at this point is output as the segment state transition matrix corresponding to this behavior segment. This matrix depicts the latent state evolution of the user within the time window of this behavior segment. The state transition path leading to transformation is significantly strengthened by the application of key behavior weight coefficients, enabling the model to more accurately identify high-value user behavior patterns.

[0091] In one optional embodiment, multiple candidate delivery actions are generated based on the current delivery environment state. Each candidate delivery action is substituted into a Hidden Markov Model for state evolution simulation to predict the state transition path and behavior pattern drift amplitude of each candidate delivery action. Candidate delivery actions whose behavior pattern drift amplitude exceeds a preset drift threshold and whose state transition path terminates at a transformation state are selected as delivery actions to be verified, including:

[0092] Extract user characteristics, time characteristics, and content characteristics from the current delivery environment to generate multiple candidate delivery actions;

[0093] Select the hidden Markov model corresponding to the most recent behavior segment, take each candidate delivery action as the initial observation input, execute the forward algorithm to calculate the hidden state sequence corresponding to each candidate delivery action, and extract the transition between adjacent hidden states in the hidden state sequence as the state transition path.

[0094] Traverse the state transition paths of each candidate delivery action, determine whether the terminating hidden state of the state transition path corresponds to the transformation behavior observation, and filter out the candidate delivery actions whose state transition paths terminate at the transformation state.

[0095] The state transition path of each candidate action is matched with the fragment state transition matrix of the adjacent behavior fragments, and the magnitude of behavior pattern drift caused by each candidate action is predicted based on the degree of difference in path transition probabilities.

[0096] Candidate campaign actions that filter behavioral pattern drift amplitudes exceeding a preset drift threshold are selected as campaign actions to be verified. Candidate campaign actions that simultaneously satisfy the condition that the state transition path terminates at the conversion state and the behavioral pattern drift amplitude exceeds the preset drift threshold are selected as campaign actions to be verified.

[0097] In one specific implementation, the candidate campaign generation stage needs to comprehensively consider multi-dimensional features. Specifically, user features are extracted from the current campaign environment, including historical click-through rates, dwell time distribution, and interest tag vectors; time features are extracted, including current time period, weekday, and holiday identifiers; and content features are extracted, including media material type, creative element combination, and copy keywords. Based on these features, an action generator is constructed, which uses permutation and combination to generate a set of candidate campaign actions. Each candidate campaign action includes specific parameters such as campaign time period, material number, and target audience tags.

[0098] In the state evolution simulation phase, the Hidden Markov Model (HMM) corresponding to the behavior segment closest to the current time is selected as the evolutionary basis. The set of observed states in this model includes user behaviors such as browsing, clicking, adding to cart, and conversion, while the set of hidden states represents the user's intrinsic interest level. Candidate actions are transformed into observation sequences as input. A forward algorithm is executed to calculate the probability distribution of hidden states at each time step under this observation sequence, and the optimal hidden state sequence is obtained through Viterbi decoding. For example, a candidate action might generate a hidden state sequence of "low interest → medium interest → high interest → conversion intention." The transition relationships between adjacent hidden states are extracted, namely "low interest to medium interest," "medium interest to high interest," and "high interest to conversion intention," forming the state transition path of the candidate action.

[0099] When determining the termination state of a state transition path, it checks whether the set of observed states corresponding to the hidden state at the end of the path contains a transition behavior. Specifically, a mapping table between hidden states and observed states is established. If the emission probability of the transition behavior in the observation probability distribution of the terminating hidden state is greater than 0.6, the path is determined to terminate at a transition state. All candidate actions are traversed, and actions that meet this condition are marked as valid candidates.

[0100] The behavior pattern drift amplitude is calculated using a path matching method. The segment state transition matrix between two adjacent behavior segments is extracted and denoted as P. t-1 and P t The state transition path of the candidate delivery action is represented as a transition sequence. For each transition relation in the path, query its value in P. t-1and P t The transition probabilities are calculated, and the absolute values ​​of the probability differences are summed. For example, the transition from "low interest to medium interest" is calculated in P... t-1 The probability is 0.3 in P. t The probability is 0.5, and the difference is 0.2. The difference values ​​of all transitions along the path are summed and normalized to obtain the behavioral pattern drift magnitude of the candidate action.

[0101] The screening process employs a dual-criteria approach. A preset drift threshold of 0.25 is set to filter out candidate actions whose behavioral pattern drift exceeds this threshold. Simultaneously, a conversion state termination determination is applied to these actions. Only candidate actions that simultaneously meet both conditions—"the state transition path terminates at the conversion state" and "the behavioral pattern drift is greater than 0.25"—are designated as pending verification actions. This dual-screening mechanism ensures that the actions are both exploratory and possess the potential to achieve conversion.

[0102] In one optional embodiment, the state transition paths of each candidate delivery action are traversed, and it is determined whether the terminating hidden state of the state transition path corresponds to a transformation behavior observation. Candidate delivery actions whose state transition paths terminate at a transformation state are selected, including:

[0103] Extract user behavior sequences containing conversion behaviors from historical campaign data, and statistically analyze the last observed behavior type before the conversion occurred.

[0104] In the Hidden Markov Model, the observation probability of the last observed behavior before the transformation behavior is generated in each hidden state is calculated, and the hidden state whose observation probability exceeds the preset probability threshold is marked as the transformation state.

[0105] Traverse the state transition paths of each candidate action and extract the terminating hidden state of the state transition path.

[0106] Determine whether the terminating hidden state belongs to the set of marked transition states, and filter out candidate delivery actions whose state transition paths terminate at the transition state.

[0107] In one specific implementation, during the candidate campaign filtering process, it is necessary to establish a correspondence between conversion states and hidden states in a Hidden Markov Model. User behavior sequences containing conversion behaviors are extracted from historical campaign data. These conversion behaviors can be explicit business conversion actions such as completing a purchase, submitting an order, or registering as a member. For each sequence containing a conversion behavior, the type of the last observed behavior before the conversion occurs is statistically analyzed; for example, the last observed behavior before conversion might be "clicking the details page" or "adding to cart." By statistically analyzing a large number of conversion sequences, the distribution characteristics of the observed behaviors before conversion can be obtained.

[0108] Within the framework of a Hidden Markov Model, the observation probability of the last observed behavior before the transformation behavior is generated in each hidden state is calculated. Specifically, for hidden state s... i Observation probability This indicates that a specific prior observation behavior is generated in this hidden state. The probability is calculated using the model's observation matrix, which records the probability distribution of each hidden state generating various observation behaviors. The calculated observation probability is compared to a preset probability threshold, which is set according to business requirements, typically ranging from 0.6 to 0.8. When the hidden state... When the observation probability of generating the pre-transformation observation behavior exceeds the threshold, the hidden state is marked as a transformation state, forming a set of transformation states.

[0109] For each candidate delivery action, its state evolution simulation generates a state transition path, which consists of a series of hidden state nodes. The state transition paths corresponding to each candidate delivery action are traversed, and the terminating hidden state of the path is extracted—that is, the hidden state node finally reached by the path. It is determined whether this terminating hidden state belongs to the marked set of conversion states through a set member judgment operation. If the terminating hidden state is in the set of conversion states, it means that the state evolution path guided by the candidate delivery action has finally reached a state capable of generating pre-conversion observation behaviors, indicating that the delivery action has the potential to facilitate user conversion.

[0110] Based on the filtering criteria of behavioral pattern drift amplitude, candidate campaign actions that simultaneously meet two conditions—a behavioral pattern drift amplitude exceeding a preset drift threshold and a state transition path terminating at a conversion state—are selected as campaign actions to be validated. The behavioral pattern drift amplitude is calculated by comparing the state transition matrices of adjacent behavioral segments, and the norm of the difference between matrix elements can be used as a metric. The preset drift threshold is set based on historical campaign performance data to ensure that campaign actions that can trigger significant changes in user behavior patterns and lead to conversion goals are selected. This filtering mechanism ensures that the campaign strategy is both exploratory and conversion-oriented, avoiding the blind execution of unverified campaign actions and improving the efficiency and safety of campaign strategy optimization.

[0111] In one optional embodiment, an execution observation window is set up. Within this window, real-time data streams of the current deployment environment are collected, and the data distribution offset between the current data stream and the path anchor point distribution is calculated. When the data distribution offset is greater than or equal to a preset safety threshold, negative feedback is recorded, and action reversal is triggered. When the execution observation window ends and the data distribution offset is less than the safety threshold, the deployment action to be verified is executed, including:

[0112] Set the execution observation window duration according to the state transition path corresponding to the action to be verified. Collect real-time data stream of the current delivery environment state by time step within the execution observation window and extract user behavior features from the real-time data stream.

[0113] From historical deployment data, extract historical observation data corresponding to each hidden state node in the state transition path of the deployment action to be verified, and construct the conditional probability distribution of each hidden state node as the path anchor point distribution set.

[0114] The real-time data stream is mapped to the hidden state node sequence of the state transition path of the action to be verified according to time steps. The distribution deviation between the real-time observation data of each time step and the path anchor point distribution of the corresponding hidden state node is calculated. The distribution deviation of each time step is accumulated by time weighting to obtain the data distribution offset.

[0115] When the data distribution offset is greater than or equal to the preset safety threshold, negative feedback is recorded and action rollback is triggered. When the observation window ends and the data distribution offset is less than the safety threshold, the action to be verified is executed.

[0116] In one specific implementation, after determining the action to be verified, a pre-execution security monitoring mechanism needs to be established. The system determines the execution observation window length based on the length of the state transition path corresponding to the action to be verified. The state transition path includes all hidden state nodes traversed from the current state to the expected transition state; a longer path means a longer observation time is required. Specifically, if the state transition path contains n hidden state nodes, the execution observation window length is set to n multiplied by the unit time step, which can be set to 5 minutes to 30 minutes.

[0117] Within the observation window, the system collects real-time data streams of the current deployment environment at fixed time steps. These real-time data streams include, but are not limited to, metrics such as user click-through rate (CTR), dwell time, bounce rate, and interaction depth. User behavior features extracted from the real-time data streams must be consistent with the observed variables defined in the Hidden Markov Model (HMM), for example, combining CTR and dwell time to form a two-dimensional feature vector.

[0118] To establish a benchmark, the system retrieves historical observation data associated with each hidden state node in the state transition path of the deployment action to be verified from historical deployment data. For each hidden state node in the path, observation data samples of that node at all historical occurrence times are collected, and a conditional probability distribution for that hidden state node is constructed using kernel density estimation or a Gaussian mixture model. This conditional probability distribution is the path anchor point distribution. The path anchor point distributions of all hidden state nodes constitute the path anchor point distribution set.

[0119] During the observation window, the system maps the real-time data collected at each time step to the sequence of hidden state nodes in the state transition path of the action to be verified, in chronological order. The mapping rule is: the real-time observation data at time step t corresponds to the t-th hidden state node in the path. The distribution deviation between the real-time observation data at each time step and the path anchor point distribution of the corresponding hidden state node is calculated. The distribution deviation is quantified using Wasserstein distance or Kullback-Leibler divergence; a larger value indicates a more severe deviation between the real-time data and the historical normal pattern.

[0120] The data distribution offset is obtained by accumulating the distribution deviations at each time step using a time-weighted method. The time-weighting mechanism assigns higher weights to deviations closer to the current time, and the weighting coefficients can adopt an exponential decay form. Where T is the current time and t is the historical time step, This is the attenuation coefficient. The data distribution offset is calculated as the cumulative sum of all weighted deviations.

[0121] After calculating the data distribution offset at each time step within the observation window, it is immediately compared with a preset safety threshold. When the data distribution offset is greater than or equal to the preset safety threshold, it is determined that the delivery environment has deviated from the expected trajectory and poses a risk. The system immediately records this negative feedback event, including the trigger time, offset value, current state characteristics, etc., and triggers the action rollback mechanism, stopping the execution of the delivery action to be verified and restoring to the previous stable delivery strategy. When the execution observation window ends and the data distribution offset remains less than the safety threshold throughout the period, it is confirmed that the delivery environment conforms to the expected evolution path, and the system officially executes the delivery action to be verified, applying it to the actual media information delivery.

[0122] like Figure 2 As shown, the process of state transition and verification for the deployment action is illustrated.

[0123] In one optional embodiment, mapping the real-time data stream to the hidden state node sequence of the state transition path of the action to be verified by time steps includes:

[0124] The real-time data stream is divided into ordered real-time observation data sequences according to the acquisition time step;

[0125] Extract the hidden state node sequence from the state transition path of the action to be verified, and obtain the length of the hidden state node sequence and the expected dwell time of each hidden state node.

[0126] Construct a time alignment matrix between the real-time observation data sequence and the hidden state node sequence, and calculate the observation probability matching degree between each time step in the real-time observation data sequence and each hidden state node in the hidden state node sequence.

[0127] Based on the observation probability matching degree and the expected dwell time of the hidden state node, dynamic time warping is performed to calculate the optimal time alignment path;

[0128] Based on the optimal time alignment path, each time step in the real-time observation data sequence is mapped to the corresponding hidden state node in the hidden state node sequence, thus establishing the mapping relationship between time steps and hidden state nodes.

[0129] In one specific implementation, within the execution observation window, the observed values ​​in the real-time data stream need to be aligned with the predicted state transition path of the action to be verified, thereby determining whether the actual execution process deviates from the expected path. Specifically, the real-time data stream is divided into multiple time steps according to the time sequence of data collection. Each time step corresponds to an observation data point, which reflects the user's behavioral characteristics at that moment, such as click-through rate, dwell time, and interaction depth. This division forms an ordered sequence of real-time observation data, the length of which depends on the duration of the execution observation window and the data collection frequency.

[0130] The hidden state node sequence is extracted from the state transition path of the action to be verified. This sequence records the expected hidden state changes that the user's behavior will undergo in the Hidden Markov Model. The total length of this hidden state node sequence is obtained, and the expected dwell time of each hidden state node is read from the training parameters of the Hidden Markov Model. This expected value represents the average dwell time of the user in a specific hidden state and is an empirical value derived from historical delivery data.

[0131] Since the number of time steps in the real-time observation data sequence is usually inconsistent with the number of nodes in the hidden state node sequence, and the temporal distribution of actual user behavior may differ from the model prediction, a time alignment matrix needs to be constructed to establish a mapping relationship between the two. The row indices of this alignment matrix correspond to each time step in the real-time observation data sequence, and the column indices correspond to each hidden state node in the hidden state node sequence. The observation probability matching degree of each element in the matrix is ​​calculated, specifically through the observation probability distribution function of the hidden Markov model: for the observation data at the i-th time step and the j-th hidden state node, the probability value of that hidden state node generating that observation data is calculated as the matching degree score.

[0132] A dynamic time warping algorithm is used to find the optimal time alignment path. This algorithm searches the time alignment matrix for a path with the highest cumulative matching degree from the top left to the bottom right. Constraints are imposed during the search: the path must be monotonically increasing and continuous; each hidden state node is allowed to correspond to multiple consecutive time steps to reflect differences in dwell time, but the cumulative number of corresponding time steps should be close to the expected dwell time of that hidden state node. The cumulative matching degree matrix is ​​calculated using dynamic programming. At each position, path scores from the left, top, and top left are compared, and the predecessor node with the highest cumulative score is selected. Simultaneously, soft constraints are applied to path selection based on the expected dwell time to avoid a hidden state node corresponding to too many or too few time steps.

[0133] Based on the optimal time alignment path obtained through backtracking, a mapping relationship is established between each time step in the real-time observation data sequence and the corresponding hidden state node in the hidden state node sequence. This mapping relationship indicates which expected hidden state each real-time observation data should belong to, providing a state-level benchmark for subsequent calculation of data distribution offset. Through this mapping relationship, the real-time observation data can be located in the hidden state space, and then accurately compared with the path anchor point distribution to determine whether the actual execution trajectory deviates from the predicted path.

[0134] The media information delivery strategy optimization system based on reinforcement learning according to embodiments of the present invention includes:

[0135] The status acquisition unit is used to acquire the current deployment environment status and historical deployment data;

[0136] The behavior analysis unit is used to extract user behavior sequences from historical delivery data, detect mutation points in user behavior sequences and divide them into multiple behavior segments based on mutation points, construct hidden Markov models for each behavior segment to obtain the segment state transition matrix, and compare the state transition matrices of adjacent segments to quantify the drift amplitude of behavior patterns.

[0137] The action generation unit is used to generate multiple candidate action for the current delivery environment state, substitute each candidate action into the Hidden Markov Model to simulate the state evolution, predict the state transition path and behavior pattern drift of each candidate action, and select the candidate action whose behavior pattern drift exceeds the preset drift threshold and whose state transition path terminates in the transformation state as the action to be verified.

[0138] The safety verification unit is used to set the execution observation window, collect the real-time data stream of the current deployment environment status within the execution observation window and calculate the data distribution offset between the data distribution and the path anchor point distribution. When the data distribution offset is greater than or equal to the preset safety threshold, negative feedback is recorded and the action is triggered to revert. When the execution observation window ends and the data distribution offset is less than the safety threshold, the deployment action to be verified is executed.

[0139] The model update unit is used to collect execution results and calculate reward values. Based on the reinforcement learning mechanism, the reward values ​​are used to update the parameters, drift threshold, and safety threshold of the Hidden Markov Model.

[0140] A third aspect of the present invention provides an electronic device, comprising:

[0141] processor;

[0142] Memory used to store processor-executable instructions;

[0143] The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0144] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0145] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0146] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A media information delivery strategy optimization method based on reinforcement learning, characterized in that, include: Obtain the current deployment environment status and historical deployment data; Extract user behavior sequences from historical delivery data, detect mutation points in user behavior sequences and divide them into multiple behavior segments based on mutation points, construct hidden Markov models for each behavior segment to obtain segment state transition matrices, and compare adjacent segment state transition matrices to quantify the drift magnitude of behavior patterns. Multiple candidate delivery actions are generated based on the current delivery environment. Each candidate delivery action is substituted into a hidden Markov model to simulate the state evolution, predict the state transition path and behavior pattern drift of each candidate delivery action, and select the candidate delivery actions whose behavior pattern drift exceeds the preset drift threshold and whose state transition path terminates at the transformation state as delivery actions to be verified. Set up an execution observation window, collect real-time data streams of the current deployment environment status within the execution observation window, and calculate the data distribution offset between the data distribution and the path anchor point distribution. When the data distribution offset is greater than or equal to a preset safety threshold, record negative feedback and trigger action rollback. When the execution observation window ends and the data distribution offset is less than the safety threshold, execute the deployment action to be verified. The execution results are collected to calculate the reward value, and the hidden Markov model parameters, drift threshold, and safety threshold are updated based on the reinforcement learning mechanism using the reward value.

2. The method according to claim 1, characterized in that, User behavior sequences are extracted from historical campaign data. Abrupt points in these sequences are detected and segmented into multiple behavior segments. Hidden Markov models are constructed for each behavior segment to obtain its state transition matrix. The drift magnitude of behavior patterns is quantified by comparing the state transition matrices of adjacent segments. Extract user behavior sequences arranged in chronological order from historical data, use a sliding window to scan the user behavior sequences to obtain multiple behavior windows, and calculate the behavior type distribution entropy and behavior time interval variance within each behavior window; The length of the sliding window is dynamically adjusted based on the distribution entropy of the behavior type of the current behavior window. Calculate the difference in behavior type distribution entropy and the difference in behavior temporal interval variance between adjacent behavior windows to determine the comprehensive difference between windows. Based on the comprehensive difference sequence between windows, identify local maxima. Mark the window boundaries corresponding to the local maxima as mutation points. Use the mutation points as the dividing boundaries to divide the user behavior sequence into multiple behavior segments. For each behavior segment, a hidden Markov model is constructed to obtain the segment state transition matrix corresponding to the behavior segment; Extract the state transition matrix of adjacent behavior segments, calculate the difference in state transition probability and the difference in transition path structure between the state transition matrices, and obtain the quantized value of the behavior pattern drift amplitude by weighted summation.

3. The method according to claim 2, characterized in that, For each behavior segment, a Hidden Markov Model is constructed to obtain the segment state transition matrix corresponding to the behavior segment, including: The sequence of user interaction behavior types within a behavior segment is used as an observation sequence. The number of behavior types in the observation sequence is counted to determine the number of hidden states in the Hidden Markov Model. Initialize the state transition probability matrix and observation probability matrix of the Hidden Markov Model; The observation sequence is traversed to identify observation locations containing transformation behaviors, and behavioral subsequences of a preset length are extracted from each observation location as key behavioral subsequences. The frequency of occurrence of the key behavior subsequence in the observation sequence is counted, and the ratio of the frequency of occurrence to the total length of the observation sequence is calculated to obtain the key behavior weight coefficient. The state transition probability matrix and the observation probability matrix are iteratively updated using the expectation-maximization algorithm. During each update of the state transition probability matrix, the hidden state transition path corresponding to the key action subsequence in the hidden state space is calculated, and a key action weight coefficient is applied to the state transition probabilities in the hidden state transition path. The iteration stops when the parameter changes of the state transition probability matrix and the observation probability matrix are less than a preset convergence threshold, and the current state transition probability matrix is ​​output as the segment state transition matrix corresponding to the behavior segment.

4. The method according to claim 1, characterized in that, Multiple candidate delivery actions are generated based on the current delivery environment. Each candidate action is then substituted into a Hidden Markov Model (HMM) to simulate state evolution, predicting the state transition path and behavioral pattern drift magnitude of each candidate action. Candidate actions whose behavioral pattern drift magnitude exceeds a preset drift threshold and whose state transition path terminates at a transformation state are selected as delivery actions to be verified, including: Extract user characteristics, time characteristics, and content characteristics from the current delivery environment to generate multiple candidate delivery actions; Select the hidden Markov model corresponding to the most recent behavior segment, take each candidate delivery action as the initial observation input, execute the forward algorithm to calculate the hidden state sequence corresponding to each candidate delivery action, and extract the transition between adjacent hidden states in the hidden state sequence as the state transition path. Traverse the state transition paths of each candidate delivery action, determine whether the terminating hidden state of the state transition path corresponds to the transformation behavior observation, and filter out the candidate delivery actions whose state transition paths terminate at the transformation state. The state transition path of each candidate action is matched with the fragment state transition matrix of the adjacent behavior fragments, and the magnitude of behavior pattern drift caused by each candidate action is predicted based on the degree of difference in path transition probabilities. Candidate campaign actions that filter behavioral pattern drift amplitudes exceeding a preset drift threshold are selected as campaign actions to be verified. Candidate campaign actions that simultaneously satisfy the condition that the state transition path terminates at the conversion state and the behavioral pattern drift amplitude exceeds the preset drift threshold are selected as campaign actions to be verified.

5. The method according to claim 4, characterized in that, Traverse the state transition paths of each candidate delivery action, determine whether the terminating hidden state of the state transition path corresponds to a transformation behavior observation, and filter out candidate delivery actions whose state transition paths terminate in a transformation state, including: Extract user behavior sequences containing conversion behaviors from historical campaign data, and statistically analyze the last observed behavior type before the conversion occurred. In the Hidden Markov Model, the observation probability of the last observed behavior before the transformation behavior is generated in each hidden state is calculated, and the hidden state whose observation probability exceeds the preset probability threshold is marked as the transformation state. Traverse the state transition paths of each candidate action and extract the terminating hidden state of the state transition path. Determine whether the terminating hidden state belongs to the set of marked transition states, and filter out candidate delivery actions whose state transition paths terminate at the transition state.

6. The method according to claim 1, characterized in that, Set up an execution observation window. Within the observation window, collect real-time data streams of the current deployment environment and calculate the data distribution offset between the data distribution and the path anchor point distribution. When the data distribution offset is greater than or equal to a preset safety threshold, record negative feedback and trigger action reversal. When the execution observation window ends and the data distribution offset is less than the safety threshold, execute the deployment action to be verified, including: Set the execution observation window duration according to the state transition path corresponding to the action to be verified. Collect real-time data stream of the current delivery environment state by time step within the execution observation window and extract user behavior features from the real-time data stream. From historical deployment data, extract historical observation data corresponding to each hidden state node in the state transition path of the deployment action to be verified, and construct the conditional probability distribution of each hidden state node as the path anchor point distribution set. The real-time data stream is mapped to the hidden state node sequence of the state transition path of the action to be verified according to time steps. The distribution deviation between the real-time observation data of each time step and the path anchor point distribution of the corresponding hidden state node is calculated. The distribution deviation of each time step is accumulated by time weighting to obtain the data distribution offset. When the data distribution offset is greater than or equal to the preset safety threshold, negative feedback is recorded and action rollback is triggered. When the observation window ends and the data distribution offset is less than the safety threshold, the action to be verified is executed.

7. The method according to claim 6, characterized in that, The sequence of hidden state nodes that maps the real-time data stream to the state transition path of the action to be verified by time step includes: The real-time data stream is divided into ordered real-time observation data sequences according to the acquisition time step; Extract the hidden state node sequence from the state transition path of the action to be verified, and obtain the length of the hidden state node sequence and the expected dwell time of each hidden state node. Construct a time alignment matrix between the real-time observation data sequence and the hidden state node sequence, and calculate the observation probability matching degree between each time step in the real-time observation data sequence and each hidden state node in the hidden state node sequence. Based on the observation probability matching degree and the expected dwell time of the hidden state node, dynamic time warping is performed to calculate the optimal time alignment path; Based on the optimal time alignment path, each time step in the real-time observation data sequence is mapped to the corresponding hidden state node in the hidden state node sequence, thus establishing the mapping relationship between time steps and hidden state nodes.

8. A media information delivery strategy optimization system based on reinforcement learning, used to implement the method of any one of claims 1-7, characterized in that, include: The status acquisition unit is used to acquire the current deployment environment status and historical deployment data; The behavior analysis unit is used to extract user behavior sequences from historical delivery data, detect mutation points in user behavior sequences and divide them into multiple behavior segments based on mutation points, construct hidden Markov models for each behavior segment to obtain the segment state transition matrix, and compare the state transition matrices of adjacent segments to quantify the drift amplitude of behavior patterns. The action generation unit is used to generate multiple candidate action for the current delivery environment state, substitute each candidate action into the Hidden Markov Model to simulate the state evolution, predict the state transition path and behavior pattern drift of each candidate action, and select the candidate action whose behavior pattern drift exceeds the preset drift threshold and whose state transition path terminates in the transformation state as the action to be verified. The safety verification unit is used to set the execution observation window, collect the real-time data stream of the current deployment environment status within the execution observation window and calculate the data distribution offset between the data distribution and the path anchor point distribution. When the data distribution offset is greater than or equal to the preset safety threshold, negative feedback is recorded and the action is triggered to revert. When the execution observation window ends and the data distribution offset is less than the safety threshold, the deployment action to be verified is executed. The model update unit is used to collect execution results and calculate reward values. Based on the reinforcement learning mechanism, the reward values ​​are used to update the parameters, drift threshold, and safety threshold of the Hidden Markov Model.

9. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 7.

Images