Live broadcast e-commerce data analysis processing method and system based on reinforcement learning

Abstract

The invention discloses a reinforcement learning-based live broadcast e-commerce data analysis processing method and system. The method comprises the following steps of S1, collecting and preprocessing e-commerce data in a live broadcast platform; s2, the main channel extracts behavior semantics, and the auxiliary channel generates a comparison sample and jointly optimizes a coding weight; s3, a double-layer strategy network of a self-adaptive pruning mechanism is constructed, the upper layer generates a behavior intention, and the lower layer outputs an action decision; s4, adjusting pruning frequency according to channel activeness and strategy gradient fluctuation, updating a network structure and outputting parameters; s5, constructing a time difference reward function, performing compression processing on the state feature sequence in combination with a clustering comparison guidance method, and updating strategy network parameters; and S6, generating a multi-round decision instruction, and adjusting live broadcast content styles, commodity recommendation sequences and user interaction trigger nodes in real time. According to the invention, efficient modeling and strategy optimization of live broadcast e-commerce behaviors are realized, and recommendation accuracy and response efficiency are improved.

Description

Technical Field

[0001] This invention relates to the field of live-streaming e-commerce, and in particular to a data analysis and processing method and system for live-streaming e-commerce based on reinforcement learning. Background Technology

[0002] In the current live-streaming e-commerce technology system, the mining and analysis of user behavior data has become crucial for achieving precision marketing, personalized recommendations, and content optimization. However, most existing live-streaming e-commerce platforms use rule-driven, collaborative filtering, or static models to analyze and recommend user data. These methods generally suffer from three significant problems: first, insufficient ability to model the temporal characteristics of complex, multi-dimensional user behavior data; second, poor real-time model response, unable to adapt to rapid changes in user interests; and third, a lack of sustainable learning mechanisms in the recommendation strategies, failing to achieve automatic optimization based on user feedback.

[0003] Especially in the live-streaming e-commerce environment, user click behavior, dwell time, interaction frequency, and product conversion data exhibit strong dynamics and non-linear correlations. Traditional analysis methods based on static rules or offline models cannot accurately capture fluctuations in user interest during live streams. Furthermore, while existing deep learning methods have made some progress in feature extraction, they lack the environmental feedback mechanism found in reinforcement learning. This prevents recommendation systems from effectively evaluating rewards and adjusting strategies based on actual user behavior, impacting the effectiveness of personalized recommendations and commercial returns.

[0004] On the other hand, in practical recommendation deployments, the existence of high-dimensional behavioral state spaces makes model training convergence difficult. Some studies have attempted to introduce state compression and model distillation techniques to reduce computational burden, but these methods fail to incorporate the real-time feedback mechanism of live-streaming scenarios and fail to effectively integrate the correlation between behavioral intent modeling and action decision-making, resulting in a fragmented recommendation logic. Furthermore, existing pruning techniques are mostly used for model compression in general neural networks, without dynamically adjusting the policy network structure based on policy gradient fluctuations and channel activity, lacking optimized pruning mechanisms for policy learning scenarios.

[0005] Therefore, how to provide a data analysis and processing method and system for live e-commerce based on reinforcement learning is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] One objective of this invention is to propose a data analysis and processing method and system for live-streaming e-commerce based on reinforcement learning. This invention fully integrates contrastive learning, policy gradient optimization, and adaptive pruning mechanisms, constructs a dual-channel state encoding network and a dual-layer policy network, and describes in detail the processing procedures in user behavior modeling, behavior intent extraction, action decision generation, and policy structure self-evolution. It has the advantages of high modeling accuracy, fast policy response, and stable training convergence.

[0007] A data analysis and processing method for live-stream e-commerce based on reinforcement learning according to an embodiment of the present invention includes the following steps:

[0008] S1. Collect e-commerce data from the live streaming platform and preprocess it;

[0009] S2. Input the preprocessed e-commerce data into a dual-channel encoding network. The main channel extracts behavioral semantic features, the auxiliary channel constructs contrast samples across time windows, and the contrast loss function is used to jointly optimize the encoding weights of the two channels, outputting a state feature sequence.

[0010] S3. Construct a two-layer policy network with an adaptive pruning mechanism. The upper layer generates behavioral intention information based on the state feature sequence, and the lower layer makes action decisions based on the behavioral intention information.

[0011] S4. During the training of the policy network, the pruning mechanism is used to remove channels from inactive neural units. The pruning frequency is dynamically adjusted according to the policy gradient fluctuation threshold. The policy network structure is updated and the parameter set is output.

[0012] S5. Obtain the correspondence between user behavior and action decision, construct a time difference reward function, combine clustering comparison guidance method to compress the state feature sequence, replace the state feature sequence as training sample, and update the policy network parameters.

[0013] S6. Based on the updated two-layer strategy network, generate multi-round action decision instructions and adjust the live broadcast content style, product recommendation sorting and user interaction trigger nodes in real time.

[0014] Optionally, the e-commerce data includes user click frequency, page dwell time, back-end customer service conversations, and product browsing records.

[0015] Optionally, the preprocessing includes deduplication and noise reduction, format unification, outlier removal, missing value completion, and data standardization.

[0016] Optionally, the two-layer policy network consists of an upper-layer intent extraction network and a lower-layer action generation network. The intent extraction network uses a gated recurrent unit structure to receive the state feature sequence input and extracts the behavioral intent vector through time step expansion. The action generation network uses a multi-layer fully connected structure, combined with residual connections and activation function processing, to generate policy actions. The two-layer structure maintains the stability of the information transmission channel through explicit mapping. The policy network sets shared parameter constraints during the initialization phase and updates the parameter weights of the two layers through joint optimization of the objective function during training.

[0017] Optionally, S2 specifically includes:

[0018] S21. The preprocessed e-commerce data is grouped according to the user dimension. Four types of behavioral features are extracted from each user's behavior sequence: click frequency, page dwell time, product browsing history and semantic encoding value of back-end customer service dialogue. A unified format behavioral feature matrix is ​​constructed.

[0019] S22. Construct a dual-channel encoding network based on the behavior feature matrix. The main channel adopts a two-layer temporal convolution structure. The first layer is used to extract local behavior change features, and the second layer is used to extract behavior trend features across time periods. Perform frame-by-frame convolution processing on each set of behavior feature vectors to generate a semantic vector sequence for each user in the current time period.

[0020] S23. Input the semantic vector sequence output by the main channel into the auxiliary channel. The auxiliary channel divides the semantic sequence into windows according to a fixed-length sliding window strategy to construct a comparison sample set, where samples within the window are positive samples and samples across the window are negative samples.

[0021] S24. The comparison sample set is encoded by a multi-layer fully connected neural network in the auxiliary channel to generate the corresponding comparison sample vector sequence. The semantic vector output by the main channel is used as the anchor point and the comparison vector output by the auxiliary channel is used as the target. A positive and negative sample comparison structure is established through a similarity constraint mechanism. Batch sample comparison matching is performed and global difference measurement is carried out in each round of training.

[0022] S25. After the main channel and auxiliary channel have completed their respective encoding outputs, perform dual-channel joint optimization training. Set the reconstruction objective of the main channel to maintain semantic stability, set the contrast objective of the auxiliary channel to maintain sample discriminability, and introduce a consistency measure between the dual-channel encoding vectors as the basis for structural loss.

[0023] S26. During joint training, a hyperparameter weighting mechanism is used to optimize the loss function, and the parameters of the convolutional layer in the main channel and the parameters of the fully connected layer in the auxiliary channel are jointly adjusted respectively.

[0024] S27. After each training cycle, the weight parameters of the main channel and the auxiliary channel are updated synchronously, the difference index of the state encoding results in the two training cycles is recorded, and the state feature sequence is output.

[0025] Optionally, S3 specifically includes:

[0026] S31. Based on the state feature sequence, a two-layer policy network is constructed. The two-layer policy network consists of an upper-layer intention extraction network and a lower-layer action generation network. The intention extraction network adopts a gated recurrent unit structure. After inputting each state feature vector, the corresponding behavioral intention vector is generated through the state update mechanism. The behavioral intention encoding is used to express the user's live content preference and interaction frequency in the current state. All the behavioral intention vectors corresponding to the users constitute a behavioral intention matrix.

[0027] S32. Input the behavioral intent matrix into the lower-level action generation network. Use a multi-layer fully connected structure and combine the residual mechanism to perform non-linear mapping on each behavioral intent vector to generate the corresponding action vector set.

[0028] S33. Set up a candidate set of clipped channels, establish channel index relationships for all hidden units in the action generation network, mark the parameter trajectory of each clipped channel, and dynamically monitor the coupling strength between the behavioral intent component corresponding to each channel and the output action during the action generation process.

[0029] S34. During the training cycle of the policy network, perform gradient sensitivity evaluation and behavior contribution evaluation once for each channel in the channel candidate set, count the key participation of each channel in the current action generation process, and quantify the score by combining it with the average activation rate in the previous training round.

[0030] S35. Based on the scoring and ranking results, structural removal is performed on the channels with the lowest scores, while the remaining channels continue to participate in subsequent training and action generation. All scores and removal actions are recorded uniformly after the current training round and used as a reference for the next round of training scheduling strategy.

[0031] S36. After completing the adaptive channel pruning operation, regenerate the action for each behavior intent vector to obtain the pruned and optimized action vector set.

[0032] Optionally, S4 specifically includes:

[0033] S41. During the training of the policy network, the activation behavior of all channels in the network is periodically recorded to generate a channel activity monitoring sequence. The average activation rate of each channel in the current training period is calculated, and a fixed threshold is set as the basis for judging channel activity.

[0034] S42 When the activation rate of a certain channel is consistently lower than the threshold during consecutive training cycles, it is marked as an inactive channel and added to the set of channels to be removed. At the same time, the gradient update trajectory during the training process of the policy network is recorded, and a policy gradient change sequence is established according to the training rounds.

[0035] S43. Based on the change magnitude between adjacent rounds in the policy gradient change sequence, determine the stability of the current policy network training phase. When the gradient change rate is within the set fluctuation range, increase the trigger frequency of pruning actions; otherwise, decrease the frequency.

[0036] S44. By statistically analyzing the stable proportion of gradient changes between each round, and combining it with the configurable maximum pruning execution frequency, calculate the pruning frequency parameter allowed in the current training phase, and use it as the basis for determining whether to perform pruning operations in each round of training.

[0037] S45. Determine whether to perform channel removal operation based on the pruning frequency parameter. If the execution conditions are met, select the channel index that currently meets the removal strategy from the set of channels to be removed and perform physical removal operation.

[0038] S46. After pruning, the network structure is reconstructed, the connection layer structure is adjusted, activation path remapping is performed on the retained neurons, and the parameter set of the policy network is output.

[0039] Optionally, S5 specifically includes:

[0040] S51. During the training process of the policy network, record the index relationship between each state feature vector and the corresponding output action, construct a set of state-action matching pairs, and collect three feedback data items within the set data sampling period: number of user clicks, product browsing records, and page dwell time.

[0041] S52. Based on the three feedback data, calculate the reward feedback value of each state-action pair, and combine it with the difference in behavioral response between the previous state and the next state to form a time difference reward estimate, which is used to characterize the degree of influence of the current policy behavior on the user, and introduce the difference in state evaluation results between the target policy network and the current policy network as a reference value.

[0042] S53. Based on the state-action matching relationship and the corresponding reward results, the state feature vector is compressed using a clustering-guided method: all state feature vectors are clustered and classified, the cluster centers are extracted as the compressed state representations, and the original state feature vectors are used as new training samples. The network training is re-executed based on the new training samples, and the policy network parameter set is updated.

[0043] Optionally, the target policy network and the current policy network are two parameter sets used for stable training in reinforcement learning. The target policy network periodically copies parameters from the current policy network, while the two-layer policy network is responsible for generating policy intentions and action decisions in layers. The target policy network and the current policy network can be deployed in a two-layer structure for parameter updates and comparative learning at different levels.

[0044] According to an embodiment of the present invention, a live-stream e-commerce data analysis and processing system based on reinforcement learning includes:

[0045] The data processing module is used to collect e-commerce data from the live streaming platform and perform preprocessing.

[0046] The feature extraction module is used to input the preprocessed e-commerce data into the dual-channel encoding network. The main channel extracts behavioral semantic features, the auxiliary channel constructs contrast samples across time windows, and the contrast loss function is used to jointly optimize the encoding weights of the two channels, outputting a state feature sequence.

[0047] The strategy modeling module is used to construct a two-layer policy network with an adaptive pruning mechanism. The upper layer generates behavioral intention information based on the state feature sequence, and the lower layer makes action decisions based on the behavioral intention information.

[0048] The adjustment and update module is used to remove inactive neural networks during the training process of the policy network by using the pruning mechanism, dynamically adjust the pruning frequency according to the policy gradient fluctuation threshold, update the policy network structure and output the parameter set.

[0049] The clustering compression module is used to obtain the correspondence between user behavior and action decision, construct a time difference reward function, and combine a clustering comparison guidance method to compress the state feature sequence, replace the state feature sequence as training samples, and update the policy network parameters.

[0050] The decision analysis module is used to generate multi-round action decision instructions based on the updated two-layer policy network, and adjust the style of live broadcast content, product recommendation ranking and user interaction trigger nodes in real time.

[0051] The beneficial effects of this invention are:

[0052] First, this invention constructs a dual-channel state coding network, using the main channel to extract semantic features of user behavior and the auxiliary channel to generate comparative samples across time windows. By using a comparative loss function for joint optimization, the state feature representation becomes more stable and discriminative, effectively improving the ability to model the dynamics of user interests and solving the problem that traditional methods cannot capture the behavioral evolution process.

[0053] Secondly, in terms of policy network design, the invention adopts a two-layer structure to distinguish between behavioral intention extraction and action decision generation, and introduces an adaptive pruning mechanism to dynamically remove inactive neural channels during training. At the same time, the pruning frequency is controlled according to the fluctuation amplitude of the policy gradient. Thus, while ensuring the policy expressive ability, the model training efficiency and inference performance are significantly improved, avoiding the problems of rigid structure and poor generalization ability in existing policy networks.

[0054] Finally, by constructing a reward function based on time difference and combining it with a clustering comparison guidance method to compress the state feature sequence, the training samples were simplified and effectively replaced, which improved the learning stability of the policy network in continuous feedback and its ability to adapt to changes in user behavior. This fundamentally solved the problems of existing methods relying on a large number of uncompressed samples and delayed feedback response during the training phase. Attached Figure Description

[0055] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0056] Figure 1 This is a flowchart of a data analysis and processing method for live-streaming e-commerce based on reinforcement learning proposed in this invention.

[0057] Figure 2 This is a flowchart of the strategy network training and pruning update process for a reinforcement learning-based live e-commerce data analysis and processing method proposed in this invention.

[0058] Figure 3 This is a module structure diagram of a reinforcement learning-based live-streaming e-commerce data analysis and processing system proposed in this invention. Detailed Implementation

[0059] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0060] refer to Figure 1-2 A data analysis and processing method for live-streaming e-commerce based on reinforcement learning includes the following steps:

[0061] S1. Collect e-commerce data from the live streaming platform and preprocess it;

[0062] S2. Input the preprocessed e-commerce data into a dual-channel encoding network. The main channel extracts behavioral semantic features, the auxiliary channel constructs contrast samples across time windows, and the contrast loss function is used to jointly optimize the encoding weights of the two channels, outputting a state feature sequence.

[0063] S3. Construct a two-layer policy network with an adaptive pruning mechanism. The upper layer generates behavioral intention information based on the state feature sequence, and the lower layer makes action decisions based on the behavioral intention information.

[0064] S4. During the training of the policy network, the pruning mechanism is used to remove channels from inactive neural units. The pruning frequency is dynamically adjusted according to the policy gradient fluctuation threshold. The policy network structure is updated and the parameter set is output.

[0065] S5. Obtain the correspondence between user behavior and action decision, construct a time difference reward function, combine clustering comparison guidance method to compress the state feature sequence, replace the state feature sequence as training sample, and update the policy network parameters.

[0066] S6. Based on the updated two-layer strategy network, generate multi-round action decision instructions and adjust the live broadcast content style, product recommendation sorting and user interaction trigger nodes in real time.

[0067] In this embodiment, the e-commerce data is obtained from the client interaction logs, server behavior records and back-end customer service system of the live streaming platform. The data includes multi-source heterogeneous data such as user click frequency, page dwell time, back-end customer service dialogue and product browsing records. A unified timestamp mechanism is used for alignment processing and the data is archived by user ID.

[0068] In this embodiment, the preprocessing process includes: performing unified timestamp correction and format standardization on the collected user click frequency, page dwell time, backend customer service dialogue and product browsing record data; using a rule-based deduplication algorithm to delete duplicate data items; using the Z-score method to detect and remove outliers exceeding a set threshold; using the K-nearest neighbor interpolation method to complete missing fields; performing maximum and minimum value normalization on numerical features; and constructing standard semantic vectors for text-based customer service dialogue records through operations such as word segmentation, stop word removal and stemming to ensure that all feature dimensions have a consistent data structure and numerical distribution.

[0069] In this embodiment, S2 specifically includes:

[0070] S21. Obtain the preprocessed e-commerce data and construct the behavioral feature matrix M = [m i,j ] u×d Where u represents the number of users, d represents the feature dimension of each user, and m i,j This represents the behavioral feature value of the i-th user in the j-th dimension, where the feature dimension consists of page dwell time, click frequency, customer service dialogue semantic encoding value, and product browsing numerical index.

[0071] S22. Construct a dual-channel encoding network. Input the behavior feature matrix M into the main channel, and use a two-layer temporal convolutional network to extract local and global semantic features, outputting the semantic vector sequence E of the main channel. (p) ,in This represents the main channel semantic vector of the i-th user;

[0072] S23. Construct a comparison sample vector sequence E based on the auxiliary channel. (a)A sliding time window mechanism is used to generate positive and negative sample pairs, and the contrastive loss function L is calculated based on the semantic vector sequence. con The contrastive loss function is defined as follows:

[0073]

[0074] in, Let sim(x,y) represent the auxiliary channel comparison sample vector of the i-th user, and let sim(x,y) represent the cosine similarity function, satisfying... τ represents the temperature parameter used to adjust the sensitivity of the similarity distribution, log2(·) represents the logarithmic function, and exp(·) represents the natural exponential function with base e.

[0075] S24, Based on semantic vector sequence E (p) Compared with the sample vector sequence E (a) Perform joint training optimization, using loss function L. total The loss function is defined as follows: Dual-channel parameter weight updates are performed.

[0076]

[0077] Among them, L rec The reconstruction loss is represented to maintain semantic stability, and α, β, and γ are weighting coefficients of the hyperparameters. This represents the Euclidean distance between the encodings of the i-th user's main channel and auxiliary channel, used to constrain encoding consistency;

[0078] S25, Output state feature sequence S, where s i ∈S represents the fusion state feature of the i-th user, satisfying

[0079] In this embodiment, S3 specifically includes:

[0080] S31. Based on the state feature sequence S, construct a two-layer policy network. The two-layer policy network consists of an upper-layer intent extraction network and a lower-layer action generation network. The intent extraction network adopts a gated recurrent unit structure, and each state feature vector s is input. i Then, the corresponding behavioral intent vector h is generated through a state update mechanism. i The behavioral intent encoding is used to express the user's preferences for live streaming content and frequency of interaction in the current state;

[0081] S32. In the lower-level action generation network, based on the behavioral intent vector h i Construct a set of action vectors A, where each action vector a i ∈A represents the policy action generated for user i in the current live streaming scenario;

[0082] S33. To optimize the generalization ability and inference efficiency of the policy network, a pruned candidate channel set C is designed, where each channel c... j ∈C represents the channel weight parameter of the j-th removable unit in the policy network;

[0083] S34. Calculate the adaptive pruning scoring function and perform pruning strategy selection on the candidate channel set. The pruning scoring function is defined as follows:

[0084]

[0085] Where P represents the overall pruning score in the current training round, n is the number of channels, J is the objective function of the current policy, λ1, λ2, and λ3 are the pruning weight coefficients, and σ represents the Sigmoid function. Let j be the j-th component in the behavioral intent vector. Let δ be the j-th component in the action vector. j This represents the average activation rate of the current channel during the previous training round.

[0086] S35. Based on the minimization result of the pruning scoring function, determine the set of channel indices to be removed. Then, a channel masking operation is performed on the corresponding network structure, and the action vector set A′ is recalculated based on the retained network structure, where each action vector a′ i ∈A′ represents the action decision output by the policy network after pruning.

[0087] In this embodiment, S4 specifically includes:

[0088] S41. During the training of the policy network, a channel activity monitoring matrix V is constructed, where v j ∈V represents the average activation rate of the j-th channel within a training epoch. A minimum activation rate threshold θ is set, where v is satisfied. j When <θ, channel j is determined to be an inactive channel and added to the set of channel indices to be removed, C′. Simultaneously, a policy gradient update trajectory sequence G is constructed, where g... t ∈G represents the norm of the gradient of the policy network in the t-th training iteration;

[0089] S42. Update the trajectory sequence G based on the policy gradient, and calculate the pruning frequency function to guide the adjustment frequency of pruning actions during the training period. It is defined as follows:

[0090]

[0091] Where F represents the recommended pruning trigger frequency for the current training phase, η is the maximum allowable pruning frequency, T is the number of training rounds completed, and g t-1Let denot be the norm of the gradient of the policy network in the (t-1)th training iteration, and let ∈ be the gradient variability threshold. This is a Boolean indicator function; its value is 1 if the condition within the parentheses is true, and 0 otherwise.

[0092] S43. Determine the channel removal step size in this round of training based on the pruning frequency function, and perform the channel removal operation in the structure graph of the policy network in combination with the set of channel indices to be removed, C′.

[0093] S44. After completing the structural adjustment, reinitialize the channel weight update strategy, perform batch normalization correction on the remaining neural units, and output the updated strategy network parameter set under convergence conditions.

[0094] In this embodiment, S5 specifically includes:

[0095] S51. During the action decision-making process of the policy network, record the index relationship between each state input and the corresponding action output, and construct a set of behavior-action matching pairs D = {(s i ,a′ i )}, where s i Let a′ be the state feature vector of the i-th user. i The output of the action decision after pruning optimization;

[0096] S52. Collect feedback indicator data for each matching pair in actual live streaming behavior, set a reward sampling period Δt, and obtain three data items within this period: number of user clicks, product click conversion rate, and page dwell time.

[0097] S53. Calculate the time difference reward function based on feedback indicator data to measure the actual return of the current strategy action on user behavior, defined as follows:

[0098]

[0099] in, Let r be the time difference reward value in the i-th state. i s is the instantaneous reward value for the current period, γ is the reward decay factor, and s i+1 Let be the state feature vector of the (i+1)th user, 'a' be the maximum action among all possible actions, Q(s,a; θ) be the state-action value function of the current policy network, and θ be the set of parameters of the main network. - The target network parameter set;

[0100] S54. Based on the behavior-action matching set and the corresponding time difference reward value, the original state feature sequence is compressed using a clustering comparison-guided method:

[0101] The K-means clustering algorithm is used to cluster the state feature set to generate a set of compression centers C. k ={c1,c2,…,c k}, where k is the preset compression dimension, and all state vectors are mapped to their nearest center point;

[0102] The mapped state representation is used as a new training sample input into the policy network, replacing the original state feature sequence;

[0103] The set of policy network parameters is updated based on the new compression state and reward results in each round of training.

[0104] refer to Figure 3 A data analysis and processing system for live-streaming e-commerce based on reinforcement learning, comprising:

[0105] The data processing module is used to collect e-commerce data from the live streaming platform and perform preprocessing.

[0106] The feature extraction module is used to input the preprocessed e-commerce data into the dual-channel encoding network. The main channel extracts behavioral semantic features, the auxiliary channel constructs contrast samples across time windows, and the contrast loss function is used to jointly optimize the encoding weights of the two channels, outputting a state feature sequence.

[0107] The strategy modeling module is used to construct a two-layer policy network with an adaptive pruning mechanism. The upper layer generates behavioral intention information based on the state feature sequence, and the lower layer makes action decisions based on the behavioral intention information.

[0108] The adjustment and update module is used to remove inactive neural networks during the training process of the policy network by using the pruning mechanism, dynamically adjust the pruning frequency according to the policy gradient fluctuation threshold, update the policy network structure and output the parameter set.

[0109] The clustering compression module is used to obtain the correspondence between user behavior and action decision, construct a time difference reward function, and combine a clustering comparison guidance method to compress the state feature sequence, replace the state feature sequence as training samples, and update the policy network parameters.

[0110] The decision analysis module is used to generate multi-round action decision instructions based on the updated two-layer policy network, and adjust the style of live broadcast content, product recommendation ranking and user interaction trigger nodes in real time.

[0111] Example 1:

[0112] To verify the feasibility of this invention in practice, it was applied to a popular product sales scenario on a live-streaming e-commerce platform. The goal was to improve user dwell time, product click-through rate, and overall conversion efficiency, while addressing the problems of delayed user behavior recognition, slow response of recommendation strategies, and low model training efficiency in existing technologies. This live-streaming scenario lasts for 6 hours daily, primarily featuring fast-moving consumer goods, fashion, and home furnishing products, and encompassing a diverse range of user types, including short-term visitors, regular followers, and repeat buyers.

[0113] In this scenario, the system deployed in this invention first collects data from the live streaming platform, including user browsing behavior, dwell time, bullet screen interactions, customer service dialogue summaries, and product click behavior. The data preprocessing module then performs operations such as format unification, deduplication and noise reduction, outlier removal, and feature normalization. After preprocessing, the data is fed into a dual-channel encoding network. The main channel extracts high-dimensional behavioral semantic features, while the auxiliary channel generates contrast samples using a sliding window approach to construct a semantic consistency target. Joint training is then performed using a contrastive loss algorithm, ultimately outputting a state vector sequence with significant expressive power.

[0114] During the strategy generation phase, the system employs a two-layer policy network. The upper layer extracts user intent vectors under different behavioral states and aggregates them in conjunction with live stream content dimensions (product type, host style, and activity rhythm). The lower layer generates specific action decision outputs based on these intents, including adjustments to the live stream content rhythm, restructuring of product recommendation rankings, and interactive prompt triggering strategies. The entire policy network utilizes an adaptive pruning mechanism during training, real-time monitoring of channel activity and controlling the pruning frequency based on policy gradient fluctuations. This effectively reduces redundant model parameters and improves inference response speed.

[0115] To verify the effectiveness of this invention, a comparative experiment was conducted with two existing mainstream methods: a static recommendation method based on collaborative filtering and a single-channel deep behavioral modeling network. Controlled variable experiments were performed using the three methods in 20 live streams, monitoring metrics including average user dwell time, product click-through rate, recommended product conversion rate, interaction response rate, and the training cycle of the strategy network.

[0116] Table 1 shows that, after adopting the method of this invention, the average user dwell time increased by 27.3% compared to the collaborative filtering method and by 14.5% compared to the single-channel model; the product click-through rate increased by 22.6% and 11.8%, respectively; the final conversion rate of recommended products increased by 19.2% and 9.7%, respectively; the average user interaction response rate increased by more than 16%; and the training cycle of the policy network was shortened by 31.4%, significantly reducing system training costs and deployment delays. Therefore, this method significantly improves training efficiency and behavioral feedback response capabilities while enhancing recommendation quality.

[0117] Table 1 Comparison of the effects of the method of the present invention and existing methods in the live-streaming e-commerce scenario.

[0118]

[0119] As shown in Table 1, by combining contrastive learning encoding, policy pruning and tuning, and behavioral intent modeling, reinforcement learning policy networks can not only achieve more accurate and real-time recommendation decisions in e-commerce live streaming scenarios, but also significantly reduce model training and inference costs, improve the overall system operating efficiency, and bring significant benefits to the user stickiness and sales conversion of live streaming e-commerce platforms.

[0120] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A data analysis and processing method for live-streaming e-commerce based on reinforcement learning, characterized in that, Includes the following steps: S1. Collect e-commerce data from the live streaming platform and preprocess it; S2. Input the preprocessed e-commerce data into a dual-channel encoding network. The main channel extracts behavioral semantic features, the auxiliary channel constructs contrast samples across time windows, and the contrast loss function is used to jointly optimize the encoding weights of the two channels, outputting a state feature sequence. S3. Construct a two-layer policy network with an adaptive pruning mechanism. The upper layer generates behavioral intention information based on the state feature sequence, and the lower layer makes action decisions based on the behavioral intention information. S4. During the training of the policy network, the pruning mechanism is used to remove channels from inactive neural units. The pruning frequency is dynamically adjusted according to the policy gradient fluctuation threshold. The policy network structure is updated and the parameter set is output. S5. Obtain the correspondence between user behavior and action decision, construct a time difference reward function, combine clustering comparison guidance method to compress the state feature sequence, replace the state feature sequence as training sample, and update the policy network parameters. S6. Based on the updated two-layer strategy network, generate multi-round action decision instructions and adjust the live broadcast content style, product recommendation sorting and user interaction trigger nodes in real time.

2. The method for analyzing and processing live-streaming e-commerce data based on reinforcement learning according to claim 1, characterized in that, The e-commerce data includes user click frequency, page dwell time, back-end customer service conversations, and product browsing history.

3. The method for analyzing and processing live-streaming e-commerce data based on reinforcement learning according to claim 1, characterized in that, The preprocessing includes deduplication and noise reduction, format unification, outlier removal, missing value completion, and data standardization.

4. The data analysis and processing method for live-streaming e-commerce based on reinforcement learning according to claim 1, characterized in that, The dual-layer policy network consists of an upper-layer intent extraction network and a lower-layer action generation network. The intent extraction network uses a gated recurrent unit structure to receive state feature sequence input and extracts behavioral intent vectors through time step expansion. The action generation network uses a multi-layer fully connected structure, combined with residual connections and activation function processing, to generate policy actions. The two-layer structure maintains the stability of the information transmission channel through explicit mapping. The policy network sets shared parameter constraints during the initialization phase and updates the parameter weights of the two layers through joint optimization of the objective function during training.

5. The method for analyzing and processing live-streaming e-commerce data based on reinforcement learning according to claim 1, characterized in that, S2 specifically includes: S21. The preprocessed e-commerce data is grouped according to the user dimension. Four types of behavioral features are extracted from each user's behavior sequence: click frequency, page dwell time, product browsing history and semantic encoding value of back-end customer service dialogue. A unified format behavioral feature matrix is ​​constructed. S22. Construct a dual-channel encoding network based on the behavior feature matrix. The main channel adopts a two-layer temporal convolution structure. The first layer is used to extract local behavior change features, and the second layer is used to extract behavior trend features across time periods. Perform frame-by-frame convolution processing on each set of behavior feature vectors to generate a semantic vector sequence for each user in the current time period. S23. Input the semantic vector sequence output by the main channel into the auxiliary channel. The auxiliary channel divides the semantic sequence into windows according to a fixed-length sliding window strategy to construct a comparison sample set, where samples within the window are positive samples and samples across the window are negative samples. S24. The comparison sample set is encoded by a multi-layer fully connected neural network in the auxiliary channel to generate the corresponding comparison sample vector sequence. The semantic vector output by the main channel is used as the anchor point and the comparison vector output by the auxiliary channel is used as the target. A positive and negative sample comparison structure is established through a similarity constraint mechanism. Batch sample comparison matching is performed and global difference measurement is carried out in each round of training. S25. After the main channel and auxiliary channel have completed their respective encoding outputs, perform dual-channel joint optimization training. Set the reconstruction objective of the main channel to maintain semantic stability, set the contrast objective of the auxiliary channel to maintain sample discriminability, and introduce a consistency measure between the dual-channel encoding vectors as the basis for structural loss. S26. During joint training, a hyperparameter weighting mechanism is used to optimize the loss function, and the parameters of the convolutional layer in the main channel and the parameters of the fully connected layer in the auxiliary channel are jointly adjusted respectively. S27. After each training cycle, the weight parameters of the main channel and the auxiliary channel are updated synchronously, the difference index of the state encoding results in the two training cycles is recorded, and the state feature sequence is output.

6. The method for analyzing and processing live-streaming e-commerce data based on reinforcement learning according to claim 1, characterized in that, S3 specifically includes: S31. Based on the state feature sequence, a two-layer policy network is constructed. The two-layer policy network consists of an upper-layer intention extraction network and a lower-layer action generation network. The intention extraction network adopts a gated recurrent unit structure. After inputting each state feature vector, the corresponding behavioral intention vector is generated through the state update mechanism. The behavioral intention encoding is used to express the user's live content preference and interaction frequency in the current state. All the behavioral intention vectors corresponding to the users constitute a behavioral intention matrix. S32. Input the behavioral intent matrix into the lower-level action generation network. Use a multi-layer fully connected structure and combine the residual mechanism to perform non-linear mapping on each behavioral intent vector to generate the corresponding action vector set. S33. Set up a candidate set of clipped channels, establish channel index relationships for all hidden units in the action generation network, mark the parameter trajectory of each clipped channel, and dynamically monitor the coupling strength between the behavioral intent component corresponding to each channel and the output action during the action generation process. S34. During the training cycle of the policy network, perform gradient sensitivity evaluation and behavior contribution evaluation once for each channel in the channel candidate set, count the key participation of each channel in the current action generation process, and quantify the score by combining it with the average activation rate in the previous training round. S35. Based on the scoring and ranking results, structural removal is performed on the channels with the lowest scores, while the remaining channels continue to participate in subsequent training and action generation. All scores and removal actions are recorded uniformly after the current training round and used as a reference for the next round of training scheduling strategy. S36. After completing the adaptive channel pruning operation, regenerate the action for each behavior intent vector to obtain the pruned and optimized action vector set.

7. The method for analyzing and processing live-streaming e-commerce data based on reinforcement learning according to claim 1, characterized in that, S4 specifically includes: S41. During the training of the policy network, the activation behavior of all channels in the network is periodically recorded to generate a channel activity monitoring sequence. The average activation rate of each channel in the current training period is calculated, and a fixed threshold is set as the basis for judging channel activity. S42 When the activation rate of a certain channel is consistently lower than the threshold during consecutive training cycles, it is marked as an inactive channel and added to the set of channels to be removed. At the same time, the gradient update trajectory during the training process of the policy network is recorded, and a policy gradient change sequence is established according to the training rounds. S43. Based on the change magnitude between adjacent rounds in the policy gradient change sequence, determine the stability of the current policy network training phase. When the gradient change rate is within the set fluctuation range, increase the trigger frequency of pruning actions; otherwise, decrease the frequency. S44. By statistically analyzing the stable proportion of gradient changes between each round, and combining it with the configurable maximum pruning execution frequency, calculate the pruning frequency parameter allowed in the current training phase, and use it as the basis for determining whether to perform pruning operations in each round of training. S45. Determine whether to perform channel removal operation based on the pruning frequency parameter. If the execution conditions are met, select the channel index that currently meets the removal strategy from the set of channels to be removed and perform physical removal operation. S46. After pruning, the network structure is reconstructed, the connection layer structure is adjusted, activation path remapping is performed on the retained neurons, and the parameter set of the policy network is output.

8. The method for analyzing and processing live-streaming e-commerce data based on reinforcement learning according to claim 1, characterized in that, S5 specifically includes: S51. During the training process of the policy network, record the index relationship between each state feature vector and the corresponding output action, construct a set of state-action matching pairs, and collect three feedback data items within the set data sampling period: number of user clicks, product browsing records, and page dwell time. S52. Based on the three feedback data, calculate the reward feedback value of each state-action pair, and combine it with the difference in behavioral response between the previous state and the next state to form a time difference reward estimate, which is used to characterize the degree of influence of the current policy behavior on the user, and introduce the difference in state evaluation results between the target policy network and the current policy network as a reference value. S53. Based on the state-action matching relationship and the corresponding reward results, the state feature vector is compressed using a clustering-guided method: all state feature vectors are clustered and classified, the cluster centers are extracted as the compressed state representations, and the original state feature vectors are used as new training samples. The network training is re-executed based on the new training samples, and the policy network parameter set is updated.

9. The method for analyzing and processing live-streaming e-commerce data based on reinforcement learning according to claim 8, characterized in that, The target policy network and the current policy network are two sets of parameters used for stable training in reinforcement learning. The target policy network periodically copies parameters from the current policy network, while the two-layer policy network is responsible for generating policy intentions and action decisions in layers. The target policy network and the current policy network can be deployed in a two-layer structure for parameter updates and comparative learning at different levels.

10. A reinforcement learning-based live-streaming e-commerce data analysis and processing system, comprising executing the reinforcement learning-based live-streaming e-commerce data analysis and processing method according to any one of claims 1 to 9, characterized in that, include: The data processing module is used to collect e-commerce data from the live streaming platform and perform preprocessing. The feature extraction module is used to input the preprocessed e-commerce data into the dual-channel encoding network. The main channel extracts behavioral semantic features, the auxiliary channel constructs contrast samples across time windows, and the contrast loss function is used to jointly optimize the encoding weights of the two channels, outputting a state feature sequence. The strategy modeling module is used to construct a two-layer policy network with an adaptive pruning mechanism. The upper layer generates behavioral intention information based on the state feature sequence, and the lower layer makes action decisions based on the behavioral intention information. The adjustment and update module is used to remove inactive neural networks during the training process of the policy network by using the pruning mechanism, dynamically adjust the pruning frequency according to the policy gradient fluctuation threshold, update the policy network structure and output the parameter set. The clustering compression module is used to obtain the correspondence between user behavior and action decision, construct a time difference reward function, and combine a clustering comparison guidance method to compress the state feature sequence, replace the state feature sequence as training samples, and update the policy network parameters. The decision analysis module is used to generate multi-round action decision instructions based on the updated two-layer policy network, and adjust the style of live broadcast content, product recommendation ranking and user interaction trigger nodes in real time.

Images