Sequence recommendation data weight optimization system and method based on reinforcement learning

Abstract

The invention discloses a sequence recommendation data weight optimization system and method based on reinforcement learning, and aims to solve the problem of insufficient model performance caused by unreasonable data weight distribution in sequence recommendation. The system comprises a strategy network, a memory buffer, a performance evaluation module and a weight updating module, the strategy network generates continuous weight adjustment actions, memory buffer storage states (sample feature embedding and current weight), actions and other empirical data through a multi-level fusion Actor architecture, the performance evaluation generates reward signals based on Recall (at) K and NDCG (at) K, and the weight updating adopts a PPO algorithm to achieve stable iteration; the method comprises the following steps: acquiring data, extracting features through a pre-training model, and initializing weights; constructing an MDP to convert weight optimization into a sequence decision; and iteratively generating a weight adjustment action, cutting a function to limit the weight to [0.1, 1.0], training the temporary model by Bernoulli sampling, and updating the strategy network based on rewards. According to the method, data weight dynamic optimization is realized, and the performance and generalization of the recommendation model are improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a data weight optimization system and method for sequence recommendation based on reinforcement learning. Background Technology

[0002] Sequence recommendation systems, as an important application of artificial intelligence technology in the consumer electronics field, can accurately predict users' subsequent interests based on their historical interaction behavior. They play an irreplaceable and crucial role in scenarios such as smart TV content delivery, personalized recommendations for mobile applications, and product guidance on e-commerce platforms. With the continuous evolution of deep learning technology, the architecture of sequence recommendation models has been constantly iterated and upgraded. From the early GRU4Rec model based on recurrent neural networks (RNNs), to the SASRec model that relies on attention mechanisms to achieve accurate sequence feature capture, and then to the new generation of technical solutions that integrate large language models into the recommendation framework to mine rich semantic associations and strengthen the representation of user and item features, the model's ability to characterize user preferences and the accuracy of recommendations have been continuously improved.

[0003] However, existing sequence recommendation systems generally face core challenges related to data quality in practical applications. Raw data collected from user interactions often contains a large amount of noise and redundant information. This low-value data directly affects the accuracy of recommendation results and user experience, and also significantly increases the computational cost of model training and iteration. Especially in the scenario of fine-tuning large language models, massive amounts of low-quality data lead to inefficient fine-tuning processes and a surge in computational resource consumption, severely restricting the large-scale application of large language models in the recommendation field. To address data quality issues, various data subset selection methods have been proposed in the existing technical field, but all have significant limitations and are difficult to meet the needs of practical recommendation scenarios: Influence function methods determine importance by evaluating the contribution of individual samples, but the calculated scores are too concentrated, making it difficult to distinguish the true contribution of samples; Gradient-based methods such as GraNd and EL2N can capture local error signals, but may not reflect the global contribution of data samples throughout the training process; Existing reinforcement learning methods such as DVRL are essentially still based on supervised learning paradigms and cannot fully utilize the true reinforcement learning framework to dynamically adapt to global feedback; Large model-dependent methods, such as DEALRec, require the use of large language models to calculate influence scores, and in long text scenarios, the computation time increases superlinearly with the input length, seriously affecting practicality.

[0004] As the application scenarios of recommender systems continue to expand, the scale of training data is growing exponentially. However, not all training samples contribute positively to model performance. Large amounts of noisy and redundant data not only increase the computational burden but may also reduce the model's generalization ability. Especially in the fine-tuning of large language models, the computational cost is exorbitant, making the selection of high-quality samples from massive datasets a critical problem that urgently needs to be solved. Traditional data selection methods often rely on static importance scores, which cannot adapt to dynamically changing training processes and performance requirements. This limitation is particularly pronounced in complex recommender scenarios.

[0005] To address this issue, this invention proposes a data weight optimization system and method based on reinforcement learning for sequence recommendation, aiming to solve the problem of insufficient model performance caused by unreasonable data weight allocation in sequence recommendation. Summary of the Invention

[0006] In view of the above-mentioned deficiencies of the prior art, the present invention at least solves the following technical problems: 1. Existing data subset selection methods lack adaptability and cannot dynamically adjust the importance weight of samples based on global performance feedback. This results in low-quality selected data subsets, making it difficult to filter out noise and redundant data, and also unable to accurately retain high-information samples. Ultimately, this restricts the performance improvement of recommendation models (including small recommendation models and large language models). 2. The importance scores calculated based on the traditional influence function method are too concentrated, which cannot effectively distinguish the true contribution value of different samples, and the accuracy of the division between high-quality samples and low-value samples is low. 3. Although existing reinforcement learning methods (such as DVRL) introduce reinforcement learning signals for network updates, they are still essentially based on the supervised learning paradigm, lack true Markov sequence decision-making ability, cannot model the temporal correlation between weight adjustment and long-term recommendation performance, and are difficult to achieve global optimization. 4. Existing methods that rely on large language models (such as DEALRec) need to use large language models to calculate the importance of sample influence scores. In long text recommendation scenarios, the computation time increases superlinearly with the input length, resulting in high computational cost and poor practicality. 5. Gradient-based methods (such as GraNd and EL2N) can only capture the local error signal of the sample in a specific training stage. They cannot reflect the global contribution of the sample in the whole training process, which can easily lead to data selection getting stuck in local optima and affect the generalization ability of the model.

[0007] This invention discloses a data weight optimization system for sequence recommendation based on reinforcement learning, comprising: The policy network module is used to receive the feature embeddings and current weights of the training samples and generate continuous weight adjustment actions; A memory buffer module is used to store experience data during the training process. The experience data includes state, action, reward and action log probability. The state is the concatenation data of sample feature embedding and current weight. The performance evaluation module is used to evaluate the performance of the recommendation model under different weight configurations on the validation set, and generate a reward signal based on the performance improvement. The weight update module is used to employ a near-end policy optimization algorithm, which limits the policy update magnitude through a pruning alternative objective function, thereby achieving stable iterative updates of the policy network module parameters. Furthermore, the policy network module adopts a multi-level feature fusion Actor network architecture, specifically including: The feature extraction network employs a two-layer neural network equipped with LayerNorm normalization and Dropout regularization mechanisms to extract sequence feature representations from training samples. The weighted feature processing network is used to independently encode the current weight information of a sample; A sample proportioning processing network is used to independently encode sample usage proportion information; A feature fusion network is used to integrate the sequence feature representation, encoded weight information, and encoded proportion information to output fused features; Action mean network and action variance network are used to receive the fused features, output the normal distribution parameters of the weight adjustment actions, and sample the continuous weight adjustment amount from the normal distribution; Furthermore, when the performance evaluation module generates the reward signal, the calculated recommendation metrics include Recall@K and NDCG@K, where... Specifically: Calculate the improvement amount of a single indicator ,in, For the performance of time step t, The performance at time t+1 after updating the weights is given, where m is the recommended metric. Calculate the comprehensive reward signal Where M is the set of recommendation indicators, The importance weights of each indicator are given, with Recall@5, Recall@10, NDCG@5, and NDCG@10 having weights of 0.40, 0.30, 0.30, and 0.20, respectively. Furthermore, when the weight update module executes the near-end policy optimization algorithm, it calculates the probability ratio and the advantage function to estimate the network parameters of the optimization policy through the pruned alternative objective function. The advantage function adopts the immediate reward approximation. In each iteration, multiple rounds of near-end policy optimization updates are performed on the same batch of empirical data, and a KL divergence threshold of 0.015 is preset as an early stopping condition. If the KL divergence exceeds the threshold, the current round of updates is stopped. Furthermore, the weight feature processing network is a fully connected network; the sample ratio processing network is also a fully connected network, and the encoding dimension is consistent with the output dimension of the weight feature processing network. When fine-tuning large models, a hierarchical sampling method is used to select samples from high-weight intervals. The high-weight intervals for the Games dataset are [0.9, 1.0), and the high-weight intervals for the MicroLens-50K dataset are [0.85, 1.0). This invention also discloses a method for optimizing data weights in sequence recommendation based on reinforcement learning, comprising the following steps: S1: Obtain a training dataset containing user interaction sequences and target items, use a pre-trained model to extract feature embeddings of the samples, and initialize the sample weight vector, with the initial value of the sample weight set to a medium value. S2: Construct a Markov decision process, defining the state as the concatenation of sample feature embeddings and current weights, the action as the weight adjustment amount output by the policy network, and the reward as the performance improvement value of the pre-trained model on the validation set, transforming data weight optimization into a sequence decision problem; S3: In each optimization iteration, the policy network generates weight adjustment actions based on the current state, and the updated weights are restricted to [0.1, 1.0] by a pruning function. Then, Bernoulli sampling is performed based on the updated weights to generate a training subset and train a temporary recommendation model. S4: Evaluate the performance of the temporary recommendation model on the validation set and calculate the performance improvement value; the performance improvement value is obtained based on a weighted improvement of Recall@K and NDCG@K, where The weights for each indicator are Recall@5=0.40, Recall@10=0.30, NDCG@5=0.30, and NDCG@10=0.20, respectively. S5: Based on the performance improvement value as a reward signal, update the policy network parameters using the near-end policy optimization algorithm; repeat steps S3-S4 until the policy network converges or reaches the preset maximum number of iterations, stop optimization and output the final sample weight configuration. Furthermore, the initial values ​​of the sample weights in step S1 are set to medium values, specifically: the sample weights are initialized using a Xavier normal distribution, and the gain coefficient is set to 0.01; the weights of the LayerNorm layer of the policy network are initialized to 1.0, and the bias is initialized to 0. Furthermore, the Bernoulli sampling based on the updated weights described in step S3 specifically involves: each sample being selected into the training subset with probability based on its current weight; if no sample is selected after sampling, the sample with the highest current weight is forcibly selected to ensure that the training subset is not empty; the actual sample usage ratio after sampling is recorded, and the actual sample usage ratio is input into the sample ratio processing network of the policy network for independent encoding, and participates in feature fusion together with the sample feature embedding and the encoded current weights to assist in generating weight adjustment actions; Furthermore, the policy network described in step S2 adopts a multi-level feature fusion architecture, specifically as follows: Sample sequence feature embeddings are extracted using a two-layer neural network equipped with LayerNorm normalization and Dropout regularization; The current weight information of the sample is independently encoded through a fully connected network; The sample usage ratio information is independently encoded through a fully connected network, and the encoding dimension is consistent with the weight information encoding dimension. The sequence features are embedded, the encoded weight information and the encoded proportion features are concatenated, and after batch normalization, they are input into the action mean network and the action variance network, which respectively output the normal distribution mean and variance of the weight adjustment action. Furthermore, the training of the temporary recommendation model described in step S3 specifically involves: The temporary recommendation model is trained using a Bayesian personalized ranking loss function, which is defined as follows: ,in, and These are the predicted preference scores of user u for positive sample i and negative sample j, respectively; the temporary recommendation model is subjected to a fixed number of iterative training rounds.

[0008] This invention achieves at least the following technical effects: 1. Based on the "Data Weight Optimization Framework Based on Reinforcement Learning", data weight optimization is modeled as a Markov decision process and the weights are dynamically adjusted through the PPO algorithm. The PPO framework ensures the stability of policy updates through the substitution objective function of pruning, avoiding the training fluctuation problem that is prone to occur in traditional reinforcement learning methods. At the same time, through the state space design of "sample feature embedding + current weight", the policy network can fully understand the intrinsic features and current importance of the samples, and generate more accurate weight adjustment policies. This not only enables small recommendation models to surpass the performance of training with the full amount of data with only a carefully selected subset (such as 95% of the Games dataset and 85% of the MicroLens-50K dataset), but also enables large language models to achieve the same effect as fine-tuning with the full amount of data with only 2% of the original training data, completely getting rid of the limitations of static weight schemes. 2. By using the "multi-dimensional information fusion Actor network architecture", a variety of weight distributions are generated by combining sample feature embedding, current weight and proportion information. This can distinguish the true contribution value of different samples in a fine-grained manner, and significantly improve the effectiveness of data selection and the final performance of the recommendation model. 3. With the "complete Markov decision process and PPO stable optimization mechanism", the weight adjustment has time-dependent and global optimization capabilities, avoiding the local optimum problem of supervised learning paradigm and ensuring that the selection of data subsets is in line with long-term performance goals; 4. By adopting a "lightweight proxy model scoring mechanism" and using SASRec, which has fewer parameters, as the proxy model, the sample selection time for the Games dataset was reduced from 13.3 hours to 3.5 hours and the MicroLens dataset was reduced from 29.4 hours to 2.8 hours. The advantages of long text scenarios are more prominent, and the deployability is greatly improved. 5. By using a "reward function based on global performance metrics", the weighted improvement of Recall@K and NDCG@K on the validation set is used as the reward to ensure that the weight adjustment always aims to improve the overall recommendation performance, avoid local signal misleading, and further improve the accuracy and reliability of data selection.

[0009] The data weight optimization method and system provided in this application embodiment enable the recommendation system to automatically identify high-value training samples, significantly reduce the amount of training data while ensuring or improving performance, thereby reducing computational costs, shortening training time, and demonstrating wide practical value in various application scenarios. Attached Figure Description

[0010] Figure 1 This is a schematic diagram of the architecture of the reinforcement learning-based sequence recommendation data weight optimization system of the present invention; Figure 2 This is a schematic diagram of the policy network module structure of the data weight optimization system for sequence recommendation based on reinforcement learning according to the present invention; Figure 3 This is a schematic diagram of the basic process of the reinforcement learning-based sequence recommendation data weight optimization method of the present invention; Figure 4 This is a schematic diagram of the extended process of the reinforcement learning-based sequence recommendation data weight optimization method of the present invention; Figure 5 This is a schematic diagram of the internal structure of a computer device to which the reinforcement learning-based sequence recommendation data weight optimization method of the present invention is applicable. Detailed Implementation

[0011] The following description, with reference to the accompanying drawings, illustrates several preferred embodiments of the present invention to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.

[0012] In the accompanying drawings, components with the same structure are indicated by the same numerical designation, and components with similar structures or functions are indicated by similar numerical designations. The dimensions and thicknesses of each component shown in the drawings are arbitrary, and the present invention does not limit the dimensions and thicknesses of each component. To make the illustrations clearer, the thickness of some components has been appropriately exaggerated in the drawings.

[0013] This invention proposes a data weight optimization system and method based on reinforcement learning for sequence recommendation, aiming to solve the problems of poor adaptability, insufficient global optimization capability, and high computational complexity of existing data subset selection methods for sequence recommendation. By dynamically optimizing sample weights through a reinforcement learning framework, the performance of recommendation models (including small recommendation models and large language models) is improved while reducing computational costs.

[0014] In one embodiment, the present invention provides a data weight optimization system based on reinforcement learning, such as... Figure 1 As shown, this system employs a reinforcement learning framework to dynamically optimize the importance weights of training samples. It achieves adaptive weight adjustment within the Proximal Policy Optimization (PPO) framework using the policy gradient method. The system comprises a policy network module, a memory buffer module, a performance evaluation module, and a weight update module. The policy network module generates weight adjustment policies based on the current state of the samples, employing an Actor network architecture and including sub-modules for feature extraction, weight processing, and action generation. The memory buffer module stores empirical data during training and supports batch policy updates. The performance evaluation module assesses model performance under different weight configurations on the validation set and generates reward signals for reinforcement learning. The weight update module optimizes the policy network parameters based on the PPO algorithm, ensuring the stability and effectiveness of weight adjustments.

[0015] like Figure 2 As shown, the architecture of the policy network fully considers the characteristics of recommendation data. The network input includes the feature embeddings of the samples and their current weights. High-level features are extracted through multiple fully connected layers and activation functions, and the final output is the mean and standard deviation of the weight adjustments, used to generate actions that follow a normal distribution. This design enables the network to learn complex feature-weight mapping relationships and generate reasonable weight adjustment strategies.

[0016] This system employs a reinforcement learning framework to dynamically optimize the importance weights of training samples. It achieves adaptive weight adjustment within the proximal policy optimization (PPO) framework using the policy gradient method, where: The policy network module generates weight adjustment actions based on the feature representation of the samples and the current weights. This module employs an innovative multi-level feature fusion architecture, comprising four core components: a feature extraction network, a weight feature processing network, a sample ratio processing network, and a feature fusion network. The feature extraction network uses a two-layer neural network structure, with each layer equipped with LayerNorm normalization and Dropout regularization mechanisms, to perform deep feature extraction on the input sample embeddings. The weight feature processing network and the sample ratio processing network independently encode the current weight information and the sample usage ratio, respectively. The feature fusion network intelligently integrates multi-dimensional information, and finally outputs the complete distribution parameters of the weight adjustment policy through separate action mean and action variance networks, achieving precise control over the magnitude and uncertainty of weight adjustments. The memory buffer module is used to store experience data during training, including information such as state, action, reward and action log probability, to provide data support for batch updates of the policy network; The performance evaluation module is used to evaluate the performance of models trained with different weights on the validation set, calculate the degree of improvement of multiple recommendation metrics (such as Recall@K, NDCG@K), and generate reward signals for reinforcement learning. The weight update module is used to update the policy network parameters based on the PPO algorithm, and the stability of the policy update is ensured by using a pruned alternative objective function.

[0017] In another embodiment, the execution flow of the system includes: Initialize the weights of all training samples to a moderate value to ensure that there is enough data for model learning in the initial stage; Extract feature representations from the training data and use the output of the last layer of the recommendation model as the semantic features of the samples; In each optimization iteration, the policy network generates weight adjustment actions based on the current state and updates the sample weights; Bernoulli sampling is performed based on the updated weights to generate a training subset, a temporary model is trained, and its performance is evaluated. Improved computational performance is used as a reward signal, and the policy network is updated using the PPO algorithm. Repeat the above process until convergence or the maximum number of iterations is reached.

[0018] In another embodiment, the present invention provides a data weight optimization method based on reinforcement learning; This method dynamically adjusts sample weights during the training process of the recommender system, where: The computer device used to perform this method receives a training dataset containing user interaction sequences and target items; The embedding representation of each sample is obtained through the feature extraction module; Initialize the weight vectors and construct the reinforcement learning environment; At each time step, a training subset is generated by sampling based on the current weight distribution; Evaluate the performance of the model trained using this subset and calculate the reward signal; The weight allocation strategy is updated based on the cumulative reward optimization strategy network.

[0019] The data weight optimization system adopted in this embodiment has three advantages: First, it achieves adaptive optimization of sample weights through a reinforcement learning framework, which is more flexible and adaptable than static methods. Second, it highlights important samples while ensuring data diversity through a continuous weight mechanism and Bernoulli sampling. Third, it guides weight adjustment through global performance feedback, ensuring that the optimization objective is consistent with the final performance index and avoiding local optima.

[0020] The data weight optimization system and method disclosed in this embodiment achieve the following: First, dynamic optimization of weights is achieved through a reinforcement learning framework, avoiding the limitations of static scoring methods; second, fine-grained sample importance assessment is provided through a continuous weight allocation mechanism, improving the accuracy of data selection; and third, global performance indicators are used as reward signals to ensure that weight optimization always aims to improve overall recommendation performance.

[0021] To further refine the above data weight optimization method, based on the same inventive concept, another embodiment provides a data weight optimization method for a recommendation system based on reinforcement learning, the method comprising: Establish a data weight optimization framework based on reinforcement learning (RLWORec), in which: The system models the data selection problem as a Markov decision process, dynamically adjusts sample weights through a policy network, uses model performance improvement as a reward signal, and uses the PPO algorithm to iteratively optimize the weight allocation strategy.

[0022] In another embodiment, the core components of the method include: The policy network is used to generate weight adjustment actions based on sample features and current weights, and a multilayer perceptron architecture is used to implement complex feature-weight mapping. A performance evaluation mechanism is used to evaluate the effects of different weight configurations on the validation set and calculate multi-metric weighted improvement as a reward. The PPO optimization algorithm is used to stably update the policy network through a pruning alternative objective function to prevent excessive policy changes. A data selection strategy is used to select high-quality subsets of data based on optimized weights, supporting small model training and large model fine-tuning.

[0023] In another embodiment, the policy network generates actions in the following manner: Extract the sequence features of the samples as the basic representation; Encodes the current weights and sample usage ratio information; Fuse multi-dimensional features to generate hidden representations; Output the mean and standard deviation parameters of the action distribution; The specific weight adjustment amount is obtained by sampling from the normal distribution.

[0024] In one embodiment, the performance evaluation includes: Calculate multiple metrics such as Recall@K and NDCG@K on the validation set; A comprehensive reward is obtained by weighted averaging the improvements in each indicator; Use exponential moving averages to update cumulative performance and reduce evaluation noise.

[0025] In another embodiment, the data selection strategy includes: For small models, the optimal selection ratio is found through grid search; For large models, stratified sampling is used to select samples from high-weight intervals; Adjust the weight thresholds based on the characteristics of the dataset to balance quality and diversity.

[0026] In another embodiment, the system exhibits the following technical effects in actual deployment: Smaller models outperform the full dataset using optimized subsets of data. Large models only require 2% of the data to achieve results close to or even surpassing those of the full dataset; Compared to traditional methods, performance is improved by 5-30%, and computational costs are reduced by more than 50 times.

[0027] It is important to emphasize that the user information (such as user interaction history, preference data, etc.) and related data (such as data used for training and evaluation) used in this application were obtained only after obtaining authorization from the user or relevant parties. Furthermore, the entire process of data collection, storage, and processing strictly complies with relevant laws and regulations to ensure its legality and compliance.

[0028] In specific embodiments, such as Figure 3 As shown, the device can perform a data weighting optimization process, including the following steps: Step 202: The device constructs a reinforcement learning environment based on the training dataset and initializes the sample weight vector. Where N is the number of training samples, and each weight Initialize to a moderate value to ensure that there are enough samples for training in the initial stage.

[0029] During implementation, the device first loads the training data for the recommendation system, including the user's historical interaction sequences and target items. For the sequence recommendation task, each sample is represented as... ,in For the user's historical interaction sequence, For the next interaction, the device uses a pre-trained recommendation model (such as SASRec) to extract feature representations for each sample. This feature captures semantic information about user behavior. The weights are initialized to moderate rather than random values ​​because training small models typically requires a large amount of data, thus avoiding insufficient training data in the initial stage due to excessively low weights.

[0030] Furthermore, during the initialization phase, the device prepares the necessary data structures for evaluation, including score matrices for the training, validation, and test sets. These matrices are stored in a compressed sparse row format for efficient calculation of recommendation metrics. Simultaneously, the device creates a mapping from user IDs to sample indices, facilitating subsequent sample selection based on weights.

[0031] Step 204: The device constructs a Markov Decision Process (MDP), defines the state space, action space, and reward function, and transforms the data selection problem into a sequence decision problem.

[0032] In implementation, state Defined as the concatenation of sample features and current weights, i.e.

[0033] in, It is a sample Feature embedding, The sample at time step t Weights. Actions This represents the amount of adjustment to the weights, determined by the policy network. Rewards are generated based on the current state. This is defined as an improvement in model performance; the specific calculation method will be detailed in subsequent steps. This MDP modeling approach enables the system to make optimal decisions based on historical information and the current state.

[0034] Furthermore, the device employs a continuous motion space design, where motion values ​​are obtained through normal distribution sampling, and the distribution parameters are output by the policy network. The weight update rule is as follows:

[0035] in, To ensure that the sample maintains the minimum selection probability, Upper bound. This design ensures data diversity while allowing for higher weighting of important samples.

[0036] Step 206: In each optimization iteration, the device uses the policy network to generate weight adjustment actions, update sample weights, and train the model based on the new weights.

[0037] In this implementation, the device first inputs the sample status of the current batch into the policy network. The policy network comprises four main components: a feature extraction network, a weight feature processing network, a sample ratio processing network, and a feature fusion network (the action generation network is integrated after the feature fusion network). The feature extraction network maps sample embeddings to hidden representations, the weight feature processing network encodes the current weight information, and the two are fused together and output as parameters of the action distribution through the action generation network. The device samples the specific weight adjustment amounts from this distribution and updates the weights of the corresponding samples.

[0038] Furthermore, based on the updated weights, the device generates a training subset D' through Bernoulli sampling. Specifically, each sample With probability The sample is selected into a subset. To avoid an empty set, when no sample is selected, the device will force the selection of the sample with the highest weight. Subsequently, the device uses this subset to train a temporary recommendation model, using the Bayesian Personalized Ranking (BPR) loss function, iterating for a specified number of rounds (e.g., 5 rounds). This short-term training strategy can effectively evaluate the quality of weight allocation while ensuring efficiency.

[0039] In this embodiment, the system employs an adaptive exploration strategy to balance exploration and utilization. The exploration coefficient (exploration_scale) is initially set to 1.0 and gradually decreases at a decay rate of 0.995 to avoid over-exploration leading to weight fluctuations in the later stages of training, ultimately maintaining a minimum value of 0.3. This design allows the system to fully explore in the early stages of training, discovering various possible weight configurations, and gradually converging to the optimal strategy as training progresses. When the system detects that the sample utilization rate is too low (e.g., below 30%), it automatically increases the exploration intensity to ensure sufficient samples participate in training.

[0040] The weights are initialized using a Xavier normal distribution with a gain coefficient of 0.01. This small gain design ensures that the output amplitude of the policy network is small in the early stages of training, avoiding training instability caused by drastic weight changes. For the LayerNorm layer, the weights are initialized to 1.0 and the biases to 0 to guarantee the identity mapping property in the initial stage.

[0041] Specifically, after generating the training subset, the device records the actual proportion of samples used. This information serves as additional input to the policy network, helping it learn the relationship between sample proportion and performance. When the sample proportion is too low (e.g., below 30%), the policy network automatically adjusts the action distribution, tending to increase the weights of more samples to ensure training stability.

[0042] In another embodiment, such as Figure 4 As shown, the data weight optimization process also includes: Step 208: The device evaluates the performance of the temporary model on the validation set and calculates the improvement of multiple recommendation metrics as a reward signal; Step 210: Update the policy network parameters using the PPO algorithm and optimize the weight allocation strategy.

[0043] The performance evaluation employs a weighted combination of multiple metrics, including Recall@K and NDCG@K. ).set up Indicates the performance at time step t. Let m represent the performance after updating the weights, then the improvement of metric m is... The overall reward is calculated as follows:

[0044] Where M is the set of indicators, The importance weights for each indicator are defined. In the experiment, the weights of the core indicators Recall@5, Recall@10, NDCG@5, and NDCG@10 were set to 0.40, 0.30, 0.30, and 0.20, respectively, reflecting the emphasis on performance in high-ranking positions.

[0045] Next, the device updates the policy network using the PPO algorithm. PPO limits the policy update magnitude through a pruning-based alternative objective function, ensuring training stability. The objective function is defined as:

[0046] in, It's a probability ratio. It is the estimation of the advantage function. This refers to the pruning parameter (e.g., 0.2). Due to the high state dimension and the global nature of the reward signal, this embodiment adopts a simplified approach, directly using the immediate reward. As an advantage estimate, i.e. Although this approximation is crude, it provides a stable update signal in practice.

[0047] The following will provide supplementary explanations of the solutions in the embodiments of this application, in conjunction with specific formulas: In this embodiment, the dataset D is represented as ,in For the user's historical interaction sequence, The goal of sequence recommendation is to learn a function. This enables it to accurately predict For recommendation models using BPR loss, the training objective is:

[0048] in, and Let represent the predicted preference scores of user u for positive sample i and negative sample j, respectively. This is the sigmoid function.

[0049] In this embodiment, the objective function of reinforcement learning is defined as maximizing the cumulative reward:

[0050] Among them, trajectory The probability is:

[0051] The policy gradient can be derived as follows:

[0052] in, This is the cumulative reward starting from time step t.

[0053] In this embodiment, the policy network A multilayer perceptron architecture is adopted. Given an input state... The network first processes the data through a feature extraction layer:

[0054] Weight information is processed through a separate layer:

[0055] Sample proportion information (if available) is also encoded:

[0056] The three features are fused to generate action distribution parameters:

[0057] The mean parameter is calculated as follows:

[0058] in, This is a scaling factor used to control the amplitude of the motion.

[0059]

[0060] in, and These are the lower and upper bounds of the logarithmic standard deviation, respectively, used to prevent numerical instability caused by excessively small or large variances.

[0061] Actions follow a normal distribution Mid-sampling, weights updated as follows:

[0062] In this embodiment, the core of the PPO algorithm is the alternative objective function for pruning. The probability ratio is defined as follows:

[0063] The cutting target is:

[0064] Considering the characteristics of this task (high-dimensional state space, global reward signal), a simplified advantage estimation method is adopted. .

[0065] It should be noted that in actual implementation, each iteration will perform multiple rounds of PPO updates on the same batch of data (e.g., 4 rounds), and set a KL divergence threshold (e.g., 0.015) as an early stopping condition to prevent the policy update from being too large and causing training instability.

[0066] In this embodiment, the specific implementation of data selection includes two scenarios. For small model training, the device generates data subsets with different proportions, and the optimal selection proportion is found through grid search. Experimental results show that the optimal proportion for the Games dataset is 95%, and for the MicroLens-50K dataset it is 85%, indicating that even for small models, appropriate data selection can improve performance. For fine-tuning of large models, a hierarchical sampling strategy is adopted. Let the weight distribution be... Weighting interval The data is divided into multiple sub-intervals, and samples are taken proportionally from each sub-interval. Compared to directly selecting the top-K samples, this method better preserves data diversity and improves performance by 2-5%.

[0067] In another embodiment, after completing weight optimization, the device performs data subset selection and model training. For small model scenarios, the device evaluates the performance under different selection ratios (5% to 100%, with a step size of 5%) and selects the ratio with the highest Recall@5 on the validation set as the optimal configuration. For large model fine-tuning scenarios, the device adopts a hierarchical sampling strategy, selecting samples within the high-weight interval.

[0068] In practice, stratified sampling sets different thresholds based on the characteristics of the dataset. For example, the Games dataset uses the range [0.9, 1.0), while the MicroLens-50K dataset uses the range [0.85, 1.0). This differentiated strategy considers the characteristics of different datasets: the Games dataset has relatively concentrated user behavior, requiring stricter filtering; the MicroLens dataset has higher content diversity, and appropriately relaxing the threshold helps maintain diversity. Experiments show that stratified sampling can improve performance by 2-5% compared to greedy selection (directly selecting the sample with the highest weight), validating the importance of maintaining data diversity.

[0069] Furthermore, during large model fine-tuning, the device constructs a few-shot learning task based on selected samples. Specifically, 1024 selected samples (approximately 2% of the total data) are formatted into an input format suitable for a large language model, including context, user history, and prediction targets. Using parameter-efficient fine-tuning techniques such as LoRA, only a small number of model parameters are updated, significantly reducing computational and storage overhead.

[0070] To verify the effectiveness of the RLWORec scheme in this application, the following comparative experiments illustrate its performance advantages in scenarios involving small model training and large model fine-tuning: The experiments used two publicly available datasets, Games and MicroLens-50K, and employed Recall@K and NDCG@K as evaluation metrics. All experiments were conducted on an NVIDIA RTX 3090 GPU, with each experiment repeated multiple times (e.g., 3 times) and the average value taken to ensure the reliability of the results. On the MicroLens-50K dataset, the existing DEALRec method relies on large language models to calculate influence scores, resulting in huge overhead that scales with the input length, and the sample selection process is particularly time-consuming; while the method of this invention only relies on the fast training and evaluation of the lightweight SASRec proxy model, which significantly reduces the time cost of sample selection. Table 1 shows a comparison of the data selection performance with related techniques after training based on the above parameters: Table 1: Experimental Results of Data Subset Selection in Small Model Scenarios

[0071] The methods in related technologies include: (1) Influence Score: Based on the influence function, the influence of a sample on the model performance is estimated. The importance is evaluated by approximating the influence of removing a single sample on the loss function.

[0072] (2) GraNd: Selects samples based on the gradient norm in the early stages of training, believing that samples with a large gradient norm contain more information.

[0073] (3) EL2N: Uses the L2 norm of the error vector to identify important samples. The larger the error, the more important the sample is considered.

[0074] (4) Random: Randomly select samples as baselines for comparison.

[0075] (5) Full Data: Use all training data as a performance upper bound reference.

[0076] As shown in Table 1, the proposed RLWORec achieves the best performance across all metrics, making it the only method consistently outperforming the full dataset. On the Games dataset, Recall@10 is improved by 2.96%, and NDCG@10 by 1.53%; on the MicroLens-50K dataset, Recall@10 is improved by 9.43%, and NDCG@10 by 3.57%. Traditional importance scoring methods (Influence Score, GraNd, EL2N) perform poorly, even worse than random selection, indicating that these methods designed for classification tasks are not suitable for recommendation scenarios. This further validates the effectiveness of this invention in small model training scenarios.

[0077] To fully verify the applicability of this invention to both small recommendation models and large language models, the following table 2 presents the Few-shot experimental results for large model fine-tuning scenarios. The specific results are as follows: Table 2: Experimental Results of Few-shot Fine-tuning in Large Model Scenarios

[0078] The methods in related technologies include: (1) Random: Randomly select 1024 samples for fine-tuning as the baseline comparison method.

[0079] (2) DEALRec: A data selection method based on an influence function specifically designed for recommendation systems. This method assesses importance by calculating the impact of each sample on the model's loss function and uses a large language model to calculate an effort score, attempting to identify the samples that contribute the most to the model's learning.

[0080] (3) DConRec: A recommendation data selection method based on the idea of ​​dataset condensation. This method attempts to generate selection probabilities that can represent the entire dataset and select important samples by minimizing the distribution difference between the original dataset and the compressed dataset.

[0081] (4) Full Data: Fine-tuning using the complete training dataset represents the upper bound of the performance of traditional methods, but it is also the most computationally expensive option.

[0082] As shown in Table 2, RLWORec achieves performance close to or exceeding that of the full dataset using only 2% of the data. On the Games dataset, Recall@10 reaches 0.0189, a 32.2% improvement compared to random selection and an 18.9% improvement compared to DEALRec. The advantage is even more pronounced on the MicroLens-50K dataset, with a 13.1% improvement in Recall@20 and an 11.3% improvement in NDCG@20. This fully demonstrates the effectiveness of RLWORec in identifying high-quality training samples.

[0083] Against the backdrop of the rapid development of large language models, the high cost of fine-tuning (fine-tuning with full data requires significant computing power and time) has become a major factor restricting their widespread application. The efficient data selection solution provided by this invention allows large language models to quickly adapt to recommendation tasks with only 1024 samples (approximately 2% of the original data), without relying on the full dataset. This significantly reduces the deployment threshold and operating costs of large models, while also substantially reducing computational overhead, demonstrating significant commercial value and market potential.

[0084] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise expressly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders.

[0085] In another embodiment, the present invention also provides a computer device including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement all the steps of the above-described data weight optimization system and method.

[0086] In another embodiment, the detailed structure of the above-described computer device is as follows: Figure 5As shown, this refined architecture includes a processor, memory, input / output interfaces (I / O interfaces), and a communication interface: the processor, memory, and I / O interfaces are interconnected via a system bus, while the communication interface relies on the I / O interfaces to connect to the system bus; the processor undertakes the device's computation and control functions; the memory includes main memory and non-volatile storage media, where main memory provides the execution environment for the operating system and program execution, and the non-volatile storage media stores the operating system, programs, and database; the database is specifically used to store training data, model parameters, and weight information; the I / O interface is used to realize data interaction between peripherals and the processor, and the communication interface is used to establish communication with external terminals via a network; when the processor runs the program stored in the non-volatile storage media, it can execute a data weight optimization method based on reinforcement learning.

[0087] It should be noted that, Figure 5 The diagram shown is merely a structural illustration of the part related to the solution in this application and does not limit the computer device to which this invention applies. In specific implementations, the computer device may contain more or fewer components than shown in the diagram, or some components may be combined, or even different device configurations may be used.

[0088] In another embodiment, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described data weight optimization method.

[0089] In another embodiment, the present invention also provides a computer program product comprising a computer program that, when executed by a processor, enables the aforementioned data weight optimization processing task.

[0090] It should be understood that those skilled in the art can use computer program instructions to drive the relevant hardware to complete the method flow in the above embodiments. The relevant program can be stored on a non-volatile computer-readable storage medium and implement the corresponding functional steps when executed.

[0091] The embodiments of this invention can be widely applied to various sequence recommendation scenarios, such as smart TVs, mobile applications, e-commerce platforms, short video recommendations, news pushes, online advertising, online education, and social platforms, demonstrating significant practical value in reducing training costs and improving model performance. These application scenarios cover the core areas of the modern digital economy, from traditional product recommendations to emerging content distribution, from enterprise-level personalized services to consumer-level entertainment applications, where the technical solutions of this invention can play a crucial role. Especially against the backdrop of the rapid development of the mobile internet and the Internet of Things, the demand for efficient data processing from edge computing devices is increasing, making the lightweight data optimization solution provided by this invention of significant practical importance.

[0092] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A reinforcement learning based sequence recommendation data weight optimization system, characterized in that, The method comprises the following steps: A policy network module is configured to receive a feature embedding of a training sample and a current weight, and generate a continuous weight adjustment action; A memory buffer module is configured to store experience data in a training process, wherein the experience data comprises a state, an action, a reward, and a logarithmic probability of the action, and the state is spliced data of the feature embedding of the sample and the current weight; A performance evaluation module is configured to evaluate the performance of a recommendation model under different weight configurations on a validation set, and generate a reward signal based on the performance improvement; A weight update module is configured to use a proximal policy optimization algorithm to limit the policy update range by a clipped surrogate objective function, and realize stable iterative update of parameters of the policy network module.

2. The reinforcement learning-based sequence recommendation data weight optimization system of claim 1, wherein, The policy network module adopts a multi-level feature fusion Actor network architecture, which specifically comprises: A feature extraction network configured to extract a sequence feature representation of a training sample by using a double-layer neural network equipped with a LayerNorm normalization mechanism and a Dropout regularization mechanism; A weight feature processing network configured to independently encode current weight information of a sample; A sample proportion processing network configured to independently encode sample usage proportion information; A feature fusion network configured to integrate the sequence feature representation, the encoded weight information, and the encoded proportion information, and output fused features; An action mean network and an action variance network configured to receive the fused features, output normal distribution parameters of a weight adjustment action, and sample a continuous weight adjustment amount from the normal distribution.

3. The reinforcement learning-based sequence recommendation data weight optimization system of claim 1, wherein, The performance evaluation module generates a reward signal, and the recommended indicators calculated include Recall@K and NDCG@K, wherein , specifically: Computing single metric improvement where, is the performance at time step t, is the performance at time step t+1 after updating the weights, m is the recommended metric; Computing a composite reward signal where M is a set of recommendation metrics, are the importance weights of each metric, and the weights of Recall@5, Recall@10, NDCG@5, NDCG@10 are 0.40, 0.30, 0.30, 0.20, respectively.

4. The reinforcement learning-based sequence recommendation data weight optimization system of claim 1, wherein, When the weight update module executes the proximal policy optimization algorithm, it calculates a probability ratio and an advantage function estimation to optimize the policy network parameters by using a clipped surrogate objective function; the advantage function uses an immediate reward approximation, a same batch of experience data is updated by multiple rounds of proximal policy optimization in each iteration, and a KL divergence threshold of 0.015 is preset as an early stopping condition; if the KL divergence exceeds the threshold, the current round of update is stopped.

5. The reinforcement learning-based sequence recommendation data weight optimization system of claim 2, wherein, The weight feature processing network is a fully connected network; the sample proportion processing network is also a fully connected network, and the encoding dimension is consistent with the output dimension of the weight feature processing network.

6. A sequence recommendation data weight optimization method based on reinforcement learning, characterized in that, The method comprises the following steps: S1: obtaining a training data set containing a user interaction sequence and a target item, extracting a feature embedding of a sample by using a pre-trained model, initializing a sample weight vector, and setting an initial value of the sample weight to a medium value; S2: constructing a Markov decision process, defining a state as spliced data of a sample feature embedding and a current weight, an action as a weight adjustment amount output by a policy network, and a reward as a performance improvement value of a pre-trained model on a validation set, and converting data weight optimization into a sequence decision problem; S3: in each optimization iteration, generating a weight adjustment action based on a current state by using a policy network, limiting the updated weight to [0.1, 1.0] by using a clipping function, and generating a training subset and training a temporary recommendation model based on the updated weight; S4: evaluating the performance of the temporary recommendation model on the validation set, and calculating a performance improvement value. The performance improvement value is obtained based on weighted improvement of Recall@K and NDCG@K, wherein , and the weights of each index are Recall@5=0.40, Recall@10=0.30, NDCG@5=0.30, and NDCG@10=0.20, respectively. S5: updating the policy network parameters based on the performance improvement value as a reward signal using a proximal policy optimization algorithm; repeating steps S3-S4 until the policy network converges or a preset maximum number of iterations is reached, stopping optimization and outputting a final sample weight configuration. 7.The reinforcement learning based sequence recommendation data weight optimization method of claim 6, wherein, In step S1, the sample weight initial value is set to a medium value, specifically: the sample weight is initialized using Xavier normal distribution, and the gain coefficient is set to 0.01; the LayerNorm layer weight of the policy network is initialized to 1.0, and the bias is initialized to 0. 8.The reinforcement learning based sequence recommendation data weight optimization method of claim 6, wherein, In step S3, the Bernoulli sampling is performed based on the updated weight, specifically: each sample is selected into the training subset with the current weight as the probability; If no sample is selected after sampling, the sample with the highest current weight is forcibly selected; the actual sample usage ratio after sampling is recorded, and the actual sample usage ratio is input into the sample ratio processing network of the policy network for independent coding, and the current weight after embedding and coding of the sample feature is jointly used for feature fusion to assist in generating a weight adjustment action. 9.The reinforcement learning based sequence recommendation data weight optimization method of claim 6, wherein, In step S2, the policy network uses a multi-level feature fusion architecture, specifically: A double-layer neural network equipped with LayerNorm normalization and Dropout regularization is used to extract sample sequence feature embedding; The current weight information of the sample is independently coded through a fully connected network; The sample usage ratio information is independently coded through a fully connected network, and the coding dimension is consistent with the weight information coding dimension; The sequence feature embedding, coded weight information, and coded ratio feature are spliced, and after batch normalization processing, they are input into the action mean network and the action variance network to output the normal distribution mean and variance of the weight adjustment action, respectively. 10.The reinforcement learning based sequence recommendation data weight optimization method of claim 6, wherein, In step S3, the temporary recommendation model is trained, specifically: training the interim recommendation model using a Bayesian personalized ranking loss function defined as wherein, and are the predicted preference scores of user u for positive sample i and negative sample j, respectively; and performing a specified number of iterations of training on the interim recommendation model, fixed.

Images