Resource allocation method based on reinforcement learning

Abstract

This application relates to the field of network optimization technology using reinforcement learning, and discloses a reinforcement learning method, system, and storage medium for virtual prize allocation. The method includes: analyzing latency fluctuation distribution by acquiring user request frequency and network bandwidth indicators, identifying bottlenecks in high-concurrency scenarios, and training a reinforcement learning model using historical data records to dynamically adjust the resource allocation scheme; improving the priority processing and low-latency processing of business data by optimizing the resource allocation scheme and network environment change trends; and optimizing the accuracy and convergence speed of the reinforcement learning model using adaptive prediction parameters and neural network proxies to eliminate network jitter and latency differences under high concurrency. This application improves the fairness and stability of online lottery systems.

Description

Technical Field

[0001] This application relates to the field of network optimization technology in reinforcement learning, and in particular to a reinforcement learning method, system, and storage medium for virtual prize allocation. Background Technology

[0002] Online lottery systems have wide applications in digital entertainment, social interaction, and marketing, especially in promotional activities and user incentives, where they can attract a large number of users and enhance platform activity. With the rapid development of internet technology, the application scenarios for online lottery systems are also increasing, becoming an important means of enhancing user engagement and brand influence. Particularly in e-commerce platforms, social media, and various online events, lottery activities typically involve a large number of participants, making ensuring fairness and stability a key challenge in system design.

[0003] However, in high-concurrency scenarios, online lottery systems face performance bottlenecks caused by a surge in user requests. This often leads to system response delays and uneven resource allocation during the lottery process, ultimately affecting user experience and the fairness of reward distribution. During peak periods, user requests increase dramatically, especially at the start of promotional activities, when user traffic can surge in a short time, placing extremely high demands on system resource allocation. Current systems often use methods such as increasing server resources or optimizing database queries to alleviate the pressure from high concurrency, but these methods are mostly static and difficult to cope with dynamically changing user demands, resulting in the inability to achieve stable and rapid resource scheduling even during peak periods.

[0004] Traditional resource allocation methods are primarily based on preset rules, failing to account for instantaneous changes in user request frequency and network bandwidth. This static allocation approach often cannot flexibly adjust to real-time user behavior and network conditions. For example, in some high-concurrency scenarios, some users may experience higher latency due to network bandwidth limitations or geographical differences, while other users may receive unnecessary priority due to over-allocation of resources, leading to resource waste. This imbalance in resource allocation not only exacerbates system latency fluctuations but may also raise user concerns about the fairness of the lottery, severely impacting user experience and brand image.

[0005] Existing solutions often employ fixed rules or simple load balancing algorithms for resource allocation, but these methods cannot dynamically adapt to changes in the network environment and user behavior. Even increasing server resources or optimizing data cannot fundamentally solve the latency problem under high concurrency, especially when user request frequency and network bandwidth fluctuate significantly. Traditional methods cannot adjust resource allocation in a timely manner, resulting in the system failing to maintain stability and fairness during peak periods.

[0006] Therefore, how to dynamically adjust resource allocation based on real-time changes in user requests in high-concurrency scenarios to ensure that all users can obtain a fair and low-latency lottery experience has become a core problem that urgently needs to be solved in the design of current online lottery systems. Summary of the Invention

[0007] This application provides a reinforcement learning method, system, and storage medium for virtual prize allocation, which can improve the fairness and stability of online lottery systems.

[0008] Firstly, this application provides a reinforcement learning method for virtual prize allocation, the method comprising:

[0009] S1. Obtain the user request frequency and network bandwidth indicators of the online lottery system, extract the peak and valley load characteristics, and further obtain the latency fluctuation distribution;

[0010] S2. Based on the delay fluctuation distribution, a real-time monitoring strategy is adopted to analyze the delay of the processing stage and determine the bottleneck point in the high-concurrency scenario;

[0011] S3. Obtain historical data records corresponding to the bottleneck point, train and learn user behavior fluctuation patterns through reinforcement learning model, and obtain adaptive prediction parameters.

[0012] S4. Based on the adaptive prediction parameters, dynamically adjust the resource allocation scheme and generate a network environment change trend based on the adjusted resource allocation scheme;

[0013] S5. Based on the network environment change trend, extract the optimized path from the historical data records to obtain the resource allocation adjustment update sequence;

[0014] S6. Adjust the priority of business data according to the update sequence, obtain the intermediate result of low-latency processing based on the adjusted business data, organize the allocation logic according to the intermediate result of low-latency processing, and determine the stable index of event trigger allocation.

[0015] S7. Feed the stability index back to the reinforcement learning model, determine whether it is necessary to iterate the adaptive prediction parameters, and further optimize the reinforcement learning model based on the adaptive prediction parameters to obtain the dynamic network environment optimization configuration.

[0016] Secondly, this application provides a reinforcement learning system for virtual prize allocation, the system comprising:

[0017] The load feature extraction module is used to obtain the user request frequency and network bandwidth indicators of the online lottery system, extract the load features during peak and valley periods, and further obtain the latency fluctuation distribution.

[0018] The bottleneck point location module is used to analyze the delay of the processing link based on the delay fluctuation distribution and to determine the bottleneck point in the high-concurrency scenario by adopting a real-time monitoring strategy.

[0019] The parameter adaptation module is used to obtain historical data records corresponding to the bottleneck point, learn the user behavior fluctuation pattern through reinforcement learning model training, and obtain adaptive prediction parameters.

[0020] The resource dynamic adjustment module is used to dynamically adjust the resource allocation scheme according to the adaptive prediction parameters, and generate a network environment change trend based on the adjusted resource allocation scheme.

[0021] The optimized path extraction module is used to extract optimized paths from the historical data records based on the network environment change trend, and obtain the update sequence of resource allocation adjustment;

[0022] The allocation logic organization module is used to adjust the priority of business data according to the update sequence, obtain the intermediate result of low-latency processing based on the adjusted business data, organize the allocation logic based on the intermediate result of low-latency processing, and determine the stable index of event-triggered allocation.

[0023] The model iteration and optimization module is used to feed the stability index back to the reinforcement learning model, determine whether it is necessary to iterate the adaptive prediction parameters, and further optimize the reinforcement learning model based on the adaptive prediction parameters to obtain the dynamic network environment optimization configuration.

[0024] Thirdly, this application provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the aforementioned reinforcement learning method for virtual prize allocation.

[0025] The technical solution provided in this application offers an innovative solution to the problems of uneven resource allocation, system latency fluctuations, and fairness in high-concurrency scenarios in existing technologies, offering significant benefits. By introducing a reinforcement learning model, the resource allocation scheme can be dynamically adjusted based on real-time user request frequency and network bandwidth changes, enhancing the system's adaptability to sudden traffic fluctuations compared to traditional static resource allocation methods. Secondly, the reinforcement learning model, through a combination of adaptive prediction and network environment change trends, ensures that all users have a fair chance to participate in the lottery under high concurrency, avoiding user dissatisfaction caused by high latency or uneven resource allocation. Furthermore, this solution reduces latency fluctuations and network jitter, improves system response speed and stability, and continuously improves the accuracy and efficiency of resource allocation through real-time feedback mechanisms and model optimization, thereby ensuring the stable operation of the system under large-scale concurrency. This method intelligently optimizes resource allocation, adapting to the network environments of different users, resolving latency differences caused by network variations, and ultimately significantly improving user experience and platform trust. In summary, this technical solution, through intelligent optimization technology, not only solves the resource allocation problem in high-concurrency environments but also improves the fairness, stability, and efficiency of online lottery systems. Attached Figure Description

[0026] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 A flowchart of the reinforcement learning method for allocating virtual prizes in this application;

[0028] Figure 2 This is a flowchart illustrating the adaptive resource allocation and trend prediction process in this application.

[0029] Figure 3 A graph showing the convergence process of the prediction parameters of the reinforcement learning method used to allocate virtual prizes in this application;

[0030] Figure 4 A chart showing the optimization effect of a lottery activity on an e-commerce platform for the purposes of this application;

[0031] Figure 5 A schematic diagram of the reinforcement learning system for allocating virtual prizes in this application. Detailed Implementation

[0032] This application provides a reinforcement learning method, system, and storage medium for virtual prize distribution. The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" or "having" and any variations thereof are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0033] For ease of understanding, the specific process of the embodiments of this application is described below. Please refer to [link / reference]. Figure 1 One embodiment of a reinforcement learning method for virtual prize allocation in this application includes:

[0034] Step S1: Obtain the user request frequency and network bandwidth indicators of the online lottery system, extract the peak and valley load characteristics, and further obtain the latency fluctuation distribution.

[0035] In one specific embodiment, the process of performing step S1 may specifically include the following steps:

[0036] (1) Collect real-time user request frequency and network bandwidth indicators of the online lottery system to generate load data sequence;

[0037] (2) Perform data preprocessing on the load data sequence to extract peak and valley features from the time series;

[0038] (3) Calculate the statistical distribution of load changes based on peak and valley characteristics;

[0039] (4) Establish a correlation model between load and delay based on statistical distribution to obtain the delay fluctuation distribution.

[0040] Specifically, a load data sequence is generated by collecting real-time user request frequency and network bandwidth metrics from the online lottery system. A data acquisition module is set up in the system to collect user request frequency and network bandwidth metrics once per second. For example, assuming the system receives 100 user requests per second and the network bandwidth is 20Mbps during a certain period, these data will form a load data sequence, such as [100, 120, 110, 95, 100, 125, ...]. For the collected load data sequence, a smoothing filter is applied to remove noise and ensure data stationarity. Then, a peak detection algorithm is used to identify local maxima and minima in the time series, extracting peak and trough features of load changes. For example, given the load data sequence [100, 120, 110, 95, 100, 125], the peak detection algorithm extracts the local maxima of 120 and 125, and the local minimum of 95, as peak and trough features, respectively.

[0041] Based on the extracted peak and trough characteristics, the statistical distribution of load changes is calculated to generate a delay fluctuation distribution. The difference between the peak and trough values ​​is used as the amplitude of the change. For example, if the calculated amplitude is [20, 15, 25, 30, 20], these differences can be divided into multiple intervals, and the frequency of each interval is calculated using a histogram method. Assuming the frequency distribution shows that 80% of the changes are concentrated in the 20-30 interval, it indicates that most delay fluctuations occur within this range. To more accurately reflect delay fluctuations, the probability distribution of each amplitude interval can be calculated using the following formula:

[0042]

[0043] in, This represents a specific range of load variation. It represents the probability within a certain range. It is the number of samples within that interval. This represents the total number of all samples. Based on the statistical distribution of these load changes, a correlation model between load and latency is further established. For example, assuming a linear relationship between load variation and latency fluctuation, regression analysis can be used to fit the relationship between the two, resulting in the following regression equation:

[0044]

[0045] in, Indicates a time delay change. Indicates the magnitude of load change. and , where is the regression coefficient, representing the impact of load changes on latency. By fitting historical data, a predictive model for latency fluctuations can be obtained, thus effectively capturing the distribution of latency fluctuations. For example, if in an online lottery event, the load data sequence is [100, 120, 110, 95, 100, 125], and the latency data is [200ms, 250ms, 220ms, 210ms, 240ms], the regression equation obtained after fitting through regression analysis is: According to this regression model, when the load change is 25, the delay change can be predicted to be 250ms, which further provides a scientific basis for optimizing resource allocation in the system.

[0046] Step S2: Based on the latency fluctuation distribution, use a real-time monitoring strategy to analyze the latency of the processing stage and identify the bottleneck points in high-concurrency scenarios.

[0047] In one specific embodiment, the process of performing step S2 may specifically include the following steps:

[0048] (1) Set the real-time monitoring time window based on the distribution of delay fluctuations;

[0049] (2) Based on the time window, collect the delay data during the processing;

[0050] (3) Perform statistical analysis on the delay data to obtain the comprehensive service delay score, and determine whether the comprehensive service delay score exceeds the preset delay threshold;

[0051] (4) If the overall service delay score exceeds the delay threshold, computing resources will be allocated first, and a resource allocation instruction will be generated;

[0052] (5) Determine the bottleneck points in high-concurrency scenarios based on resource allocation instructions.

[0053] Specifically, by analyzing the peak and trough characteristics in the latency fluctuation distribution, the time window is set to a dynamic range of 5 to 30 seconds to capture real-time changes. For example, the system may experience significant latency fluctuations during periods of high load, and the time window will automatically adjust accordingly. Specifically, the start and end points of the time window are calculated based on the peak data in the latency fluctuation distribution to ensure that the window covers high-load periods. For example, if latency fluctuations are large during a peak period, the time window will automatically extend to 20 seconds to monitor and respond to load fluctuations. Simultaneously, the length of the time window is adjusted according to network bandwidth fluctuations; when network bandwidth fluctuates significantly, the window length may increase to ensure it can adapt to latency changes caused by bandwidth fluctuations.

[0054] Through a set time window, the system collects latency data for each request in real time and records it to form a data sequence. This latency data reflects the processing time of each user request from initiation to completion, forming a real-time latency dataset. By statistically analyzing this data, the system can calculate the average and standard deviation of the latency and compare them with a preset latency threshold. If the average latency exceeds the threshold of 100 milliseconds, the system judges it as abnormal latency, identifying a potential performance bottleneck. To calculate the average latency, all latency data are summed and then divided by the number of samples to obtain the average latency value. To calculate the standard deviation, the sum of the squared differences between each latency value and the average, the square root of which is then divided by the number of samples minus one, yields the degree of latency fluctuation. The calculation of the standard deviation helps determine whether the latency fluctuation is stable; large fluctuations may indicate the presence of unstable factors in the system.

[0055] When latency exceeds a threshold, the system prioritizes the allocation of computing resources and generates corresponding resource allocation instructions. For example, when the average latency exceeds a preset threshold, the system allocates computing resources (such as CPU) from idle servers to the processing stage to speed up response. During resource allocation, latency data points exceeding the threshold are identified, and the required additional resources are calculated. This calculation typically determines the amount of additional resources to be allocated based on the percentage of latency exceeding the threshold multiplied by the percentage of the total resource pool. The system generates specific resource allocation instructions, specifying the types of resources to be allocated and their priorities, ensuring rapid response in high-concurrency scenarios.

[0056] Step S3: Obtain historical data records corresponding to the bottleneck point, train the user behavior fluctuation pattern through reinforcement learning model, and obtain adaptive prediction parameters.

[0057] In one specific embodiment, the process of performing step S3 may specifically include the following steps:

[0058] (1) Retrieve historical data records related to the bottleneck from the database;

[0059] (2) Construct the state space of user behavior based on historical data records;

[0060] (3) Based on the state space, an action selection strategy is defined through a reinforcement learning model, and user behavior fluctuation patterns are trained and learned.

[0061] (4) Generate a reward function based on the user behavior fluctuation pattern;

[0062] (5) Optimize the action selection strategy based on the reward function to obtain adaptive prediction parameters.

[0063] Specifically, in this embodiment, by retrieving historical data records related to bottlenecks from the database, the system can comprehensively understand past user behavior and system response. This historical data includes key features such as user request frequency, latency fluctuations, and network bandwidth changes, helping the system identify potential bottlenecks during high concurrency. For example, by analyzing past lottery activity data, the system can identify that during certain high-load periods, a surge in user request frequency leads to a significant increase in system latency. Using this historical data, the system constructs a state space representing user behavior. In the state space, each user behavior is abstracted into a multi-dimensional vector, encompassing features such as user request frequency, network latency, and bandwidth. For example, the system might represent user behavior as a three-dimensional vector: [request frequency, latency, bandwidth], where the value of each dimension represents a specific indicator at the current moment. By combining these different features into a state space, the system can quantify each different user behavior scenario.

[0064] Based on the constructed state space, the reinforcement learning model defines an action selection policy to determine which resource allocation strategy the system should adopt under different user behavior patterns. For example, the reinforcement learning model selects the appropriate action based on the current state (such as request frequency, latency, bandwidth, etc.), such as "increasing CPU resource allocation," "adjusting bandwidth," or "maintaining the current resource configuration." To achieve this goal, the system employs the Q-learning algorithm, learning the optimal action strategy for a specific state through continuous interaction with the environment. In this process, the Q-value represents the expected reward of taking a specific action in a given state. The system optimizes the action selection policy by repeatedly updating the Q-value, gradually approaching the optimal solution. Specifically, the Q-value update formula is:

[0065]

[0066] in, Indicates the state Select action The current value, It's the learning rate. It's an instant reward. It is a discount factor. In Indicates a new state Any one of the possible actions from all available actions. This indicates that these actions correspond to The model, through multiple rounds of learning and adjustment, can accurately predict and optimize resource allocation strategies. To guide the system in selecting the optimal strategy, a reward function is designed. The goal of the reward function is to maximize system performance, reduce latency fluctuations, and optimize resource utilization efficiency. The reward function can be expressed as:

[0067]

[0068] in, Represents the reward value. It is a time-delayed fluctuation. This is the maximum allowable delay. It is the amount of resources used. and It is a regulating factor. In this way, the system encourages the selection of strategies that can reduce latency fluctuations and make rational use of resources, ensuring efficient resource allocation.

[0069] Based on feedback from the reward function, the system further optimizes its action selection strategy, ultimately obtaining a set of adaptive prediction parameters. By continuously optimizing these parameters, the system can adaptively adjust resource allocation according to different network conditions and fluctuations in user behavior. For example, if the system detects excessive latency fluctuations and insufficient resource allocation during a high-concurrency period, the reinforcement learning model will adjust the prediction parameters based on historical data and current feedback, making resource allocation more accurate and efficient. The optimized parameters can be updated using the following formula:

[0070]

[0071] in, These are the current prediction parameters. These are the updated parameters. The reward is obtained through actual calculation. It is the reward value predicted by the model. It is the learning step size. Through this adaptive optimization process, the system can continuously learn and adapt to fluctuations in user behavior, achieving dynamic optimization and precise allocation of network resources.

[0072] Taking an online lottery system as an example, the system retrieves historical data records related to bottlenecks from a database. This data includes user request frequency per second, latency, and bandwidth. Assume that historical data shows that during high-concurrency periods (such as promotional events), the request frequency can reach 200 times per second, latency fluctuates between 50-200 milliseconds, and bandwidth fluctuates between 10-50 Mbps. Based on this historical data, the system constructs a state space for user behavior, including three features: request frequency, latency, and bandwidth. This forms a three-dimensional vector representing the user behavior state, such as [200, 150, 30], representing the user request frequency, latency, and bandwidth at a given moment. To learn how to optimize resource allocation in high-concurrency scenarios, the system uses a Deep Q-Network (DQN) as the reinforcement learning model architecture. This architecture includes a deep neural network whose input is the user behavior state space, and whose output is the Q-value for each possible action. The neural network learns the mapping relationship between states and actions through multiple convolutional and fully connected layers.

[0073] During model training, the reinforcement learning model interacts with the environment, selects actions based on the current state, and adjusts its strategy based on the reward value from the environment. The system defines three possible actions: "allocate more computing resources," "increase bandwidth," or "maintain the current resource allocation." During training, the network progressively optimizes its weights through backpropagation to minimize the gap between the weights and the expected reward. Through multiple rounds of training, the model learns how to make optimal resource allocation decisions based on real-time changes in user behavior patterns.

[0074] As training progresses, the system adjusts resource allocation based on the learned strategies and continuously optimizes the model's behavior through adaptive prediction parameters to adapt to different concurrent requests and network conditions. The trained model can adjust resource configuration in real-time during practical applications, ensuring low latency and high fairness, thereby improving the performance and user experience of the online lottery system.

[0075] Step S4: Dynamically adjust the resource allocation scheme according to the adaptive prediction parameters, and generate the network environment change trend based on the adjusted resource allocation scheme.

[0076] In one specific embodiment, the process of performing step S4 may specifically include the following steps:

[0077] (1) Generate an initial resource allocation scheme based on adaptive prediction parameters, wherein the initial scheme is used to adjust the allocation ratio of computing resources and transmission resources;

[0078] (2) During the off-peak period, based on the initial plan, reduce the resource consumption of the transmission link and obtain a new resource allocation plan;

[0079] (3) Implement a new resource allocation plan, and at the same time, monitor the real-time changes in the network environment;

[0080] (4) Generate the change trend of the network environment according to the real-time changes.

[0081] Specifically, an initial plan for resource allocation is generated based on the adaptive prediction parameters. In the initial plan, according to the analysis of historical data, the computing resources (such as CPU and memory) and transmission resources (such as bandwidth) are allocated at a ratio of 60% and 40%. The goal of this initial plan is to ensure that during high concurrency, the computing resources can sufficiently support the computing requirements of the prize allocation logic, while the transmission resources are moderately allocated according to the bandwidth fluctuations to avoid resource waste and delays caused by over-allocation of bandwidth. During the low peak period of the activity, the system adjusts the resources according to the initial plan to reduce the resource occupancy in the transmission link. For example, during the low peak period, the user request frequency drops to 50 times per second, and the bandwidth usage also decreases to 10 Mbps. At this time, the system reduces the bandwidth resource allocation by 30% according to the initial plan, adjusts the bandwidth allocation ratio to 30%, and uses the remaining resources to enhance the computing power.

[0082] On this basis, the system executes a new resource allocation plan and monitors the changes in the network environment in real time. For example, after the bandwidth usage in the low peak period decreases, the system continues to monitor the changes in bandwidth and request frequency, and finds that the bandwidth usage has a gradually rising trend, reaching 20 Mbps, and at the same time the request frequency rises again to 120 times per second. The system dynamically adjusts the resource allocation plan according to these real-time changes, moderately increases the bandwidth ratio, and slightly reduces the proportion of computing resources, ensuring that the system can respond to new requests in a timely manner. < / /

[0083] Based on these real-time changes, the system generates the change trend of the network environment. For example, the system monitors that the bandwidth increases from 10 Mbps in the low peak period to 20 Mbps, and the request frequency gradually returns to the normal level. Through this trend analysis, the system predicts that within the next 30 minutes, the bandwidth demand will continue to gradually increase and may reach 30 Mbps. Therefore, the system adjusts the resource allocation plan and prepares in advance to gradually allocate the bandwidth resources from the computing resources to the transmission resources to ensure that the system can handle the upcoming peak period. This dynamic resource adjustment method ensures the efficient operation of the system during high concurrency periods through the feedback of real-time data and trend prediction, maximizes resource utilization and reduces delays. Refer to Figure 2 , which shows the flowchart of resource adaptive allocation and trend prediction.

[0084] Step S5: According to the change trend of the network environment, extract the optimization path from the historical data records to obtain the update sequence of resource allocation adjustment.

[0085] In a specific embodiment, the process of executing step S5 may specifically include the following steps:

[0086] (1) Based on the changing trends of the network environment, filter historical data records that match the current environment and define them as reference historical records;

[0087] (2) Extract historical optimization paths for user behavior fluctuations based on reference historical records;

[0088] (3) Based on the historical optimization path, the state value of the reinforcement learning model is updated using the Q-value update rule;

[0089] (4) Store the historical optimization path in the experience replay buffer for reuse in subsequent batch sampling by the reinforcement learning model;

[0090] (5) Generate an update sequence for resource allocation adjustment based on the updated state value.

[0091] Specifically, the system filters historical data records that match the current environment based on real-time monitored network environment trends (such as bandwidth changes and user request frequency changes), defining these records as reference historical records. For example, during a promotional event, the system detected that bandwidth usage increased from 20Mbps to 40Mbps, and the request frequency increased from 50 times per second to 200 times per second, which matches similar high-concurrency scenarios in the past. Therefore, the system extracts historical data records from the database under similar past scenarios, such as peak changes in bandwidth usage and changes in request frequency, as reference historical records. Based on the reference historical records, the system extracts historical optimization paths for user behavior fluctuations. For example, the historical records show that during a certain event, as bandwidth usage gradually increased, the system addressed latency fluctuations by increasing the proportion of computing resources (e.g., from 30% to 50%). Through analysis of these historical records, the system extracts an optimization path, indicating how to adjust the ratio of computing resources to transmission resources to achieve optimal system performance during a gradual increase in bandwidth.

[0092] Based on these historical optimization paths, the system uses a Q-value update rule to update the state values ​​of the reinforcement learning model. The Q-values ​​for different actions in the current state are updated according to past actions and reward values. For example, in a historical optimization path, if a bandwidth increase leads to a moderate increase in computing resources resulting in a decrease in latency, the system updates the Q-value based on this feedback, enabling the reinforcement learning model to learn appropriate resource adjustment strategies in similar situations. These historical optimization paths and the Q-value-updated state values ​​are stored in an experience replay buffer for reuse in subsequent batch sampling by the reinforcement learning model. The system can learn from past experience, avoiding training from scratch each time, thus accelerating the model's learning process. Whenever the system detects a similar high-concurrency scenario, the model can retrieve the relevant optimization path from the experience replay buffer for rapid adjustment and optimization.

[0093] Based on the updated state values, the system generates an update sequence for resource allocation adjustments. For example, suppose that at a certain point, the system predicts that bandwidth demand will continue to increase, and the proportion of computing resources needs to be adjusted accordingly. Based on the updated Q value, the system decides to increase computing resources from the current 40% to 50% and appropriately increase bandwidth allocation. Through this series of dynamic adjustments, the system can optimize resource allocation in real time under high concurrency scenarios, ensuring low latency, high efficiency, and fairness.

[0094] Step S6: Adjust the priority of business data according to the update sequence, obtain the intermediate results of low-latency processing based on the adjusted business data, organize the allocation logic based on the intermediate results of low-latency processing, and determine the stable indicators of event trigger allocation.

[0095] In one specific embodiment, the process of performing step S6 may specifically include the following steps:

[0096] (1) Adjust the priority of business data according to the update sequence. Business data is the user lottery request packet generated during the online lottery process;

[0097] (2) Generate low-latency processing intermediate results based on the adjusted business data;

[0098] (3) Based on the intermediate results and combined with the random sampling mechanism, generate an event triggering allocation scheme;

[0099] (4) Calculate the allocation ratio for user points redemption using the event-triggered allocation scheme;

[0100] (5) Generate stable indicators for event-triggered allocation based on the allocation ratio. The stable indicators are used to verify the reliability of low-latency processing.

[0101] Specifically, the system adjusts the priority of business data based on the update sequence. Business data refers to user lottery request packets generated during the online lottery process. For example, in a promotional activity, user A's lottery request timestamp is T1, and user B's timestamp is T2. If user A's request has a higher priority (e.g., user A purchased a promotional item), then user A's request will be processed first, thereby reducing its latency. The system adjusts the processing order of each request based on this information to ensure that high-priority requests are responded to as early as possible.

[0102] The adjusted business data enters a low-latency processing phase, where the system quickly processes these requests and generates intermediate results. For example, the system generates a winning result for user A ("Congratulations on winning") and prepares to send it to the user. This intermediate result will be ready in a short time, ensuring that user experience is not affected by excessive delays.

[0103] The generated intermediate results trigger an event allocation scheme through a random sampling mechanism. Assuming there are 1000 user requests, the system randomly selects 100 users for processing based on request priority and resource availability. This mechanism ensures that every user has a chance to participate in the lottery while preventing some users from consuming excessive resources due to frequent requests. The system calculates the allocation ratio for user points redemption based on the triggered event. For example, if user A has a higher priority, the system may calculate a higher redemption ratio for them (e.g., 1:10), while the redemption ratio for other lower-priority users is 1:5. This ratio will be dynamically adjusted to ensure that, with limited resources, higher-priority users receive more rewards. The system generates stability metrics based on the allocation ratio to verify the reliability of low-latency processing. The system verifies the stability of the current resource allocation using historical data, ensuring that under high concurrency, the system can effectively handle a large number of requests with acceptable latency. Through these stability metrics, the system can optimize subsequent resource allocation decisions, ensuring the smooth operation of the lottery activity.

[0104] Step S7: Feed the stability index back to the reinforcement learning model, determine whether it is necessary to iterate the adaptive prediction parameters, and further optimize the reinforcement learning model based on the adaptive prediction parameters to obtain the optimized configuration of the dynamic network environment.

[0105] In one specific embodiment, the process of performing step S7 may specifically include the following steps:

[0106] (1) Use the difference normalization method to process the stable index, obtain feedback data, and update the reward function based on the feedback data;

[0107] (2) Determine whether iterative training is needed based on the updated reward function. If so, adjust the training parameters of the reinforcement learning model to obtain new adaptive prediction parameters.

[0108] (3) Optimize the prediction accuracy of the reinforcement learning model based on the new adaptive prediction parameters;

[0109] (4) Generate an initial strategy for optimizing the configuration of the dynamic network environment based on the optimized reinforcement learning model;

[0110] (5) Optimize the convergence speed of the initial strategy through a neural network agent, and determine whether the convergence speed meets the preset convergence conditions. If so, generate the final dynamic network environment optimization configuration.

[0111] (6) By optimizing the configuration of the dynamic network environment, the network jitter and latency differences of users under high concurrency are eliminated, thereby achieving the fairness of the online lottery system.

[0112] Specifically, the system uses interpolation normalization to process stability metrics, obtain feedback data, and update the reward function. For example, suppose that during high-concurrency periods, the system's stability metrics show large latency fluctuations, ranging from 50ms to 300ms. After interpolation normalization, these latency fluctuations are converted into standardized values ​​between 0 and 1, which are fed back into the reward function. This adjusts the parameters of the reward function, enabling future resource allocation to reduce latency fluctuations. Based on the updated reward function, the system determines whether further training of the reinforcement learning model is needed. If the adjustment of the reward function significantly improves system performance, the system adjusts the training parameters of the reinforcement learning model, such as the learning rate or discount factor, to obtain new adaptive prediction parameters. For example, suppose the model finds that latency has decreased by 15%, indicating that further model optimization will help improve system performance, thus prompting training iterations. Based on the new adaptive prediction parameters, the reinforcement learning model will optimize prediction accuracy, enabling more precise calculation of resource allocation ratios to better cope with future network changes. The system then generates an initial strategy for dynamic network environment optimization, such as prioritizing bandwidth resources over computing resources during high-load periods to reduce latency. Next, the system accelerates the convergence speed of the initial strategy through a neural network agent, ensuring that the strategy can adapt to the new network environment in a short time. Once the convergence speed reaches a preset standard (e.g., convergence within 5 seconds), the system generates the final dynamic network environment optimization configuration, adjusts resource allocation in real time, eliminates network jitter and latency differences, and ensures the fairness and stability of the online lottery system in high-concurrency scenarios.

[0113] Taking an online lottery event on an e-commerce platform as an example, during high-concurrency lottery periods, the system uses difference normalization to process stable indicators monitored in real time and updates the reward function accordingly. During the event, the system detected significant fluctuations in user request latency, ranging from 50ms to 300ms. After normalization, these fluctuations are converted into standardized feedback values ​​between 0 and 1, which serve as input to the reward function, adjusting reward parameters to optimize subsequent resource allocation strategies. The updated reward function guides the system to optimize resource allocation ratios and reduce latency. Further training of the reinforcement learning model revealed that the adjusted reward function reduced latency fluctuations by approximately 15%. Based on this feedback, the model automatically adjusts training parameters, improving the prediction accuracy for bandwidth and computing resources. Figure 3The figure illustrates the convergence process of the prediction parameters. The system generates an initial strategy based on the new adaptive prediction parameters, such as prioritizing resource allocation to users with higher bandwidth demands during high-concurrency periods. By accelerating the convergence speed of the initial strategy through a neural network agent, the system can quickly adapt to high-concurrency scenarios, meeting the preset standard within 5 seconds. The system generates a dynamic network optimization configuration and adjusts resource allocation in real time throughout the lottery event, eliminating latency differences and network jitter among users and ensuring the fairness of the lottery. Through this optimization, the system successfully maintained stable latency performance among 5000 concurrent users, improving the user experience, and no user missed the lottery opportunity due to latency issues. (Reference) Figure 4 The figure shows an analysis of the optimization effect of a lottery activity on an e-commerce platform.

[0114] The reinforcement learning method for virtual prize allocation in the embodiments of this application has been described above. The reinforcement learning system for virtual prize allocation in the embodiments of this application is described below. Please refer to [link / reference]. Figure 5 One embodiment of the reinforcement learning system for virtual prize allocation in this application includes:

[0115] The load feature extraction module is used to obtain the user request frequency and network bandwidth indicators of the online lottery system, extract the load features during peak and valley periods, and further obtain the latency fluctuation distribution.

[0116] The bottleneck point location module is used to analyze the latency of the processing link based on the latency fluctuation distribution and adopt a real-time monitoring strategy to identify the bottleneck point in high-concurrency scenarios.

[0117] The parameter adaptation module is used to obtain historical data records corresponding to bottleneck points, learn user behavior fluctuation patterns through reinforcement learning model training, and obtain adaptive prediction parameters.

[0118] The resource dynamic adjustment module is used to dynamically adjust the resource allocation scheme based on adaptive prediction parameters, and generate network environment change trends based on the adjusted resource allocation scheme.

[0119] The optimized path extraction module is used to extract optimized paths from historical data records based on the changing trends of the network environment, and obtain the update sequence of resource allocation adjustment.

[0120] The allocation logic organization module is used to adjust the priority of business data according to the update sequence, obtain the intermediate results of low-latency processing based on the adjusted business data, organize the allocation logic based on the intermediate results of low-latency processing, and determine the stable indicators of event-triggered allocation.

[0121] The model iteration and optimization module is used to feed the stability index back to the reinforcement learning model, determine whether it is necessary to iterate the adaptive prediction parameters, and further optimize the reinforcement learning model based on the adaptive prediction parameters to obtain the optimized configuration of the dynamic network environment.

[0122] Through the collaborative efforts of the aforementioned components, the online lottery system can monitor and dynamically adjust resource allocation in real time, effectively addressing load fluctuations in high-concurrency scenarios and ensuring system stability and fairness. The load feature extraction module collects user request frequency and network bandwidth data in real time, identifying load peaks and troughs to generate latency fluctuation distributions. Based on these fluctuation distributions, the bottleneck point location module identifies key bottlenecks in high-concurrency scenarios, ensuring the system can accurately pinpoint the core issues affecting performance. Next, the parameter adaptation module analyzes historical data of bottleneck points using a reinforcement learning model, learns user behavior fluctuation patterns, and generates adaptive prediction parameters, providing a scientific basis for subsequent resource allocation. The resource dynamic adjustment module uses these adaptive prediction parameters to adjust the resource allocation scheme in real time and predicts future network environment changes, providing continuously optimized resource allocation strategies for the system. The optimization path extraction module extracts the optimal resource allocation path based on historical data and current network trends, generating an update sequence to further optimize resource allocation. The allocation logic processing module adjusts the priority of business data according to the update sequence, ensuring that high-priority user requests are processed first during high concurrency and generating low-latency intermediate results, thereby guaranteeing system fairness. The model iteration and optimization module feeds stability metrics back to the reinforcement learning model, checks the effectiveness of adaptive prediction parameters, and further optimizes the model. Ultimately, it generates a dynamic network environment optimization configuration, ensuring stable system operation under high load and providing a fair user experience. Through automated resource scheduling and optimization, the entire system not only improves the efficiency of the lottery process but also minimizes latency differences, enhancing user engagement and satisfaction.

[0123] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the steps of the reinforcement learning method for virtual prize allocation.

[0124] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0125] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0126] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for resource allocation based on reinforcement learning, characterized in that, include: S1. Obtain the user request frequency and network bandwidth indicators of the online lottery system, extract the peak and valley load characteristics, and further obtain the latency fluctuation distribution; S2. Based on the latency fluctuation distribution, set a real-time monitoring time window; based on the time window, collect latency data during the processing; perform statistical analysis on the latency data to obtain a comprehensive service latency score, and determine whether the comprehensive service latency score exceeds the preset latency threshold; if the comprehensive service latency score exceeds the latency threshold, prioritize the allocation of computing resources and generate resource allocation instructions; based on the resource allocation instructions, determine the bottleneck point in the high-concurrency scenario. S3. Retrieve historical data records related to the bottleneck from the database; construct the state space of user behavior based on the historical data records. Based on the state space, an action selection strategy is defined through a reinforcement learning model, and user behavior fluctuation patterns are trained and learned; a reward function is generated based on the user behavior fluctuation patterns. The action selection strategy is optimized based on the reward function to obtain adaptive prediction parameters; S4. Generate an initial resource allocation scheme based on adaptive prediction parameters. The initial scheme is used to adjust the allocation ratio of computing resources and transmission resources. During off-peak periods, reduce the resource consumption of transmission links based on the initial scheme and obtain a new resource allocation scheme. Implement the new resource allocation scheme and monitor real-time changes in the network environment. Generate a network environment change trend based on the real-time changes. S5. Based on the trend of network environment changes, filter historical data records that match the current environment and define them as reference historical records; extract historical optimization paths of user behavior fluctuations based on reference historical records; update the state values ​​of the reinforcement learning model using the Q-value update rule based on the historical optimization paths; store the historical optimization paths in the experience replay buffer for subsequent batch sampling and reuse by the reinforcement learning model; generate an update sequence for resource allocation adjustment based on the updated state values. S6. Adjust the priority of business data according to the update sequence, obtain the intermediate results of low-latency processing based on the adjusted business data, organize the allocation logic based on the intermediate results of low-latency processing, and determine the stable indicators of event trigger allocation. S7. The stability index is processed using the difference normalization method to obtain feedback data, and the reward function is updated based on the feedback data. Based on the updated reward function, it is determined whether iterative training is needed. If so, the training parameters of the reinforcement learning model are adjusted to obtain new adaptive prediction parameters. The prediction accuracy of the reinforcement learning model is optimized based on the new adaptive prediction parameters. An initial strategy for dynamic network environment optimization is generated based on the optimized reinforcement learning model. The convergence speed of the initial strategy is optimized through a neural network agent, and it is determined whether the convergence speed meets the preset convergence conditions. If so, the final dynamic network environment optimization configuration is generated. Through dynamic network environment optimization configuration, network jitter and latency differences among users under high concurrency are eliminated, achieving fairness in the online lottery system.

2. The resource allocation method based on reinforcement learning according to claim 1, characterized in that, S1 includes: Collect real-time user request frequency and network bandwidth metrics from the online lottery system to generate load data sequences; Preprocess the load data sequence to extract peak and valley features from the time series; Based on peak and valley characteristics, calculate the statistical distribution of load changes; establish a correlation model between load and latency based on the statistical distribution to obtain the latency fluctuation distribution.

3. The resource allocation method based on reinforcement learning according to claim 2, characterized in that, S6 includes: The priority of business data is adjusted according to the update sequence. Business data is the user lottery request packet generated during the online lottery process. Generate low-latency intermediate results based on the adjusted business data; Based on the intermediate results and a random sampling mechanism, an event-triggered allocation scheme is generated; the allocation ratio for user points redemption is calculated using the event-triggered allocation scheme. Based on the allocation ratio, a stable metric for event-triggered allocation is generated, which is used to verify the reliability of low-latency processing.

4. A resource allocation system based on reinforcement learning, used to implement the resource allocation method based on reinforcement learning as described in any one of claims 1 to 3, characterized in that, include: The load feature extraction module is used to obtain the user request frequency and network bandwidth indicators of the online lottery system, extract the load features during peak and valley periods, and further obtain the latency fluctuation distribution. The bottleneck point location module is used to analyze the latency of the processing link based on the latency fluctuation distribution and adopt a real-time monitoring strategy to identify the bottleneck point in high-concurrency scenarios. The parameter adaptation module is used to obtain historical data records corresponding to bottleneck points, learn user behavior fluctuation patterns through reinforcement learning model training, and obtain adaptive prediction parameters. The resource dynamic adjustment module is used to dynamically adjust the resource allocation scheme based on adaptive prediction parameters, and generate network environment change trends based on the adjusted resource allocation scheme. The optimized path extraction module is used to extract optimized paths from historical data records based on the changing trends of the network environment, and obtain the update sequence of resource allocation adjustment. The allocation logic organization module is used to adjust the priority of business data according to the update sequence, obtain the intermediate results of low-latency processing based on the adjusted business data, organize the allocation logic based on the intermediate results of low-latency processing, and determine the stable indicators of event-triggered allocation. The model iteration and optimization module is used to feed the stability index back to the reinforcement learning model, determine whether it is necessary to iterate the adaptive prediction parameters, and further optimize the reinforcement learning model based on the adaptive prediction parameters to obtain the optimized configuration of the dynamic network environment.

5. A computer-readable storage medium storing instructions thereon, characterized in that, When the instructions are executed by the processor, they implement the resource allocation method based on reinforcement learning as described in any one of claims 1 to 3.

Images