Online gambling machine classification method and device suitable for delayed feedback environment

Abstract

The invention discloses a gambling machine online classification method and device suitable for a delayed feedback environment, and the method comprises the following steps: 1), receiving a current input sample, predicting a classification result according to a classifier, and requesting single-point feedback; 2) single-point feedback of the round is received, and gradient estimation is carried out by using a bandit method; 3) storing the estimated approximate gradient into a gradient barrel; 4) after gradient barrel collection is completed, on the basis of gradient information in the gradient barrel, a meta algorithm is combined with a plurality of expert algorithms to update classifier parameters; according to the method, limited gradient feedback is processed by utilizing a bandit and a gradient barrel, and real-time classification of streaming input data can be effectively realized under the condition that gradient information is incomplete or cannot be directly acquired.

Description

Technical Field

[0001] The present invention belongs to the field of online learning in machine learning, and in particular relates to an online classification method and device in a dynamic environment. Background Art

[0002] Online classification is a common machine learning technique suitable for dynamically classifying massive amounts of streaming data. When processing each piece of data, the system first analyzes its features and predicts its category. It then updates the model based on the actual category, continuously optimizing its predictive capabilities. This method offers strong adaptability and real-time performance, making it particularly suitable for scenarios with constantly changing data distribution. With the rapid development of big data, online classification has been widely used in various internet fields, including spam detection, targeted advertising, and personalized e-commerce recommendations. However, traditional online classification methods generally rely on complete gradient information and assume immediate feedback upon request. This idealistic assumption often fails to hold true in real-world scenarios. For example, in high-frequency applications such as online routing, personalized recommendations, and financial risk control, the system often struggles to obtain complete feedback for each decision round. For example, in online routing, while the system can observe the overall loss (path time) after a packet arrives at its destination, it cannot capture the real-time congestion status of each link in the network during transmission, nor can it reconstruct the form of the entire loss function. This lack of information directly affects the model's ability to accurately estimate gradients during parameter updates, thereby reducing learning efficiency and ultimate prediction accuracy. In applications such as advertising and recommendation systems, user behavior feedback often has significant delays. For example, actions such as clicking, browsing, or purchasing may not occur until a long time later. Initially, the system can only observe whether the behavior occurs, but cannot obtain the entire path or intermediate process of the behavior.

[0003] In this delayed feedback scenario, after a model makes a decision, it often needs to wait several rounds or longer before receiving feedback related to that decision and updating it accordingly. This feedback delay can arise from a variety of real-world factors, including communication delays, delayed user behavior, privacy protection policies, or limited computing resources. Furthermore, another common constraint in delayed feedback environments is the "low dimensionality" or "non-differentiability" of the feedback information. Many real-world systems do not return the complete form of the loss function or its gradient information, but only provide a single feedback value for a specific decision outcome.

[0004] To address the information constraints in delayed feedback scenarios, bandit optimization frameworks have gained increasing attention. Bandit optimization methods rely solely on a single loss value in the current round, without requiring a complete loss function or gradient information, making them naturally suitable for such limited feedback scenarios. When bandit optimization methods are combined with the delayed feedback setting, a delayed feedback-constrained bandit optimization problem emerges. In this setting, the model only receives delayed loss feedback in each round, resulting from a decision made several rounds ago, lacking immediately available feedback information.

[0005] Therefore, how to design an online classification method that can adapt to information lag and learn efficiently in the scenario of delayed feedback has become an important technical difficulty that needs to be urgently solved in this field. Summary of the Invention

[0006] The present invention aims to overcome the shortcomings of existing online classification methods that generally rely on complete gradients or immediate feedback and are difficult to adapt to dynamic environments with feedback delays and limited information. An online classification method and device for slot machines suitable for delayed feedback scenarios are proposed. Under the condition of only receiving delayed single-point loss feedback, the present invention can still achieve effective model updates and accurate classification, thereby improving the adaptability and robustness of the system in complex real-world environments.

[0007] The first aspect of the present invention relates to an online classification method for slot machines in a delayed feedback environment, which is used in scenarios such as online advertising, online routing, and personalized recommendations. Specifically, first, the current input sample is received and predicted by a classifier, while the single-point loss value for that round is requested. Then, the slot machine method is used to perform gradient approximation on the single-point loss value received in this round and store it in a gradient bucket for subsequent cumulative analysis. Finally, when the information in the gradient bucket reaches the upper limit, an adaptive optimization mechanism is initiated that includes a meta-method and multiple expert methods: wherein the multiple expert methods use different learning rates to simulate and adapt to different types of environmental changes, and update their respective parameters via gradient descent; the meta-method integrates the outputs of each expert method in each round, dynamically adjusts the weights based on their recent classification performance in the delayed feedback environment, and determines the final parameter update direction based on the weighted results, thereby achieving effective optimization of the classifier.

[0008] The specific steps of the meta-method are:

[0009] Step 100, input the current sample;

[0010] Step 101, setting the step size parameter α;

[0011] Step 102, setting the number N of expert methods;

[0012] Step 103, set the learning rate η of each expert method i ;

[0013] Step 104: Initialize the weight of each expert method

[0014] Step 105: In each round t=1, 2, ..., T, perform the following steps, where T represents the total number of rounds:

[0015] Step 106: Receive the output of each expert method

[0016] Step 107, calculate classifier parameters

[0017] Step 108, predicting a classification result according to the classifier;

[0018] Step 109, randomly select a unit vector s t ;

[0019] Step 110: Randomly perturb the classifier parameters x t =z t +s t ;

[0020] Step 111, requesting the perturbed single point gradient feedback f t (x t );

[0021] Step 112: Receive the current round single point gradient feedback set {f k (x k )|k∈F t} and sent to various experts, where F t Represents the gradient set received in this round;

[0022] Step 113: Construct a gradient estimator Where δ represents the perturbation radius, and n represents the feature dimension of the input;

[0023] Step 114: construct a linear approximation function l according to the gradient estimator. k (z)= <g t ,zz t >

[0024] Step 115: Update the weight of each expert method

[0025] The specific steps of each expert method are:

[0026] Step 200, setting the disturbance radius δ;

[0027] Step 201, setting the gradient bucket size K;

[0028] Step 202, set the classifier parameter index τ=1;

[0029] Step 203: Initialize the classifier parameters of each expert

[0030] Step 204: Initialize all gradient buckets P i , where i∈{1,2,...,T / K}, T / K represents the number of gradient buckets

[0031] Step 205 , performing the following steps in each interval m=1, 2, ..., T / K, where T represents the total number of rounds;

[0032] Step 206: Initialize the gradient bucket set collected in this interval

[0033] Step 207, in each interval t=(m-1)K+1,...,mK, perform the following steps;

[0034] Step 208: Sent to metamethod;

[0035] Step 209: Receive a single-point gradient feedback set {f k (x k )|k∈F t};

[0036] Step 210: Feedback the single point gradient f k (x k ) Add the corresponding gradient bucket P i ,in

[0037] Step 211, ending the steps within the interval;

[0038] Step 212: add the full gradient buckets in the interval to set A m ;

[0039] Step 213, use A m All gradient buckets P i Update gradients where π X [·] represents the projection operator;

[0040] Step 214, update the classifier parameter index τ=τ+1;

[0041] The setting method of the step size parameter α in step 101 is as follows: γ=Kn 2 M 2 / δ2 +K 2 G 2 , where d represents the maximum delay between sending a gradient request and receiving the gradient, D represents the diameter of the feasible region X of the classifier parameters, and G is any value that makes the following equation true: M is any value such that:

[0042] The number N of expert methods in step 102 is set as follows:

[0043] The learning rate η of each expert method in step 103 i The setting method is

[0044] The disturbance radius δ in step 200 is set as follows:

[0045] The gradient bucket size K in step 201 is set as K=nT 1 / 2 ;

[0046] The projection operator Π in step 212 X The specific definition of [·] is Π X [u]=argmin v∈W ‖uv‖,u∈X.

[0047] A second aspect of the present invention relates to an online classification device for slot machines in a delayed feedback environment, comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the online classification method for slot machines in a delayed feedback environment of the present invention.

[0048] The innovation of this invention lies in the following: This invention proposes an online classification method and device for slot machines suitable for delayed feedback environments. The core innovation lies in combining single-point feedback approximation and delayed feedback compensation with a multi-expert adaptive optimization mechanism, which solves the performance degradation problem of traditional online classification methods when feedback is delayed and incomplete. Specifically, it includes: (1) using gradient buckets to accumulate historical loss information to compensate for the information loss caused by delayed feedback; (2) designing a multi-expert collaboration framework, in which experts with different learning rates adapt to dynamic changes in the environment; (3) introducing a meta-method to dynamically adjust expert weights to ensure that the classifier can still optimize the update direction under delayed feedback.

[0049] The working principle of this invention (analysis of the reasons for the beneficial effects): This method first uses gradient buckets to approximate storage and cumulative analysis of delayed single-point loss values ​​to address the incompleteness and lag of feedback information. Subsequently, an adaptive optimization mechanism is activated: multiple expert methods run in parallel, using different learning rates to capture environmental trends and update parameters through gradient descent. The meta-method dynamically adjusts the weights of each expert based on their recent performance in the delayed feedback environment, and weightedly integrates the outputs to ultimately determine the optimization direction of the classifier. This mechanism enables the system to maintain rapid adaptability and stable classification performance in complex scenarios with limited feedback.

[0050] Beneficial effects of the present invention: Compared with the prior art, the present invention can achieve adaptive online classification when only delayed and incomplete feedback information is received, and is suitable for complex dynamic environments with uncertain feedback and limited information. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a flow chart of the overall method of the present invention.

[0052] Figure 2 It is a flowchart of the meta-method of the present invention;

[0053] Figure 3 This is a workflow diagram of the expert method of the present invention.

[0054] Figure 4 It is a schematic diagram of the device of the present invention. DETAILED DESCRIPTION

[0055] The present invention will be further described below with reference to specific examples. It should be understood that the following examples are only used to illustrate the technical solutions of the present invention, rather than to limit the present invention.

[0056] Example 1

[0057] like Figure 1 ,This embodiment relates to an online classification method for slot machines in a delayed feedback environment, comprising the following steps;

[0058] S1. Receive the current input sample, and have the classifier predict it, and request the single-point loss value of this round;

[0059] S2. Use the bandit method to perform gradient approximation on the single-point loss value received in this round and store it in the gradient bucket for subsequent cumulative analysis;

[0060] S3. When the information in the gradient bucket reaches its upper limit, an adaptive optimization mechanism consisting of a meta-method and multiple expert methods is initiated. The multiple expert methods use different learning rates to simulate and adapt to different types of environmental changes, and update their respective parameters via gradient descent. The meta-method integrates the outputs of each expert method in each round, dynamically adjusts the weights based on their recent classification performance in a delayed feedback environment, and determines the final parameter update direction based on the weighted results, thereby effectively optimizing the classifier.

[0061] like Figure 2 , the specific steps of the meta-method are:

[0062] Step 100, input the current sample;

[0063] Step 101, setting the step size parameter α;

[0064] Step 102, setting the number N of expert methods;

[0065] Step 103, set the learning rate η of each expert method i ;

[0066] Step 104: Initialize the weight of each expert method

[0067] Step 105: In each round t=1, 2, ..., T, perform the following steps, where T represents the total number of rounds:

[0068] Step 106: Receive the output of each expert method

[0069] Step 107, calculate classifier parameters

[0070] Step 108, predicting a classification result according to the classifier;

[0071] Step 109, randomly select a unit vector s t ;

[0072] Step 110: Randomly perturb the classifier parameters x t =z t +s t ;

[0073] Step 111, requesting the perturbed single point gradient feedback f t (x t );

[0074] Step 112: Receive the current round single point gradient feedback set {f k (x k )|k∈F t} and sent to various experts, where Ft Represents the gradient set received in this round;

[0075] Step 113: Construct a gradient estimator Where δ represents the perturbation radius, and n represents the feature dimension of the input;

[0076] Step 114: construct a linear approximation function l according to the gradient estimator. k (z)= <g t ,zz t >

[0077] Step 115: Update the weight of each expert method

[0078] like Figure 3 ,The specific steps of each expert method are:

[0079] Step 200, setting the disturbance radius δ;

[0080] Step 201, setting the gradient bucket size K;

[0081] Step 202, set the classifier parameter index τ=1;

[0082] Step 203: Initialize the classifier parameters of each expert

[0083] Step 204: Initialize all gradient buckets P i , where i∈{1,2,...,T / K}, T / K represents the number of gradient buckets

[0084] Step 205 , performing the following steps in each interval m=1, 2, ..., T / K, where T represents the total number of rounds;

[0085] Step 206: Initialize the gradient bucket set collected in this interval

[0086] Step 207, in each interval t=(m-1)K+1,...,mK, perform the following steps;

[0087] Step 208: Sent to metamethod;

[0088] Step 209: Receive a single-point gradient feedback set {f k (x k )|k∈F t};

[0089] Step 210: Feedback the single point gradient f k (x k ) Add the corresponding gradient bucket P i,in

[0090] Step 211, ending the steps within the interval;

[0091] Step 212: add the full gradient buckets in the interval to set A m ;

[0092] Step 213, use A m All gradient buckets P i Update gradients where π X [·] represents the projection operator;

[0093] Step 214, update the classifier parameter index τ=τ+1;

[0094] The setting method of the step size parameter α in step 101 is as follows: γ=Kn 2 M 2 / δ 2 +K 2 G 2 , where d represents the maximum delay between sending a gradient request and receiving the gradient, D represents the diameter of the feasible region X of the classifier parameters, and G is any value that makes the following equation true: M is any value such that:

[0095] The number N of expert methods in step 102 is set as follows:

[0096] The learning rate η of each expert method in step 103 i The setting method is

[0097] The disturbance radius δ in step 200 is set as follows:

[0098] The gradient bucket size K in step 201 is set as K=nT 1 / 2 ;

[0099] The projection operator Π in step 212 X The specific definition of [·] is Π X [u]=argmin v∈W ‖uv‖,u∈X.

[0100] Example 2

[0101] like Figure 4 This embodiment relates to an online classification device for slot machines in a delayed feedback environment, comprising a memory and one or more processors. The memory stores executable code. When the one or more processors execute the executable code, they are used to implement the online classification method for slot machines in a delayed feedback environment of embodiment 1.

[0102] Example 3

[0103] This embodiment relates to an online advertising recommendation method that utilizes the online gambling machine classification method for a delayed feedback environment of Example 1. When a user enters a page, the system must instantly determine which advertisement to display. This embodiment utilizes the "online gambling machine classification method for a delayed feedback environment" proposed in this invention for advertisement selection. This method achieves efficient model updates and classification decisions even when receiving only delayed single-point loss feedback.

[0104] The method of this embodiment includes the following steps:

[0105] S1. Receive the current input sample, and have the classifier predict it, and request the single-point loss value of this round;

[0106] S2. Use the bandit method to perform gradient approximation on the single-point loss value received in this round and store it in the gradient bucket for subsequent cumulative analysis;

[0107] S3. When the information in the gradient bucket reaches its upper limit, an adaptive optimization mechanism consisting of a meta-method and multiple expert methods is initiated. The multiple expert methods use different learning rates to simulate and adapt to different types of environmental changes, and update their respective parameters via gradient descent. The meta-method integrates the outputs of each expert method in each round, dynamically adjusts the weights based on their recent classification performance in a delayed feedback environment, and determines the final parameter update direction based on the weighted results, thereby effectively optimizing the classifier.

[0108] S4. In online ad recommendation systems, based on the aforementioned classification prediction, gradient approximation, and adaptive optimization mechanisms, the core of the system's handling of delayed feedback lies in recording user-ad features for each ad display. When delayed user click feedback is obtained, it is matched with historical features and loss is calculated. Gradient approximation is performed using a bandit approach, triggering an adaptive optimization mechanism to update the model. This design enables the system to effectively address feedback delays in real-world scenarios, maintaining real-time recommendation capabilities while ensuring the model can continuously learn from delayed feedback and optimize its recommendations.

[0109] The workflow of the metamethod is as follows Figure 1As shown in the figure. First, the concatenation vector of user features and candidate ad features for the current ad recommendation round is input as the input sample for this round. User features include static attributes such as gender, age, and region, and dynamic attributes such as ad click-through rate, recently viewed and clicked ads, interest tags, and recent conversion behaviors (such as purchases, registrations, or downloads). Candidate ad features include the ad's industry category, ad creative type, historical click-through rate, and bid information.

[0110] Next, determine the total number of recommendation rounds T and set the step size parameter γ=Kn 2 M 2 / δ 2 +K 2 G 2 , number of experts and the learning rate of each expert method where d represents the maximum delay between sending a gradient request and receiving the gradient, D represents the diameter of the feasible region X of the classifier parameters, and G is any value such that the following equation holds: M is any value such that:

[0111] Then, in each recommendation round, the meta-method receives the output of each expert method And calculate the classifier parameters Finally, advertisement recommendations are made based on the output of the classifier;

[0112] Finally, due to the delay in user behavior feedback, the meta-method cannot immediately obtain the ads that users actually clicked in the recommendation round, and therefore cannot immediately obtain information feedback for this round. t Add a random perturbation x t =z t +s t , where s t is a unit vector with a random direction; after that, the meta-method requests a single-point loss feedback f t (x t ); Then the meta-method receives the feedback set F that arrives in this round t , sent to each expert; finally, the meta-method updates the weight of each expert method where l k (z) is the gradient estimator The constructed linear approximation function l k (z)= <g t ,zz t >.

[0113] The workflow of each expert method is as follows Figure 2 As shown. First, initialize the perturbation radius And the gradient bucket upper limit K = nT 1 / 2 Then, the total number of recommendation rounds T is divided into T / K intervals. In each recommendation round within the interval, the gradient bucket is collected and communicated with the meta-method. At the end of each interval, the expert classifier parameters are updated using gradient descent based on the gradient buckets collected within the interval. in Represents the classifier parameters obtained in the current interval, P i Represents the collection of gradient buckets, Π X [·] represents the projection operator.

[0114] The contents described in the embodiments of this specification are merely an enumeration of the implementation forms of the inventive concept. The scope of protection of the present invention should not be regarded as limited to the specific forms described in the embodiments. The scope of protection of the present invention also extends to equivalent technical means that can be conceived by those skilled in the art based on the inventive concept.

Claims

1. A method for online classification of slot machines in a delayed feedback environment, characterized by: The method includes the following steps: S1. Receive the current input sample, and have the classifier predict it, and request the single-point loss value of this round; S2. Use the bandit method to perform gradient approximation on the single-point loss value received in this round and store it in the gradient bucket for subsequent cumulative analysis; S3. When the information in the gradient bucket reaches its upper limit, an adaptive optimization mechanism consisting of a meta-method and multiple expert methods is initiated. The multiple expert methods use different learning rates to simulate and adapt to different types of environmental changes, and update their respective parameters via gradient descent. The meta-method integrates the outputs of each expert method in each round, dynamically adjusts the weights based on their recent classification performance in a delayed feedback environment, and determines the final parameter update direction based on the weighted results, thereby effectively optimizing the classifier.

2. The online classification method for slot machines in a delayed feedback environment according to claim 1, characterized in that: The specific steps of the meta-method described in step S3 are: Step 100, input the current sample; Step 101, setting the step size parameter α; Step 102, setting the number N of expert methods; Step 103, set the learning rate η of each expert method i ; Step 104: Initialize the weight of each expert method Step 105: In each round t=1, 2, ..., T, perform the following steps, where T represents the total number of rounds: Step 106: Receive the output of each expert method Step 107, calculate classifier parameters Step 108, predicting a classification result according to the classifier; Step 109, randomly select a unit vector s t ; Step 110: Randomly perturb the classifier parameters x t =z t +s t ; Step 111, requesting the perturbed single point gradient feedback f t (x t ); Step 112: Receive the current round single point gradient feedback set {f k (x k )|k∈F t } and sent to various experts, where F t Represents the gradient set received in this round; Step 113: Construct a gradient estimator Where δ represents the perturbation radius, and n represents the feature dimension of the input; Step 114: construct a linear approximation function l according to the gradient estimator. k (z)= <g t ,zz t > Step 115: Update the weight of each expert method The specific steps of the expert method described in step S3 are: Step 200, setting the disturbance radius δ; Step 201, setting the gradient bucket size K; Step 202, set the classifier parameter index τ=1; Step 203: Initialize the classifier parameters of each expert Step 204: Initialize all gradient buckets P i , where i∈{1,2,...,T / K}, T / K represents the number of gradient buckets Step 205 , performing the following steps in each interval m=1, 2, ..., T / K, where T represents the total number of rounds; Step 206: Initialize the gradient bucket set collected in this interval Step 207, in each interval t=(m-1)K+1,...,mK, perform the following steps; Step 208: Sent to metamethod; Step 209: Receive a single-point gradient feedback set {f k (x k )|k∈F t }; Step 210: Feedback the single point gradient f k (x k ) Add the corresponding gradient bucket P i ,in Step 211, ending the steps within the interval; Step 212: add the full gradient buckets in the interval to set A m ; Step 213, use A m All gradient buckets P i Update gradients where π X [·] represents the projection operator; Step 214: Update the classifier parameter index τ=τ+1.

3. The online classification method for slot machines in a delayed feedback environment according to claim 2, wherein: The setting method of the step size parameter α in step 101 is γ=Kn 2 M 2 / δ 2 +K 2 G 2 , where d represents the maximum delay between sending a gradient request and receiving the gradient, D represents the diameter of the feasible region X of the classifier parameters, and G is any value that makes the following equation true: M is any value such that:

4. The online classification method for slot machines in a delayed feedback environment according to claim 2, wherein: The number of expert methods N in step 102 is set as follows:

5. The online classification method for slot machines in a delayed feedback environment according to claim 2, wherein: The learning rate η of each expert method in step 103 i The setting method is 6. The online classification method for slot machines in a delayed feedback environment according to claim 2, wherein: The disturbance radius δ in step 200 is set as 7. The online classification method for slot machines in a delayed feedback environment according to claim 2, wherein: The gradient bucket size K in step 201 is set as K=nT 1 / 2 .

8. The online classification method for slot machines in a delayed feedback environment according to claim 2, wherein: The projection operator Π in step 212 X The specific definition of [·] is Π X [u]=argmin v∈W ‖uv‖,u∈X.

9. An online classification device for slot machines in a delayed feedback environment, characterized in that: The method comprises a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, the method is used to implement the online classification method for slot machines in a delayed feedback environment according to any one of claims 1 to 8.

Images