Reinforcement learning scheduling optimization method and device based on adaptive fairness constraint

Abstract

This application provides a method and apparatus for optimizing ride-hailing order scheduling based on adaptive fairness constraints using reinforcement learning, belonging to the field of ride-hailing order scheduling technology. The method includes: acquiring pressure index data of the ride-hailing market at the current time step, fairness evaluation index data corresponding to the current time step and multiple historical time steps; determining historical statistical features based on the fairness evaluation index data corresponding to multiple historical time steps; determining a target fairness threshold based on the pressure index data and historical statistical features; determining the degree of constraint violation of the target fairness threshold based on the fairness evaluation index data corresponding to the current time step, and using Lagrange constraint reinforcement learning to determine the parameters of a neural network model used for ride-hailing order scheduling, and determining the updated neural network model based on the parameters of the neural network model; and scheduling ride-hailing orders based on the updated neural network model. This application can optimize ride-hailing order scheduling.

Description

Technical Field

[0001] This application belongs to the field of ride-hailing order scheduling technology, and more specifically, it relates to a reinforcement learning scheduling optimization method and device based on adaptive fairness constraints. Background Technology

[0002] The order dispatch system of ride-hailing platforms is a core technology in the field of intelligent mobility. Its goal is to efficiently allocate passenger orders to drivers in order to achieve multiple objectives such as maximizing platform revenue, minimizing passenger waiting time, and equalizing driver income.

[0003] With the development of reinforcement learning technology, more and more ride-hailing platforms are adopting reinforcement learning-based order scheduling methods, learning optimal scheduling strategies through the interaction between agents and the environment. However, in traditional reinforcement learning, fairness constraint methods typically use fixed threshold settings. The ride-hailing market is highly volatile, making fixed fairness constraint thresholds insufficient to meet its needs. For example, during peak demand periods, a pre-set fixed fairness constraint threshold may be too strict, leading to decreased scheduling efficiency, while during off-peak periods, it may be too lenient, failing to effectively guarantee fair driver income. This makes it difficult for existing traditional constraint methods to achieve a dynamic balance between efficiency and fairness. Summary of the Invention

[0004] The purpose of this application is to provide a reinforcement learning scheduling optimization method and apparatus based on adaptive fairness constraints to optimize ride-hailing order scheduling.

[0005] A first aspect of this application provides a reinforcement learning scheduling optimization method based on adaptive fairness constraints, comprising: Obtain stress index data for the ride-hailing market at the current time step, as well as fairness assessment index data for the current time step and multiple historical time steps. Among them, stress index data is used to characterize the supply and demand tension in the current ride-hailing market, and fairness assessment index data is used to characterize the degree of income inequality among all ride-hailing drivers in their corresponding time steps. Historical statistical features are determined based on fairness assessment indicators from multiple historical time steps. These historical statistical features are used to characterize the overall distribution pattern of income for all ride-hailing drivers across multiple historical time steps. Determine the target fairness threshold based on stress index data and historical statistical characteristics; The degree of constraint violation of the target fairness threshold is determined based on the fairness assessment index data corresponding to the current time step; Based on the objective fairness threshold and the degree of constraint violation, the parameters of the neural network model used for ride-hailing order scheduling are determined using Lagrange constraint reinforcement learning, and the updated neural network model is determined based on the parameters of the neural network model. The scheduling of ride-hailing orders is based on a neural network model with updated parameters.

[0006] A second aspect of this application provides a reinforcement learning scheduling optimization apparatus based on adaptive fairness constraints, comprising: The data acquisition module is used to acquire the stress index data of the ride-hailing market at the current time step, as well as the fairness assessment index data corresponding to the current time step and multiple historical time steps. Among them, the stress index data is used to characterize the supply and demand tension of the current ride-hailing market, and the fairness assessment index data is used to characterize the degree of income inequality among all ride-hailing drivers in their corresponding time steps. The statistical feature determination module is used to determine historical statistical features based on fairness assessment index data corresponding to multiple historical time steps; historical statistical features are used to characterize the overall distribution pattern of income of all ride-hailing drivers in multiple historical time steps; The fairness threshold determination module is used to determine the target fairness threshold based on stress index data and historical statistical characteristics; The constraint violation degree determination module is used to determine the degree of constraint violation of the target fairness threshold based on the fairness assessment index data corresponding to the current time step; The model determination module is used to determine the parameters of the neural network model for ride-hailing order scheduling based on the objective fairness threshold and the degree of constraint violation using Lagrangian constraint reinforcement learning, and to determine the updated neural network model based on the parameters of the neural network model. The order scheduling module is used to schedule ride-hailing orders based on the updated neural network model.

[0007] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of the reinforcement learning scheduling optimization method based on adaptive fairness constraints described above.

[0008] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the reinforcement learning scheduling optimization method based on adaptive fairness constraints described above.

[0009] The beneficial effects of the reinforcement learning scheduling optimization method and apparatus based on adaptive fairness constraints provided in this application are as follows: This application embodiment acquires pressure index data characterizing the tightness of market supply and demand, as well as driver income fairness assessment index data across multiple time steps. This allows for the accurate capture of real-time supply and demand dynamics and historical income fairness distribution patterns in the ride-hailing market. Based on the historical statistical characteristics determined by the aforementioned data, the overall distribution trend of driver income can be reflected. Combined with market pressure indicators, a target fairness threshold is dynamically set, replacing the traditional fixed threshold. This solves the problem that a fixed threshold, being too strict during peak market periods, leads to decreased scheduling efficiency, while being too lenient during off-peak periods fails to guarantee driver income fairness. Simultaneously, by calculating the degree of constraint violation, a constraint reference is provided for Lagrange constraint reinforcement learning. This allows this application embodiment to optimize the parameters of the scheduling neural network model based on dynamic thresholds and the degree of violation. The updated model executes scheduling decisions, adapting the strategy to the real-time market state and optimizing ride-hailing order scheduling. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A flowchart illustrating a reinforcement learning scheduling optimization method based on adaptive fairness constraints provided in an embodiment of this application; Figure 2 A structural block diagram of a reinforcement learning scheduling optimization device based on adaptive fairness constraints provided in an embodiment of this application; Figure 3 This is a schematic block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0012] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0013] To make the objectives, technical solutions, and advantages of this application clearer, the following description will be provided in conjunction with the accompanying drawings and specific embodiments.

[0014] Please refer to Figure 1 , Figure 1The flowchart of a reinforcement learning scheduling optimization method based on adaptive fairness constraints provided in an embodiment of this application can be executed by an electronic device. The method may include: S101-S106.

[0015] S101: Obtain the stress index data of the ride-hailing market at the current time step, as well as the fairness assessment index data corresponding to the current time step and multiple historical time steps.

[0016] In this embodiment, the pressure index data is used to characterize the current supply and demand tension in the ride-hailing market.

[0017] In one embodiment, the pressure index data for the current time step can be obtained in the following way: Obtain the supply-demand ratio, average dynamic price adjustment ratio, weather impact factor, and time period factor corresponding to the current time step; determine the pressure indicator data for the current time step based on the supply-demand ratio, average dynamic price adjustment ratio, weather impact factor, and time period factor corresponding to the current time step.

[0018] The supply-demand ratio refers to the ratio of the number of orders to the number of drivers in the current time step. The average dynamic fare adjustment rate is the arithmetic mean of the fare adjustment rates of ride-hailing platforms in the current time step. These rates are calculated in real-time by the ride-hailing platform using its built-in algorithm, or set by the platform's operators based on experience. The fare adjustment rate is used to adjust order prices. For example, for a journey from point A to point B, if the calculated price based on distance is 20 yuan, then with a fare adjustment rate of 1.2, the actual order price would be 24 yuan. The supply-demand ratio and the average dynamic fare adjustment rate can be obtained from the ride-hailing platform.

[0019] In this embodiment, the weather impact factor can be obtained by mapping the current weather conditions through a meteorological data interface. The more severe the weather, the greater the travel demand and the smaller the supply, resulting in a larger factor. For example: sunny day = 1.0, light rain = 1.2, heavy rain = 1.5, snowy day = 1.8, extreme weather = 2.0. The time period factor can be obtained by mapping the current time. For example, during the morning rush hour (7:00-9:00), the time period factor = 1.3; during the evening rush hour (17:00-19:00), the time period factor = 1.4; during the late night (23:00-6:00), the time period factor = 0.8; and for other times, the time period factor = 1.0.

[0020] In this embodiment, the supply-demand ratio (demand_supply_ratio), average dynamic price adjustment multiplier (avg_surge_multiplier), weather factor (weather_factor), and time factor (time_factor) corresponding to the current time step can be multiplied to obtain the pressure index data for the current time step: P_market=demand_supply_ratio×avg_surge_multiplier×weather_factor×time_factor.

[0021] In this embodiment, the fairness assessment index data is used to characterize the degree of income inequality among all ride-hailing drivers at their corresponding time steps.

[0022] In one embodiment, the fairness assessment metric for each time step can be the Gini coefficient of the income of all ride-hailing drivers. Taking the current time step as an example: Assuming the total number of ride-hailing drivers in the current time step is n, the cumulative income of the i-th driver is income_i, and the cumulative income of the j-th driver is income_j, then the formula for calculating the Gini coefficient G is: , This represents the average income of all ride-hailing drivers.

[0023] In this embodiment, to reflect income differences among different driver groups, the fairness assessment indicator data can be determined as the coefficient of variation of the income of all drivers, or the fairness assessment indicator data can be determined as a combination of the Gini coefficient and at least one group coefficient of variation. In this embodiment, the calculation process of the coefficient of variation can be as follows: Drivers are divided into multiple groups based on their attributes, which include at least one of the following: years of service, service area, vehicle type, and online time.

[0024] Based on driving experience: New driver (<3 months), Regular driver (3-12 months), Experienced driver (>12 months) Classified by service area: drivers in core urban areas, drivers in suburban areas, and drivers operating across regions. Classified by vehicle type: Express driver, Premium driver, Private car driver Based on online time: Full-time drivers (average >8 hours per day), Part-time drivers (average <8 hours per day) The coefficient of variation (CV) for group equity is calculated according to any division as follows: ,in, The standard deviation of the average income of each group This represents the average income of all ride-hailing drivers.

[0025] S102: Determine historical statistical characteristics based on fairness assessment index data corresponding to multiple historical time steps.

[0026] In this embodiment, historical statistical features are used to characterize the overall distribution pattern of income for all ride-hailing drivers across multiple historical time steps. In this embodiment and subsequent embodiments, the fairness assessment indicator is determined as the Gini coefficient of income for all ride-hailing drivers.

[0027] In one embodiment, the fairness assessment index data corresponding to multiple historical time steps can be statistically calculated to obtain the historical statistical features, such as the historical mean. Historical standard deviation and historical best value .

[0028] S103: Determine the target fairness threshold based on stress index data and historical statistical characteristics.

[0029] In one embodiment, determining the target fairness threshold based on stress index data and historical statistical characteristics includes: Market adjustment factors are determined based on stress index data; market adjustment factors are positively correlated with stress index data. Historical adaptation factors are determined based on historical statistical characteristics. These factors characterize the degree of deviation between historical fairness assessment index data and historical optimal fairness assessment index data at multiple historical time steps. The target fairness threshold is determined based on a pre-set benchmark fairness threshold, market adjustment factor, and historical adaptation factor.

[0030] In this embodiment, the market adjustment factor is expressed as: Historical adaptation factors are expressed as ,in, The current time step's pressure indicator data. This is the market sensitivity coefficient, used to control the degree of influence of market pressure on the threshold. It can be selected in the range of [0.05, 0.2] through grid search to ensure that the threshold is relaxed to a maximum of 1.1 times under extreme supply and demand conditions (e.g., P_market=2.0). It can take the value 0.1; The historical fitness coefficient is used to control the influence of historical performance on the threshold. It can be selected within the range of [0.01, 0.1] using a grid search. It can take the value 0.05.

[0031] In this embodiment, the target fairness threshold can be determined based on the following formula: ,in, Indicates the target fairness threshold; The base threshold represents the fairness constraint standard under normal conditions in the ride-hailing market, and its value can be 0.30.

[0032] In this embodiment, the calculated target fairness threshold can also be... Crop the data, that is, when the calculated threshold is less than... Time to take greater than Time to take Otherwise, keep the original value. and It can be set based on experience.

[0033] S104: Determine the degree of constraint violation of the target fairness threshold based on the fairness assessment index data corresponding to the current time step.

[0034] In this embodiment, the degree of constraint violation refers to the quantitative calculation result of the positive deviation of the fairness evaluation index data at the current time step from the target fairness threshold, based on the target fairness threshold as the judgment benchmark. It represents the extent to which the fairness evaluation index exceeds the target threshold. When it does not exceed the threshold, the value is 0. It is the core penalty quantification parameter for constructing the loss function in subsequent Lagrangian constraint reinforcement learning.

[0035] In one embodiment, determining the degree of constraint violation of the target fairness threshold based on the fairness assessment index data corresponding to the current time step includes: Calculate the difference between the fairness assessment index data corresponding to the current time step and the target fairness threshold; determine the larger value between the difference and 0 as the degree of constraint violation of the target fairness threshold.

[0036] S105: Based on the objective fairness threshold and the degree of constraint violation, the parameters of the neural network model used for ride-hailing order scheduling are determined using Lagrange constraint reinforcement learning, and the updated neural network model is determined based on the parameters of the neural network model.

[0037] In this embodiment, Lagrange-constrained reinforcement learning is an optimization method that introduces constraints into reinforcement learning, aiming to maximize cumulative rewards while satisfying specific constraints. The neural network model may contain a policy network and a value network. In this embodiment, the policy network is a feedforward neural network responsible for the decision-making output of ride-hailing order scheduling actions. The value network is a feedforward neural network responsible for the quantitative evaluation of the state value of ride-hailing scheduling. The parameters of the policy network determine the probability distribution output of order-driver matching, and the parameters of the value network determine the estimation result of the state value.

[0038] In this embodiment, the target fairness threshold can be used as the constraint benchmark and the degree of constraint violation as the penalty coefficient, which are embedded in the construction of the Lagrange function and the loss function. Starting from the initial parameters of the neural network model, the model parameters are updated in each iteration through training with Lagrange constraint reinforcement learning until the iteration result meets the preset convergence condition to determine the final parameters of the ride-hailing order scheduling neural network model. Based on these parameters, the original neural network model is replaced to generate a neural network model with updated parameters.

[0039] S106: Scheduling ride-hailing orders based on a neural network model with updated parameters.

[0040] In this embodiment, the policy network in the neural network model can output the probability distribution of each driver accepting the order, and complete the state value auxiliary evaluation through the value network; based on the probability distribution, the optimal order-driver matching result is determined, and the actual ride-hailing order scheduling is completed.

[0041] As can be seen from the above, this application embodiment, by acquiring pressure index data characterizing the tightness of market supply and demand, and driver income fairness assessment index data at multiple time steps, can accurately capture the real-time supply and demand dynamics and historical income fairness distribution patterns of the ride-hailing market. Based on the historical statistical characteristics determined by the above data, the overall distribution trend of driver income can be reflected. Combined with market pressure indicators, a target fairness threshold is dynamically formulated, replacing the traditional fixed threshold. This solves the problem that a fixed threshold that is too strict during market peaks leads to decreased scheduling efficiency, while one that is too lenient during off-peak periods fails to guarantee driver income fairness. At the same time, by calculating the degree of constraint violation, a constraint reference is provided for Lagrange constraint reinforcement learning. This allows this application embodiment to optimize the parameters of the scheduling neural network model based on dynamic thresholds and the degree of violation. The updated model executes scheduling decisions, adapting the strategy to the real-time market state and optimizing ride-hailing order scheduling.

[0042] In one embodiment of this application, Lagrange-constrained reinforcement learning is implemented based on a pre-constructed Lagrange function and a loss function; Based on the objective fairness threshold and the degree of constraint violation, Lagrange constraint reinforcement learning is used to determine the parameters of the neural network model used for ride-hailing order scheduling, including: An initial strategy for obtaining initial parameters to characterize a neural network model; Based on the objective fairness threshold and the degree of constraint violation, the initial policy is iterated using the Lagrangian function and the loss function until the preset convergence condition is reached; The parameters of the neural network model are determined based on the strategy after each iteration; In this process, the target fairness threshold is updated after each iteration.

[0043] In this embodiment, the initial strategy, i.e., the decision rule for order-driver matching, is parameterized by a neural network. Its input is the current state (e.g., order characteristics, driver characteristics, market state), and its output is the probability distribution of each driver being assigned the order. The preset convergence condition can be that the fluctuation range of the loss function value is less than a preset fluctuation threshold after N consecutive iterations.

[0044] In one embodiment, the Lagrangian function contains Lagrange multipliers and at least one constraint term, such as a Gini coefficient constraint and / or a waiting time constraint. A target fairness threshold can be embedded as a constraint benchmark in the construction of the Lagrangian function; for example, the constrained optimization problem can first be transformed into a Lagrangian form, such as the original constrained optimization problem being: ,in, Representation strategy, This indicates that the multi-objective comprehensive reward corresponding to the strategy can be calculated based on the multi-objective reward function of Lagrange-constrained reinforcement learning. This indicates the strategy among all candidate scheduling strategies that maximizes the expected value of the multi-objective comprehensive reward R_total. "Subject to" is an abbreviation for "constraint". Indicating in strategy The expected value of the Gini coefficient generated by the operating system varies depending on the strategy, which leads to different order allocation results and thus different driver income distributions and Gini coefficients. This is the Gini coefficient constraint threshold, which is also the target fairness threshold. Representation Strategy Average passenger waiting time This is the waiting time constraint threshold. In this embodiment, This refers to the average passenger waiting time, calculated based on order spatiotemporal process data after the scheduling strategy π is executed in a simulation environment, historical playback environment, or real business environment. Specifically, the waiting time for a single order can be calculated based on timestamps such as order initiation time, driver order acceptance time, and driver arrival time at the pick-up point. Then, the average waiting time over a scheduling cycle is calculated to obtain the average passenger waiting time. In the simulation environment, the order initiation time and driver stage time can be customized. The driver's arrival time at the pick-up point can be determined based on distance and speed. The speed of each driver can be set to an average value or can also be customized.

[0045] Next, the constrained optimization problem described above is transformed into Lagrangian form, and the Lagrangian function can be: ,in, For Lagrange functions, It is a Lagrange multiplier.

[0046] In one embodiment, the degree of constraint violation can be used as a penalty coefficient and embedded in the construction of the loss function. For example, the loss function is: ,in, Represents the loss function. Let m represent the standard proximal policy optimization (PPO) loss, where m represents the number of constraint terms. This represents the degree of violation of the m-th constraint, or the extent of constraint violation.

[0047] ,in, The probability ratio of the policy after the current iteration to the policy after the previous iteration in the same state-action pair is calculated by forward propagation of the neural network, yielding the output probabilities of the old and new policies. A is the advantage function estimate, representing the difference between the actual return and the value network estimate. This is the pruning parameter, which can be set to 0.2 to limit the magnitude of the policy update.

[0048] In this embodiment, the policy network takes the state as input and outputs an action probability distribution to determine which driver an order is assigned to. The value network takes the state as input and outputs a state value to estimate the expected cumulative reward in the current state, which is used to calculate the advantage function to reduce the variance of the policy gradient.

[0049] In this embodiment, the Gini coefficient constraint threshold (target fairness threshold) and waiting time constraint threshold can be updated after each iteration based on the adaptive threshold module. Specifically, the adaptive threshold module receives historical fairness indicators, historical waiting time indicators, and current market status data, and updates the Gini coefficient constraint threshold and waiting time constraint threshold according to a preset update rule.

[0050] For example, the Gini coefficient constraint threshold It can be updated based on historical Gini coefficient window statistics and current market pressure index. In one implementation, the target fairness threshold can be recalculated based on historical mean, historical best value, standard deviation, supply-demand ratio, dynamic price adjustment ratio, weather factor, time factor, etc., and the result can be cropped into a preset interval to obtain the updated Gini coefficient constraint threshold.

[0051] Waiting time constraint threshold There are two possible implementation methods: First, use a fixed service threshold, such as a preset 5 minutes. Second, use a method similar to... A similar adaptive update method dynamically adjusts based on historical waiting time statistics and current market pressure.

[0052] In one embodiment of this application, after each iteration, in addition to updating the Gini coefficient constraint threshold and the waiting time constraint threshold, the Lagrange multipliers are also updated. Specifically, this can be implemented in the following manner: Calculate the actual values ​​of the policy in each dimension constraint term after this iteration; update the Lagrange multipliers based on the preset dual learning rate, the preset threshold of each dimension constraint term, and the actual values ​​of each dimension constraint term.

[0053] In this embodiment, the dual ascent method can be used to update the Lagrange multipliers, for example... , ,in, To control the dual learning rate, the step size for updating the multipliers can be set to 0.001 to 0.01, and determined through parameter tuning on the validation set. This represents the actual value of the m-th constraint under the strategy after this iteration, for example... , This represents the Gini coefficient at the current time step. This represents the threshold of the m-th constraint term, for example... .

[0054] As can be seen from the above, the embodiments of this application embed multiple constraint terms, such as the Gini coefficient and passenger waiting time, into the Lagrangian function, transforming the constrained multi-objective optimization problem into an unconstrained optimization problem. Combined with a loss function that incorporates the degree of constraint violation, the policy iteration always prioritizes constraint satisfaction, solving the problems of difficult-to-adjust penalty coefficients and easy constraint failure in traditional methods. Simultaneously, relying on the pruning mechanism of the PPO algorithm to limit the policy update magnitude, and combining it with a value network to reduce gradient variance, the stability and accuracy of parameter iteration are guaranteed. During the iteration process, fairness and waiting time thresholds are dynamically updated, allowing constraints to adapt to the dynamic changes in the ride-hailing market. This overcomes the limitations of fixed fairness thresholds, achieving multi-objective collaborative optimization of platform revenue, driver income fairness, and passenger experience, thereby improving the adaptability and optimization effect of the scheduling strategy.

[0055] In one embodiment of this application, the parameters of the neural network model used for ride-hailing order scheduling are determined based on the strategy after each iteration, including: The policy is scored after each iteration based on the Pareto optimality method to obtain the target policy with the highest score; The parameters of the neural network model used for ride-hailing order scheduling are determined based on the target strategy.

[0056] In this embodiment, for the two strategies and Let its performance on each objective be as follows: , ,in These represent the platform revenue, driver fairness, and passenger experience corresponding to Strategy A, respectively. Let A represent platform revenue, driver fairness, and passenger experience corresponding to strategy B, respectively. The condition under which strategy A dominates strategy B is: No inferior to in all objectives And superior to at least one objective .

[0057] In this embodiment, the strategy and Performance across various objectives is measured by statistically analyzing the results after deploying the strategy in simulation, historical playback, or test traffic environments. For example, for a candidate strategy... The following metrics can be calculated within an evaluation period: platform revenue metric m_p( Examples include average order value and order completion rate; driver fairness index m_d( ), such as the Gini coefficient, coefficient of variation, or their transformed fairness score; passenger experience index m_x( ), such as average waiting time or its normalized score. Then, the above statistical results are used to construct the strategy. The performance vector is used for Pareto dominance determination and subsequent strategy selection.

[0058] In this embodiment, when a new strategy is generated, the following update logic is executed: Check whether the new strategy is dominated by an existing strategy. If a strategy dominates the new strategy, discard the new strategy. If the new strategy is not dominated, then: add the new strategy to the Pareto strategy set, and remove the strategy dominated by the new strategy from the Pareto strategy set.

[0059] In this embodiment, the relevant operators of the ride-hailing platform can select the strategy most suitable for the current business needs by setting preference weight vectors corresponding to platform revenue, driver fairness, and passenger experience. The score can be obtained by weighting platform revenue, driver fairness, and passenger experience with their respective weights. Finally, the strategy with the highest score is marked as the target strategy, and the parameters represented by the target strategy are used as the parameters of the neural network model.

[0060] In one embodiment of this application, a soft switching mechanism is adopted to avoid the impact of policy switching on user experience: ,in This represents the hybrid strategy actually deployed, where the new strategy is used with probability γ and the old strategy is used with probability (1-γ) during decision-making. This indicates the current online operating strategy. Indicate the target strategy, This represents the switching coefficient, which gradually increases from 0 to 1 (e.g., 0.1 per hour).

[0061] This embodiment allows for a smooth transition to the new strategy through a gradual switch, while providing operators with a rollback window so that the switch can be stopped promptly if the new strategy proves ineffective.

[0062] In one embodiment of this application, the multi-objective reward function can be: ,in, Indicates a multi-objective comprehensive reward. This represents the platform's revenue rewards after normalization. This indicates a fair reward for drivers after normalization. This represents the passenger experience reward after normalization. This represents the weighting coefficient, and the value of the weighting coefficient can be set by the user, for example... , , .

[0063] Platform Rewards ,in Indicates the order price. This is an indicator variable that shows whether an order has been completed, and it takes the value 0 or 1. It takes the value 1 when the order is successfully assigned to a driver and the service is completed, and 0 when the order times out and is not matched. This indicates the total number of orders.

[0064] Fair rewards for drivers The driver fairness reward is calculated based on the Gini coefficient and uses a negative reward format. This represents the Gini coefficient at the current time step. A lower Gini coefficient indicates a more equal distribution of income (desired direction). Since the goal of reinforcement learning is to maximize rewards, the Gini coefficient needs to be negative, so that reducing the Gini coefficient is equivalent to increasing the reward value.

[0065] Passenger Experience Rewards , This represents the average passenger waiting time. This indicates the reference waiting time, for example, set to 5 minutes.

[0066] The aforementioned platform revenue rewards, driver fairness rewards, and passenger experience rewards are normalized before the calculation of the multi-objective comprehensive reward is performed.

[0067] As can be seen from the above, the embodiments of this application first construct a normalized multi-objective reward function, and then weight and integrate three types of rewards: platform revenue, driver fairness, and passenger experience. This approach takes into account the interests of all three parties from the source of reinforcement learning optimization, thus solving the problem of interest imbalance in traditional single-objective optimization. Next, the Pareto optimal method is used to screen non-dominated strategies, construct an optimal strategy set, and eliminate inferior strategies to ensure strategy quality. Finally, a soft strategy switching mechanism is adopted to gradually integrate the old and new strategies through the switching coefficient, thereby avoiding experience fluctuations caused by sudden changes in scheduling modes.

[0068] Corresponding to the reinforcement learning scheduling optimization method based on adaptive fairness constraints in the above embodiments, Figure 2 This is a structural block diagram of a reinforcement learning scheduling optimization device based on adaptive fairness constraints, provided as an embodiment of this application. For ease of explanation, only the parts relevant to the embodiment of this application are shown. References Figure 2 The reinforcement learning scheduling optimization device 20 based on adaptive fairness constraints includes: a data acquisition module 21, a statistical feature determination module 22, a fairness threshold determination module 23, a constraint violation degree determination module 24, a model determination module 25, and an order scheduling module 26.

[0069] The data acquisition module 21 is used to acquire the stress index data of the ride-hailing market at the current time step, the fairness assessment index data corresponding to the current time step and multiple historical time steps respectively; the stress index data is used to characterize the supply and demand tension of the current ride-hailing market, and the fairness assessment index data is used to characterize the degree of income inequality among all ride-hailing drivers in their corresponding time step. The statistical feature determination module 22 is used to determine historical statistical features based on the fairness assessment index data corresponding to multiple historical time steps; the historical statistical features are used to characterize the overall distribution pattern of income of all ride-hailing drivers in multiple historical time steps; Fairness threshold determination module 23 is used to determine the target fairness threshold based on stress index data and historical statistical characteristics; The constraint violation degree determination module 24 is used to determine the degree of constraint violation of the target fairness threshold based on the fairness assessment index data corresponding to the current time step; The model determination module 25 is used to determine the parameters of the neural network model for ride-hailing order scheduling based on the objective fairness threshold and the degree of constraint violation using Lagrange constraint reinforcement learning, and to determine the updated neural network model based on the parameters of the neural network model. The order scheduling module 26 is used to schedule ride-hailing orders based on the updated neural network model.

[0070] In one embodiment of this application, the fairness threshold determination module 23 is specifically used to determine the market adjustment factor based on stress index data; the market adjustment factor is positively correlated with the stress index data; Historical adaptation factors are determined based on historical statistical characteristics. These factors characterize the degree of deviation between historical fairness assessment index data and historical optimal fairness assessment index data at multiple historical time steps. The target fairness threshold is determined based on a pre-set benchmark fairness threshold, market adjustment factor, and historical adaptation factor.

[0071] In one embodiment of this application, the constraint violation degree determination module 24 is specifically used to calculate the difference between the fairness evaluation index data corresponding to the current time step and the target fairness threshold. The larger of the difference and 0 is used to determine the degree of constraint violation of the target fairness threshold.

[0072] In one embodiment of this application, Lagrange-constrained reinforcement learning is implemented based on a pre-constructed Lagrange function and a loss function; The model determination module 25 is specifically used to obtain the initial strategy for the initial parameters used to characterize the neural network model; Based on the objective fairness threshold and the degree of constraint violation, the initial policy is iterated using the Lagrangian function and the loss function until the preset convergence condition is reached; The parameters of the neural network model are determined based on the strategy after each iteration; In this process, the target fairness threshold is updated after each iteration.

[0073] In one embodiment of this application, the Lagrange function contains Lagrange multipliers and at least one constraint term; different constraint terms are used to constrain the strategy obtained during the iteration process from different dimensions; The model determination module 25 is also used to calculate the actual values ​​of the constraint terms of the strategy in each dimension after this iteration. The Lagrange multipliers are updated based on the preset dual learning rate, the preset thresholds of each dimension constraint term, and the actual values ​​of each dimension constraint term.

[0074] In one embodiment of this application, whether the preset convergence condition is met is determined based on the loss function value after each iteration; the model determination module 25 is further used to obtain the violation amount of the policy in each dimension constraint after the current iteration, and the probability ratio of the policy after the current iteration to the policy after the previous iteration on the same state-action pair. The loss function value after this iteration is determined based on the violation amount and the probability ratio.

[0075] In one embodiment of this application, the model determination module 25 is further configured to score the policy after each iteration based on the Pareto optimality method to obtain the target policy with the highest score. The parameters of the neural network model used for ride-hailing order scheduling are determined based on the target strategy.

[0076] See Figure 3 , Figure 3 This is a schematic block diagram of an electronic device provided according to an embodiment of this application. Figure 3 The electronic device 300 in this embodiment may include one or more processors 301, one or more input devices 302, one or more output devices 303, and one or more memories 304. The processors 301, input devices 302, output devices 303, and memories 304 communicate with each other via a communication bus 305. The memories 304 store computer programs, including program instructions. The processors 301 execute the program instructions stored in the memories 304. Specifically, the processors 301 are configured to invoke the program instructions to perform the functions of each module / unit in the above-described device embodiments, for example... Figure 2 The functions of the data acquisition module 21, statistical feature determination module 22, fairness threshold determination module 23, constraint violation degree determination module 24, model determination module 25, and order scheduling module 26 are shown.

[0077] It should be understood that, in the embodiments of this application, the processor 301 may be a central processing unit (CPU), but it may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0078] Input device 302 may include a touchpad, a fingerprint sensor (for collecting the user's fingerprint information and fingerprint orientation information), a microphone, etc., and output device 303 may include a display (LCD, etc.), a speaker, etc.

[0079] The memory 304 may include read-only memory and random access memory, and provides instructions and data to the processor 301. A portion of the memory 304 may also include non-volatile random access memory. For example, the memory 304 may also store device type information.

[0080] In specific implementations, the processor 301, input device 302, and output device 303 described in the embodiments of this application can execute the implementation method described in the reinforcement learning scheduling optimization method based on adaptive fairness constraints provided in the embodiments of this application, or they can execute the implementation method of the electronic device described in the embodiments of this application, which will not be repeated here.

[0081] In another embodiment of this application, a computer-readable storage medium is provided. This computer-readable storage medium stores a computer program, which includes program instructions. When executed by a processor, the program instructions implement all or part of the processes in the methods described above. Alternatively, the computer program can instruct related hardware to complete the process. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include any entity or device capable of carrying computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0082] The computer-readable storage medium can be an internal storage unit of the electronic device in any of the foregoing embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device. Furthermore, the computer-readable storage medium can include both internal and external storage units of the electronic device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0083] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0084] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the electronic devices and units described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0085] In the several embodiments provided in this application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces or units, or it may be an electrical, mechanical, or other form of connection.

[0086] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0087] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0088] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0089] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A reinforcement learning scheduling optimization method based on adaptive fairness constraints, characterized in that, include: The system acquires stress index data for the ride-hailing market at the current time step, as well as fairness assessment index data for the current time step and multiple historical time steps. The stress index data is used to characterize the supply and demand tension in the current ride-hailing market, and the fairness assessment index data is used to characterize the degree of income inequality among all ride-hailing drivers at their corresponding time steps. Historical statistical features are determined based on the fairness assessment index data corresponding to the multiple historical time steps; the historical statistical features are used to characterize the overall distribution pattern of the income of all ride-hailing drivers in the multiple historical time steps. The target fairness threshold is determined based on the stress index data and the historical statistical characteristics. The degree of constraint violation of the target fairness threshold is determined based on the fairness assessment index data corresponding to the current time step; Based on the target fairness threshold and the degree of constraint violation, the parameters of the neural network model used for ride-hailing order scheduling are determined using Lagrange constraint reinforcement learning, and the updated neural network model is determined based on the parameters of the neural network model. The scheduling of ride-hailing orders is based on the neural network model updated with the parameters.

2. The reinforcement learning scheduling optimization method based on adaptive fairness constraints as described in claim 1, characterized in that, The determination of the target fairness threshold based on the stress index data and the historical statistical characteristics includes: A market adjustment factor is determined based on the aforementioned pressure index data; the market adjustment factor is positively correlated with the aforementioned pressure index data. Historical adaptation factors are determined based on the aforementioned historical statistical characteristics; these historical adaptation factors are used to characterize the degree of deviation of the historical fairness assessment index data corresponding to the multiple historical time steps from the historical optimal fairness assessment index data. The target fairness threshold is determined based on a preset benchmark fairness threshold, the market adjustment factor, and the historical adaptation factor.

3. The reinforcement learning scheduling optimization method based on adaptive fairness constraints as described in claim 1, characterized in that, The determination of the degree of constraint violation of the target fairness threshold based on the fairness assessment index data corresponding to the current time step includes: Calculate the difference between the fairness assessment index data corresponding to the current time step and the target fairness threshold; The larger of the difference and 0 is determined as the degree of constraint violation of the target fairness threshold.

4. The reinforcement learning scheduling optimization method based on adaptive fairness constraints as described in claim 1, characterized in that, The Lagrange-constrained reinforcement learning is implemented based on a pre-constructed Lagrange function and loss function; The parameters of the neural network model used for ride-hailing order scheduling, determined using Lagrange constraint reinforcement learning based on the target fairness threshold and the degree of constraint violation, include: An initial strategy for obtaining initial parameters to characterize the neural network model; Based on the target fairness threshold and the degree of constraint violation, the initial policy is iterated using the Lagrange function and the loss function until the preset convergence condition is reached; The parameters of the neural network model are determined based on the strategy after each iteration; The target fairness threshold is updated after each iteration.

5. The reinforcement learning scheduling optimization method based on adaptive fairness constraints as described in claim 4, characterized in that, The Lagrange function contains Lagrange multipliers and at least one constraint term; different constraint terms are used to constrain the strategy obtained during the iteration process from different dimensions. The Lagrange multipliers are updated after each iteration in the following manner: Calculate the actual values ​​of the policy constraints in each dimension after this iteration; The Lagrange multipliers are updated based on the preset dual learning rate, the preset thresholds of each dimension constraint term, and the actual values ​​of each dimension constraint term.

6. The reinforcement learning scheduling optimization method based on adaptive fairness constraints as described in claim 5, characterized in that, Whether the preset convergence condition has been met is determined based on the loss function value after each iteration; The process of determining the loss function value after each iteration includes: Obtain the violation of constraints in each dimension of the policy after this iteration, and the probability ratio of the policy after this iteration to the policy after the previous iteration on the same state-action pair; The loss function value after this iteration is determined based on the violation amount and the probability ratio.

7. The reinforcement learning scheduling optimization method based on adaptive fairness constraints as described in claim 4, characterized in that, The process of determining the parameters of the neural network model used for ride-hailing order scheduling based on the strategy after each iteration includes: The strategy after each iteration is scored based on the Pareto optimality method to obtain the target strategy with the highest score. The parameters of the neural network model used for ride-hailing order scheduling are determined based on the target strategy.

8. A reinforcement learning scheduling optimization device based on adaptive fairness constraints, characterized in that, include: The data acquisition module is used to acquire the stress index data of the ride-hailing market at the current time step, the fairness assessment index data corresponding to the current time step and multiple historical time steps respectively; wherein, the stress index data is used to characterize the supply and demand tension of the current ride-hailing market, and the fairness assessment index data is used to characterize the degree of income inequality among all ride-hailing drivers in their corresponding time steps. The statistical feature determination module is used to determine historical statistical features based on the fairness assessment index data corresponding to the multiple historical time steps; the historical statistical features are used to characterize the overall distribution pattern of the income of all ride-hailing drivers in the multiple historical time steps. A fairness threshold determination module is used to determine a target fairness threshold based on the stress index data and the historical statistical characteristics; The constraint violation degree determination module is used to determine the degree of constraint violation of the target fairness threshold based on the fairness assessment index data corresponding to the current time step; The model determination module is used to determine the parameters of the neural network model for ride-hailing order scheduling based on the target fairness threshold and the degree of constraint violation using Lagrange constraint reinforcement learning, and to determine the updated neural network model based on the parameters of the neural network model. The order scheduling module is used to schedule ride-hailing orders based on the updated neural network model with the parameters mentioned above.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images