Order supply and demand matching method and system based on near-end strategy optimization algorithm

Abstract

The invention relates to the technical field of order supply and demand management, in particular to an order supply and demand matching method and system based on a near-end strategy optimization algorithm, and the method comprises the steps: obtaining order data and supplier data in real time; processing the obtained order data and supplier data, and constructing a multi-dimensional state vector including spatial-temporal features, supply and demand distribution features, historical matching features and real-time state indexes; on the basis of the multi-dimensional state vector, calculation is carried out through a near-end strategy optimization PPO algorithm, and matching probability distribution of the order and each candidate supplier is generated; according to the matching probability distribution, generating a matching decision in combination with an epsilon-greedy strategy to select a target supplier; sending a distribution instruction to the target supplier to execute order distribution operation; and recording interaction data of each distribution operation and storing the interaction data to an experience playback buffer area. Real-time fluctuation of a supply-demand relation can be quickly responded, a matching strategy is adjusted in time, and stable operation of the system in a complex scene is ensured.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of order supply and demand management, in particular to an order supply and demand matching method and system based on a proximal policy optimization algorithm. BACKGROUND

[0002] In the field of order supply and demand matching, the prior art has obvious limitations. Traditional algorithms such as greedy algorithm and linear programming usually rely on a single indicator such as distance or time for local optimal matching, which is difficult to cope with real-time dynamic scenarios with dramatic fluctuations in order volume (such as a 400% increase during a big promotion), leading to unbalanced resource scheduling, order backlog and low completion rate. The rule-based heuristic method requires manual design of rules, which has poor flexibility and adaptability and cannot adapt to complex and variable business environments.

[0003] In machine learning methods, simple neural network models lack the ability to dynamically optimize based on real-time feedback and are prone to suboptimal solutions. Reinforcement learning schemes such as DQN algorithm have overestimation problems and are difficult to handle multiple-to-multiple continuous action spaces; the A2C algorithm training process is unstable and requires a large number of samples to converge, which cannot meet the real-time matching efficiency requirements.

[0004] In addition, existing methods focus on single target optimization (such as shortest distance or fastest response) and ignore global resource balancing, often leading to overloading of some suppliers and idling of others, resulting in resource waste and service degradation. SUMMARY

[0005] To solve the above problems, the application provides an order supply and demand matching method and system based on a proximal policy optimization algorithm, which is applied to real-time dynamic order matching in multiple scenarios such as e-commerce logistics, food delivery, and online car dispatching.

[0006] In a first aspect, the application provides an order supply and demand matching method based on a proximal policy optimization algorithm, comprising the following steps: S1, real-time acquisition of order data and supplier data, the order data including order ID, geographic location, service type, time requirement and priority, and the supplier data including supplier ID, current location, service capacity, service status and historical service score; S2, processing the acquired order data and supplier data to construct a multi-dimensional state vector containing space-time features, supply and demand distribution features, historical matching features and real-time state indicators; S3, based on the multi-dimensional state vector, calculating the matching probability distribution of orders and each candidate supplier by a proximal policy optimization (PPO) algorithm; wherein the policy network and value network of the PPO algorithm are pre-trained; S4. Based on the matching probability distribution, generate a matching decision using an ε-greedy strategy to select a target supplier; send an allocation instruction to the target supplier to execute the order allocation operation; and record the interactive data of each allocation operation to the experience replay buffer. S5. Periodically or when the amount of data in the experience replay buffer reaches a preset threshold, randomly sample a batch of trajectory data from the buffer; based on the sampled data, iteratively update the parameters of the policy network and the value network.

[0007] By acquiring detailed order and supplier data in real time, a comprehensive and accurate information foundation is provided for subsequent matching decisions, enabling these decisions to fully consider various factors and improve the accuracy and rationality of the matching process. Integrating information from multiple dimensions, such as spatiotemporal features, supply and demand distribution characteristics, historical matching features, and real-time status indicators, into a multi-dimensional state vector allows the model to more comprehensively perceive the order supply and demand environment, providing richer information support for strategy decisions and improving matching accuracy and adaptability. The Proximal Policy Optimization (PPO) algorithm is used to generate the matching probability distribution. The PPO algorithm has advantages such as stable training and high sample efficiency, effectively solving the problems of unstable training and low sample efficiency in traditional reinforcement learning algorithms, thus improving the system's matching efficiency and stability. Combining an ε-greedy policy to generate matching decisions allows for the utilization of the current optimal matching scheme to a certain extent, while also discovering new and potentially better matching schemes through random exploration. This balances exploration and utilization, preventing the policy from prematurely converging to a local optimum, and improving the system's adaptability and matching effect.

[0008] As a further limitation of the technical solution of the present invention, in S2, the step of processing the acquired order data and supplier data to construct a multi-dimensional state vector containing spatiotemporal characteristics, supply and demand distribution characteristics, historical matching characteristics, and real-time status indicators includes: S21. Clean the acquired raw order data and supplier data, remove invalid and outlier values, and standardize the numerical data to the [0,1] range. S22. Extract the latitude and longitude coordinates of the order and supplier's geographical location, and combine them with the order generation timestamp to encode and generate spatiotemporal features, including weekday / weekend and morning / evening peak features; S23. Using a preset geographical area grid as the unit, calculate the order density and supplier density in each grid in real time, and calculate the ratio between the two as a supply and demand distribution characteristic that reflects the degree of regional supply and demand imbalance. S24. Extract historical matching success rate, average response time and order cancellation rate data for different regions and time periods within a preset time window from the historical matching record database to obtain historical matching features; S25. Extract real-time status indicators from suppliers, including current load rate, online status, and real-time waiting time and timeout warning level for orders; S26. The spatiotemporal features, supply and demand distribution features, historical matching features, and real-time status indicators are spliced ​​and fused to generate a multi-dimensional state vector.

[0009] Using a pre-defined geographical grid as the unit, the system calculates the order density and supplier density within each grid in real time, and the ratio of these two densities serves as a supply-demand distribution characteristic. This intuitively reflects the degree of supply-demand imbalance in different regions, providing the model with crucial information about regional supply and demand. This helps the model prioritize regions with more severe supply-demand imbalances during the matching process, optimizing resource allocation and improving overall matching efficiency and resource utilization. Historical matching success rates, average response times, and order cancellation rates for different regions and time periods within a pre-defined time window are extracted from the historical matching record database to obtain historical matching characteristics. This allows the model to learn from historical experience, understand matching effects and problems in different scenarios, and avoid repeating past errors in current matching decisions, thereby improving matching success rates and efficiency. Furthermore, based on patterns and trends in historical data, the system can predict and adjust future order supply and demand conditions.

[0010] As a further limitation of the technical solution of the present invention, based on the multidimensional state vector, a probabilistic probability distribution of matching between orders and each candidate supplier is generated by calculating the PPO algorithm through a near-end strategy optimization; specifically: The multidimensional state vector is input into a pre-trained policy network. After forward propagation, the policy network outputs a probability distribution vector, where each dimension of the vector corresponds to a candidate supplier, and its value represents the matching probability of assigning the current order to that supplier. The policy network adopts a multilayer perceptron (MLP) architecture, where the number of neurons in the input layer is the same as the dimension of the multidimensional state vector, the number of neurons in the output layer is the same as the number of candidate suppliers, and the output layer uses the Softmax activation function to normalize the output value to a matching probability distribution. While the policy network generates the matching probability distribution, a multidimensional state vector is also input into the value network of the near-end policy optimization algorithm; the value network is a deep neural network used to output a scalar value evaluation value for the current system state.

[0011] While the policy network generates the matching probability distribution, a multi-dimensional state vector is also input into the value network of the near-end policy optimization algorithm. The value network outputs a scalar value assessment of the current system state, which can provide an overall value assessment of the current order supply and demand state. This provides a reference for the policy network's decision-making, helping the system to consider the value of the current state while considering the matching probability, thereby making a more comprehensive and optimized matching decision and further improving the system's matching effect and performance.

[0012] As a further limitation of the technical solution of the present invention, the following steps are performed before inputting the state vector into the policy network: Based on the order's timeliness requirements, geographical location, and the supplier's service capabilities and status, a set of candidate suppliers that meet the basic matching criteria is selected from all suppliers. The probability distribution generated by the policy network is only for suppliers in the candidate supplier set; The basic matching conditions include one or more of the following conditions: (a) The supplier is currently in a serviceable state; (b) The geographical distance between the supplier's current location and the order's destination is less than a first preset threshold; (c) The supplier’s current load is below a second preset threshold of its maximum service capacity; (d) The supplier possesses the qualifications required for the type of service specified in the order; (e) The supplier’s historical service rating is higher than the third preset threshold.

[0013] Before inputting the state vector into the policy network, a set of candidate suppliers that meet the basic matching conditions is selected from all suppliers based on the order's timeliness requirements, geographical location, and the supplier's service capabilities and status. This can quickly eliminate suppliers that obviously do not meet the requirements, narrow the matching range, reduce the amount of computation, improve matching efficiency, and also help improve the accuracy of matching.

[0014] As a further limitation of the technical solution of the present invention, in S4, the step of generating a matching decision based on the matching probability distribution and combining it with an ε-greedy strategy to select a target supplier; and sending an allocation instruction to the target supplier to execute the order allocation operation includes: S41. Generate a random number ρ between 0 and 1; If ρ≤ε, then the exploration strategy is executed, and a supplier is randomly selected from the set of candidate suppliers; If ρ>ε, then the utilization strategy is executed, and the supplier with the highest probability value in the matching probability distribution is selected; S42. Send an order matching request to the supplier selected in S41 and receive its response; If the response is "accept", the matching operation is complete. If the response is timeout or rejection, the supplier is removed from the current candidate supplier set, and S41 is repeated until a match is found or the candidate supplier set is empty.

[0015] As a further limitation of the technical solution of the present invention, the step of recording the interactive data of each allocation operation to the experience replay buffer in S4 includes: Record the multidimensional state vector constructed before this allocation operation occurs; Record the action performed by S4 this time, that is, the unique identifier assigned to the supplier of the order; The receiving system calculates the immediate reward based on the result of this allocation operation using a preset multi-objective reward function R; whereby the multi-objective reward function R is:

[0016] In the formula, This is a reward sub-item based on order response time and completion time; This is a reward sub-item based on the supplier's load balancing level; This is a reward sub-item based on user ratings and order cancellation rates; These are the weighting coefficients for each reward sub-item; After the allocation operation is completed, data is collected again after a preset time interval to construct a new multidimensional state vector, which is the next state vector after the action is performed. Store the quadruple data <state, action, reward, next state> as an experience tuple in the experience replay buffer.

[0017] As a further limitation of the technical solution of the present invention, the step of iteratively updating the parameters of the policy network and the value network includes: The generalized advantage estimation algorithm is used to calculate the advantage value of actions; the pruning objective function is used to limit the range of change of the probability ratio between the old and new policies, so as to constrain the parameter update magnitude of the policy network; and the stochastic gradient descent algorithm is used to iteratively update the policy network and value network parameters of the PPO algorithm based on the constrained objective function.

[0018] As a further limitation of the technical solution of the present invention, the advantage value of the action is calculated through generalized advantage estimation and updated using the pruning objective function constraint strategy, specifically including: Based on trajectory data sampled from the experience replay buffer, the generalized dominance estimation algorithm is used to calculate the dominance function value of the action at each time step. ; Based on the advantage function value, the pruning objective function of PPO is calculated to update the policy network parameters. The objective function expression is as follows:

[0019] The probability ratio between the old and new strategies. For pruning hyperparameters; By minimizing the value network prediction value Compared with the actual return estimate The mean squared error loss function is used to update the value network parameters. ϕ The loss function expression is:

[0020] Using the stochastic gradient descent algorithm, a joint loss function is generated by combining the pruning objective function and the loss function. This joint loss function is then used to update the parameters of the policy network and the value network in a single operation. The joint loss function is as follows:

[0021] in, The weighting factor for the value loss item. The weighting coefficient for the entropy reward term. For strategy In state The entropy is used to encourage exploration and prevent strategies from prematurely converging to local optima.

[0022]

[0023] It is the action space, which contains all possible actions. , It is in state Next, according to the strategy Select Action The probability of.

[0024] As a further limitation of the technical solution of the present invention, the generalized dominance estimation algorithm calculates the dominance value using the following formula:

[0025] In the formula, γ This is a discount factor used to weigh the importance of current and future returns; These are GAE parameters used to control the tradeoff between bias and variance; For the first Step-time differential error; Actual return estimates in the step of updating value network parameters The objective is calculated using time difference learning, and its expression is:

[0026] From the old value network parameters The calculation is as follows: State Input into the old value network. Use the old value network parameters. Calculation state value .

[0027] Secondly, the technical solution of the present invention also provides an order supply and demand matching system based on a near-end strategy optimization algorithm, comprising: The data acquisition module is used to acquire order data and supplier data in real time. The order data includes order ID, geographical location, service type, timeliness requirements and priority. The supplier data includes supplier ID, current location, service capability, service status and historical service rating. The state construction module is used to process the acquired order data and supplier data to construct a multi-dimensional state vector that includes spatiotemporal features, supply and demand distribution features, historical matching features, and real-time state indicators. The strategy decision module, based on the multidimensional state vector, calculates the matching probability distribution between the order and each candidate supplier using the near-end strategy optimization (PPO) algorithm; wherein, the strategy network and value network of the PPO algorithm are pre-trained. The matching execution module is used to generate a matching decision based on the matching probability distribution and an ε-greedy strategy to select a target supplier; send an allocation instruction to the target supplier to execute the order allocation operation; and record the interactive data of each allocation operation to the experience replay buffer. The training optimization module is used to periodically or when the amount of data in the experience replay buffer reaches a preset threshold, randomly sample a batch of trajectory data from the buffer; and iteratively update the parameters of the policy network and the value network based on the sampled data.

[0028] As can be seen from the above technical solutions, this application has the following advantages: it solves the problems of poor dynamic adaptability and unstable reinforcement learning training in traditional algorithms, while overcoming the overestimation problem of traditional reinforcement learning algorithms such as DQN and the instability and low sample efficiency of the A2C training process, ensuring stable model training and efficient operation, and reducing training time and resource consumption. The PPO algorithm, by pruning the objective function, effectively avoids performance degradation caused by excessive policy updates, ensuring the stability of model training and better adapting to the dynamic changes in order supply and demand. By integrating multi-dimensional features such as spatiotemporal, supply and demand distribution, historical matching, and real-time status indicators, the model can more comprehensively perceive the order supply and demand environment, providing richer information support for decision-making and improving matching accuracy and efficiency.

[0029] The training process comprehensively considers multiple objectives such as efficiency, fairness, and stability. By appropriately setting the reward function, the model is guided to pursue efficient matching while also considering supplier load balancing and service quality, avoiding overload on some suppliers while others are idle. Through multi-dimensional state construction and intelligent strategy decision-making, orders and suppliers are matched more accurately, reducing order backlogs and cancellations caused by unreasonable matching, and significantly improving order completion rates. Real-time perception of order and supplier status information enables rapid matching decisions, shortening the time from order generation to allocation and reducing average response time. Optimized global resource allocation balances supplier load, preventing resource idleness and waste.

[0030] It can quickly respond to real-time fluctuations in supply and demand, and adjust matching strategies in a timely manner when faced with sudden increases or decreases in order volume or dynamic changes in suppliers, ensuring stable operation of the system in complex scenarios. Attached Figure Description

[0031] To more clearly illustrate the technical solution of this application, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a flowchart illustrating the method provided in an embodiment of the present invention.

[0033] Figure 2 This is an overall structural diagram of the system provided in an embodiment of the present invention. Detailed Implementation

[0034] To make the purpose, features, and advantages of this application more apparent and understandable, specific embodiments and accompanying drawings will be used to clearly and completely describe the technical solution protected by this application. Obviously, the embodiments described below are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0035] Unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this application and in the specification of this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0036] like Figure 1 As shown, this embodiment of the invention provides an order supply and demand matching method based on a near-end strategy optimization algorithm, including the following steps: S1. Real-time acquisition of order and supplier data; In this embodiment of the invention, the collected order data includes key information such as order ID, geographical location, service type (e.g., fresh food express delivery, regular delivery), delivery time requirement (e.g., 30-minute delivery), and priority. Supplier data includes supplier ID, current location, service capacity (e.g., remaining delivery capacity, number of orders that can be accepted), service status (busy / idle), and historical service ratings. This data provides a comprehensive information foundation for subsequent matching decisions.

[0037] S2. Process the acquired order data and supplier data to construct a multi-dimensional state vector that includes spatiotemporal features, supply and demand distribution features, historical matching features, and real-time status indicators; In this embodiment of the invention, this step specifically includes: S21. Clean the acquired raw order data and supplier data, remove invalid and outlier values, and standardize the numerical data to the [0,1] range. S22. Extract the latitude and longitude coordinates of the order and supplier's geographical location, and combine them with the order generation timestamp to encode and generate spatiotemporal features, including weekday / weekend and morning / evening peak features; S23. Using a preset geographical area grid as the unit, calculate the order density and supplier density in each grid in real time, and calculate the ratio between the two as a supply and demand distribution characteristic that reflects the degree of regional supply and demand imbalance. S24. Extract historical matching success rate, average response time and order cancellation rate data for different regions and time periods within a preset time window from the historical matching record database to obtain historical matching features; S25. Extract real-time status indicators from suppliers, including current load rate, online status, and real-time waiting time and timeout warning level for orders; S26. The spatiotemporal features, supply and demand distribution features, historical matching features, and real-time status indicators are spliced ​​and fused to generate a multi-dimensional state vector.

[0038] In other words, spatiotemporal features: The geographical locations of orders and suppliers are precisely encoded into latitude and longitude coordinates, and combined with timestamp information to generate spatiotemporal features reflecting the characteristics of different time periods, such as weekdays / weekends, morning peak hours / evening peak hours. For example, during peak nighttime hours of e-commerce promotions, this feature allows the model to perceive the specificities of order demand and thus adjust the matching strategy.

[0039] Supply and demand distribution characteristics: By calculating the ratio of order density to supplier density within a region, the degree of supply and demand imbalance can be analyzed. For example, during the lunch rush hour in a business district, the system can quickly identify when the order density is several times higher than the supplier density, and then optimize the matching strategy accordingly.

[0040] Historical matching features: Information such as successful / failed cases and average response time are extracted from historical matching records, serving as important empirical references for policy learning. For example, if a certain region has a low success rate in historical nighttime order delivery, the model can accordingly reduce the weight of orders allocated to that region during that time period.

[0041] Real-time status indicators: Incorporating real-time status indicators such as the supplier's current load rate and order timeout warning level, the model can perceive the system's operating status in real time, providing a more accurate basis for dynamic matching.

[0042] S3. Based on the multidimensional state vector, the matching probability distribution between the order and each candidate supplier is generated by calculating the PPO algorithm through the near-end strategy optimization; wherein, the policy network and value network of the PPO algorithm are pre-trained. It's important to note that the Proximal Policy Optimization (PPO) algorithm is based on an Actor-Critic architecture. The Actor network takes the state vector output by the state building module as input and, through complex neural network calculations, outputs the probability distribution of all possible matching actions (i.e., allocating an order to a supplier). This probability distribution provides a quantitative basis for matching decisions, guiding the system to select the optimal allocation scheme. The Critic network evaluates the value of the current state and predicts the expected return after taking a certain action. Its output provides feedback to the Actor network, helping it continuously optimize the matching strategy to achieve higher returns.

[0043] This step specifically includes: The multidimensional state vector is input into a pre-trained policy network. After forward propagation, the policy network outputs a probability distribution vector, where each dimension of the vector corresponds to a candidate supplier, and its value represents the matching probability of assigning the current order to that supplier. The policy network adopts a multilayer perceptron (MLP) architecture, where the number of neurons in the input layer is the same as the dimension of the multidimensional state vector, the number of neurons in the output layer is the same as the number of candidate suppliers, and the output layer uses the Softmax activation function to normalize the output value to a matching probability distribution. While the policy network generates the matching probability distribution, a multidimensional state vector is also input into the value network of the near-end policy optimization algorithm; the value network is a deep neural network used to output a scalar value evaluation value for the current system state.

[0044] It should be further explained that, based on the timeliness requirements of the order, geographical location, and the service capabilities and status of the supplier, a set of candidate suppliers that meet the basic matching conditions is selected from all suppliers. The probability distribution generated by the policy network is only for suppliers in the candidate supplier set; The basic matching conditions include one or more of the following conditions: (a) The supplier is currently in a serviceable state; (b) The geographical distance between the supplier's current location and the order's destination is less than a first preset threshold; (c) The supplier’s current load is below a second preset threshold of its maximum service capacity; (d) The supplier possesses the qualifications required for the type of service specified in the order; (e) The supplier’s historical service rating is higher than the third preset threshold.

[0045] Another point to explain is the specific training steps for the policy network and value network in the PPO algorithm: (1) Randomly sample a batch of trajectory data from the experience playback buffer. Each trajectory data contains the following: state : A multidimensional state vector that includes spatiotemporal features, supply and demand distribution features, historical matching features, and real-time state indicators.

[0046] action In state The selected supplier.

[0047] award The immediate reward calculated based on the multi-objective reward function.

[0048] Next state : Execute action The new state after that.

[0049] (2) Perform necessary preprocessing on the sampled data, such as normalization and standardization, to improve the stability and efficiency of training.

[0050] The GAE algorithm is used to calculate each time step. t Action advantage value ;

[0051] γ This is a discount factor used to weigh the importance of current and future returns; These are GAE parameters used to control the tradeoff between bias and variance; For the first Step-time differential error; (3) For each sampled data point, calculate the probability ratio of the new and old strategies. ;

[0052] It is in state Next, according to the strategy Select Action The probability of.

[0053] Based on the dominance value Probability ratio between old and new strategies Calculate the PPO pruning objective function. Using the stochastic gradient descent algorithm, based on the pruning objective function... Update the parameters of the policy network .

[0054] (4) Calculate the mean squared error loss function of the value network based on the sampled data. Using the stochastic gradient descent algorithm, based on the value network loss function... Update the parameters ϕ of the value network.

[0055] (5) Combine the pruning objective function and value network loss function Generate joint loss function Using the stochastic gradient descent algorithm, based on the joint loss function... The parameters of the policy network and the value network are updated jointly in one go.

[0056] (6) For the same batch of sampled trajectory data, perform multiple epochs of iterative updates. In each iteration, recalculate the advantage function value and the objective function, and perform parameter updates until the preset number of updates is reached.K This is to fully explore the value of the sampling data and improve the efficiency of sample utilization.

[0057] S4. Based on the matching probability distribution, generate a matching decision using an ε-greedy strategy to select a target supplier; send an allocation instruction to the target supplier to execute the order allocation operation; and record the interactive data of each allocation operation to the experience replay buffer. The ε-Greedy strategy is a commonly used exploration-exploitation balance strategy in reinforcement learning. It introduces randomness into action selection, allowing it to utilize currently known optimal actions while also exploring unknown ones, thus preventing the policy from prematurely converging to a local optimum. The core idea is to... t With probability 1− ε Choose the action that the current strategy deems optimal (exploitation), based on probability. ε Randomly select an action (exploration). ε It is a parameter between 0 and 1, representing the probability of exploration.

[0058] S5. Periodically or when the amount of data in the experience replay buffer reaches a preset threshold, randomly sample a batch of trajectory data from the buffer; based on the sampled data, iteratively update the parameters of the policy network and the value network.

[0059] In some embodiments, in step S4, the step of generating a matching decision based on the matching probability distribution and an ε-greedy strategy to select a target supplier, and sending an allocation instruction to the target supplier to execute the order allocation operation includes: S41. Generate a random number ρ between 0 and 1; If ρ≤ε, then the exploration strategy is executed, and a supplier is randomly selected from the set of candidate suppliers; If ρ>ε, then the utilization strategy is executed, and the supplier with the highest probability value in the matching probability distribution is selected; S42. Send an order matching request to the supplier selected in S41 and receive its response; If the response is "accept", the matching operation is complete. If the response is timeout or rejection, the supplier is removed from the current candidate supplier set, and S41 is repeated until a match is found or the candidate supplier set is empty.

[0060] Furthermore, the step in S4 of recording the interactive data of each allocation operation to the experience replay buffer includes: Record the multidimensional state vector constructed before this allocation operation occurs; Record the action performed by S4 this time, that is, the unique identifier assigned to the supplier of the order; The receiving system calculates the immediate reward based on the result of this allocation operation using a preset multi-objective reward function R; whereby the multi-objective reward function R is:

[0061] In the formula, This is a reward sub-item based on order response time and completion time; This is a reward sub-item based on the supplier's load balancing level; This is a reward sub-item based on user ratings and order cancellation rates; These are the weighting coefficients for each reward sub-item; It should be noted that the efficiency reward sub-item The calculation is negatively correlated with the order's response time and completion time; the shorter the response time and completion time, the better. The higher the value;

[0062] The first term in the formula measures the difference between the actual response time and the maximum allowable response time, and is multiplied by a weighting factor for response time. The closer the actual response time is to the maximum allowable response time, the lower the reward; the closer the actual response time is to the maximum allowable response time, the higher the reward.

[0063] The second term in the formula measures the difference between the actual completion time and the maximum allowed completion time, multiplied by a weighting coefficient for completion time. The closer the actual completion time is to the maximum allowed completion time, the lower the reward; conversely, the higher the actual completion time, the higher the reward. This is the weighting coefficient for response time. This represents the maximum allowable response time. This represents the actual response time. The weighting factor for completion time. This is the maximum allowed completion time. This refers to the actual completion time.

[0064] Fair Rewards Sub-item The calculation is negatively correlated with the load variance of all suppliers within the system; the more balanced the load, the smaller the variance. The higher the value;

[0065] In the formula, The weighting coefficients for load balancing. This is the standard deviation function, used to measure the distribution of load. This represents the current load for each supplier.

[0066] Quality Award Sub-items The calculation is positively correlated with user ratings and negatively correlated with order cancellation rates; higher user ratings correspond to lower cancellation rates. The higher the value, the better.

[0067]

[0068] In the formula, the first term measures the difference between the actual average user rating and the baseline user rating, multiplied by a weighting coefficient for user ratings. If the actual average user rating is higher than the baseline user rating, the reward is positive; if it is lower, the reward is negative. The second term measures the order cancellation rate, multiplied by a weighting coefficient for order cancellation rate. A higher order cancellation rate results in a lower reward; a lower order cancellation rate results in a higher reward. Weighting coefficients for user ratings. Average user rating. Basic user ratings are benchmark values ​​used to measure user satisfaction. This is the weighting coefficient for the order cancellation rate. This refers to the order cancellation rate.

[0069] After the allocation operation is completed, data is collected again after a preset time interval to construct a new multidimensional state vector, which is the next state vector after the action is performed. Store the quadruple data <state, action, reward, next state> as an experience tuple in the experience replay buffer.

[0070] In this embodiment of the invention, the step of iteratively updating the parameters of the policy network and the value network includes: The generalized advantage estimation algorithm is used to calculate the advantage value of actions; a pruning objective function is used to limit the range of change in the probability ratio between the old and new policies, thereby constraining the parameter update magnitude of the policy network; and a stochastic gradient descent algorithm is used to iteratively update the policy network and value network parameters of the PPO algorithm based on the constrained objective function. In practice, the update steps are largely the same as the training steps described above.

[0071] By minimizing the value network prediction value Compared with the actual return estimate The mean squared error loss function is used to update the value network parameters. ϕ The loss function expression is:

[0072] Using the stochastic gradient descent algorithm, a joint loss function is generated by combining the pruning objective function and the loss function. This joint loss function is then used to update the parameters of the policy network and the value network in a single operation. The joint loss function is as follows:

[0073] in, The weighting factor for the value loss item. The weighting coefficient for the entropy reward term. For strategy In state The entropy is used to encourage exploration and prevent strategies from prematurely converging to local optima.

[0074]

[0075] It is the action space, which contains all possible actions. , It is in state Next, according to the strategy Select Action The probability of.

[0076] The state space is { , }, the action space is { , }.Strategy It can be defined as follows: In state Down, =0.7, =0.3.

[0077] In state Down, =0.4, =0.6.

[0078] In state Next, strategy There is a 70% probability of choosing an action. There is a 30% probability of choosing an action. In state Next, strategy There is a 40% probability of choosing an action. There is a 60% probability of choosing an action. .

[0079] In state Entropy below for:

[0080] The generalized dominance estimation algorithm calculates the dominance value using the following formula:

[0081] In the formula, where, γThis is a discount factor used to weigh the importance of current and future returns; These are GAE parameters used to control the tradeoff between bias and variance; For the first The time difference error of the step.

[0082] The estimated actual return in the value update step The objective is calculated using time difference learning, and its expression is:

[0083] From the old value network parameters The calculation is as follows: State Input into the old value network. Use the old value network parameters. Calculation state value .

[0084] like Figure 2 As shown, this embodiment of the invention also provides an order supply and demand matching system based on a near-end strategy optimization algorithm, including: The data acquisition module is used to acquire order data and supplier data in real time. The order data includes order ID, geographical location, service type, timeliness requirements and priority. The supplier data includes supplier ID, current location, service capability, service status and historical service rating. The state construction module is used to process the acquired order data and supplier data to construct a multi-dimensional state vector that includes spatiotemporal features, supply and demand distribution features, historical matching features, and real-time state indicators. The strategy decision module, based on the multidimensional state vector, calculates the matching probability distribution between the order and each candidate supplier using the near-end strategy optimization (PPO) algorithm; wherein, the strategy network and value network of the PPO algorithm are pre-trained. The matching execution module is used to generate a matching decision based on the matching probability distribution and an ε-greedy strategy to select a target supplier; send an allocation instruction to the target supplier to execute the order allocation operation; and record the interactive data of each allocation operation to the experience replay buffer. The training optimization module is used to periodically or when the amount of data in the experience replay buffer reaches a preset threshold, randomly sample a batch of trajectory data from the buffer; and iteratively update the parameters of the policy network and the value network based on the sampled data.

[0085] In some embodiments, the state construction module includes: The data cleaning unit is used to clean the acquired raw order data and supplier data, remove invalid and outlier values, and standardize the numerical data to the [0,1] range. The spatiotemporal feature extraction unit is used to extract the latitude and longitude coordinates of the geographical location of the order and the supplier, and combine them with the order generation timestamp to encode and generate spatiotemporal features, including weekday / weekend and morning / evening peak features; The supply and demand distribution characteristic calculation unit is used to calculate the order density and supplier density in each grid in real time, based on a preset geographical area grid, and calculate the ratio between the two as a supply and demand distribution characteristic that reflects the degree of regional supply and demand imbalance. The historical matching feature extraction unit is used to extract historical matching success rate, average response time and order cancellation rate data for different regions and time periods within a preset time window from the historical matching record database to obtain historical matching features; The real-time status indicator extraction unit is used to extract the supplier's real-time status indicators, including the current load rate, online status, and real-time waiting time and timeout warning level of orders. The feature fusion unit is used to splice and fuse the spatiotemporal features, supply and demand distribution features, historical matching features and real-time status indicators to generate a multi-dimensional state vector.

[0086] In some embodiments, the strategy decision module includes: A policy network is used to input the multidimensional state vector into a pre-trained policy network. After forward propagation, the policy network outputs a probability distribution vector, where each dimension of the vector corresponds to a candidate supplier, and its value represents the matching probability of assigning the current order to that supplier. The policy network adopts a multilayer perceptron (MLP) architecture, where the number of neurons in its input layer is the same as the dimension of the multidimensional state vector, the number of neurons in its output layer is the same as the number of candidate suppliers, and the output layer uses a softmax activation function to normalize the output value to a matching probability distribution. A value network is used to input multidimensional state vectors into a proximal policy optimization algorithm; the value network is a deep neural network used to output a scalar value assessment of the current system state.

[0087] In some embodiments, the matching execution module further includes: The candidate supplier screening unit is used to select a set of candidate suppliers that meet basic matching conditions from all suppliers based on order timeliness requirements, geographical location, and the supplier's service capabilities and status; the probability distribution generated by the strategy network applies only to suppliers in this candidate supplier set; the basic matching conditions include one or more of the following conditions: (a) The supplier is currently in a serviceable state; (b) The geographical distance between the supplier's current location and the order's destination is less than a first preset threshold; (c) The supplier’s current load is below a second preset threshold of its maximum service capacity; (d) The supplier possesses the qualifications required for the type of service specified in the order; (e) The supplier’s historical service rating is higher than the third preset threshold.

[0088] In some embodiments, the matching execution module further includes: The random number generation unit is used to generate a random number ρ between 0 and 1; The matching decision unit is used to generate a matching decision based on the random number ρ and the matching probability distribution, combined with an ε-greedy strategy, to select a target supplier; specifically: If ρ≤ε, then the exploration strategy is executed, and a supplier is randomly selected from the set of candidate suppliers; If ρ>ε, then the utilization strategy is executed, and the supplier with the highest probability value in the matching probability distribution is selected; The allocation instruction sending unit is used to send an order matching request to the selected supplier and receive its response; if the response is acceptance, the matching operation is completed; if the response is timeout or rejection, the supplier is removed from the current candidate supplier set, and the matching decision process is repeated until the matching is successful or the candidate supplier set is empty.

[0089] In some embodiments, the matching execution module further includes: The data recording unit is used to record the interactive data of each allocation operation into the experience replay buffer; specifically, it includes: Record the multidimensional state vector constructed before this allocation operation occurs; Record the action performed by the matching decision unit, i.e., the unique identifier assigned to the supplier of the order; The receiving system calculates the immediate reward based on the result of this allocation operation using a preset multi-objective reward function R; whereby the multi-objective reward function R is:

[0090] In the formula, This is a reward sub-item based on order response time and completion time; This is a reward sub-item based on the supplier's load balancing level; This is a reward sub-item based on user ratings and order cancellation rates; These are the weighting coefficients for each reward sub-item; After the allocation operation is completed, data is collected again after a preset time interval to construct a new multidimensional state vector, which is the next state vector after the action is performed. Store the quadruple data <state, action, reward, next state> as an experience tuple in the experience replay buffer.

[0091] In some embodiments, the training optimization module includes: The advantage function calculation unit is used to calculate the advantage value of an action using a generalized advantage estimation algorithm. The pruning objective function unit is used to limit the range of change in the probability ratio between the old and new policies by pruning the objective function, thereby constraining the parameter update magnitude of the policy network; The network parameter update unit is used to iteratively update the policy network and value network parameters of the PPO algorithm based on the constrained objective function using the stochastic gradient descent algorithm.

[0092] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An order supply and demand matching method based on a near-end strategy optimization algorithm, characterized in that, Includes the following steps: S1. Real-time acquisition of order data and supplier data, wherein the order data includes order ID, geographical location, service type, timeliness requirements and priority, and the supplier data includes supplier ID, current location, service capability, service status and historical service rating; S2. Process the acquired order data and supplier data to construct a multi-dimensional state vector that includes spatiotemporal features, supply and demand distribution features, historical matching features, and real-time status indicators; S3. Based on the multidimensional state vector, the matching probability distribution between the order and each candidate supplier is generated by calculating the PPO algorithm through the near-end strategy optimization; wherein, the policy network and value network of the PPO algorithm are pre-trained. S4. Based on the matching probability distribution, generate a matching decision using an ε-greedy strategy to select a target supplier; send an allocation instruction to the target supplier to execute the order allocation operation; and record the interactive data of each allocation operation to the experience replay buffer. S5. Periodically or when the amount of data in the experience replay buffer reaches a preset threshold, randomly sample a batch of trajectory data from the buffer; based on the sampled data, iteratively update the parameters of the policy network and the value network.

2. The order supply and demand matching method based on the near-end strategy optimization algorithm according to claim 1, characterized in that, In S2, the steps of processing the acquired order data and supplier data to construct a multi-dimensional state vector containing spatiotemporal features, supply and demand distribution features, historical matching features, and real-time status indicators include: S21. Clean the acquired raw order data and supplier data, remove invalid and outlier values, and standardize the numerical data to the [0,1] range. S22. Extract the latitude and longitude coordinates of the order and supplier's geographical location, and combine them with the order generation timestamp to encode and generate spatiotemporal features, including weekday / weekend and morning / evening peak features; S23. Using a preset geographical area grid as the unit, calculate the order density and supplier density in each grid in real time, and calculate the ratio between the two as a supply and demand distribution characteristic that reflects the degree of regional supply and demand imbalance. S24. Extract historical matching success rate, average response time and order cancellation rate data for different regions and time periods within a preset time window from the historical matching record database to obtain historical matching features; S25. Extract real-time status indicators from suppliers, including current load rate, online status, and real-time waiting time and timeout warning level for orders; S26. The spatiotemporal features, supply and demand distribution features, historical matching features, and real-time status indicators are spliced ​​and fused to generate a multi-dimensional state vector.

3. The order supply and demand matching method based on the near-end strategy optimization algorithm according to claim 2, characterized in that, Based on the multidimensional state vector, the PPO algorithm is optimized using a near-end strategy to calculate and generate the matching probability distribution between orders and each candidate supplier; specifically: The multidimensional state vector is input into a pre-trained policy network. After forward propagation, the policy network outputs a probability distribution vector, where each dimension of the vector corresponds to a candidate supplier, and its value represents the matching probability of assigning the current order to that supplier. The policy network adopts a multilayer perceptron (MLP) architecture, where the number of neurons in the input layer is the same as the dimension of the multidimensional state vector, the number of neurons in the output layer is the same as the number of candidate suppliers, and the output layer uses the Softmax activation function to normalize the output value to a matching probability distribution. While the policy network generates the matching probability distribution, a multidimensional state vector is also input into the value network of the near-end policy optimization algorithm; the value network is a deep neural network used to output a scalar value evaluation value for the current system state.

4. The order supply and demand matching method based on the near-end strategy optimization algorithm according to claim 3, characterized in that, Before inputting the state vector into the policy network, perform the following steps: Based on the order's timeliness requirements, geographical location, and the supplier's service capabilities and status, a set of candidate suppliers that meet the basic matching criteria is selected from all suppliers. The probability distribution generated by the policy network is only for suppliers in the candidate supplier set; The basic matching conditions include one or more of the following conditions: (a) The supplier is currently in a serviceable state; (b) The geographical distance between the supplier's current location and the order's destination is less than a first preset threshold; (c) The supplier’s current load is below a second preset threshold of its maximum service capacity; (d) The supplier possesses the qualifications required for the type of service specified in the order; (e) The supplier’s historical service rating is higher than the third preset threshold.

5. The order supply and demand matching method based on the near-end strategy optimization algorithm according to claim 4, characterized in that, In S4, a matching decision is generated based on the matching probability distribution and combined with an ε-greedy strategy to select a target supplier; The steps of sending an allocation instruction to the target supplier to execute the order allocation operation include: S41. Generate a random number ρ between 0 and 1; If ρ≤ε, then the exploration strategy is executed, and a supplier is randomly selected from the set of candidate suppliers; If ρ>ε, then the utilization strategy is executed, and the supplier with the highest probability value in the matching probability distribution is selected; S42. Send an order matching request to the supplier selected in S41 and receive its response; If the response is "accept", the matching operation is complete. If the response is timeout or rejection, the supplier is removed from the current candidate supplier set, and S41 is repeated until a match is found or the candidate supplier set is empty.

6. The order supply and demand matching method based on the near-end strategy optimization algorithm according to claim 5, characterized in that, The steps in S4 to record the interactive data of each allocation operation to the experience replay buffer include: Record the multidimensional state vector constructed before this allocation operation occurs; Record the action performed by S4 this time, that is, the unique identifier assigned to the supplier of the order; The receiving system calculates the immediate reward based on the result of this allocation operation using a preset multi-objective reward function R; whereby the multi-objective reward function R is: In the formula, This is a reward sub-item based on order response time and completion time; This is a reward sub-item based on the supplier's load balancing level; This is a reward sub-item based on user ratings and order cancellation rates; These are the weighting coefficients for each reward sub-item; After the allocation operation is completed, data is collected again after a preset time interval to construct a new multidimensional state vector, which is the next state vector after the action is performed. Store the quadruple data <state, action, reward, next state> as an experience tuple in the experience replay buffer.

7. The order supply and demand matching method based on the near-end strategy optimization algorithm according to claim 6, characterized in that, The steps for iteratively updating the parameters of the policy network and the value network include: The generalized advantage estimation algorithm is used to calculate the advantage value of actions; the pruning objective function is used to limit the range of change of the probability ratio between the old and new policies, so as to constrain the parameter update magnitude of the policy network; and the stochastic gradient descent algorithm is used to iteratively update the policy network and value network parameters of the PPO algorithm based on the constrained objective function.

8. The order supply and demand matching method based on the near-end strategy optimization algorithm according to claim 7, characterized in that, The advantage value of an action is calculated using generalized advantage estimation, and the policy is updated by constraining the pruning objective function. Specifically, this includes: Based on trajectory data sampled from the experience replay buffer, the generalized dominance estimation algorithm is used to calculate the dominance function value of the action at each time step. ; Based on the advantage function value, the pruning objective function of PPO is calculated to update the policy network parameters. The objective function expression is as follows: The probability ratio between the old and new strategies. For pruning hyperparameters; By minimizing the value network prediction value Compared with the actual return estimate The mean squared error loss function is used to update the value network parameters. ϕ The loss function expression is: Using the stochastic gradient descent algorithm, a joint loss function is generated by combining the pruning objective function and the loss function. This joint loss function is then used to update the parameters of the policy network and the value network in a single operation. The joint loss function is as follows: in, The weighting factor for the value loss item. The weighting coefficient for the entropy reward term. For strategy In state The entropy below is used to encourage exploration and prevent strategies from prematurely converging to local optima; It is the action space, which contains all possible actions. , It is in state Next, according to the strategy Select Action The probability of.

9. The order supply and demand matching method based on the near-end strategy optimization algorithm according to claim 8, characterized in that, The generalized dominance estimation algorithm calculates the dominance value using the following formula: In the formula, where, γ This is a discount factor used to weigh the importance of current and future returns; These are GAE parameters used to control the tradeoff between bias and variance; For the first The time difference error of the step.

10. An order supply and demand matching system based on a near-end strategy optimization algorithm, characterized in that, include: The data acquisition module is used to acquire order data and supplier data in real time. The order data includes order ID, geographical location, service type, timeliness requirements and priority. The supplier data includes supplier ID, current location, service capability, service status and historical service rating. The state construction module is used to process the acquired order data and supplier data to construct a multi-dimensional state vector that includes spatiotemporal features, supply and demand distribution features, historical matching features, and real-time state indicators. The strategy decision module, based on the multidimensional state vector, calculates the matching probability distribution between the order and each candidate supplier using the near-end strategy optimization (PPO) algorithm; wherein, the strategy network and value network of the PPO algorithm are pre-trained. The matching execution module is used to generate a matching decision based on the matching probability distribution and an ε-greedy strategy to select a target supplier; send an allocation instruction to the target supplier to execute the order allocation operation; and record the interactive data of each allocation operation to the experience replay buffer. The training optimization module is used to periodically or when the amount of data in the experience replay buffer reaches a preset threshold, randomly sample a batch of trajectory data from the buffer; and iteratively update the parameters of the policy network and the value network based on the sampled data.

Images