A crossing scheduling method and system based on reinforcement learning

Abstract

The application relates to the technical field of artificial intelligence, and discloses a road crossing scheduling method and system based on reinforcement learning, wherein the road crossing scheduling method based on reinforcement learning comprises the following steps: acquiring road crossing scheduling scene data, generating a scene complexity evaluation result; processing historical operation data of a dispatcher, generating a personnel capability and expertise portrait; acquiring reinforcement learning decision process data, generating decision explanation information; generating a man-machine task allocation scheme based on scene complexity and personnel expertise; processing reinforcement learning decision data, quantifying a decision uncertainty index; acquiring dispatcher feedback data, updating a reinforcement learning model; processing man-machine cooperation process data, optimizing a man-machine division strategy; the application can accurately identify a decision situation that is difficult for an AI system to determine, and timely introduces human intervention; by continuously integrating human professional knowledge, the decision quality and robustness of the system in a complex scene are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, more specifically, it relates to a crossing scheduling method and system based on reinforcement learning. BACKGROUND

[0002] With the rapid development of logistics industry and transportation system, crossing scheduling system is facing an increasingly complex operating environment and growing efficiency demand. The traditional crossing scheduling method mainly relies on manual experience for decision-making. In the face of complex scenarios such as a large number of vehicles requesting to pass at peak time, different types of goods needing differentiated processing, and frequent occurrence of unexpected situations, it is often difficult to make timely and optimal scheduling decisions, resulting in crossing congestion, long waiting time, and low resource utilization.

[0003] With the application of artificial intelligence technology, some crossing scheduling systems have begun to introduce automated decision support functions. However, the existing human-machine hybrid decision-making mode has three main technical problems: lack of clear task allocation mechanism, leading to unclear decision-making responsibilities, and lack of clear responsibility boundaries between the system and human dispatchers; the decision-making system based on artificial intelligence usually behaves as a "black box", lacking explainability, making it difficult for human dispatchers to understand and trust the decisions made by the system; unable to dynamically adjust human-machine division according to scene complexity and personnel expertise, leading to waste of human resources in simple scenarios and inability to fully utilize the experience and judgment ability of human experts in complex scenarios.

[0004] In addition, the existing crossing scheduling system performs poorly in handling highly uncertain scenarios, is difficult to effectively identify the boundaries of AI system decision-making ability, and cannot introduce human intervention in time; at the same time, it also lacks effective mechanisms to continuously learn and optimize from the professional knowledge and operation feedback of human dispatchers, making it difficult for the system performance to improve over time. These problems seriously affect the efficiency and reliability of the crossing scheduling system and limit the effectiveness of human-machine collaboration. SUMMARY

[0005] The present application provides a crossing scheduling method and system based on reinforcement learning, which solves the technical problems of unclear human-machine task allocation, lack of explainability of AI decision-making, inability to dynamically adjust human-machine division according to scene complexity, difficulty in identifying the boundaries of AI decision-making ability, and lack of continuous learning and optimization from human professional knowledge in related technologies.

[0006] The present application provides a crossing scheduling method based on reinforcement learning, comprising the following steps:

[0007] Obtain crossing scheduling scene data, generate scene complexity evaluation results through a scene complexity evaluation model;

[0008] The historical operation data of the dispatch personnel is processed to generate personnel portrait information through a personnel capability and expertise portrait system;

[0009] Based on the scene complexity evaluation result and the personnel portrait information, a man-machine task allocation scheme is generated through a man-machine task adaptive allocation mechanism;

[0010] According to the man-machine task allocation scheme, decision data of a reinforcement learning system is obtained, and decision explanation information is generated through an interpretable AI framework;

[0011] Based on the decision explanation information, low-confidence decisions are identified through a decision uncertainty quantification algorithm;

[0012] For low-confidence decisions, dispatch personnel feedback data is collected, and a reinforcement learning model is updated through a human feedback enhanced learning system;

[0013] Based on the updated model running result, man-machine collaboration process data is analyzed, and a man-machine collaboration efficiency evaluation and optimization system is used to optimize man-machine division strategy.

[0014] In a preferred embodiment, the scene complexity evaluation model further comprises:

[0015] A feature extraction module is used to extract a multi-dimensional scene feature vector from original intersection dispatch scene data;

[0016] A complexity calculation module calculates the scene complexity score through an activation function based on the output of the neural network;

[0017] An uncertainty evaluation module calculates the uncertainty index through an activation function based on the output of the neural network.

[0018] In a preferred embodiment, the personnel capability and expertise portrait system converts the historical operation records of the dispatch personnel into a multi-dimensional capability vector based on collaborative filtering and clustering analysis algorithms, wherein each dimension represents the dispatch personnel's score in a specific capability.

[0019] In a preferred embodiment, the working process of the collaborative filtering and clustering analysis algorithm includes:

[0020] Collecting the historical operation records of all dispatch personnel;

[0021] Calculating the similarity matrix between dispatch personnel;

[0022] Identifying similar dispatch personnel groups based on the similarity matrix;

[0023] Using the behavior patterns of similar dispatch personnel to infer the target dispatch personnel's scores in each capability dimension;

[0024] Using clustering algorithms to divide the dispatch personnel into different expertise groups;

[0025] Generating a final capability vector based on clustering results and personal history data.

[0026] In a preferred embodiment, the interpretable AI framework comprises:

[0027] A feature importance extraction module for calculating the contribution of each feature in the reinforcement learning model to the decision-making process;

[0028] An attention mechanism module for identifying key influencing factors in the decision-making process;

[0029] An explanation generation module for decomposing the state-action value function into a weighted combination of key features;

[0030] A text transformation module for converting numerical explanations into human-understandable language descriptions.

[0031] In a preferred embodiment, the human-machine task adaptive allocation mechanism comprises:

[0032] An input processing module for receiving and preprocessing scene complexity, uncertainty indicators, and dispatcher capability vectors;

[0033] A multi-objective function module for defining decision quality, response time, and collaboration efficiency, among other optimization objectives;

[0034] A task division module for generating task sets assigned to the AI system and human dispatchers;

[0035] A permission management module for assigning different levels of decision-making authority to the AI system and human dispatchers.

[0036] In a preferred embodiment, the decision uncertainty quantification algorithm is based on Bayesian uncertainty estimation, calculating the confidence interval of the reinforcement learning system's value estimate for actions in a state, and identifying low-confidence decisions by comparing with a preset threshold.

[0037] In a preferred embodiment, the human feedback enhanced learning system constructs a parameterized reward function based on human preference data, trains by minimizing human preference prediction error, and optimizes the reinforcement learning strategy using the reward function.

[0038] In a preferred embodiment, the human-machine collaboration efficiency evaluation and optimization system calculates a comprehensive efficiency score based on decision accuracy, average response time, system stability, and other indicators, and adjusts human-machine division parameters through gradient optimization methods to maximize collaboration efficiency.

[0039] In a preferred embodiment, a reinforcement learning-based crossing dispatching system for implementing a reinforcement learning-based crossing dispatching method, comprising:

[0040] a scene complexity evaluation module configured to obtain intersection scheduling scene data and generate a scene complexity evaluation result by using a scene complexity evaluation model;

[0041] a personnel capability and expertise profiling module configured to process historical operation data of dispatch personnel and generate personnel profile information by using a personnel capability and expertise profiling system;

[0042] a human-machine task adaptive allocation module configured to generate a human-machine task allocation scheme by using a human-machine task adaptive allocation mechanism based on the scene complexity evaluation result and the personnel profile information;

[0043] an interpretable AI module configured to obtain decision data of a reinforcement learning system according to the human-machine task allocation scheme and generate decision explanation information by using an interpretable AI framework;

[0044] a decision uncertainty quantification module configured to identify low-confidence decisions by using a decision uncertainty quantification algorithm based on the decision explanation information;

[0045] a human feedback enhanced learning module configured to collect dispatch personnel feedback data for the low-confidence decisions and update a reinforcement learning model by using a human feedback enhanced learning system;

[0046] a human-machine collaboration efficiency evaluation and optimization module configured to analyze human-machine collaboration process data based on the updated model running result and optimize a human-machine division strategy by using a human-machine collaboration efficiency evaluation and optimization system.

[0047] The present application has the following advantages:

[0048] The context-aware human-machine task allocation mechanism can dynamically adjust the decision boundary according to the scene complexity and personnel expertise, overcoming the limitations of traditional fixed duty division. In simple and highly regular scenarios, the system automatically increases the decision weight of the AI system, reducing the burden of manual scheduling; in complex and highly uncertain scenarios, the system appropriately increases the decision weight of the human dispatcher, fully utilizing the experience and judgment ability of human experts. This dynamic task allocation mechanism significantly improves the adaptability and resource utilization efficiency of the overall intersection scheduling system.

[0049] The interpretable AI framework solves the problem of lack of trust caused by traditional "black box" AI decision-making by providing decision reasons and reasoning processes. The system can present the decision logic in a way that is easy for humans to understand, including key influencing factors, weights of each factor, and their influence paths on the final decision, so that the dispatch personnel can quickly understand and evaluate the rationality of the system's suggestions. This transparent mechanism significantly improves the trust of human dispatchers in the system and reduces the friction cost of human-machine cooperation.

[0050] The decision uncertainty quantification algorithm can accurately identify the decision-making situation that is difficult for the AI system to determine, and timely introduce human intervention. The system calculates the confidence interval of different decision options, accurately locates the high uncertainty area in AI decision-making, avoids the safety risks and efficiency losses that may be caused by wrong decisions, reduces the invalid supervision burden of human dispatchers, and realizes the optimal allocation of human and machine resources.

[0051] The human feedback enhanced learning mechanism improves the decision-making quality and robustness of the system in complex scenarios by continuously integrating human professional knowledge. The system can learn from the operation correction and preference feedback of the dispatchers, gradually optimize the reinforcement learning model, and make the decision-making mode of the AI system closer to the level of human experts. At the same time, the human-machine cooperation efficiency evaluation and optimization mechanism continuously monitors and improves the human-machine division strategy, automatically identifies the bottleneck link in human-machine cooperation, and adjusts the task allocation parameters and cooperation process accordingly, forming a closed-loop optimization system to ensure the long-term stable growth of the crossing scheduling efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 is a flowchart of a crossing scheduling method based on reinforcement learning of the present application;

[0053] Figure 2 is a line graph of the complexity evaluation results of the logistics center crossing scene of the present application;

[0054] Figure 3 is a stacked column chart of the human-machine task allocation scheme based on scene complexity of the present application;

[0055] Figure 4 is a grouped bar chart of the implementation effect comparison of the explainable AI framework of the present application;

[0056] Figure 5 is a grouped column chart of the overall efficiency improvement effect of the system of the present application;

[0057] Figure 6 is a grouped bar chart of the human-machine cooperation quality improvement effect of the present application;

[0058] Figure 7 is a scatter plot of the relationship between decision uncertainty and decision accuracy of the present application. DETAILED DESCRIPTION

[0059] The subject matter described herein will now be discussed with reference to example implementations. It should be understood that discussions of these implementations are merely provided to enable those skilled in the art to better understand so as to best use the subject matter described herein, and variations of elements, functions, and arrangements of elements can be substituted for those illustrated and described herein without departing from the scope of the present specification. Various examples can omit, substitute, or add various procedures or components as appropriate, and the descriptions and representations of various examples are used merely to provide illustrative examples for discussion and should not be taken as limiting. Additionally, features described as taking place in a particular order may

[0060] A reinforcement learning-based intersection scheduling method is disclosed in at least one embodiment of the present invention, as shown in Figure 1 comprising the following steps:

[0061] Step 1, obtain intersection scheduling scene data, generate scene complexity evaluation results through a scene complexity evaluation model;

[0062] Real-time data of the intersection scheduling scene is obtained, including multi-dimensional data such as the number of passing objects, passing demand types, historical congestion situations, and weather conditions. These data are processed through the scene complexity evaluation model to output the complexity score and uncertainty level of the current scheduling scene. The scene complexity score is used to quantify the difficulty of the current intersection scheduling task, providing a basis for subsequent human-machine task allocation.

[0063] Before performing scene complexity evaluation, the original data needs to be preprocessed, including normalizing numerical features (such as the number of passing objects, waiting time) to scale them to the [0, 1] interval; converting categorical features (such as weather conditions, road conditions) into numerical representations through one-hot encoding; and periodically encoding time features (such as time of day, day of week) to ensure that the features are dimensionally consistent and suitable for neural network processing.

[0064] The scene complexity evaluation model includes the following components:

[0065] Feature extraction module: responsible for extracting a multi-dimensional scene feature vector from the original intersection scheduling scene data:

[0066] ;

[0067] wherein, represents a set of feature vectors containing multiple scene features; , , represents the , , th scene feature, such as the number of waiting vehicles, average waiting time, and specific feature values; represents the total number of extracted scene features.

[0068] Neural network processing module: composed of an input layer, multiple hidden layers and an output layer, using a multi-layer perceptron structure to process feature vectors;

[0069] Complexity calculation module: based on the output of the neural network, through a special activation function and weight matrix , calculate the scene complexity score ;

[0070] Uncertainty evaluation module: based on the output of the neural network, through a special activation function and weight matrix , calculate the uncertainty index .

[0071] where, represents the complexity score mapping function, which converts the vector output by the neural network through the sigmoid activation function into a complexity score between 0 and 1, and the specific expression is:

[0072] ;

[0073] where z represents the value of the neural network output after linear transformation; represents the base of the natural logarithm; represents the sigmoid function expression; represents the complexity score mapping function.

[0074] And represents the uncertainty index mapping function, which converts the neural network output into a non-negative uncertainty index through the softplus activation function, and the specific expression is:

[0075] ;

[0076] where, represents the neural network output value; represents the base of the natural logarithm; represents the natural logarithm function; represents the softplus function expression; represents the uncertainty index mapping function.

[0077] This function ensures that the uncertainty index is always positive.

[0078] Further, the scene complexity evaluation model also includes a time series processing module, which integrates time series analysis functions, and through analysis of historical scheduling data, periodic patterns and abnormal situations are identified, making the scene complexity evaluation results more in line with actual situations.

[0079] The working process of the multi-layer perceptron comprises the following steps:

[0080] receiving an input feature vector wherein represents a set of feature vectors containing multiple scene features;

[0081] For each hidden layer, the output of the layer is calculated according to the forward propagation rule, wherein the hidden layer represents an intermediate layer between the input layer and the output layer in the neural network.

[0082] weighted sum of inputs and wherein, represents a weight matrix of the complexity score; represents a bias term of the complexity score; represents a weight matrix of the uncertainty indicator; represents a bias term of the uncertainty indicator;

[0083] applying an activation function and obtaining a scene complexity score and an uncertainty indicator wherein, represents a complexity score mapping function; represents an uncertainty indicator mapping function; represents a scene complexity score; represents an uncertainty indicator.

[0084] As shown in Figure 2 , the changing trends of the scene complexity score and the uncertainty indicator at different time points in a day of the logistics center are shown. Through the chart, it can be intuitively seen that the scene complexity reaches the highest value at the morning and evening peak hours (9:00 and 17:00), and is lower at noon and at night. The uncertainty indicator is positively correlated with the scene complexity, but the fluctuation amplitude is smaller. These data provide an important basis for human-machine task allocation.

[0085] Step 2, process the historical operation data of the dispatcher, and generate personnel portrait information through the personnel ability and expertise portrait system;

[0086] The historical operation data of the dispatcher is analyzed, including decision types, decision quality, processing speed, etc. dimensions, and these data are processed through the personnel ability and expertise portrait system to output personnel portrait information representing the decision preferences and expertise fields of the dispatcher. The personnel portrait information contains the processing ability and expertise degree of the dispatcher in different types of dispatching tasks, providing a basis for subsequent task allocation.

[0087] Before processing historical operational data from dispatchers, decision quality scores are standardized to ensure consistency in scoring standards across different evaluators. Decision types are labeled and coded, converting them into numerical features. Simultaneously, time-related indicators such as processing speed undergo logarithmic transformation to reduce the impact of outliers. Furthermore, missing values ​​need to be appropriately imputed to ensure the integrity and consistency of historical data.

[0088] The personnel competency and expertise profiling system includes the following components:

[0089] Historical data processing module: Responsible for cleaning, standardizing, and structuring the historical operation records of scheduling personnel.

[0090] ;

[0091] in, Indicates the first A collection of historical operation records of each dispatcher; , , They represent the first The first dispatcher's , , One historical operation record; This indicates the total number of historical operation records.

[0092] Collaborative filtering module: Infers the capability characteristics of individual schedulers based on the behavioral patterns of similar schedulers;

[0093] Clustering analysis module: Groups dispatchers and identifies the behavioral characteristics of different dispatcher groups;

[0094] Ability Scoring Module: Generates Multidimensional Ability Vectors

[0095] ;

[0096] in, Indicates the first A capability vector for each scheduler; , , They represent the first , , The dispatcher in the first Scores across each capability dimension; Indicates the dispatcher's ID number; Indicates the number of the capability dimension; This represents the total number of capability dimensions.

[0097] Further, the personnel capability and expertise profiling system also includes an active learning module that periodically pushes different types of dispatch tasks to dispatch personnel to collect their performance data, making up for the possible lack of comprehensive coverage of capability dimensions in historical operation data.

[0098] The working process of collaborative filtering and clustering analysis algorithm includes the following steps:

[0099] Collecting historical operation records of all dispatch personnel;

[0100] Calculating the similarity matrix between dispatch personnel;

[0101] Identifying similar dispatch personnel groups based on the similarity matrix;

[0102] Using the behavior patterns of similar dispatch personnel to infer the target dispatch personnel's scores in each capability dimension;

[0103] Applying clustering algorithms (such as K-means) to divide dispatch personnel into different expertise groups;

[0104] Generating the final capability vector based on clustering results and individual historical data.

[0105] As shown in Figure 3 , the decision weight allocation between the AI system and human dispatchers under different scenario complexity is shown. It can be seen that as the scenario complexity decreases, the decision weight of the AI system gradually increases, while the decision weight of the human dispatcher gradually decreases. In high complexity scenarios, human dispatchers have 80% decision weight; while in low complexity scenarios, AI systems have 85% decision weight. This dynamic allocation mechanism effectively balances the efficiency of human-machine collaboration.

[0106] Step 3, based on the scenario complexity evaluation results and personnel portrait information, generate a human-machine task allocation scheme through a human-machine task adaptive allocation mechanism;

[0107] Obtain the decision data of the reinforcement learning system in the crossing dispatch process, including state features, selected actions, expected returns, etc. Process these data through an interpretable AI framework to output intuitive decision explanations and reasoning path information, including decision content, decision explanation, key feature contribution, and decision confidence, etc. Decision explanation information enables human dispatchers to understand the decision logic of the AI system, improving system transparency and credibility.

[0108] In the reinforcement learning decision-making process, state features need to be normalized to ensure that features of different dimensions (such as vehicle quantity and time indicators) can be compared fairly. Meanwhile, the discrete action space is one-hot encoded to make it suitable for neural network processing. For continuous values such as expected returns, the Min-Max normalization method is used to map them to a standard interval that is easy to understand.

[0109] wherein the interpretable AI framework comprises the following components:

[0110] Feature importance extraction module: responsible for calculating the contribution of each feature in the reinforcement learning model to the decision-making process;

[0111] Attention mechanism module: based on the state-action value function, identifies key influencing factors in the decision-making process;

[0112] Explanation generation module: decomposes the state-action value function into a weighted combination of key features:

[0113] ;

[0114] wherein, represents the value estimate of taking action in state ; represents the current system state; represents the available actions; represents the weight coefficient of the th feature; represents the influence function of the th feature; represents the total number of features; represents the weighted sum of all features;

[0115] Text conversion module: converts numerical explanations into human-understandable language descriptions.

[0116] wherein, is calculated as the difference between the feature value and the baseline value multiplied by the feature sensitivity coefficient, with the specific expression being:

[0117] ;

[0118] wherein, represents the influence function of the th feature; represents the current value of the th feature; represents the baseline reference value of the th feature; represents the a sensitivity coefficient of the feature, used to adjust the influence strength of the feature on the decision; a function representing the specific influence degree of the first feature on the action ; a current state of the system; a selectable action of the system.

[0119] Further, the interpretable AI framework also includes a natural language generation module that automatically adjusts the technical depth and detail level of the explanation according to the professional level of the dispatcher, improving the understandability of the decision explanation.

[0120] The explanation generation process based on the attention mechanism includes the following steps:

[0121] extracting the state-action value function in the reinforcement learning model ;

[0122] calculating the contribution of each feature to the state-action value ;

[0123] calculating the importance weight for each feature ;

[0124] selecting the most explanatory features according to the weight size;

[0125] generating explanation text in the form of "due to the value of feature A being x, the value of feature B being y, the system recommends performing action a".

[0126] As Figure 4 shown, the changes in various indicators before and after the implementation of the interpretable AI framework are shown. It can be seen that after the implementation, the understanding degree of the dispatcher for the AI decision is improved from 35% to 92%, the acceptance rate of the AI suggestion is improved from 42% to 88%, the proportion of AI decisions being manually vetoed is reduced from 38% to 12%, the trust score of the dispatcher for the AI system is improved from 5.2 to 8.7, and the frequency of decision conflicts is reduced from 8.5 times per day to 2.1 times. These data fully verify the effectiveness of the interpretable AI framework.

[0127] Step 4, according to the human-machine task allocation scheme, obtain the decision data of the reinforcement learning system, and generate decision explanation information through the interpretable AI framework;

[0128] The scene complexity evaluation result and the personnel capability portrait information are taken as inputs, the data are processed through a man-machine task adaptive allocation mechanism, and a man-machine task allocation scheme under a current dispatching scene is output. The allocation scheme clearly defines the decision-making range and authority of the AI system and the human dispatcher, including time period division, main responsible personnel assignment, AI system and human dispatcher authority allocation, and allocation basis, etc., and realizes dynamic task division based on scene and personnel characteristics.

[0129] In the task allocation process, due to the different dimensions and distribution characteristics of the scene complexity , the uncertainty index and the dispatching personnel capability vector , it is necessary to standardize them to be in the same numerical interval. For the dispatching personnel capability vector, dimension alignment processing is needed to ensure that its dimension characteristics match the task characteristics, so as to facilitate the calculation of the adaptation degree between the task and the personnel.

[0130] The man-machine task adaptive allocation mechanism includes the following components:

[0131] Input processing module: receiving and preprocessing scene complexity , uncertainty index and dispatching personnel capability vector ;

[0132] Multi-objective function module: defining multiple optimization objectives such as decision quality, response time and collaboration efficiency;

[0133] Task division module: generating a task allocation function:

[0134] ;

[0135] Wherein, represents the task allocation function; represents the scene complexity score; represents the uncertainty index; represents the dispatching personnel capability vector; represents the task set allocated to the AI system; represents the task set allocated to the human dispatcher; represents the decision-making authority level of the AI system; represents the decision-making authority level of the human dispatcher;

[0136] Authority management module: allocating different levels of decision-making authority to the AI system and the human dispatcher.

[0137] Wherein, the task allocation function is a composite decision function, which evaluates the scene complexity , uncertainty index and dispatcher capability vector to determine the optimal task allocation scheme.

[0138] The specific calculation method is as follows: first, construct the task difficulty matrix:

[0139] ;

[0140] Among them, denotes the task difficulty matrix; denotes the function of mapping scene complexity and uncertainty to specific task difficulty; denotes the scene complexity score; denotes the uncertainty index.

[0141] Then, combined with the dispatcher capability vector, calculate the matching degree matrix:

[0142] ;

[0143] Among them, denotes the matching degree matrix; denotes the function of calculating the matching degree of task difficulty and personnel capability; denotes the task difficulty matrix; denotes the dispatcher capability vector.

[0144] Then, based on the matching degree matrix, select the most suitable dispatcher, and determine the decision authority level of the AI system and the human dispatcher according to the scene complexity; finally, generate a complete task allocation scheme including time period division, main responsible person, authority allocation and allocation basis.

[0145] Further, the human-machine task adaptive allocation mechanism also includes a gradual authority transfer module, which allows dynamic adjustment of the decision authority of the AI system and the human dispatcher during the dispatch process according to the situation changes, rather than one-time determination of the unchangeable task allocation.

[0146] The working process of the multi-objective optimization algorithm includes the following steps:

[0147] Construct a multi-objective function, including dimensions such as decision quality, response time and coordination efficiency;

[0148] Assign weights to each objective function to form a weighted comprehensive optimization target;

[0149] Generate a candidate set of possible task allocation schemes;

[0150] Calculate the value of each objective function for each candidate scheme;

[0151] Based on the Pareto optimality principle, screen out non-dominated schemes;

[0152] The final allocation result is selected from the non-dominated schemes with the highest comprehensive score.

[0153] As Figure 5 shown, the comparison data of overall efficiency before and after the system implementation is shown. It can be seen that after the implementation, the average waiting time of vehicles is reduced from 45 minutes to 18 minutes, the daily average handling cargo throughput is increased from 8000 tons to 10500 tons, the number of dispatchers is reduced from 8 to 5, and the response time of emergency situations is shortened from 15 minutes to 4 minutes. These data show that the method significantly improves the operation efficiency and resource utilization of the logistics center.

[0154] Step 5, based on the decision explanation information, identify low confidence decisions through decision uncertainty quantification algorithm;

[0155] Analyzing the decision data generated by the reinforcement learning system, including the value estimates of each candidate action and their distribution, processing these data through the decision uncertainty quantification algorithm, and outputting the uncertainty index that identifies low confidence decisions. This uncertainty index is used to identify decision-making situations that are difficult for the AI system to determine, and timely introduce human dispatcher's judgment and intervention.

[0156] In the uncertainty quantification process, to ensure that the outputs of different value estimators can be effectively compared, all value function estimates are standardized by Z-score to convert them to standard normal distribution; at the same time, in view of the distribution characteristics difference of different state-action pairs, RobustScaling method is used to reduce the influence of outliers, to ensure the accuracy and stability of uncertainty evaluation.

[0157] The working process of the decision uncertainty quantification algorithm includes the following steps:

[0158] Collecting multiple value estimates of the reinforcement learning system for action in state ;

[0159] Calculating the statistical distribution characteristics of value estimates, including mean and standard deviation ;

[0160] According to the specified confidence level , determine the corresponding parameters ;

[0161] Calculate the uncertainty index:

[0162] ;

[0163] Where, represents the action taken in state a decision uncertainty indicator at the time t; a standard deviation of the state-action value function ; a critical value of the standard normal distribution at the confidence level ; denotes the current state of the system; denotes the system's selectable actions; denotes the confidence level parameter.

[0164] The uncertainty indicator is compared with a preset threshold to identify low-confidence decisions.

[0165] Further, the decision uncertainty quantification process also employs an ensemble learning method, including the following steps:

[0166] Train multiple value function estimators, each using a different subset of training data or structural parameters;

[0167] Collect the prediction results of all value function estimators for the same state-action pair;

[0168] Calculate the variance of these prediction results as another measure of uncertainty;

[0169] Weighted fusion of the standard deviation-based uncertainty indicator and the ensemble variance-based uncertainty indicator to form the final uncertainty assessment.

[0170] As shown in Figure 6 , the comparative data of human-machine collaboration quality before and after the system is implemented is shown. It can be seen that after the implementation, the job satisfaction of the dispatchers is improved from 5.5 to 8.6, the system suggestion adoption rate is improved from 45% to 88%, the number of human-machine decision conflicts is reduced from 8.5 times per day to 2.1 times, the human-machine collaboration satisfaction is improved from 4.8 to 8.5, and the average time of human-machine handover decision is reduced from 85 seconds to 25 seconds. These data show that the method effectively improves the quality and efficiency of human-machine collaboration.

[0171] Step 6, collect dispatcher feedback data for low-confidence decisions, and update the reinforcement learning model through the human feedback reinforcement learning system;

[0172] Collect the dispatcher's corrections, preferences, and evaluation data for the AI system's decisions, process these data through the human feedback reinforcement learning system, and output an updated reinforcement learning model that integrates human expertise. This reinforcement learning model combines the experience of human dispatchers and the computing power of AI systems, improving the system's decision-making quality in complex scenarios.

[0173] Before processing the dispatcher feedback data, the dispatcher's subjective preference needs to be standardized to reduce the influence of individual bias; the paired comparison data is converted by the Bradley-Terry model to express the relative preference as a probability form; at the same time, the qualitative feedback is converted into numerical features by text encoding, so that it can be input into the neural network together with quantitative data for processing.

[0174] Among them, the human feedback enhanced learning system includes the following components:

[0175] Feedback collection module: responsible for obtaining and preprocessing the decision modification, preference and evaluation data of dispatchers;

[0176] Reward model module: based on human feedback data, construct a parameterized reward function:

[0177] ;

[0178] Among them, represents the parameterized reward function; represents the set of learnable parameters of the reward function; represents the current state of the system; represents the action that the system can choose; represents the neural network function with as the parameter.

[0179] Policy optimization module: update the reinforcement learning policy according to the reward function:

[0180] ;

[0181] Among them, represents the updated new policy; represents the policy that maximizes the objective function among all possible policies ; represents the expected value; represents the state-action pair sampled according to the policy ; represents the reward function with parameter ; represents the current state of the system; represents the action that the system can choose.

[0182] Knowledge integration module: integrate human professional knowledge with the computational model of AI system.

[0183] Among them, is a parameterized function that maps state-action pairs to reward values, which is implemented as a multi-layer neural network, and the function is trained by minimizing the prediction error of human preference, and the specific form is:

[0184] ;

[0185] wherein, represents a reward function parameterized by ; represents a set of learnable parameters of the neural network, including weight matrices and bias terms; represents the current state vector of the system, containing various features of the crossing scheduling scenario; represents the action vector that the system can choose, containing specific scheduling decision options; represents a neural network model parameterized by , which is used to map the state-action pair to a reward value; represents the input vector obtained by concatenating the state vector and the action vector in the feature dimension, as the input of the neural network.

[0186] Further, the human feedback enhanced learning system also includes an active query module, which strategically selects the most informationally valuable decision request to schedule personnel to provide feedback, reducing the cost of obtaining human expert knowledge while maximizing the learning benefit of feedback.

[0187] The working process of the preference-based reinforcement learning method includes the following steps:

[0188] Collecting preference data of scheduling personnel on different decision choices of the AI system, usually in the form of pairwise comparison;

[0189] Constructing a dataset , wherein, represents the i-th state of the system; represents the i-th candidate action; represents the j-th candidate action; represents the probability that the scheduling personnel prefers action to in state ;

[0190] Based on these preference data, train the reward model ;

[0191] Using standard reinforcement learning methods (such as PPO, SAC, etc.), optimize the policy based on the reward model;

[0192] Integrate the optimized policy into the existing reinforcement learning system to update the decision model.

[0193] For example, Figure 7As shown, the relationship between the decision uncertainty indicator and the decision accuracy rate is shown. It can be seen that there is a clear negative correlation between the two: as the decision uncertainty indicator increases, the decision accuracy rate gradually decreases. When the uncertainty indicator is less than 0.2, the decision accuracy rate can be maintained above 80%; while when the uncertainty indicator exceeds 0.5, the decision accuracy rate rapidly decreases to below 50%. This relationship verifies the effectiveness of the decision uncertainty quantification algorithm, providing a scientific basis for timely introduction of human intervention.

[0194] Step 7, based on the updated model running results, analyze the human-machine collaboration process data, and optimize the human-machine division strategy through the human-machine collaboration efficiency evaluation and optimization system;

[0195] Record and analyze the whole process data of human-machine collaboration, including decision allocation, decision response time, decision accuracy rate, etc. Process these data through the human-machine collaboration efficiency evaluation and optimization system, and output the improved human-machine division strategy. The human-machine division strategy is continuously monitored and adjusted to continuously optimize the efficiency and quality of human-machine collaboration.

[0196] In the process of human-machine collaboration efficiency evaluation, since the decision accuracy rate (percentage), response time (time unit) and system stability (coefficient of variation) have different dimensions and numerical ranges, they need to be standardized and then combined with weights. The response time is transformed by reciprocal transformation, so that it is consistent with other indicators (all the better the better); At the same time, the weight of each index is normalized to ensure that the total weight is 1, so that the comprehensive efficiency score has a clear interpretation.

[0197] Among them, the human-machine collaboration efficiency evaluation and optimization system includes the following components:

[0198] Data monitoring module: continuously collect multi-dimensional performance data in human-machine collaboration process;

[0199] Multi-index evaluation module: based on decision accuracy rate , average response time and system stability , calculate the comprehensive efficiency score:

[0200] ;

[0201] Among them, represents the decision accuracy rate, i.e. the proportion of correct decisions to the total number of decisions; represents the average response time, i.e. the average time interval from receiving the dispatch request to making a decision; represents the reciprocal of the average response time, which is used to convert the time indicator to a positive indicator; represents the system stability, i.e. the degree of performance fluctuation of the system during continuous operation; represents the weight coefficient of decision accuracy rate; represents the weight coefficient of response time; represents the weight coefficient of system stability; represents the comprehensive efficiency score, used to measure the overall performance of the system;

[0202] Parameter optimization module: using gradient optimization and other methods, adjust the key parameters of human-machine division, maximize the collaborative efficiency;

[0203] Strategy updating module: apply the optimized division parameters to the actual scheduling system.

[0204] Further, the human-machine collaborative efficiency evaluation and optimization system also includes a meta-learning module, which can learn from past optimization experience, accelerate the optimization process in new scenarios, and improve the system's adaptability to environmental changes.

[0205] The working process of the gradient optimization method includes the following steps:

[0206] Initialize human-machine division parameters, such as the decision weight distribution ratio of AI system and human dispatcher;

[0207] Based on the current parameters, evaluate the collaborative efficiency indicators of the system ;

[0208] Calculate the gradient relative to each parameter;

[0209] Update the parameters along the gradient direction, so that increases;

[0210] Repeat the steps until the convergence condition is reached or the maximum number of iterations is reached;

[0211] Output the optimized human-machine division strategy parameters.

[0212] Application examples of this embodiment:

[0213] Application scenario description:

[0214] This application example is based on the implementation of a crossing scheduling system in a large e-commerce logistics center. The logistics center covers an area of about 150,000 square meters, has 48 loading and unloading crossing points, and handles an average of 8,000 tons of goods per day, with more than 120 vehicles entering and leaving every hour during peak periods. The logistics center has six main functional areas (receiving area, sorting area, storage area, picking area, packaging area, and shipping area), and the logistics transportation between areas is mainly completed by forklifts, AGV cars, and conveyor belt systems.

[0215] Before implementing the method, the logistics center faces the following problems:

[0216] Peak crossing congestion is serious, with an average waiting time of more than 45 minutes;

[0217] Manual scheduling decision-making is inefficient, scheduling personnel are under great pressure, and decision-making quality is unstable;

[0218] Simple AI scheduling systems have been tried, but lack of explainability, scheduling personnel lack trust in system decision-making, resulting in system recommendations often being ignored or overturned;

[0219] The division of human and machine responsibilities is not clear, causing decision-making conflicts and unclear responsibilities.

[0220] Core step implementation instance:

[0221] Step 1 instance: Obtain crossing scheduling scenario data and generate scenario complexity evaluation results;

[0222] In this application example, the logistics center has deployed a comprehensive data collection system, including 48 crossing cameras, vehicle recognition systems, RFID tag readers, and weather stations, etc. The raw data collected by the system in real time is shown in Table 1:

[0223] Table 1: Crossing scheduling scenario raw data example;

[0224]

[0225] The system preprocesses these raw data, including normalization, one-hot encoding, and periodic encoding, to generate standardized feature vectors.

[0226] For example, divide the number of passing objects by the upper limit of crossing capacity (50) to get the normalized value 0.64; convert the weather condition "light rain" to [0, 0, 1, 0, 0] (corresponding to [sunny, cloudy, light rain, heavy rain, snow]) through one-hot encoding; convert the current time point (14:30) to [sin(2π14.5 / 24), cos(2π14.5 / 24)] = [0.38, -0.92] through periodic encoding.

[0227] The processed feature vectors are input into the scenario complexity evaluation model, which consists of 3 layers of fully connected neural networks with 64 and 32 hidden layer neurons respectively, using ReLU activation function. The model output is processed through sigmoid activation function and softplus activation function to get scenario complexity score and uncertainty index respectively, and the scenario complexity evaluation result example is shown in Table 2:

[0228] Table 2: Scenario complexity evaluation result example;

[0229]

[0230] Step 3 Example: Obtain reinforcement learning decision-making process data and generate decision explanation information;

[0231] In this application example, the reinforcement learning system is based on a Double DQN architecture, the state space contains multiple dimensions of features such as crossing state, vehicle information and time information, and the action space includes vehicle allocation decisions (such as allocating a specific vehicle to a specific crossing) and scheduling priority decisions (such as adjusting the vehicle passing order). The system records decision-making process data in real time as shown in Table 3:

[0232] Table 3: Example of original data of reinforcement learning decision-making process (14:30 time point);

[0233]

[0234] The interpretable AI framework processes these decision-making data, first calculates the contribution of each feature to the Q value through the feature importance extraction module, then identifies the key influencing factors through the attention mechanism, and finally generates understandable decision explanations. An example of decision explanation information is shown in Table 4:

[0235] Table 4: Example of decision explanation information;

[0236]

[0237] These decision explanations are presented to the dispatchers through the system interface, allowing them to understand the decision-making logic of the AI system. Dispatchers can view more detailed explanations, including the specific influence path and numerical calculation process of each feature.

[0238] Step 4 Example: Generate human-machine task allocation scheme based on scenario complexity and personnel expertise;

[0239] In this application example, the system profiles the capabilities and expertise of the 5 dispatchers at the logistics center, and generates a dynamic human-machine task allocation scheme based on the scenario complexity evaluation results. An example of dispatcher capability and expertise profile is shown in Table 5:

[0240] Table 5: Example of dispatcher capability and expertise profile;

[0241]

[0242] The five dimensions of the capability vector represent: special cargo handling capability, peak period response capability, emergency handling capability, routine scheduling capability, and internal coordination capability. Based on the scenario complexity score, uncertainty index and dispatcher capability vector, the system generates a task allocation scheme through a human-machine task adaptive allocation mechanism. An example of human-machine task allocation scheme is shown in Table 6:

[0243] Table 6: Human-machine task allocation scheme example;

[0244]

[0245] In high complexity scenarios (such as 9:00-11:00 period), the system allocates most of the decision-making authority to dispatcher B with peak scheduling expertise and dispatcher C with emergency handling expertise, and the AI system only provides decision-making suggestions; while in low complexity scenarios (such as 19:00-22:00 period), the system transfers the complete decision-making authority to the AI system, only retaining the minimum supervisory intervention of dispatcher D, greatly reducing the artificial burden.

[0246] This dynamic task allocation scheme adapts to the complexity of the scene and the expertise of personnel, ensuring efficient use of decision-making resources. At the same time, the system also realizes gradual transfer of authority, allowing quick adjustment of decision-making authority allocation in emergency situations (such as sudden worsening weather conditions).

[0247] Technical effect verification:

[0248] After the application example is implemented in the logistics center, through system monitoring and data analysis, the following two key technical effects are verified:

[0249] Technical effect 1: Context-aware human-machine task allocation mechanism effect;

[0250] Before and after the implementation of this method, the human-machine resource utilization of the logistics center under different scene complexity is compared as shown in Table 7:

[0251] Table 7: Comparison of human-machine resource utilization efficiency data;

[0252]

[0253] The data shows that in high complexity scenarios, the system appropriately retains the decision-making authority of human dispatchers, but reduces the workload through more reasonable task allocation; in medium complexity scenarios, the system realizes more balanced human-machine collaboration; in low complexity scenarios, the system greatly reduces human intervention and releases human resources. At the same time, the decision-making accuracy and response time in each scenario have been significantly improved.

[0254] Technical effect 2: Explainable AI framework effect;

[0255] Before and after the implementation of this method, the understanding and acceptance of the AI system decision by the dispatchers are compared as shown in Table 8:

[0256] Table 8: Comparison of AI system decision explainability effect data;

[0257]

[0258] The data shows that, through the implementation of the interpretable AI framework, the understanding of the dispatchers on the AI system decision has been greatly improved from 35% to 92%, and the acceptance rate of the AI suggestion has also increased from 42% to 88%. At the same time, the proportion of AI decisions being manually vetoed has significantly decreased, and the trust and collaboration satisfaction scores of the dispatchers on the AI system have been greatly improved. The frequency of decision conflicts and the average time spent on human-machine handover decisions have also been significantly reduced, indicating that the improvement of system transparency effectively improves the human-machine collaboration efficiency.

[0259] The above describes the embodiments of the present application, but the embodiments are not limited to the specific implementation described above, which is only illustrative and not restrictive. Those skilled in the art can make more forms of equivalent embodiments under the inspiration of the embodiments, which are all within the protection of the embodiments.

Claims

1. A level crossing scheduling method based on reinforcement learning, characterized in that, Includes the following steps: Obtain level crossing scheduling scenario data, and generate scenario complexity assessment results using a scenario complexity assessment model; the scenario complexity assessment model also includes: The feature extraction module is used to extract multi-dimensional scene feature vectors from the original level crossing scheduling scene data; The complexity calculation module calculates the scene complexity score based on the output of the neural network and through the activation function. The uncertainty assessment module calculates uncertainty indices based on the output of a neural network and through activation functions. Process historical operation data of dispatchers and generate personnel profile information through a personnel ability and expertise profiling system; Based on the scenario complexity assessment results and personnel profile information, a human-machine task allocation scheme is generated through a human-machine task adaptive allocation mechanism; the human-machine task adaptive allocation mechanism includes: The input processing module is used to receive and preprocess scenario complexity, uncertainty indicators, and scheduler capability vectors; The multi-objective function module is used to define decision quality, response time, collaborative efficiency, and other optimization objectives. The task allocation module is used to generate task sets to be assigned to both the AI ​​system and human schedulers; The access control module is used to assign different levels of decision-making permissions to the AI ​​system and human dispatchers. Based on the human-machine task allocation scheme, decision data from the reinforcement learning system is acquired, and decision explanation information is generated using an interpretable AI framework. The interpretable AI framework includes: The feature importance extraction module is used to calculate the contribution of each feature in the reinforcement learning model to the decision. The attention mechanism module is used to identify key influencing factors in the decision-making process; The interpretation and generation module is used to decompose the state-action value function into a weighted combination of key features; The text conversion module is used to convert numerical explanations into human-understandable language descriptions; Based on decision explanation information, low-confidence decisions are identified through a decision uncertainty quantification algorithm; For low-confidence decisions, feedback data from dispatchers is collected, and the reinforcement learning model is updated through a reinforcement learning system based on human feedback. Based on the updated model's running results, the data of the human-machine collaboration process is analyzed, and the human-machine division of labor strategy is optimized through the human-machine collaboration efficiency evaluation and optimization system.

2. The level crossing scheduling method based on reinforcement learning according to claim 1, characterized in that, The personnel competence and expertise profiling system is based on collaborative filtering and clustering analysis algorithms. It converts the historical operation records of dispatchers into multi-dimensional competence vectors, where each dimension represents the dispatcher's score in a specific competence.

3. The level crossing scheduling method based on reinforcement learning according to claim 2, characterized in that, The working process of collaborative filtering and clustering analysis algorithms includes: Collect all historical operation records of dispatchers; Calculate the similarity matrix among the dispatchers; Identifying groups of similar dispatchers based on similarity matrices; Infer the target dispatcher's scores across various capability dimensions by leveraging the behavioral patterns of similar dispatchers; Clustering algorithms were used to divide dispatchers into different specialty groups; The final capability vector is generated based on the clustering results and individual historical data.

4. The level crossing scheduling method based on reinforcement learning according to claim 1, characterized in that, The decision uncertainty quantification algorithm is based on Bayesian uncertainty estimation. It calculates the confidence interval of the reinforcement learning system's value estimate of an action in a given state and identifies low-confidence decisions by comparing it with a preset threshold.

5. The level crossing scheduling method based on reinforcement learning according to claim 1, characterized in that, Human feedback reinforcement learning systems construct parameterized reward functions based on human preference data, train by minimizing human preference prediction errors, and use the reward function to optimize reinforcement learning strategies.

6. The level crossing scheduling method based on reinforcement learning according to claim 1, characterized in that, The human-machine collaboration efficiency evaluation and optimization system calculates a comprehensive efficiency score based on decision accuracy, average response time, system stability, and other indicators, and adjusts the human-machine division of labor parameters through gradient optimization methods to maximize collaboration efficiency.

7. A level crossing scheduling system based on reinforcement learning, used to execute the level crossing scheduling method based on reinforcement learning as described in any one of claims 1-6, characterized in that, include: The scenario complexity assessment module is used to acquire level crossing scheduling scenario data and generate scenario complexity assessment results through the scenario complexity assessment model. The personnel capability and expertise profiling module is used to process historical operation data of dispatch personnel and generate personnel profile information through the personnel capability and expertise profiling system. The human-machine task adaptive allocation module generates a human-machine task allocation scheme based on the scenario complexity assessment results and personnel profile information through a human-machine task adaptive allocation mechanism. The Explainable AI module is used to acquire decision data from the reinforcement learning system based on the human-machine task allocation scheme, and generate decision explanation information through the Explainable AI framework; The decision uncertainty quantification module identifies low-confidence decisions based on decision interpretation information and through a decision uncertainty quantification algorithm. The human feedback reinforcement learning module is used to collect feedback data from dispatchers for low-confidence decisions and update the reinforcement learning model through the human feedback reinforcement learning system. The Human-Machine Collaboration Efficiency Evaluation and Optimization Module is used to analyze human-machine collaboration process data based on the updated model operation results, and optimize human-machine division of labor strategies through the human-machine collaboration efficiency evaluation and optimization system.

Images