Charging robot scheduling method and storage medium

Abstract

The invention provides a charging robot scheduling method and a storage medium. The charging robot scheduling method aims at a plurality of charging requests in an intelligent parking lot, performs intelligent scheduling on a charging robot according to priorities of the charging requests and prediction of newly added charging requests, and comprises the following steps: constructing a priority potential field of a parking space-charging system based on the plurality of charging requests of a user; based on the constructed priority potential field of the parking space-charging system, modeling a multi-target tracking and priority intelligent scheduling process as a Markov decision process; based on the modeled Markov decision process, a reinforcement learning network based on the model is designed, and the reinforcement learning network comprises a dynamic network, a value-strategy network and a target network; based on the constructed reinforcement learning network, parallel training of a plurality of networks is realized by designing an interaction mode between a model of a dynamic network and a model of a strategy-value network and a collected data set.

Description

Technical Field

[0001] The present invention relates to a scheduling method for a multi-target single intelligent body, and in particular to a scheduling method and a storage medium for a charging robot. Background Art

[0002] As electric vehicles become more popular, the demand for charging infrastructure is also growing rapidly. Urban and transportation planners are increasing their investment in the construction of charging piles to meet the growing demand for electric vehicle charging. As the demand for charging equipment gradually increases, intelligent charging systems can reduce the total demand for equipment while improving charging efficiency, and popularize the charging methods of electric vehicles, so that more electric vehicle owners who do not have professional charging knowledge can achieve automatic charging with the help of intelligent systems. Therefore, the market has emerged to use mobile robots to respond to charging requests and move to designated parking spaces to operate charging equipment to complete the charging task of electric vehicles. However, this charging mode requires intelligent scheduling algorithms to determine the order in which charging requests are met when facing multiple charging requests.

[0003] The intelligent scheduling algorithms for traditional robots to respond to multiple goals mainly include first-come-first-served (scheduling in the order in which charging requests arrive), shortest job first (prioritizes scheduling of tasks with the shortest execution time), priority scheduling (allocating resources according to task priority, with high-priority tasks given priority), and preemptive scheduling (allowing higher-priority tasks to interrupt currently executing tasks to ensure that urgent tasks can be responded to in a timely manner). However, traditional scheduling usually relies on expert knowledge to design scheduling rules for different situations to achieve scheduling, and is unable to make correct responses to unknown and unconsidered situations.

[0004] Advanced heuristic optimization algorithms are also widely used in the field of intelligent scheduling, such as genetic algorithms (see non-patent literature 1) and particle swarm algorithms (see non-patent literature 2), which transform scheduling problems into optimization problems of designing fitness functions. However, these algorithms are prone to falling into local optimal solutions and are sensitive to algorithm parameters. In addition, the fitness function often only includes the waiting time of each current charging request, and there is no user-related rule design, which often causes dissatisfaction among some users.

[0005] With the development of artificial intelligence technology, artificial neural networks are gradually used for intelligent scheduling, among which the most typical one is the reinforcement learning algorithm. Model-free reinforcement learning algorithms learn the optimal strategy by interacting with the constructed environment to maximize rewards (see non-patent document 3). However, this model-free reinforcement learning algorithm has the problem of low sample efficiency caused by the time cost and computing power cost of interacting with the environment, which requires a long training time and computing power, and is prone to instability during the training process, with performance fluctuations or difficulty in converging to the optimal solution for a long time.

[0006] Prior art literature: Non-patent literature 1: Samsuria, Erlianasha, et al. "Adaptive fuzzy-genetic algorithm operators for solving mobile robot scheduling problem in job-shopFMS environment." Robotics and Autonomous Systems 176 (2024): 104683.

[0007] Non-patent literature 2: Zhang, Junqi, et al. "Moving-distance-minimized PSO formobile robot swarm." IEEE Transactions on Cybernetics 52.9 (2021): 9871-9881.

[0008] Non-patent document 3: Zhu, Aiyu, et al. "Deep reinforcement learning for real-time assembly planning in robot-based prefabricated construction." IEEE Transactions on Automation Science and Engineering 20.3 (2023): 1515-1526. Summary of the invention

[0009] The present invention aims to address the above-mentioned problems existing in the prior art and provides a charging robot scheduling method. When faced with multiple charging requests in a smart parking lot, the charging robot scheduling method schedules the charging robot to respond quickly based on an analysis of the priorities of these charging requests and a prediction of future additional charging requests.

[0010] The present invention mainly designs priority rules based on users who issue charging requests, combines a sequence-level adaptive probabilistic reasoning framework to predict new charging requests, and uses model-based reinforcement learning to establish a world state evolution and reward model to improve training efficiency and closely integrate the optimal strategy with environmental dynamics to enhance its generalization ability.

[0011] According to a first aspect of the present invention, a charging robot scheduling method is provided, which intelligently schedules the charging robot according to the priority of the charging requests and the prediction of the new charging requests for multiple charging requests in the intelligent parking lot, and the method comprises the following steps: a priority potential field construction step of the parking space-charging system, which constructs the priority potential field of the parking space-charging system based on the multiple charging requests of the user; a Markov decision process conversion step, which models the multi-target tracking and priority intelligent scheduling process as a Markov decision process based on the constructed priority potential field of the parking space-charging system; a reinforcement learning network construction step, which designs a model-based reinforcement learning network based on the modeled Markov decision process, wherein the reinforcement learning network includes a dynamic network, a value-strategy network and a target network; a reinforcement learning network training and testing step, which performs parallel training of multiple networks based on the constructed reinforcement learning network by designing the interaction between the model of the dynamic network and the model of the strategy-value network and the collected data set.

[0012] As a preferred scheme, according to the charging robot scheduling method of the present invention, the step of constructing the priority potential field of the parking space-charging system includes the step of establishing an undirected topological network and the step of designing the priority potential field of the vehicle node charging request based on the established undirected topological network.

[0013] As a preferred solution, according to the charging robot scheduling method of the present invention, in the step of establishing an undirected topological network, a parking space-charging system topological network is set Including existing Parking space vertex set and rail edge set ,in Indicates connected vertices and Bidirectional edges, parking space-charging system topology network .

[0014] As a preferred solution, according to the charging robot scheduling method of the present invention, in the step of designing the priority potential field of the vehicle node charging request, based on the current moment defined by using the parking space-charging system topology network The occupancy matrix of the parking system , defined according to the following formula (1) at position The preferred potential field strength at is: (1) The elements in formula (1) are expressed as follows: Vertex Priority Value : is the vertex exist The priority value of the charging request at the moment is determined by the time factor It is used to describe the time the vehicle waits for charging. Mean Matrix : represents the center point of the priority potential field: Covariance matrix : The user factor Depends on the user's rating and feedback, the vehicle factor Describe the vehicle's charging request. is the weight coefficient.

[0015] As a preferred solution, according to the charging robot scheduling method of the present invention, in the Markov decision process transformation step, according to the action space of the charging robot Describe the movement direction and speed of the charging robot, the state space at time k Transfer to the state space at time k+1 s k+1 The system state transfer equation is as follows: in They represent the vertex priority value of the parking space vertex, the occupancy matrix, the priority potential field strength of the charging robot’s location, and the state transition matrix of the charging robot’s energy consumption.

[0016] As a preferred scheme, according to the charging robot scheduling method of the present invention, the reinforcement learning network construction step includes the step of building a mechanism and data-driven simulation environment and the step of designing a model-based reinforcement learning network architecture based on the built simulation environment.

[0017] As a preferred solution, according to the charging robot scheduling method of the present invention, the step of building a simulation environment includes system state transfer and the design of a reward function, wherein the designed reward function is as follows: in, represents the potential field strength of the charging robot at the current time k, represents the reward value corresponding to the vertex priority value of the charging robot at the current time k, represents the reward value corresponding to the energy consumption of the charging robot at the current time k, , and is the weight coefficient.

[0018] As a preferred solution, according to the charging robot scheduling method of the present invention, The steps of designing a model-based reinforcement learning network architecture include the construction of a dynamic network model and the construction of a policy-value network and corresponding target network model.

[0019] As a preferred scheme, according to the charging robot scheduling method of the present invention, the training and testing steps of the reinforcement learning network include the following processes: self-game, data accumulation, joint training and parameter preservation, wherein self-game is performed through the existing strategy-value network model and the simulation environment to obtain multiple game data and store them, and during joint training, data is randomly extracted from the stored game data to train the parameters of the strategy-value network and the dynamic network model, and the parameters are updated in the target network.

[0020] According to a second aspect of the present invention, there is provided a non-temporary storage medium storing a computer program, which, when executed by a processor, can implement the charging robot scheduling method according to the first aspect of the present invention.

[0021] The beneficial effects of the present invention are: it can achieve long-sequence decomposition prediction of future charging requests based on the priority of each charging request, which can better capture the seasonality and trend of charging requests and achieve more robust scheduling. On this basis, the model-based reinforcement learning algorithm can improve the training efficiency and test effect of the overall scheduling strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 The overall flow chart of the charging robot scheduling method according to the present invention is illustrated.

[0023] Figure 2A A flowchart illustrating a sub-process of the step of constructing a priority potential field of a parking space-charging system in the charging robot scheduling method according to the present invention.

[0024] Figure 2B A schematic block diagram illustrating the construction of a priority potential field of a parking space-charging system in the charging robot scheduling method according to the present invention is illustrated.

[0025] Figure 3A A flowchart illustrating a sub-process of a Markov decision process transformation step in a charging robot scheduling method according to the present invention.

[0026] Figure 3B A schematic diagram illustrating the Markov process transformation of the charging robot scheduling method according to the present invention is shown.

[0027] Figure 4AA flowchart illustrating a sub-process of the step of designing a model-based reinforcement learning network in the charging robot scheduling method according to the present invention.

[0028] Figure 4B A schematic diagram of an adaptive probabilistic reasoning framework for long sequence prediction in a charging robot scheduling method according to the present invention is illustrated.

[0029] Figure 5 A block diagram illustrating the overall training process of the model-based reinforcement learning scheduling algorithm in the charging robot scheduling method according to the present invention is shown.

[0030] Figure 6 A schematic diagram illustrating a self-game through Monte Carlo tree search in a charging robot scheduling method according to the present invention is illustrated.

[0031] Figure 7 A schematic diagram illustrating the joint training of network parameters in the charging robot scheduling method according to the present invention is shown.

[0032] Figure 8 A flowchart illustrating a detailed process of the charging robot scheduling method according to the present invention. DETAILED DESCRIPTION

[0033] Exemplary embodiments of the present invention are described in detail below with reference to the accompanying drawings. It should be noted that the relative arrangement of components, numerical expressions and numerical values ​​described in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.

[0034] The charging robot scheduling method of the present invention can be implemented by a processor in the charging robot executing a computer program stored in a memory in the charging robot. As an alternative, the charging robot can also communicate with a server, and the processor in the server executes a computer program stored in the server or the cloud and feeds back the program execution results to the charging robot in real time.

[0035] In the following description of the present invention, the scheduling method of the present invention is used for scheduling a charging robot to quickly respond to multiple charging requests in a smart parking lot. In addition, the scheduling method of the present invention is not limited to this, and it can also be applied to situations where multiple requests (such as multiple cleaning requests or multiple purchase requests, etc.) in other environments other than smart parking lots need to be processed.

[0036] In addition, it should be understood that the parking lot mentioned in the present invention may refer not only to the entire parking lot, but also to each area when the parking lot is large and the entire parking lot is divided into multiple areas.

[0037] The core technology of the charging robot scheduling method of the present invention is to first establish an undirected topological graph of the parking space-charging system using the relevant theories of graph theory for the vehicle-charging system, then design a user-based priority calculation rule to describe the priority of each user's charging request, and then combine the established undirected topological graph of the parking space-charging system and the designed user priority rules to establish the priority potential field of the parking space-charging system. On this basis, the charging robot and priority intelligent scheduling process targeted by the present invention are converted into a Markov decision process, and the vertex state transition model and the topological structure transition model of the undirected topological graph are learned and inferred using a dynamic network model, wherein the latter uses an adaptive prediction probability framework based on sequence decomposition to predict new charging requests. Then, a "mechanism + data" driven simulation environment is established. The mechanism environment is established using the priority potential field derivative equation and the charging robot energy consumption equation. The data environment is established in combination with previous charging request data. The value-strategy network model is combined with the dynamic network model to interact with the simulation environment. The optimal strategy is searched for in the form of Monte Carlo tree search, and the corresponding value and reward are obtained. Data is accumulated in the form of self-game. Finally, the stored data is used to update the network parameters to improve the training efficiency and test effect of the overall scheduling strategy.

[0038] Below, refer to Figure 1 To illustrate the charging robot scheduling method according to the present invention, the method includes the following steps S100 to S400.

[0039] First, in step S100, a priority potential field of the parking space-charging system is constructed. Specifically, in step S100, based on multiple charging requests of electric vehicle users in the smart parking lot, a priority potential field of the parking space-charging system is constructed.

[0040] Next, in step S200, based on the priority potential field of the parking space-charging system constructed in step S100, the multi-target tracking and priority intelligent scheduling process is modeled as a Markov decision process.

[0041] Then, in step S300, based on the transformation of multi-target tracking and priority intelligent scheduling to Markov decision process created in step S200, a model-based reinforcement learning network (algorithm) is designed to ensure the optimal strategy and efficient training. Specifically, the reinforcement learning network includes a dynamic network that describes the vertex state and topological structure transfer function of the undirected topological network, a value-strategy network that explores the optimal scheduling strategy, and a target network for parameter update.

[0042] Finally, in step S400, based on the undirected topological network established in step S100 and the reinforcement learning network designed in step S300, the model of the dynamic network and the interaction between the model of the policy-value network and the collected data set are designed to achieve parallel training of multiple networks. Specifically, the dynamic network model is obtained by learning the prior data set, and the policy network is trained to maximize the reward and the value network is trained to maximize the expected cumulative discounted reward through interaction with the dynamic network model, and the parameters are soft-updated in the target network to verify in the test set and actual operation.

[0043] The charging robot scheduling method of the present invention performs intelligent scheduling for multiple charging requests in an intelligent parking lot according to the priorities of the charging requests and the prediction of new charging requests, so that the charging robot can respond quickly.

[0044] Below, refer to Figure 2A and Figure 2B right Figure 1 The process of constructing the parking space-charging system priority potential field in step S100 is described in detail. Figure 2A As shown, the process of constructing the parking space-charging system priority potential field specifically includes step S101 of establishing an undirected topological network and step S102 of designing the priority potential field of parking space vertex charging requests.

[0045] Specifically, refer to Figure 2B The construction schematic diagram shown is used to describe in detail the specific process of constructing the priority potential field of the parking space-charging system of the smart parking lot.

[0046] In step S101 of establishing an undirected topological network, a parking space-charging system topological diagram of the smart parking lot is set. , including existing Episodes and rail edge set ,in Indicates connected vertices and The bidirectional edge of the parking space-charging system topology (topological network) is recorded as Due to the characteristics of the charging robot rails, the topology graph is a bidirectional but non-connected graph, in which the parking space vertices can also be called parking space nodes, corresponding to the parking spaces in the physical diagram, and the edge values ​​are set as the rail distances between adjacent parking spaces.

[0047] Step S102 of designing a priority potential field for parking space vertex charging requests includes step S1021 of calculating vertex priority values ​​and step S1022 of calculating a covariance matrix based on users and vehicles.

[0048] Specifically, in step S102 of designing the priority potential field of the parking space vertex charging request, according to the parking space-charging system topology map established in step S101, first define the current time When the parking space system occupancy matrix : in Indicates the currently occupied parking space vertex, based on which the position is defined The preferred potential field strength at As shown in the following formula (1): (Formula 1) The description of each relevant element in formula (1) is as follows: (1.1) Vertex priority (priority value) : in Vertex exist The priority value of the charging request based on time. The larger the vertex priority value, the more urgent the charging request. It mainly describes the time the vehicle waits for charging: in Respectively represent the current world time and vertex The initial time when the parking space represented is occupied, Represents the parameters that control the characterization of the time factor.

[0049] (1.2) Mean matrix , which represents the center point of the priority potential field: (1.3) Covariance matrix , which is expressed as follows: The user factor and vehicle factor They are defined as follows: is the weight of two different factors to balance the impact of different priority values ​​on the overall priority value: (1.3.1) Vehicle factor :Describe the charging request for the vehicle. Specifically, after the vehicle is parked in the parking space, the camera on the rail identifies the vehicle type and obtains the battery capacity of the vehicle. , and then the vehicle factor is obtained by calculation : in It is a tuning parameter used to adjust the image of the vehicle factor function according to actual conditions.

[0050] (1.3.2) User factors : Depends on the user's rating and feedback to get: in is the maximum tolerance time of users calculated based on user feedback. The level of the user who is charging for the current request: in The current accumulated charging time of the user.

[0051] The above describes the construction process of the priority potential field of the parking space-charging system of the intelligent parking lot of the present invention. In addition, the reinforcement learning algorithm in the scheduling method for multiple targets of the charging robot of the present invention mainly relies on a Markov decision process (MDP). Specifically, based on the priority potential field of the parking space-charging system constructed in step S100, the multi-target tracking and priority intelligent scheduling process is modeled as a Markov decision process for the subsequent model-based reinforcement learning algorithm design.

[0052] Below, refer to Figure 3A and Figure 3B The flowchart of the Markov decision process transformation step S200 in the charging robot scheduling method according to the present invention is described in detail.

[0053] like Figure 3A As shown, the Markov decision process conversion step S200 includes a step S201 of transferring the priority potential field state based on the parking space-charging system priority potential field constructed in step S100 and a step S202 of establishing the system state transfer equation based on the charging robot energy consumption mechanism model.

[0054] like Figure 3B As shown in the figure, the Markov decision process transformation mainly includes core elements such as state space, action space, and reward function. The state space mainly contains the information of the undirected topological graph of the parking space-charging system, and the action space mainly contains the motion control instructions of the charging robot. The reward value is mainly obtained by the sum of the vertex priority values ​​in the current undirected graph structure and the energy consumption of the charging robot. The following is a detailed description of each core element.

[0055] First, the state space Mainly the information of the undirected topological graph of the parking space-charging system, including the vertex V, edge value E and occupancy matrix of the current undirected topological graph , Vertex priority values ​​in topology and the preferred potential field strength And the energy consumption of the current charging robot is specifically expressed as follows: in The power of the charging robot at the current moment.

[0056] Next, we describe the action space , which mainly includes the movement direction and speed of the charging robot: in They represent the current direction of the charging robot and its speed in that direction respectively.

[0057] Reward Function The sum of the vertex priority values ​​of the current undirected topology graph, the priority potential field strength of the charging robot, and the energy consumption are used to obtain: Furthermore, the specific setting of the reward function will be described in the reinforcement learning network design step S300 to be described below.

[0058] like Figure 3B As shown, the left side represents the state space at the current k moment , which consists of the vertex V, edge value E and occupancy matrix of the undirected topological graph , Vertex priority values ​​in topology and the preferred potential field strength And the energy consumption of current charging robots According to the action space of the charging robot Describe the movement direction and speed of the charging robot, state space Transfer to the state space at time k+1 s k+1 like Figure 3B Specifically, the system state transition equation is as follows: in They represent the vertex priority value of the parking space vertex, the occupancy matrix, the priority potential field strength of the charging robot's location, and the state transition matrix of the charging robot's energy consumption power. This is achieved through the priority potential field strength rule proposed in step S100 and the movement of the charging robot. When the charging robot moves to an occupied parking space, the priority value of the current parking space will be cleared, and the action space will be locked to fix the position of the charging robot to achieve the charging operation. The network model is obtained through the long sequence prediction adaptive inference framework described below. The relationship between the charging robot's motion actions and its energy consumption will be obtained through the learning described below.

[0059] The Markov decision process transformation described above will be used in the design of the model-based reinforcement learning algorithm described below. The multi-target tracking and priority intelligent scheduling problem is transformed into a reinforcement learning algorithm, and the scheduling strategy network that can minimize the priority of each vertex in the overall undirected graph network is learned.

[0060] Below, refer to Figure 4A and Figure 4B The model-based reinforcement learning network design step S300 in the charging robot scheduling method according to the present invention is described.

[0061] The model-based reinforcement learning scheduling algorithm design process of the present invention is to design a model-based reinforcement learning algorithm on the basis of the transformation design of the charging robot multi-target tracking and intelligent scheduling Markov decision process in step S200 to ensure the optimal strategy and efficient training.

[0062] like Figure 4A As shown, the model-based reinforcement learning network design step S300 of the present invention includes the following steps S301 to S302.

[0063] Step S301: Construction of a simulation environment driven by “mechanism + data”. The construction of the simulation environment mainly includes the system state transfer and the design of the reward function.

[0064] The system state transfer is displayed in an iterative form through the system state transfer equation established in step S202. It mainly includes three state transfer equations: the vertex priority of the undirected graph, the occupancy matrix, and the energy consumption power of the charging robot. The occupancy matrix cannot be described using a mechanism model, so it is established in a data-driven way. The overall state transfer equation consists of a "mechanism" state transfer model composed of the designed vertex priority state transfer and the energy consumption power state transfer of the charging robot, and a "data" state transfer model composed of the daily parking space occupancy situation in the current parking lot.

[0065] The design of the reward function is described below. Real-time reward feedback is as described in the reward function in step S200 above. The specific reward function is set as follows: in Represents the strength of the potential field formed by the vertex priority at the location of the charging robot at the current time k, so as to prompt the charging robot to solve the more urgent charging request response; Represents the reward value corresponding to the vertex priority value (vertex priority value) in the undirected topological graph at the current time k, to ensure that the charging request delay for each parking space is minimized; represents the reward value corresponding to the energy consumption power of the charging robot at the current time k; , and is the weight coefficient, which is used to measure the importance of these three reward values. The specific definitions of these parameters are shown in the following formula: Step S302: Design of model-based reinforcement learning network architecture. In order to closely integrate the obtained optimal strategy with the environment dynamic model, a model-based reinforcement learning network architecture is designed, which specifically includes the construction of a dynamic network model and a strategy-value network.

[0066] First, the construction of the dynamic network model is described. After the intelligent scheduling problem of the charging robot is converted into a reinforcement learning problem, the effective reward value data needs to be obtained by testing the strategy obtained by optimizing the policy network in the test set, which will lead to inefficient training results. In order to improve the training rate and closely combine the optimal strategy with the environmental dynamics, the present invention constructs a dynamic network model to accurately describe the state transition probability and the reward value obtained, so that the data obtained not only comes from the real data set, but also comes from a large amount of data jointly constructed by the dynamic network model and the policy model. Combined with the strategy of establishing the simulation environment in step S301, the dynamic network model is defined as: Among them, the network This can be achieved through a multi-layer perceptron. It is not a state transfer function of the Markov decision process, but a long sequence prediction adaptive probability reasoning framework is used to predict future new requests, enhancing the robustness of the model-based reinforcement learning scheduling algorithm to new requests. The overall framework diagram of the long sequence prediction adaptive probability reasoning framework and its processing process are shown in the figure. Figure 4B shown.

[0067] Refer to the following Figure 4BDescribe the details of the adaptive probabilistic reasoning framework for long-sequence prediction of the present invention. First, after analyzing the newly added charging request data, it is found that its long time series usually show long-term trends and short-term periodic fluctuations. Therefore, decomposing the time series can help the model effectively capture its internal complex time dynamics. Use a frequency-based method to decompose the time series, map the series to the frequency domain, and then use frequency masking to treat the high-frequency part as the seasonal component. , the low frequency part is used as the trend component : in represents the Fourier transform, represents the inverse Fourier transform, since Noise is likely to exist, so some filtering is required. In the present invention, the model parameters in the neural network are expressed as: in denote the seasonal encoder and trend encoder respectively, represents the seasonal trend global representation extractor, Represents a decoder.

[0068] like Figure 4B As shown, the long time series of the charging request dataset is decomposed into seasonal components. and trend component , respectively, using the seasonal encoder and Trend Encoder Seasonal Components and trend component Then, in cluster matching mode, the seasonal trend global representation extractor The coded sequence is divided into Meta-learning tasks. Based on the network parameters corresponding to the meta-learning tasks and , learning time parameters Seasonality and trend representation. Finally, in the decoder The seasonal and trend components are integrated and decoded to complete the final prediction. So far, the prediction of new charging requests has been completed.

[0069] The following describes the construction of the policy-value network. In order to better optimize the policy network parameters, the policy-value network and the corresponding target network model are established. The parameters in the two networks are trained using the stochastic gradient method, and the parameters in the target network are optimized using the soft update method. The established policy network is used to give the optimal policy according to the current system state: in Representing the current moment of the charging robot through the strategy network According to the current status Get the best action.

[0070] Value Network Used to estimate the cumulative discounted reward to be generated based on the current state: in Represents the expected value corresponding to the current state and action, Represents the neural network function that predicts the expected value. The policy network and the value network are optimized together in a policy iteration manner. The goal of the policy network is to maximize the expected cumulative discounted reward, while the goal of the value network is to make the predicted value consistent with the actual reward value. The loss function of the value network is: in The true value represents the expected value. The true value is obtained through the Markov characteristic. The loss gradient of the policy network is as follows: Policy Network pass n The exploration obtains data and stores it. For the policy network The actions taken are in the neural network function The state-action value obtained in the training is trained by gradient back propagation to update the model parameters of the policy network so that the policy network can be trained to approach the optimal.

[0071] The above describes the design process of the model-based reinforcement learning network. The following describes step S400 of training and testing the model-based reinforcement learning network.

[0072] Specifically, based on the reinforcement learning network constructed in step S300, the dynamic network model and the strategy-value network model established in step S302 are designed to interact with the collected data set to achieve parallel training of multiple networks. The training and testing step S400 of the model-based reinforcement learning network (scheduling algorithm) mainly includes four processes: self-game, data accumulation (experience replay), parameter training (joint training), and parameter preservation. Figure 5 The flowchart of the overall training process of the model-based reinforcement learning scheduling algorithm is shown.

[0073] like Figure 5As shown, a large amount of game data is obtained by playing self-games with the established strategy-value network model and the simulation environment. Then, the game data obtained through self-games is stored in the experience pool. During joint training, the collected game data (experience data) is randomly extracted from the experience pool to train the model parameters of the strategy-value network and the dynamic network, and the trained model parameters are saved, and the cycle is repeated. It mainly includes the following processes 1 to 3: Process 1- Dynamic network model pre-training: First, collect historical parking data of the parking lot and train The world state transition model represented by the network is pre-trained to accurately represent the priority potential field of the parking space-charging system undirected graph and the occupancy matrix state transition model caused by new charging requests.

[0074] Process 2-Generation of environmental interaction data: Since the charging robot can only move on the rails, the action space can be discretized into: in The action and direction selected by the charging robot in the discretized action space through the agent strategy Can be discretized into forward, backward, left and right turns, speed It can be discretized into several specific gears. The specific speed gear depends on the charging robot used, and the present invention does not impose specific restrictions. In order to obtain the optimal strategy, data is generated through the interaction between the self-game module and the real environment to provide data for subsequent model training. The present invention uses a model-based Monte Carlo tree search (MCTs) to explore and select the action space, and samples through the distribution of the number of visits to finally obtain a strategy for interaction with the environment. The dynamic network and the strategy-value network are used for search, and the self-game is performed through the Monte Carlo tree search to complete the data collection process. Figure 6 shown.

[0075] like Figure 6 As shown, each state When , the strategy probability distribution is obtained through Monte Carlo tree search , after sampling and selection, the action at the current moment is obtained , and get the current moment's reward and the state at the next moment , and complete the data collection in this cycle. The process of Monte Carlo tree search is to select the current action at each node according to the upper bound confidence algorithm after initializing the Monte Carlo tree, and expand to the next node through the dynamic network model combined with the current node and the selected action, and update the number of selections and reward values ​​of the next node, until the number of selections and reward values ​​are backtracked after expanding to the leaf node, to reconstruct and optimize the Monte Carlo tree until the maximum number of search steps is reached. The process of Monte Carlo tree search mainly includes the following sub-processes: MCTs subprocess 1-initialize MCTs: The main role of MCTs is to determine the most effective method, obtain timely rewards, and evaluate the predicted value, initialize the node , each edge stores the following information: in Indicates the number of visits to the current node. represents the optimal strategy and predicted value obtained by the current parameters of the strategy-value network, Reward value prediction and state transfer equation obtained for the current parameters of the dynamic network.

[0076] MCTs sub-process 2-strategy selection: For each node, the upper bound confidence algorithm is used to select the strategy with the highest score by combining the value-strategy network and the number of visits to the node. The specific formula is as follows: in Represents the number of times action a is selected in the s state node, Indicates the number of times all actions except a are selected at the s state node. The upper bound confidence algorithm is calculated by the first term in the formula This makes the search more inclined to action-state pairs with higher historical scores, and the latter term in the formula makes the search tend to select nodes that have not been explored yet.

[0077] in is a parameter used to adjust the balance between the efficiency and breadth of the upper confidence algorithm. If is larger, the search algorithm is more inclined to search for nodes with a high current success rate, focusing on search efficiency; on the contrary, if The larger the value, the more likely it is to choose unexplored nodes and focus on search breadth. The coordination between the two ensures the balance between the efficiency and breadth of the search tree. Therefore, the search tree is more inclined to find and explore high-value nodes that have not yet been discovered.

[0078] MCTs sub-process 3-node expansion: expand new state nodes according to the strategy selected in MCTs process 2 through the dynamic network model, and predict the reward of the next state until the leaf node of the search tree is reached Finally, the corresponding strategy and value are predicted by the strategy-value function as follows: MCTs subprocess 4-backward propagation: Through the current search trajectory, the predicted value is back-propagated to all nodes in the search path, increasing the number of visits and updating the average value of the node: in Indicates the number of times a node is visited. When the trajectory passes through the node, the number of visits increases by 1. For value estimation: in, represents the decay factor in the Markov process, Representative rewards, represents the expected value, τ represents the number of iterations, m is the last leaf node of the search tree, represents the total number of steps, and k is the current moment of mcts, represents the current number of steps.

[0079] Process 3-Joint training: Since the search time of MCTs is relatively long, in actual application, each network is trained to obtain parameters to represent the optimal strategy found. Since the state transition model in the dynamic network model has been pre-trained, the strategy network, value network, and reward network are mainly jointly trained. The overall joint training process is as follows: Figure 7 As shown, in actual application, the original observation state is collected at every time interval , and at the current moment, through the self-game process of interacting with the environment data in process 2 (environment interaction data generation), after searching and sampling for the optimal strategy using Monte Carlo tree search, perform actions in the environment , get the observation state at the next moment and timely rewards received , the above data constitute a set of moment data in the trajectory After a certain amount of data collection is completed, the joint training process is started. A track is randomly sampled from the many collected tracks, and the data of the first k moments are taken as the initial state. , Strategy Network and value network Based on the current input The existing network model parameters are used to calculate the corresponding timely rewards, strategic actions and expected values. The loss function is calculated using the timely rewards fed back by the simulator, the optimal strategy obtained by MCTs search and the corresponding expected values ​​in the collected data set. Then, the network model parameters are calculated and updated through back propagation. The above formula represents the overall loss function of joint training, c represents the regularization term of network training to ensure the sparsity of the network, Represent the loss functions of the dynamic network, policy network, and value network, respectively, which are expressed as minimizing the error between the predicted policy and the optimal policy searched by MCTs, the error between the predicted value and the target value of the value network, and minimizing the error between the predicted reward of the dynamic network model and the actual observed reward. will be the input of the policy-value network, so its gradient can be back-propagated to the dynamic network model for parameter update. Figure 8 shown.

[0080] like Figure 8 As shown in the block diagram on the right, the training and testing step S400 of the reinforcement learning scheduling algorithm of the present invention is performed on the basis of the reinforcement learning network constructed in step S300. First, the initial state of the self-game is selected in step S401. Then, in step S402, the optimal action is selected by Monte Carlo tree search. In step S403, the node of the Monte Carlo tree is expanded based on the selected optimal action. In step S404, it is determined whether the node is expanded to a leaf node. If it is determined in step S404 that it is not expanded to a leaf node (No in step S404), the processing returns to step S402, and if it is determined that it is expanded to a leaf node (Yes in step S404), the processing proceeds to step S405, and the corresponding strategy and value are predicted by the policy network until the end of the self-game. Next, in S406, the reward of this self-game exploration is traced back along the path and back-propagated back to all nodes in the search path. Then, in step S407, the data experience of this self-game is stored for data accumulation.

[0081] In step S408, it is determined whether the maximum batch of data accumulation has been reached, wherein the maximum batch of data accumulation may be set according to user requirements or actual conditions. If it is determined in S408 that the maximum batch has not been reached (No in step S408), the process returns to step S401, and if it is determined in S408 that the maximum batch has been reached (Yes in step S408), the process proceeds to step S409 to start the joint training process of parameter training.

[0082] Specifically, in step S409, the data experience of the self-game is retrieved, and the initial state of the experience is set in step S410. Next, in step S411, the strategy-value network is used to select actions to calculate the corresponding strategy actions and expected values ​​according to the existing network model parameters. In step S412, the dynamic network is used to generate rewards and perform state transitions, and the corresponding timely rewards are specifically calculated according to the existing network model parameters. Then, in step S413, the strategy actions, expected values, and timely rewards calculated in steps S411 and S412 are compared with the experience data in the data set to perform error calculation. In step S414, the calculation results in step S413 are back-propagated to the dynamic network model for parameter update, thereby performing parameter training. In step S415, it is determined whether the training of this batch is completed. If it is determined in step S415 that the training of this batch is not completed, the process returns to step S409 to continue the joint training process. If it is determined in step S415 that the training of this batch has been completed, the process returns to step S401 to start the self-game process again, and the cycle is repeated until a preset number of cycle steps is reached.

[0083] Reference above Figure 8 Describes the training and testing process of the reinforcement learning scheduling algorithm of the present invention. Figure 8 The other steps and processes in the above have been described in the previous section and will not be repeated here.

[0084] In the training and testing of the reinforcement learning scheduling algorithm of the present invention, based on the above-mentioned undirected graph construction and reinforcement learning network design, through interaction with the dynamic network, the policy network is trained to maximize the reward, and the value network is trained to maximize the expected cumulative discounted reward for parameters, and the parameters are soft-updated in the target network, and finally verified in the test set and actual operation, thereby improving the training efficiency and testing effect of the overall scheduling strategy.

[0085] In summary, the charging robot scheduling method of the present invention utilizes the formulated priority rules to construct the priority potential field of the parking space-charging system, combines deep learning to predict charging requests, and transforms the intelligent scheduling process into a Markov decision process. On this basis, a model-based reinforcement learning intelligent scheduling algorithm is designed, and the Monte Carlo tree search is used to train and test the optimal strategy. As a result, the present invention realizes the long-sequence decomposition and prediction of future charging requests based on the priority of each charging request, which can better capture the seasonality and trend of charging requests, schedule charging robots for rapid response, and achieve more robust scheduling. In addition, on this basis, the model-based reinforcement learning algorithm can improve the training efficiency and testing effect of the overall scheduling strategy.

[0086] [Other embodiments] The embodiments of the present invention may also be implemented by reading out and executing computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be more completely referred to as a "non-transitory computer-readable storage medium") to perform one or more functions in the above-mentioned embodiments, and / or a computer of a system or device including one or more circuits (e.g., an application-specific integrated circuit (ASIC)) for performing one or more functions in the above-mentioned embodiments, and the embodiments of the present invention may be implemented by, for example, reading out and executing the computer executable instructions from the storage medium by the computer of the system or device to perform one or more functions in the above-mentioned embodiments, and / or controlling the one or more circuits to perform one or more functions in the above-mentioned embodiments. The computer may include one or more processors (e.g., a central processing unit (CPU), a microprocessing unit (MPU)), and may include a network of separate computers or separate processors to read out and execute the computer executable instructions. The computer executable instructions may be provided to the computer, for example, from a network or the storage medium. The storage medium may include, for example, one or more of a hard disk, a random access memory (RAM), a read-only memory (ROM), a memory of a distributed computing system, an optical disk (such as a compact disc (CD), a digital versatile disc (DVD), or a Blu-ray Disc (BD)™), a flash memory device, a memory card, and the like.

[0087] Although the present invention has been described above with reference to exemplary embodiments, the above embodiments are only for illustrating the technical concept and features of the present invention and cannot be used to limit the protection scope of the present invention. Any equivalent variations or modifications made according to the spirit and essence of the present invention should be included in the protection scope of the present invention.

Claims

1. A charging robot scheduling method, characterized in that: For multiple charging requests in a smart parking lot, a charging robot is intelligently scheduled according to the priority of the charging requests and the prediction of new charging requests. The method includes the following steps: A priority potential field construction step (S100) of a parking space-charging system is to construct a priority potential field of the parking space-charging system based on multiple charging requests of the user; A Markov decision process conversion step (S200) is to model the multi-target tracking and priority intelligent scheduling process as a Markov decision process based on the priority potential field of the parking space-charging system constructed in the priority potential field construction step (S100) of the parking space-charging system; A reinforcement learning network construction step (S300) is to design a model-based reinforcement learning network based on the Markov decision process modeled in the Markov decision process transformation step (S200), wherein the reinforcement learning network includes a dynamic network, a value-strategy network, and a target network; The training and testing step (S400) of the reinforcement learning network is based on the reinforcement learning network constructed in the reinforcement learning network construction step (S300), and multiple networks are trained in parallel by designing the interaction between the dynamic network model and the policy-value network model and the collected data set.

2. The charging robot scheduling method according to claim 1, wherein: The priority potential field construction step (S100) of the parking space-charging system includes the step of establishing an undirected topological network (S101) and the step of designing a priority potential field of vehicle node charging requests based on the established undirected topological network (S102).

3. The charging robot scheduling method according to claim 2, wherein: In the step of establishing an undirected topological network, set the parking space-charging system topological network Including existing Parking space vertex set and rail edge set ,in Indicates connected vertices and Bidirectional edges, parking space-charging system topology network .

4. The charging robot scheduling method according to claim 3, wherein: In the step of designing a priority potential field for vehicle node charging requests, based on the current moment defined by using the parking space-charging system topology network The occupancy matrix of the parking system , according to the following formula (1) defined at position The preferred potential field strength at is: （1） The elements in formula (1) are expressed as follows: Vertex Priority Value : is the vertex exist The priority value of the charging request at the moment is determined by the time factor It is used to describe the time the vehicle waits for charging. Mean Matrix : represents the center point of the priority potential field: Covariance matrix : The user factor Depends on the user's rating and feedback, the vehicle factor Describe the vehicle's charging request. is the weight coefficient.

5. The charging robot scheduling method according to claim 1, wherein: In the Markov decision process transformation step (S200), according to the action space of the charging robot Describe the movement direction and speed of the charging robot, the state space at time k Transfer to the state space at time k+1 s k+1 The system state transfer equation is as follows: in They represent the vertex priority value of the parking space vertex, the occupancy matrix, the priority potential field strength of the charging robot’s location, and the state transition matrix of the charging robot’s energy consumption.

6. The charging robot scheduling method according to claim 1, wherein: The reinforcement learning network construction step (S300) includes the step of constructing a mechanism and data-driven simulation environment (S301) and the step of designing a model-based reinforcement learning network architecture based on the constructed simulation environment (S302).

7. The charging robot scheduling method according to claim 6, wherein: The step of building the simulation environment (S301) includes the design of system state transfer and reward function, wherein the designed reward function is as follows: in, represents the potential field strength of the charging robot at the current time k, represents the reward value corresponding to the vertex priority value of the charging robot at the current time k, represents the reward value corresponding to the energy consumption of the charging robot at the current time k, , and is the weight coefficient.

8. The charging robot scheduling method according to claim 1, wherein: The step (S302) of designing a model-based reinforcement learning network architecture includes constructing a dynamic network model and constructing a strategy-value network and a corresponding target network model.

9. The charging robot scheduling method according to claim 1, wherein: The training and testing steps (S400) of the reinforcement learning network include the following processes: self-play, data accumulation, joint training and parameter preservation, Among them, multiple game data are obtained and stored by playing self-games with the existing strategy-value network model and the simulation environment. During joint training, data are randomly extracted from the stored game data to train the parameters of the strategy-value network and the dynamic network model, and the parameters are updated in the target network.

10. A non-temporary storage medium storing a computer program, which, when executed by a processor, can implement the charging robot scheduling method according to any one of claims 1 to 9.

Images