Multi-vehicle cooperative scheduling method and system

Abstract

The invention discloses a multi-vehicle cooperative scheduling method and system, and belongs to the field of multi-vehicle cooperative construction scheduling. The method comprises the steps of obtaining a current vehicle running state and a mean field, wherein the vehicle running state comprises state information and environment information of a vehicle and all vehicles in a sensing field of the vehicle; the mean field is a mean value of actions adopted by the vehicle and all vehicles in the sensing field of the vehicle at the last moment; and inputting the current vehicle running state and the mean field into a pre-trained DDPG-MFG model to obtain the current vehicle action. According to the method, the DDPG-MFG model is adopted to determine the vehicle action, the MFG is introduced into the DDPG framework, the average behavior of all agents is considered, global behavior perception is improved, efficient coordination control of multiple vehicles is achieved, and the efficiency and safety of an intelligent traffic system are improved.

Description

Technical Field

[0001] The present invention relates to the field of multi-vehicle collaborative construction scheduling, and in particular, to a multi-vehicle collaborative scheduling method and system based on reinforcement learning. Background Art

[0002] As an advanced machine learning technology capable of handling high-dimensional states and continuous action spaces, deep reinforcement learning (DRL) has been widely applied to the research of intelligent transportation systems. DRL learns the optimal policy through interaction with the environment, can adaptively optimize the control behavior of vehicles, and improve the overall efficiency and safety of the transportation system. Among them, the deep deterministic policy gradient (DDPG) algorithm, as an advanced reinforcement learning algorithm suitable for continuous action spaces, combines a policy network (Actor) and a value network (Critic), and performs well in handling continuous control tasks such as vehicle acceleration.

[0003] The prior art has optimized the autonomous planning and scheduling of single agents from aspects such as operability and efficiency. For example, the Chinese patent with publication number CN114355980A proposes a method for autonomous navigation of unmanned aerial vehicles using DDPG. It uses the DDPG network model for autonomous navigation of quadrotor unmanned aerial vehicles. By inputting the pose state of the unmanned aerial vehicle, the ESDF map matrix, and the discrete path point vector into the feature extraction matrix, the extracted features are concatenated and input into the DDPG network to obtain the action a, and the action a is input to the control module to be converted into the actual operation of the unmanned aerial vehicle and executed. However, this patent only focuses on the autonomous navigation of unmanned aerial vehicles, lacks the collaborative ability with other agents, and only relies on the sensing data of a single agent, with limited environmental perception and no decision-making ability in multi-agent scenarios.

[0004] The prior art has proposed a new design of a single-agent obstacle avoidance algorithm model based on deep reinforcement learning (Y.Xiang, J. Wen, W. Luo and G. Xie, "Research on Collision-Free Control and Simulation of Single-Agent Based on An Improved DDPG Algorithm," 2020 35th Youth Academic Annual Conference of Chinese Association of Automation (YAC), Zhanjiang, China, 2020, pp. 552-556). Aiming at the many deficiencies of the traditional DDPG algorithm, such as a large number of training rounds and the need for a lot of learning time, an improved deep deterministic policy gradient algorithm is used to improve the efficiency of obtaining the obstacle avoidance path. However, although this technology optimizes the training efficiency, it is still limited to the single-agent scenario and only solves the specific problem of obstacle avoidance, without considering the multi-agent collaborative decision-making requirements. Summary of the Invention

[0005] The purpose of the present invention is to provide a multi-vehicle collaborative scheduling method and system to solve the technical bottleneck that traditional single-agent reinforcement learning algorithms cannot effectively handle group collaboration in multi-vehicle scenarios.

[0006] To achieve the above purpose, the present invention adopts the following technical solutions: In the first aspect, the present invention provides a multi-vehicle collaborative scheduling method, including: Obtain the current vehicle operation state and the mean field, where the vehicle operation state includes the state information and environmental information of the vehicle itself and all vehicles within its perception area; the mean field is the average value of the actions taken by the vehicle itself and all vehicles within its perception area at the previous moment; Input the current vehicle operation state and the mean field into a pre-trained DDPG-MFG model to obtain the current vehicle action.

[0007] Further, the DDPG-MFG model includes a policy network, a target policy network, a value network, and a target value network; the DDPG-MFG model is obtained according to the following steps: Take the vehicle operation state and the mean field as the input of the policy network, and the output is the vehicle action ; take the vehicle operation state the mean field and the vehicle action As the input of the value network, the output is ; Initialize the network parameters of the policy network, target policy network, value network and target value network, and construct and initialize the experience replay pool to store training samples; At each time step, the policy network is based on the vehicle running state and the mean field to output the vehicle action , and the agent selects an action according to the policy output by the policy network, executes the action and interacts with the environment in real time to obtain the immediate reward and the new state , and update the mean field according to the actions of all vehicles to ; Store the experience tuple generated by the interaction into the experience replay pool; Randomly sample a small batch of data from the experience replay pool, use the gradient descent method and the Adam optimizer to adaptively adjust the learning rate, and iteratively update the network weights according to the loss function.

[0008] Furthermore, the vehicle running state is defined as: ; where is the running state of vehicle i at the current moment, is the position of vehicle i at the current moment; is the speed of vehicle i at the current moment; is the acceleration of vehicle i at the current moment; is the road where vehicle i is located at the current moment; represents the road fork that vehicle i is driving towards at the current moment; represents the set of the position, speed and road information of the neighboring vehicle j, represents the perception neighborhood of vehicle i.

[0009] Furthermore, the mean field is calculated according to the following formula: ; where, is the mean field at the current moment, is the action taken by the vehicle at the previous moment, and N is the total number of all vehicles including itself and those within its perception neighborhood.

[0010] Furthermore, the action space of the vehicle action is defined as: ; where, and respectively represent the minimum and maximum limits of the acceleration, Represents the acceleration / braking output ratio of vehicle i, Indicates that vehicle i accelerates proportionally, Indicates that vehicle i brakes proportionally, Indicates that vehicle i coasts.

[0011] Furthermore, the reward is determined according to a reward function, and the reward function is: ; where, is the current reward of vehicle i, is the speed of vehicle i at the current moment, is the predetermined target speed of the vehicle, represents the perceived neighborhood of vehicle i at the current moment, is the speed deviation weight coefficient, is the collision risk coefficient, is used to determine whether there is a collision risk between the current vehicle i and other vehicles j within its perceived neighborhood, The specific setting of is as follows: ; ; ; ; where represents the road intersection that vehicle i is approaching at the current moment, represents the road intersection that vehicle j is approaching at the current moment, represents the vehicle collision warning distance, represents the longitude and latitude coordinates of vehicle i at the current moment, represents the longitude and latitude coordinates of vehicle j at the current moment, is the actual distance between the longitude and latitude points of vehicle i and vehicle j calculated at the current moment, m is an auxiliary calculation variable used to calculate the spherical distance between two longitude and latitude points, C is the calculated spherical angular distance, and R is the radius of the earth.

[0012] Furthermore, the policy network includes an input layer, 2 hidden layers, and an output layer; the number of neurons in each hidden layer is set to 256, the activation function uses the ReLU function, and batch processing and L2 regularization are used; the output layer uses the Tanh activation function; the target policy network adopts the same network architecture as the policy network.

[0013] Furthermore, the value network includes an input layer, 2 hidden layers, and an output layer; the number of neurons in each hidden layer is set to 256, and the activation function uses the ReLU function; the target value network adopts the same network architecture as the value network.

[0014] Further, the V2X communication technology is adopted to perform real-time interaction with surrounding vehicles and / or infrastructure, and the status information of surrounding vehicles, environmental information, and actions taken by surrounding vehicles at the previous moment are collected.

[0015] In a second aspect, the present invention provides a multi-vehicle cooperative scheduling system, including: A data acquisition module, configured to acquire the current vehicle operation state and the mean field, where the vehicle operation state includes the status information and environmental information of the vehicle itself and all vehicles within its perception field; the mean field is the average value of the actions taken by the vehicle itself and all vehicles within its perception field at the previous moment; An action determination module, configured to input the current vehicle operation state and the mean field into a pre-trained DDPG-MFG model to obtain the current vehicle action.

[0016] Further, the data acquisition module includes: An acquisition module, configured to acquire the status information and environmental information of the current vehicle itself, and perform real-time interaction with surrounding vehicles and / or infrastructure to acquire the status information, environmental information of surrounding vehicles, and actions taken by surrounding vehicles at the previous moment; A mean field calculation module, configured to calculate the average value of the actions taken by the vehicle itself and its surrounding vehicles at the previous moment to obtain the mean field.

[0017] Further, the DDPG-MFG model includes a policy network, a target policy network, a value network, and a target value network; the system further includes a model training module, configured to perform the following steps: Taking the vehicle operation state and the mean field as the input of the policy network, and the output is the vehicle action ; taking the vehicle operation state the mean field and the vehicle action as the input of the value network, and the output is ; Initializing the network parameters of the policy network, the target policy network, the value network, and the target value network, and constructing and initializing an experience replay pool to store training samples; At each time step, the policy network outputs the vehicle action based on the vehicle operation state and the mean field , the agent selects an action according to the policy output by the policy network, executes the action and interacts with the environment in real time to obtain an immediate reward and a new state , and updates the mean field to according to the actions of all vehicles; Store the experience tuples generated by the interaction into the experience replay pool; Randomly sample a small batch of data from the experience replay pool, adaptively adjust the learning rate using the gradient descent method and the Adam optimizer, and iteratively update the network weights according to the loss function.

[0018] In a third aspect, the present invention provides an electronic device, including one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for executing any of the methods according to the first aspect.

[0019] In a fourth aspect, the present invention provides a readable storage medium, on which one or more programs are stored, and the one or more programs include instructions that, when executed by a computing device, cause the computing device to execute the instructions of any of the methods according to the first aspect.

[0020] Compared with the prior art, the beneficial technical effects achieved by the present invention are:

[0021] The present invention proposes a DDPG-MFG hybrid architecture. By combining the deep deterministic policy gradient algorithm with the mean field game theory, a collaborative decision-making mechanism for multi-vehicle systems is realized. Thus, while ensuring the optimal decision-making of each individual, the operating efficiency of the overall system is effectively improved, and efficient, coordinated, and safe policy control can be achieved in a multi-agent collaborative scheduling system. The present invention also introduces a distributed data sharing mechanism based on V2X, realizing real-time information interaction between vehicles and between vehicles and infrastructure, significantly expanding the sensing range of the system, and providing comprehensive and real-time scenario information support for multi-vehicle collaborative decision-making. The real-time communication ability provided by V2X greatly enhances the environmental perception and information sharing of agents, and enhances the diversity of experience replay. The mean field game (MFG) provides stable global feedback, enabling the model to converge quickly and perform well in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a framework diagram of a multi-vehicle collaborative scheduling method of the present invention;

[0023] Figure 2 is a neural network structure diagram of the policy network adopted in the embodiment of the present invention;

[0024] Figure 3 is a neural network structure diagram of the value network adopted in the embodiment of the present invention;

[0025] Figure 4Structural diagram of a multi-vehicle collaborative scheduling system according to the present invention;

[0026] Figure 5 Structural framework diagram of the simulation system of the present invention. Specific implementation manners

[0027] The present invention will be further described below in conjunction with specific embodiments. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0028] In conjunction with Figure 1 As shown, the present invention provides a multi-vehicle collaborative scheduling method, including: Step 1, obtaining the current vehicle running state and the mean field; The current vehicle running state includes the state information and environmental information of the vehicle itself and all vehicles within its sensing area; the current mean field is the average value of the actions taken by the vehicle itself and all vehicles within its sensing area at the previous moment.

[0029] The vehicle running state includes vehicle data and road data. The vehicle data includes the speed, position, and road where the vehicle itself is located in each time step, and also includes information about surrounding vehicles and infrastructure, including the position, speed, acceleration, and road information of surrounding vehicles. The road data includes global road information and road intersection information.

[0030] In the embodiment of the present invention, the V2X communication technology is adopted to interact with surrounding vehicles and / or infrastructure in real time, collect the state information and environmental information of surrounding vehicles, and realize information sharing and state transfer between vehicles.

[0031] By real-time transmitting the state information and environmental information of the vehicle through the V2X technology, the sensing ability and decision-making ability of the vehicle are effectively enhanced, providing rich real-time information support for multi-vehicle collaborative scheduling, and enabling the intelligent agent to make decisions based on more comprehensive environmental data.

[0032] Step 2, inputting the current vehicle running state and the mean field into a pre-trained DDPG-MFG model to obtain the current vehicle action; The construction process of the DDPG-MFG model is as follows: (1) Defining the vehicle state space: For the vehicle cluster running in the entire area, the running state of each vehicle is defined as: ; where is the position of vehicle i at the current moment; is the speed of vehicle i at the current moment; is the acceleration of vehicle i at the current moment; is the road where vehicle i is located at the current moment; represents the road intersection that vehicle i is driving towards at the current moment; represents the set of information about the position, speed, and road where the neighboring vehicle j is located, represents the perception neighborhood of vehicle i.

[0033] (2) Define the vehicle action space: The action space of the vehicle is defined as follows: ; Among them, and respectively represent the minimum and maximum limits of the acceleration. A negative acceleration indicates braking. represents the acceleration / braking output ratio. When the vehicle accelerates proportionally, when the vehicle brakes proportionally, and when the vehicle coasts.

[0034] The correspondence between the value of Table 1 Correspondence between acceleration and vehicle behavior

[0035] (3) Set the reward function: By setting an effective reward function, the efficiency and risk of vehicle driving can be more effectively balanced, thereby guiding the vehicle cluster to conduct road planning.

[0036] The setting of the reward function includes two items. The first item is the speed deviation penalty term. By restricting the degree to which the current vehicle speed deviates from the target speed, the agent is encouraged to drive at the target speed. The formula is: .

[0037] The second item is the collision risk penalty term. By reducing the collision risk between the current vehicle and other vehicles within its perception range, the smooth operation of the vehicle system is ensured. The formula is set as: .

[0038] The specific setting of the reward function is as follows: ; Among them, is the current reward of vehicle i, is the speed of vehicle i at the current moment, is the predetermined target speed of the vehicle, is the speed deviation weight coefficient, which is used to measure the influence degree of the deviation between the current vehicle speed and the target speed on the current reward; represents the perception neighborhood of vehicle i at the current moment; is the collision risk coefficient, which is used to measure the influence degree of the collision risk on the reward; is used to judge whether there is a collision risk between the current vehicle i and other vehicles j within its perception range. It judges whether there is a collision risk by calculating whether there is a vehicle in the perception area of vehicle i that is driving towards the same intersection and the distance is within the collision warning distance.

[0039] The specific settings are as follows: ; ; ; ; Among them, represents the road fork that vehicle i is driving towards at the current moment, represents the road fork that vehicle j is driving towards at the current moment, represents the vehicle collision warning distance, represents the longitude and latitude coordinates of vehicle i at the current moment, represents the longitude and latitude coordinates of vehicle j at the current moment, is the actual distance between the longitude and latitude points of vehicle i and vehicle j calculated at the current moment. m is an auxiliary calculation variable used to calculate the spherical distance between two longitude and latitude points. C is the calculated spherical angular distance, and R is the radius of the earth.

[0040] The reward function comprehensively considers the efficiency and safety of vehicle driving scheduling, encourages the scheduling vehicle to run at a predetermined speed, and at the same time avoids collisions between vehicles at intersections.

[0041] (4)MFG mean field design:

[0042] DDPG is a deep reinforcement learning algorithm applicable to continuous action spaces, which combines a policy network (Actor) and a value network (Critic). In the present invention, the idea of mean field game (MFG) is introduced, that is, in a multi-agent system, the decision of each agent is not only based on its own state, but also based on the average behavior of other agents, so as to achieve a more coordinated global behavior. By updating the mean field at each time step, the average behavior of all agents is reflected. Define each network and model parameters of the model based on DDPG, and introduce the calculation of the mean field into the data flow of each branch of the model.

[0043] In MFG, the strategy of each vehicle depends not only on its own state but also on the average behavior of other vehicles. The mean field represents the average value of the actions taken by all vehicles: ; where is the mean field at the current time, is the action taken by the vehicle at the previous time, and N is the total number of the vehicle itself and all vehicles within its perceived neighborhood.

[0044] Among them, the actions taken by the surrounding vehicles at the previous time are collected through V2X communication technology.

[0045] The mean field game can enhance the global behavior perception of the scheduling model, enabling vehicles to consider the dynamics of the global traffic flow when making decisions, improving the coordination between vehicles and the global optimality of decisions.

[0046] (5) Design of the policy network (Actor network): As Figure 2 shown, the policy network receives the state and the mean field as inputs, and the output is the action of the vehicle agent. The output action of the policy network makes decisions not only based on the state of the current agent but also on the mean field of the global environment: ; where is the policy network, are the parameters of the policy network.

[0047] As Figure 2 shown, the network architecture of the policy network includes: an input layer, two hidden layers, and an output layer. The number of neurons in the hidden layer is set to 256, the rectified linear unit (ReLU) is used as the activation function, and batch processing and L2 regularization are used to avoid overfitting; the Tanh function is used as the activation function in the output layer.

[0048] At the same time, the target network of the policy network is constructed, that is, the target policy network, and the target policy network adopts the same network architecture as the policy network.

[0049] After each small-batch data training, the network parameters are updated through the gradient descent algorithm, and the Adam optimizer is used to adaptively adjust the learning rate to accelerate convergence. At the same time, the target network parameters are updated in a soft update manner. Soft update is not performed at fixed time steps, but at each step, with a smaller update amplitude. This makes the parameter changes of the target network smoother, thus maintaining the stability of the network gradient change during the reinforcement learning process and helping the algorithm converge more easily.

[0050] (6)Value network (Critic network) design: As Figure 3 shown, in the value network, the input is the state and action of the agent. At the same time, the mean field is used as an additional input, so that the value network can estimate the value of the state-action pair under the influence of the mean field, and the output is .

[0051] The specific settings of the value network adopt a design similar to that of the policy network, with 2 hidden layers, and each layer has 256 neurons. The rectified linear unit activation function is used as the activation function.

[0052] At the same time, the target network of the value network is constructed, that is, the target value network. The target value network adopts the same network architecture as the value network. The update of the parameters of the value network and the target value network adopts the same update method as the policy network.

[0053] (7)Experience replay pool design: To break the correlation between the data of the interaction between the agent and the environment, the model uses an experience replay pool to store the experience tuples of the agent.

[0054] At each network update, a small batch of data is randomly sampled from the experience replay pool for network training. The capacity of the experience replay pool is limited. When the pool is full, new experiences will replace the earliest experiences to achieve rolling storage.

[0055] The DDPG-MFG model is trained according to the following steps: Initialize the network: Initialize the network parameters of the policy network, target policy network, value network and target value network, and construct and initialize the experience replay pool to store training samples; Environment interaction: At each time step, the policy network outputs the vehicle action based on the vehicle running state and the mean field . The agent selects an action according to the policy output by the policy network, executes the action and interacts with the environment in real time to obtain the immediate reward and the new state and update the mean field according to the actions of all vehicles as ; Store experience: Store the experience tuple generated by the interaction into the experience replay pool; Parameter update: Randomly sample a small batch of data from the experience replay pool, adaptively adjust the learning rate using gradient descent and the Adam optimizer, and iteratively update the network weights according to the loss function; Repeat iteration: Continuously iterate the above process until the preset model convergence condition is met to complete the training.

[0056] As Figure 4 shown, the present invention provides a multi-vehicle collaborative scheduling system, including: A data acquisition module, configured to acquire the current vehicle running state and the mean field, where the vehicle running state includes the state information and environmental information of the vehicle itself and all vehicles within its sensing area; the mean field is the average value of the actions taken by the vehicle itself and all vehicles within its sensing area at the previous moment; An action determination module, configured to input the current vehicle running state and the mean field into a pre-trained DDPG-MFG model to obtain the current vehicle action.

[0057] Among them, the data acquisition module includes: An acquisition module, configured to acquire the state information and environmental information of the current vehicle itself, and perform real-time interaction with surrounding vehicles and / or infrastructure to acquire the state information, environmental information of surrounding vehicles, and the actions taken by surrounding vehicles at the previous moment; A mean field calculation module, configured to calculate the average value of the actions taken by the vehicle itself and surrounding vehicles at the previous moment to obtain the mean field.

[0058] In a preferred embodiment, the acquisition module adopts a V2X data module. The V2X data module performs real-time interaction with surrounding vehicles and / or infrastructure through V2X communication technology, and transmits the state information and environmental information of the vehicle in real time. The V2X data module includes a vehicle data module and a road data module, configured to acquire and process real-time traffic information to realize information sharing and state transfer between vehicles.

[0059] The vehicle data module contains the acquired vehicle data, and the vehicle data includes the speed, position, and road where the vehicle is located in each time step, and also includes the information of surrounding vehicles and infrastructure, including position, speed, acceleration, road information, etc.

[0060] The road data module includes global road information and road intersection information. The global road information includes all road numbers. By matching the road where the vehicle is located with the global road information, the road where the vehicle is currently located can be obtained. Among them, the multi-vehicle collaborative scheduling system further includes: a model training module for performing the following steps: Take the vehicle running state and the mean field as the input of the policy network, and the output is the vehicle action ; Take the vehicle running state the mean field and the vehicle action as the input of the value network, and the output is ; Initialize the network parameters of the policy network, target policy network, value network and target value network, and construct and initialize an experience replay pool to store training samples; At each time step, the policy network outputs the vehicle action based on the vehicle running state and the mean field . The agent selects an action according to the policy output by the policy network, executes the action and interacts with the environment in real time to obtain an immediate reward and a new state , and updates the mean field to according to the actions of all vehicles; Store the experience tuple generated by the interaction into the experience replay pool; Randomly sample a small batch of data from the experience replay pool, use the gradient descent method and the Adam optimizer to adaptively adjust the learning rate, and perform iterative updates of the network weights according to the loss function.

[0061] The following gives a simulation system for multi-vehicle collaborative scheduling, as Figure 5 shown, including the following steps: Step 1, establish a V2X data module; The V2X data module includes a vehicle data module and a road data module.

[0062] The vehicle data module contains vehicle data, which includes the speed, position, and road where the vehicle is located at each time step, and also includes information about surrounding vehicles and infrastructure, including position, speed, acceleration, road information, etc.

[0063] The road data module includes global road information and road intersection information. The global road information includes all road numbers. During simulation, by matching the road where the vehicle is located with the global road information, the road where the vehicle is currently located can be obtained.

[0064] The V2X data module further includes a data preprocessing module, which includes operations such as abnormal data detection and data standardization according to the input requirements of the DDPF-MFG model.

[0065] Step 2: Establish a simulation module. Define road information and vehicle model information in the simulation environment. At the same time, simulate V2X communication in the simulation environment to achieve information exchange and sharing between vehicles, and establish a vehicle state transition model.

[0066] Define the road topology structure in the simulation environment, including main roads, branch roads, roundabouts, and intersections; define vehicle attributes in the simulation environment, including the initial position, target speed, and road where the vehicle is located; at the same time, set the driving direction and destination fork of the vehicle; simulate V2X communication in the simulation environment to achieve data broadcasting and point-to-point communication between vehicles, and transmit dynamic states such as position information, speed, and acceleration.

[0067] The method for establishing the vehicle state transition model is as follows: Determine the running state of the current vehicle, including the vehicle's speed, position, and road where it is located, as well as information about surrounding vehicles and infrastructure, including position, speed, acceleration, road information, etc.; transmit the current running state of the vehicle to the DDPG-MFG model, and receive the traction control instruction output from the DDPG-MFG model, and input it into the vehicle kinematic model for vehicle kinematic calculation to obtain the next state of the vehicle.

[0068] The simulation module includes a sensor module, which is used to obtain information such as the position, speed, and acceleration of the vehicle.

[0069] Step 3: Based on the V2X data module and the simulation module, establish a multi-vehicle cooperative scheduling model, including defining the state space of the vehicle, defining the action space of the vehicle, defining the comprehensive reward function, defining the mean field game (MFG), defining the DDPG algorithm framework, designing the neural network structure of DDPG, defining the optimizer, and establishing the multi-vehicle cooperative scheduling model, namely the DDPG-MFG model. The establishment process of the multi-vehicle cooperative scheduling model is as described above.

[0070] Step 4: Determine the hyperparameters of the artificial neural network in the DDPG-MFG model and perform training until the model reaches the convergence condition to obtain the trained DDPG-MFG model. The strategy output by the trained DDPG-MFG model is the control method that meets safe and efficient multi-vehicle cooperative scheduling.

[0071] The present invention also provides an electronic device, including one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing any of the methods according to the foregoing methods.

[0072] The present invention also provides a readable storage medium, on which one or more programs are stored, and the one or more programs include instructions, and when the instructions are executed by a computing device, the computing device is caused to execute any of the foregoing methods.

[0073] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0074] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, and the combination of flows and / or blocks in the flowchart and / or block diagram can also be implemented by computer program instructions. These computer program instructions can be provided to the processors of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processors of the computer or other programmable data processing devices generate means for implementing the specified functions in one process Figure 1 one process or multiple processes and / or blocks Figure 1 or multiple blocks.

[0075] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including instruction means, and the instruction means implements the specified functions in one process Figure 1 one process or multiple processes and / or blocks Figure 1 or multiple blocks.

[0076] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide for implementing in one process Figure 1 one process or multiple processes and / or blocksFigure 1 Steps of functions specified in one or more boxes.

[0077] The above are only embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention are included within the scope of the claims of the present invention pending approval of the application.

Claims

1. A multi-vehicle collaborative scheduling method, characterized in that, Including: Obtain the current vehicle operating state and the mean field, where the vehicle operating state includes the state information and environmental information of the vehicle itself and all vehicles within its perception area; The mean field is the average of the actions taken by the vehicle itself and all vehicles within its perception area at the previous moment; Input the current vehicle operating state and the mean field into a pre-trained DDPG-MFG model to obtain the current vehicle action.

2. The multi-vehicle collaborative scheduling method according to claim 1, wherein The DDPG-MFG model includes a policy network, a target policy network, a value network, and a target value network; the DDPG-MFG model is obtained according to the following steps: Take the vehicle operating state and the mean field as the input of the policy network, and the output is the vehicle action ; Take the vehicle operating state , the mean field and the vehicle action as the input of the value network, and the output is ; Initialize the network parameters of the policy network, target policy network, value network, and target value network, and construct and initialize an experience replay pool to store training samples; At each time step, the policy network outputs vehicle actions based on the vehicle operating state and the mean field ; the agent selects actions according to the policy output by the policy network, executes the actions and interacts with the environment in real time to obtain immediate rewards and new states and updates the mean field to ; Store the experience tuples generated by the interaction into the experience replay pool; Randomly sample a small batch of data from the experience replay pool, use the gradient descent method and the Adam optimizer to adaptively adjust the learning rate, and iteratively update the network weights according to the loss function.

3. The multi-vehicle collaborative scheduling method according to claim 1 or 2, characterized in that The vehicle operating state is defined as: ； wherein is the running state of vehicle i at the current moment, is the position of vehicle i at the current moment; is the speed of vehicle i at the current moment; is the acceleration of vehicle i at the current moment; is the road where vehicle i is located at the current moment; represents the road fork that vehicle i is driving towards at the current moment; represents the set of the position, speed, and road information of the neighboring vehicle j, represents the perception neighborhood of vehicle i.

4. The multi-vehicle collaborative scheduling method according to claim 1 or 2, characterized in that, The mean field is calculated according to the following formula: ； wherein, is the mean field at the current moment, is the action taken by the vehicle at the previous moment, and N is the total number of all vehicles including the vehicle itself and its perceived neighboring vehicles.

5. The multi-vehicle collaborative scheduling method according to claim 1 or 2, characterized in that The action space of the vehicle action is defined as: ； wherein, and respectively represent the minimum and maximum limits of the acceleration, represents the acceleration / braking output ratio of vehicle i, represents that vehicle i accelerates proportionally, represents that vehicle i brakes proportionally, represents that vehicle i coasts.

6. The multi-vehicle collaborative scheduling method according to claim 2, wherein The reward is determined according to a reward function, and the reward function is: ； Among them, is the current reward of vehicle i, is the speed of vehicle i at the current moment, is the predetermined target speed of the vehicle, represents the perception neighborhood of vehicle i at the current moment, is the speed deviation weight coefficient, is the collision risk coefficient, is used to determine whether there is a collision risk between the current vehicle i and other vehicles j within its perception neighborhood, The specific setting of is as follows: ； ； ； ； Among them represents the road intersection that vehicle i is driving towards at the current moment, represents the road intersection that vehicle j is driving towards at the current moment, represents the vehicle collision warning distance, represents the longitude and latitude coordinates of vehicle i at the current moment, represents the longitude and latitude coordinates of vehicle j at the current moment, is the actual distance between the longitude and latitude points of vehicle i and vehicle j calculated at the current moment, m is an auxiliary calculation variable used to calculate the spherical distance between two longitude and latitude points, C is the calculated spherical angular distance, and R is the radius of the earth.

7. The multi-vehicle collaborative scheduling method according to claim 2, wherein The policy network includes an input layer, 2 hidden layers, and an output layer; the number of neurons in each hidden layer is set to 256, the ReLU function is used as the activation function, and batch processing and L2 regularization are used; the Tanh activation function is used in the output layer; the target policy network adopts the same network architecture as the policy network.

8. The multi-vehicle collaborative scheduling method according to claim 2, wherein, The value network includes an input layer, 2 hidden layers, and an output layer; the number of neurons in each hidden layer is set to 256, and the ReLU function is used as the activation function; the target value network adopts the same network architecture as the value network.

9. The multi-vehicle collaborative scheduling method according to claim 1, wherein, Adopt V2X communication technology to interact with surrounding vehicles and / or infrastructure in real time, and collect the state information, environmental information of surrounding vehicles, and the actions taken by surrounding vehicles at the previous moment.

10. A multi-vehicle collaborative scheduling system, characterized in that, Including: A data acquisition module for obtaining the current vehicle operating state and the mean field, where the vehicle operating state includes the state information and environmental information of the vehicle itself and all vehicles within its perception area; the mean field is the average of the actions taken by the vehicle itself and all vehicles within its perception area at the previous moment; An action determination module for inputting the current vehicle operating state and the mean field into a pre-trained DDPG-MFG model to obtain the current vehicle action.

11. The multi-vehicle collaborative scheduling system according to claim 10, wherein The data acquisition module includes: A collection module for collecting the state information and environmental information of the current vehicle itself, and interacting with surrounding vehicles and / or infrastructure in real time to collect the state information, environmental information of surrounding vehicles, and the actions taken by surrounding vehicles at the previous moment; A mean field calculation module for calculating the average of the actions taken by the vehicle itself and its surrounding vehicles at the previous moment to obtain the mean field.

12. The multi-vehicle collaborative scheduling system according to claim 10, wherein The DDPG-MFG model includes a policy network, a target policy network, a value network, and a target value network; the system further includes a model training module for performing the following steps: Take the vehicle running state and the mean field as the input of the policy network, and the output is the vehicle action ; Take the vehicle running state , the mean field and the vehicle action as the input of the value network, and the output is ; Initialize the network parameters of the policy network, target policy network, value network, and target value network, and construct and initialize an experience replay pool to store training samples; At each time step, the policy network outputs vehicle actions based on the vehicle operating state and the mean field ; the agent selects actions according to the policy output by the policy network, executes the actions and interacts with the environment in real time to obtain immediate rewards and the new state and updates the mean field according to the actions of all vehicles to be ; Store the experience tuples generated by the interaction in the experience replay pool; Randomly sample a small batch of data from the experience replay pool, use the gradient descent method and the Adam optimizer to adaptively adjust the learning rate, and iteratively update the network weights according to the loss function.

Images