Intelligent car cluster control method, device, system and storage medium

Abstract

The present invention discloses a control method, device, system and storage medium for a cluster of intelligent cars. When selecting a strategy for multi-agent reinforcement learning, the method does not seek a Nash equilibrium point, but minimizes the regret of each agent's action. At the same time, when updating the Q value, the Q value of the action with the smallest regret in the next state is used as part of the update, so that the agent can finally learn the strategy of minimizing regret, and then drive each intelligent car in the cluster of intelligent cars to perform the corresponding action. The present invention improves the strategy selection part in traditional multi-agent reinforcement learning, adopts the regret minimization strategy to select actions, and the action made by each agent is no longer greedy to select the optimal action, and the final joint action no longer satisfies the Nash equilibrium. In some specific scenarios where the Nash equilibrium cannot be well solved, it has better control solution results.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-agent reinforcement learning, and in particular to a method, device, system and storage medium for controlling a cluster of intelligent vehicles. Background Art

[0002] Smart cars play an increasingly important role in daily life, such as warehousing and logistics, automatic cleaning, rescue, express delivery, dangerous work, etc. Whether or not a smart car can navigate autonomously is the basis for completing tasks. At present, most navigation classic pathfinding technologies are used, but when the environment around the smart car is complex and changeable, it is difficult to update the map in time, making it difficult for the smart car to complete the navigation task. Reinforcement learning is an important way for new smart cars to find their way from the perspective of artificial intelligence.

[0003] When there are multiple smart cars in the same geographical environment, and the goal is to complete the same or different tasks, the planning of the cars will become more complicated. For such a cluster of smart cars, the key issue is how to achieve the collaboration or coordination of different small workshops. Traditional multi-agent reinforcement learning uses Q-learning to learn state value functions, such as Minimax-Q, Nash Q-learning, and Friend-or-Foe Q-Learning, but they use different methods to solve the Nash equilibrium strategy under different states. These algorithms perform well in situations where the Nash equilibrium itself is the optimal solution. In a multi-agent system, each agent learns to improve its own strategy by interacting with the environment to obtain reward values, so as to obtain the optimal strategy in the environment. This is multi-agent reinforcement learning. Multi-agent reinforcement learning needs to solve problems such as environmental instability, reward distribution, partial observability, and computational complexity. In multi-agent reinforcement learning, the payoff matrix of each state is generally unknown, and it is necessary to learn the state value function of each state through continuous interaction with the environment, and then learn the optimal Nash strategy through these state value functions. Regret minimization is a new concept in game theory. Regret refers to the difference between the benefit of the optimal action and the benefit of the current action. Regret minimization eliminates actions in the strategy whose regret values ​​are not the minimum, and only retains actions with the minimum regret values.

[0004] However, although Minimax-Q, Nash Q-learning, Friend-or-Foe Q-Learning and other algorithms perform well in test scenarios such as the grid world and rock-paper-scissors, the optimal solutions of the above scenarios themselves meet the Nash equilibrium. When the optimal solution of the scenario does not meet the Nash equilibrium, such as the cluster control of multiple smart cars in logistics warehousing, the solution result is not ideal.

[0005] Therefore, how to provide a control method for the collaboration between multiple individuals in a smart car cluster in a specific scenario where Nash equilibrium cannot meet the needs is a technical problem that needs to be solved urgently. The above content is only used to assist in understanding the technical solution of the present invention and does not mean that the above content is recognized as prior art. Summary of the invention

[0006] The main purpose of the present invention is to provide a method, device, system and storage medium for controlling a cluster of intelligent vehicles, aiming to solve the technical problem that the current traditional control method based on multi-agent reinforcement learning has unsatisfactory control solution results when faced with a scenario where the optimal solution does not satisfy the Nash equilibrium.

[0007] To achieve the above object, the present invention provides a smart car cluster control method, the method comprising:

[0008] Establish a multi-agent reinforcement learning model, initialize multiple agents in the environment, and initialize the values ​​of all states in the agent's Q table to 0;

[0009] Get state S and select the target action using the ∈-greedy strategy;

[0010] Drive the agent to perform the target action to transfer the environment to state S', and obtain the reward for each agent when the environment transfers to state S';

[0011] According to the reward and the Q value of the action with the smallest regret value in state S', the Q value of the corresponding action selected by each agent in state S' is updated, and the smart car is driven to perform the corresponding action according to the Q value of each agent;

[0012] Return to execute and obtain state S, and use the ∈-greedy strategy to select the target action step until state S is the terminal state.

[0013] Optionally, obtaining the Q value of the action with the minimum regret value in the state S' specifically includes:

[0014] Get the regret value of the current action space and perform iterative regret minimization on the joint action space of all agents;

[0015] According to the iterative regret minimization operation results of all agents, the Q value of the action with the smallest regret value in state S' is determined.

[0016] Optionally, the step of obtaining the regret value of the current action space specifically includes: i The regret value of the action with the smallest regret value.

[0017] Get the current joint action The regret value under , determines the regret value when agent i takes action a;

[0018] According to the regret value when agent i takes action a, the regret value of the current action space is obtained.

[0019] Optional, get the current joint action The regret value under the condition of determining the regret value step when agent i takes action a is:

[0020] The current joint action The expression of the regret value under , is:

[0021]

[0022] The expression of the regret value when the agent i takes action a is specifically:

[0023]

[0024] The expression of the regret value of the current action space is specifically:

[0025]

[0026] Among them, Regret is the value of the satisfaction of the agent before selecting an action, and a i is the action of agent i, a -i represents the actions of other agents, Q i is the state-action benefit value of the agent, A i is the set of all optional actions of agent i.

[0027] Optionally, the step of obtaining the regret value of the current action space and performing an iterative regret minimization operation on the joint action space of all agents specifically includes:

[0028] Get the regret value of the current action space and perform a regret minimization operation on the action space of each agent;

[0029] Based on the regret minimization results of each agent, a minimization operation is performed on the joint action space of all agents;

[0030] According to the regret minimization operation results of all agents, an iterative regret minimization operation is performed on the joint action space of all agents.

[0031] Optionally, during the iterative regret minimization step over the joint action space of all agents:

[0032] The regret minimization operation is performed on the action space of each agent, specifically:

[0033]

[0034] The minimization operation of the joint action space of all agents is specifically as follows:

[0035]

[0036] The iterative regret minimization operation on the joint action space of all agents is specifically as follows:

[0037]

[0038] Regret is the value of the satisfaction of the action before the car selects it, and a i is the action of agent i, a -i represents the actions of other agents, Q i is the benefit value of the agent's state-action.

[0039] Optionally, update the expression of the Q value of each agent's corresponding action in state S', specifically:

[0040]

[0041] Among them, a i is the action of agent i, a -i represents the actions of other agents, Q i is the state-action benefit value of the agent, α t ,β,r i All of them are freely adjustable parameters in machine learning training, which can be gradually optimized by traversal. RM iteratively deletes operations and gradually reduces the search space for optimizing the action of the smart car. Finally, It represents the optimal strategy after iterative deletion, that is, how the agent i should choose action a in each state s. i .

[0042] In addition, in order to achieve the above-mentioned purpose, the present invention also provides a smart car cluster control device, the device comprising:

[0043] The initialization module is used to establish a multi-agent reinforcement learning model, initialize multiple agents in the environment, and initialize the values ​​of all states in the agent's Q table to 0;

[0044] The selection module is used to obtain the state S and select the target action using the ∈-greedy strategy;

[0045] The environment transfer module is used to drive the agent to perform the target action so that the environment is transferred to the state S' and obtain the reward for each agent when the environment is transferred to the state S';

[0046] An updating module is used to update the Q value of the corresponding action selected by each agent in state S' according to the reward and the Q value of the action with the smallest regret value in state S', and drive the smart car to perform the corresponding action according to the Q value of each agent;

[0047] The loop module is used to return to the execution state S and select the target action step using the ∈-greedy strategy until the state S is the terminal state.

[0048] In addition, in order to achieve the above-mentioned purpose, the present invention also provides a smart car cluster control system, the smart car cluster control system comprising:

[0049] A plurality of smart cars, each of which performs corresponding actions according to a driving instruction sent by a smart car cluster control device;

[0050] An intelligent car cluster control device, the intelligent car cluster control device includes a memory, a processor and an intelligent car cluster control method program stored in the memory and executable on the processor, and the intelligent car cluster control method program implements the steps of the intelligent car cluster control method as described above when executed by the processor.

[0051] In addition, in order to achieve the above-mentioned purpose, the present invention also provides a storage medium, on which a smart car cluster control method program is stored. When the smart car cluster control method program is executed by a processor, the steps of the above-mentioned smart car cluster control method are implemented.

[0052] The embodiment of the present invention proposes a method, device, system and storage medium for controlling a cluster of intelligent cars. When selecting a strategy for multi-agent reinforcement learning, the method does not seek a Nash equilibrium point, but minimizes the regret of each agent's action. At the same time, when updating the Q value, the Q value of the action with the smallest regret in the next state is used as part of the update, so that the agent can eventually learn the strategy of minimizing regret, and then drive each intelligent car in the cluster of intelligent cars to perform the corresponding action. The present invention improves the strategy selection part in traditional multi-agent reinforcement learning and adopts a regret minimization strategy to select actions. The action made by each agent is no longer a greedy selection of the optimal action, and the final joint action no longer satisfies the Nash equilibrium. In some specific scenarios where the Nash equilibrium cannot be well solved, it has better control solution results. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a structural schematic diagram of an intelligent car cluster control system of the present invention;

[0054] Figure 2This is a structural schematic diagram of an intelligent car cluster control device of the present invention;

[0055] Figure 3 A schematic diagram of a flow chart of a method for controlling a cluster of intelligent vehicles according to the present invention;

[0056] Figure 4 It is a schematic diagram of the principle of the intelligent car cluster control method of the present invention;

[0057] Figure 5 It is a schematic diagram of the regret value deep network (Regret Q network) of the smart car of the present invention;

[0058] Figure 6 This is a schematic diagram of the effect of the reinforcement learning algorithm in the first type of intelligent car interaction scenario of the present invention;

[0059] Figure 7 This is a schematic diagram of the effect of the reinforcement learning algorithm in the second type of intelligent car interaction scenario of the present invention;

[0060] Figure 8 This is a schematic diagram of the effect of the reinforcement learning algorithm in the third type of intelligent car interaction scenario of the present invention;

[0061] Fig. 9 This is a schematic diagram of the effect of the reinforcement learning algorithm in the fourth type of intelligent car interaction scenario of the present invention;

[0062] Fig.10 This is a structural block diagram of an intelligent car cluster control device of the present invention.

[0063] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0064] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0065] At present, in the relevant technical field, the traditional control method based on multi-agent reinforcement learning does not produce ideal control solution results when faced with scenarios where the optimal solution does not satisfy the Nash equilibrium.

[0066] In order to solve this problem, various embodiments of the intelligent car cluster control method of the present invention are proposed. The intelligent car cluster control method provided by the present invention improves the strategy selection part in the traditional multi-agent reinforcement learning, adopts the regret minimization strategy to select actions, and the actions made by each agent are no longer greedy to select the optimal action, and the final joint action no longer satisfies the Nash equilibrium. In some specific scenarios where the Nash equilibrium cannot be well solved, it has better control solution results.

[0067] Reference Figure 1 , Figure 1 It is a schematic diagram of the structure of the intelligent car cluster control system involved in the embodiment of the present invention.

[0068] In this embodiment, the intelligent car cluster control system includes:

[0069] A plurality of smart cars 100, wherein the smart cars 100 perform corresponding actions according to the driving instructions sent by the smart car cluster control device 200;

[0070] An intelligent car cluster control device 200 includes a memory, a processor, and an intelligent car cluster control method program stored in the memory and executable on the processor. When the intelligent car cluster control method program is executed by the processor, the steps of the intelligent car cluster control method are implemented.

[0071] For smart car cluster control equipment, refer to Figure 2 , Figure 2 The figure is a schematic diagram of the structure of the intelligent vehicle cluster control device involved in the embodiment of the present invention.

[0072] The device may be a user equipment (UE) such as a mobile phone, a smart phone, a laptop, a digital broadcast receiver, a personal digital assistant (PDA), a tablet computer (PAD), a handheld device, a vehicle-mounted device, a wearable device, a computing device or other processing device connected to a wireless modem, a mobile station (MS), etc. The device may be called a user terminal, a portable terminal, a desktop terminal, etc.

[0073] Typically, the device includes: at least one processor 301, a memory 302, and a smart car cluster control method program stored in the memory and executable on the processor, wherein the smart car cluster control method program is configured to implement the steps of the smart car cluster control method as described above.

[0074] The processor 301 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 301 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 301 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 301 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. The processor 301 may also include an AI (Artificial Intelligence) processor, which is used to process operations related to the intelligent car cluster control method, so that the intelligent car cluster control method model can be trained and learned autonomously to improve efficiency and accuracy.

[0075] The memory 302 may include one or more computer-readable storage media, which may be non-transitory. The memory 302 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 302 is used to store at least one instruction, which is used to be executed by the processor 301 to implement the intelligent car cluster control method provided in the method embodiment of the present application.

[0076] In some embodiments, the terminal may further optionally include: a communication interface 303 and at least one peripheral device. The processor 301, the memory 302 and the communication interface 303 may be connected via a bus or a signal line. Each peripheral device may be connected to the communication interface 303 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 304, a display screen 305 and a power supply 306.

[0077] The communication interface 303 can be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 301 and the memory 302. The communication interface 303 is used to receive the movement trajectories and other data of multiple mobile terminals uploaded by users through the peripheral device. In some embodiments, the processor 301, the memory 302 and the communication interface 303 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 301, the memory 302 and the communication interface 303 can be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0078] The radio frequency circuit 304 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 304 communicates with the communication network and other communication devices through electromagnetic signals, so as to obtain the movement trajectory and other data of multiple mobile terminals. The radio frequency circuit 304 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 304 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 304 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 304 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0079] The display screen 305 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos and any combination thereof. When the display screen 305 is a touch display screen, the display screen 305 also has the ability to collect touch signals on the surface or above the surface of the display screen 305. The touch signal can be input to the processor 301 as a control signal for processing. At this time, the display screen 305 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 305 can be one, the front panel of the electronic device; in other embodiments, the display screen 305 can be at least two, respectively arranged on different surfaces of the electronic device or in a folding design; in some further embodiments, the display screen 305 can be a flexible display screen, arranged on a curved surface or a folding surface of the electronic device. Even, the display screen 305 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 305 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode, organic light-emitting diode).

[0080] The power supply 306 is used to power various components in the electronic device. The power supply 306 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 306 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0081] Those skilled in the art will understand that Figure 2 The structure shown in the figure does not constitute a limitation on the intelligent car cluster control device, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0082] The embodiment of the present invention provides a method for controlling a cluster of intelligent vehicles. Figure 3 , Figure 3 The figure is a flow chart of an embodiment of the intelligent vehicle cluster control method of the present invention.

[0083] In this embodiment, the intelligent vehicle cluster control method includes the following steps:

[0084] Step S100, establishing a multi-agent reinforcement learning model, initializing multiple agents in the environment, and initializing the values ​​of all states in the Q table of the agent to 0;

[0085] Step S200, obtaining state S, and selecting a target action using the ∈-greedy strategy;

[0086] Step S300, driving the agent to perform the target action so that the environment is transferred to state S', and obtaining the reward for each agent when the environment is transferred to state S';

[0087] Step S400, according to the reward and the Q value of the action with the smallest regret value in state S', update the Q value of the corresponding action selected by each agent in state S', and drive the smart car to perform the corresponding action according to the Q value of each agent;

[0088] Step S500, return to execute and obtain state S, and use the ∈-greedy strategy to select the target action step until state S is the terminal state.

[0089] Specifically, in this embodiment, if Figure 4 As shown, the intelligent car cluster control method includes the following detailed steps:

[0090] Step 1: Initialize the environment;

[0091] Model the problem that the algorithm needs to solve, such as the cluster control problem of multiple smart cars, and set the state of the model to the initial state.

[0092] Step 2: Initialize the agent in the environment.

[0093] Set the number of agents in the environment, set the relevant parameters of the agents, and set the values ​​of all states in the agent's Q table to 0.

[0094] Step 3: The agent observes the surrounding environment.

[0095] The agent obtains the state S of the current environment. If the current state is the terminal state of the environment, it returns to step 1. If it is not the terminal state, it goes to step 4.

[0096] Step 4: The agent selects the target action based on the state S using the ∈-greedy strategy.

[0097] ∈ will decay with the number of training rounds. If the value of ∈ is greater than the preset constant a, the agent will randomly select an action. Otherwise, the agent will use the regret minimization strategy to obtain the target action.

[0098] Step 5: The environment moves to the next new state;

[0099] Apply the actions of all agents obtained in step 4 to the environment, the state of the environment will change, enter a new state, and the agent will be given a corresponding reward.

[0100] Step 6: The agent receives a reward from the environment.

[0101] Depending on the actions of each agent, the environment will give corresponding rewards.

[0102] Step 7: The agent updates its own Q table;

[0103] The agent uses the regret minimization strategy to obtain the action with the smallest regret value in the new state, and uses the Q value of the action with the smallest regret value in the new state and the reward received to update the Q value of the corresponding action selected in the current state according to the Bellman equation. Return to step 2.

[0104] In a preferred embodiment, the relevant concepts of the regret minimization strategy involved in steps 4 and 7 are defined as follows:

[0105] Ac represents the action space of agent i, and s represents the state of the environment.

[0106] Indicates the current joint action The regret below is the difference between the Q value of the best action that the agent can take and the Q value of the current action.

[0107] The regret value when agent i takes action a is the maximum regret of all joint actions that include action a in the current state. It is the regret value of the agent taking action a in the worst case in the current state, that is, the regret value when other agents take the most advantageous action for them.

[0108] Indicates that the regret value of the current action space is equal to the action space A i The regret of the action with the smallest regret value in . Regret is the value of the satisfaction of the action evaluated before the car selects an action. i is the movement of car i, a -i Indicates the actions of other cars, Q i is the state-action benefit value of the car, A i is the set of all optional actions of agent i. Figure 5 shown.

[0109] Perform a regret minimization operation on the action space of an agent:

[0110]

[0111] Perform a regret minimization operation on the joint action space of all agents:

[0112]

[0113] Preferably, the update formula of step 7 is expressed as follows:

[0114]

[0115] Among them, a i is the movement of car i, a -i Indicates the actions of other cars, α t ,β,r i These are freely adjustable parameters in machine learning training, which can be gradually optimized by traversal. i is the state-action benefit value of the car. RM is the Q table remaining after the algorithm performs a screening operation on the Q table. Iterative deletion operations gradually reduce the search space for optimizing the car's actions. Finally, It represents the optimal strategy after iterative deletion, that is, how car i should choose action a in each state s i .

[0116] By introducing regret minimization in game theory into the reinforcement learning of smart car clusters, the smart cars can learn the regret minimization strategy, which is better than the algorithm for solving Nash equilibrium in four types of smart car scenarios. The present invention also proposes a new multi-agent reinforcement learning algorithm MARMQ under the independent training version, which takes into account the influence of information sharing factors on the algorithm. The deep neural network is introduced into MARMQ, and the deep version of MARMQ is proposed.

[0117] In this embodiment, in order to prove the effectiveness of the present application in solving multi-agent reinforcement learning problems, four multi-agent problems were selected to test the algorithm.

[0118] Figure 6 The training effect of this application on a type of smart car scenario is demonstrated, and compared with Nash Q-learning. The first type of scenario is characterized by: the interaction of the smart car is repeated over a long period of time. This scenario is particularly suitable for applications with fewer vehicles and closed spaces. In this scenario, the final gain of this application is about 87, and Nash Q-learning is about 7. Figure 7 The training effect of this application in the second type of smart car scene is demonstrated and compared with Nash Q-learning. The second type of scene is characterized by a large number of vehicles, but the type of cars is consistent, and the space is still closed. The benefits of the two smart cars in Nash Q-learning are both around 0, while the final benefit of this application is 4. Figure 8 The training results of this application on three types of smart car problems are shown. The third type of scenario is characterized by different types of smart cars, such as different load capacities and speeds. This application finally learned the optimal hybrid strategy. Fig. 9The training results of this application on four types of smart cars are shown and compared with MADDPG. The characteristics of the fourth type of scene are: there are a large number of smart cars and the environment is relatively open and changeable. The final benefit of this application is about -1, while the benefit of MADDPG is -2.

[0119] This embodiment provides a multi-agent reinforcement learning algorithm based on regret minimization in response to the shortcomings of multi-agent learning technology in the existing market. The selection part of the strategy of multi-agent reinforcement learning is improved. Instead of looking for the Nash equilibrium point, the regret of each agent's action is minimized. At the same time, when updating the Q value, the Q value of the action with the smallest regret in the next state is used as part of the update, so that the agent can eventually learn the strategy of regret minimization.

[0120] Specifically, the present application improves the strategy selection part in traditional multi-agent reinforcement learning, adopts the regret minimization strategy to select actions, and the actions made by each agent are no longer greedy to select the best action, and the final joint action no longer satisfies the Nash equilibrium. In some specific scenarios where Nash equilibrium cannot be solved well, the effect of the present invention is better. In addition, the regret minimization strategy is used to update the Q value, while the traditional multi-agent algorithm needs to use linear programming methods to calculate the Nash equilibrium strategy to update the Q value. The calculation of regret minimization requires less calculation than the Nash equilibrium.

[0121] Reference Fig.10 , Fig.10 This is a structural block diagram of an embodiment of the intelligent car cluster control device of the present invention.

[0122] like Fig.10 As shown, the intelligent car cluster control device proposed in the embodiment of the present invention includes:

[0123] Initialization module 10, used to establish a multi-agent reinforcement learning model, initialize multiple agents in the environment, and initialize the values ​​of all states in the Q table of the agent to 0;

[0124] A selection module 20 is used to obtain the state S and select the target action using the ∈-greedy strategy;

[0125] The environment transfer module 30 is used to drive the agent to perform the target action so that the environment is transferred to the state S', and obtain the reward for each agent when the environment is transferred to the state S';

[0126] An updating module 40 is used to update the Q value of the corresponding action selected by each agent in state S' according to the reward and the Q value of the action with the smallest regret value in state S', and drive the smart car to perform the corresponding action according to the Q value of each agent;

[0127] The loop module 50 is used to return to the execution acquisition state S and select the target action step using the ∈-greedy strategy until the state S is the terminal state.

[0128] Other embodiments or specific implementations of the intelligent car cluster control device of the present invention can refer to the above-mentioned method embodiments and will not be described in detail here.

[0129] In addition, an embodiment of the present invention further proposes a storage medium, on which a smart car cluster control method program is stored, and when the smart car cluster control method program is executed by a processor, the steps of the smart car cluster control method described above are implemented. Therefore, it will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application. As an example, program instructions can be deployed to be executed on one computing device, or on multiple computing devices located at one location, or on multiple computing devices distributed at multiple locations and interconnected by a communication network.

[0130] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the above-mentioned program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the above-mentioned storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0131] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the accompanying drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art may understand and implement it without creative work.

[0132] Through the description of the above implementation mode, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present invention, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

Claims

1. A method for controlling a cluster of intelligent vehicles, characterized in that: The method comprises: Establish a multi-agent reinforcement learning model, initialize multiple agents in the environment, and initialize the values ​​of all states in the agent's Q table to 0; Get state S and select the target action using the ∈-greedy strategy; Drive the agent to perform the target action to transfer the environment to state S', and obtain the reward for each agent when the environment transfers to state S'; According to the reward and the Q value of the action with the smallest regret value in state S', the Q value of the corresponding action selected by each agent in state S' is updated, and the smart car is driven to perform the corresponding action according to the Q value of each agent; The acquisition of the Q value of the action with the minimum regret value under the state S' specifically includes: Get the regret value of the current action space and perform iterative regret minimization on the joint action space of all agents; According to the iterative regret minimization operation results of all agents, determine the Q value of the action with the smallest regret value in state S'; The step of obtaining the regret value of the current action space specifically includes: According to the regret value when agent i takes action a, the regret value of the current action space is obtained; Get the current joint action The regret value under the condition of determining the regret value step when agent i takes action a is: The current joint action The expression of the regret value under , is: The expression of the regret value when the agent i takes action a is specifically: The expression of the regret value of the current action space is specifically: in, The action space A i The regret value of the action with the smallest regret value, a i is the action of agent i, a -i represents the actions of other agents, Q i is the benefit value of the agent's state-action, Ai is the set of all optional actions of agent i; Return to execute and obtain state S, and use the ∈-greedy strategy to select the target action step until state S is the terminal state.

2. The intelligent car cluster control method according to claim 1, characterized in that: The step of obtaining the regret value of the current action space and performing iterative regret minimization operation on the joint action space of all agents specifically includes: Get the regret value of the current action space and perform a regret minimization operation on the action space of each agent; Based on the regret minimization results of each agent, a minimization operation is performed on the joint action space of all agents; According to the regret minimization operation results of all agents, an iterative regret minimization operation is performed on the joint action space of all agents.

3. The intelligent car cluster control method according to claim 2, characterized in that: In the iterative regret minimization step of the joint action space of all agents: The regret minimization operation is performed on the action space of each agent, specifically: The minimization operation of the joint action space of all agents is specifically as follows: The iterative regret minimization operation on the joint action space of all agents is specifically as follows: Among them, a i is the action of agent i, a _i represents the actions of other agents, Q i is the benefit value of the agent's state-action.

4. The intelligent car cluster control method according to claim 3, characterized in that: Update the expression of the Q value of each agent's corresponding action in state S', specifically: Among them, a i is the action of agent i, a _i represents the actions of other agents, Q i is the state-action benefit value of the agent, α t ,β,r i These are freely adjustable parameters in machine learning training, which can be gradually optimized by traversal. It indicates how the car n should choose an action in each state s after iterative deletion, where n is the car number.

5. A smart car cluster control device, characterized in that: The device comprises: The initialization module is used to establish a multi-agent reinforcement learning model, initialize multiple agents in the environment, and initialize the values ​​of all states in the agent's Q table to 0; The selection module is used to obtain the state S and select the target action using the ∈-greedy strategy; The environment transfer module is used to drive the agent to perform the target action so that the environment is transferred to the state S' and obtain the reward for each agent when the environment is transferred to the state S'; An updating module is used to update the Q value of the corresponding action selected by each agent in state S' according to the reward and the Q value of the action with the smallest regret value in state S', and drive the smart car to perform the corresponding action according to the Q value of each agent; The acquisition of the Q value of the action with the minimum regret value under the state S' specifically includes: Get the regret value of the current action space and perform iterative regret minimization on the joint action space of all agents; According to the iterative regret minimization operation results of all agents, determine the Q value of the action with the smallest regret value in state S'; The step of obtaining the regret value of the current action space specifically includes: i The regret value of the action with the smallest regret value; According to the regret value when agent i takes action a, the regret value of the current action space is obtained; Get the current joint action The regret value under the condition of determining the regret value step when agent i takes action a is: The current joint action The expression of the regret value under , is: The expression of the regret value when the agent i takes action a is specifically: The expression of the regret value of the current action space is specifically: Among them, a i is the action of agent i, a _i represents the actions of other agents, Q i is the state-action benefit value of the agent, A i is the set of all optional actions of agent i; The loop module is used to return to the execution state S and select the target action step using the ∈-greedy strategy until the state S is the terminal state.

6. An intelligent car cluster control system, characterized in that: The intelligent car cluster control system includes: A plurality of smart cars, each of which performs corresponding actions according to a driving instruction sent by a smart car cluster control device; An intelligent car cluster control device, the intelligent car cluster control device includes a memory, a processor and an intelligent car cluster control method program stored in the memory and executable on the processor, the intelligent car cluster control method program, when executed by the processor, implements the steps of the intelligent car cluster control method as described in any one of claims 1 to 4.

7. A storage medium, characterized in that: The storage medium stores a smart car cluster control method program, and when the smart car cluster control method program is executed by the processor, the steps of the smart car cluster control method according to any one of claims 1 to 4 are implemented.

Images