Multi-agent autonomous decision-making system based on reinforcement learning

Abstract

The invention discloses a multi-agent autonomous decision-making system based on reinforcement learning, and the system comprises an image collection module, an image processing and analysis module, a calculation module, a first intelligent judgment module, a second intelligent judgment module, a trolley information collection module, and a storage module. The calculation module can calculate the road condition in front of the running of the trolley so as to ensure that the trolley can pass normally and safely, and when the trolley encounters an obstacle, the calculation module can calculate the speed and steering information of the trolley passing through the obstacle so as to avoid the influence of the obstacle on the running of the trolley. And by using the first intelligent judgment module and the second intelligent judgment module, when the trolley encounters the obstacle next time and before the calculation module, whether the obstacle can influence the operation of the trolley or not can be judged, so that the time consumed by the calculation module for re-calculation is saved, and the judgment process is greatly accelerated.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a multi-agent autonomous decision-making system based on reinforcement learning. Background Art

[0002] In factories, in order to meet the needs of modernization and save labor costs, unmanned transport vehicles are widely used.

[0003] When the unmanned transport vehicle is implemented, it will be equipped with a camera. The computing module in the vehicle will calculate the collected images to determine whether the unmanned vehicle can pass normally. When there is an obstacle in front of the vehicle, calculation is required to determine whether the vehicle can pass normally.

[0004] In the prior art, due to the limitation of the computing speed of the computing module, the use range of the car is greatly restricted. In order to solve this problem, the present application proposes a multi-agent autonomous decision-making system based on reinforcement learning. Summary of the invention

[0005] To this end, the present application provides a multi-agent autonomous decision-making system based on reinforcement learning to solve the problem that the computing module in the prior art requires a lot of time for calculation and cannot meet the operation requirements of the vehicle.

[0006] In order to achieve the above objectives, this application provides the following technical solutions:

[0007] In the first aspect, a multi-agent autonomous decision-making system based on reinforcement learning includes an image acquisition module, an image processing and analysis module, a calculation module, a first intelligent judgment module, a second intelligent judgment module, a vehicle information acquisition module, and a storage module;

[0008] The image acquisition module is used to acquire images in front of the vehicle, and the images in front of the vehicle include road surface images and images on both sides of the road surface;

[0009] The image processing and analysis module is used to process the collected image to reduce noise in the collected image and analyze obstacle information in the image;

[0010] The calculation module is used to calculate the image collected by the image acquisition module to ensure that the car can pass safely;

[0011] The control module is used to control the speed and direction of the car.

[0012] The first intelligent judgment module is used to analyze the road surface image in front of the car. When an obstacle appears on the road surface, the first intelligent judgment module can judge the obstacle so that the car can pass the obstacle safely;

[0013] The second intelligent judgment module is used to analyze obstacles on both sides of the road in front of the car. When obstacles appear on both sides of the road, the second intelligent judgment module can judge the obstacles so that the car can pass safely;

[0014] The vehicle information collection module is used to collect the vehicle's own data when it safely passes through obstacles;

[0015] The storage module can store obstacle information and the vehicle's own data information when the calculation module calculates the image collected by the image collection module and the vehicle passes through the obstacle safely.

[0016] Preferably, the obstacles include road potholes, road bumps, obstacles on the road surface, and tunnels on the road surface.

[0017] Preferably, the vehicle's own data includes the vehicle speed information, the wheel size of the vehicle, the centrifugal force of the vehicle and the size of the vehicle itself.

[0018] Preferably, after the image processing and analysis module processes and analyzes the collected image, the obstacle information is obtained, and the obstacle information includes one or more of the height, width, depth and distance of the obstacle. Each obstacle has a set of obstacle information, and each set of obstacle information is marked as T1, T2, T3, ..., Tn in sequence. Each set of obstacle information corresponds to a set of vehicle operation information, and each set of vehicle operation information is marked as M1, M2, M3, ..., Mn, and T1 corresponds to M1, T2 corresponds to M2, and Tn corresponds to Mn. The storage module stores Tn and Mn.

[0019] Preferably, the output end of the calculation module is connected to the input end of the control module. After the calculation module calculates the information of the obstacle, the information of the obstacle is Tx, and a control signal is sent to the control module to change the running state of the car so that the car can pass the obstacle safely. The running information of the car is Mx, and x is any value from 1 to n.

[0020] Preferably, the output end of the first intelligent judgment module is connected to the input end of the control module;

[0021] When the obstacle is a pothole or a bump on the road surface, the image processing and analysis module analyzes the pothole or bump information to obtain the pothole or bump information Ty1, and the first intelligent judgment module can compare Ty1 with any stored information Tz1 in the storage module;

[0022] When Ty1 is less than or equal to Tz1, the first intelligent judgment module can extract the information of Mz1 corresponding to Tz1 in the storage module, and send Mz1 to the control module, so that the car runs according to Mz1;

[0023] When Ty1 is greater than Tz1, Ty1 enters the calculation module, and the calculation module calculates so that the car can pass through the pit safely.

[0024] Preferably, the output end of the second intelligent judgment module is connected to the input end of the control module;

[0025] When the obstacle is a roadblock on both sides of the road, the image processing and analysis module analyzes the roadblock information to obtain roadblock information Ty2, and the first intelligent judgment module can compare Ty2 with any stored information Tz2 in the storage module;

[0026] When Ty2 is less than or equal to Tz2, the first intelligent judgment module can extract the information of Mz2 corresponding to Tz2 in the storage module, and send Mz2 to the control module, so that the car runs according to Mz2;

[0027] When Ty2 is greater than Tz2, Ty2 enters the calculation module, and the calculation module calculates so that the car can safely pass the roadblock.

[0028] Preferably, it also includes an alarm module, which can issue an alarm message when the running information of the vehicle passing through the obstacle is greater than the running information when passing safely.

[0029] Preferably, a speaker module is also included, and the speaker module can emit a warning sound when the vehicle is running.

[0030] Compared with the prior art, this application has at least the following beneficial effects:

[0031] 1. When this system is implemented, the calculation module can calculate the road conditions in front of the car to ensure that the car can pass normally and safely. When encountering obstacles, the calculation module can calculate the speed and steering information of the car passing the obstacle, thereby avoiding the obstacle from affecting the operation of the car. The use of the first intelligent judgment module and the second intelligent judgment module can determine whether the obstacle will affect the operation of the car before the calculation module when the car encounters an obstacle next time, thereby saving the time spent on the calculation module to recalculate and greatly speeding up the judgment process;

[0032] 2. With the continuous use of the car, more obstacle information is stored in the storage module, and the frequency of use of the computing module will be lower, which greatly reduces the computing pressure of the computing module and weakens the demand for high-performance computing modules. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more intuitively illustrate the prior art and the present application, exemplary drawings are given below. It should be understood that the specific shapes and structures shown in the drawings should not generally be regarded as limiting conditions for implementing the present application; for example, those skilled in the art are capable of easily making conventional adjustments or further optimizations to the addition / reduction / attribution division, specific shapes, positional relationships, connection methods, dimensional ratios, etc. of certain units (components) based on the technical concepts and exemplary drawings disclosed in the present application.

[0034] Figure 1 A module diagram of a multi-agent autonomous decision-making system based on reinforcement learning provided in Example 1 of the present application. DETAILED DESCRIPTION

[0035] The present application is further described below in detail through specific embodiments in conjunction with the accompanying drawings.

[0036] A multi-agent autonomous decision-making system based on reinforcement learning, comprising an image acquisition module, an image processing and analysis module, a calculation module, a first intelligent judgment module, a second intelligent judgment module, a vehicle information acquisition module and a storage module;

[0037] The image acquisition module: The image acquisition module is used to acquire images in front of the vehicle, and the images in front of the vehicle include road surface images and images on both sides of the road surface;

[0038] The image processing and analysis module: The image processing and analysis module is used to process the collected image to reduce the noise in the collected image and analyze the obstacle information in the image. When the image processing module is implemented, it can be a camera device to collect road surface images;

[0039] The calculation module: The calculation module is used to calculate the images collected by the image acquisition module to ensure that the car can pass safely. After the image is collected, when an obstacle appears on the road, the calculation module can judge the obstacle according to the running state of the car and its own state to determine whether the obstacle will affect the operation of the car;

[0040] The control module: The control module is used to control the speed of the car and the direction of the car. When the car is running on the road, the running speed of the car, whether the car turns, and the start and stop of the car are all controlled by the control module;

[0041] The first intelligent judgment module: The first intelligent judgment module is used to analyze the road surface image in front of the car. When an obstacle appears on the road surface, the first intelligent judgment module can judge the obstacle so that the car can pass the obstacle safely;

[0042] The second intelligent judgment module: The second intelligent judgment module is used to analyze obstacles on both sides of the road in front of the car. When obstacles appear on both sides of the road, the second intelligent judgment module can judge the obstacles so that the car can pass safely;

[0043] The vehicle information collection module: The vehicle information collection module is used to collect the vehicle's own data when it safely passes through obstacles;

[0044] The storage module: When the calculation module calculates the image collected by the image acquisition module and the car passes through the obstacle safely, the storage module can store the obstacle information and the car's own data information.

[0045] When this system is implemented, the calculation module can calculate the road conditions in front of the car to ensure that the car can pass normally and safely. When encountering obstacles, the calculation module can calculate the speed and steering information of the car passing the obstacle, thereby avoiding the obstacle from affecting the operation of the car. The use of the first intelligent judgment module and the second intelligent judgment module can determine whether the obstacle will affect the operation of the car before the calculation module when the car encounters an obstacle next time, thereby saving the time spent on the calculation module to recalculate and greatly speeding up the judgment process.

[0046] And with the continuous use of the car, more obstacle information is stored in the storage module, and the frequency of use of the computing module will be lower, which greatly reduces the computing pressure of the computing module and weakens the demand for high-performance computing modules.

[0047] The obstacles include road potholes or road bumps or obstacles on the road surface or tunnels on the road surface or other obstacles on the road surface.

[0048] The car's own data includes the car's speed information, the size of the car's wheels, the centrifugal force when the car passes through a curve, the car's own weight, and the car's own size.

[0049] After the image processing and analysis module processes and analyzes the collected image, the obstacle information is obtained, and the obstacle information includes one or more of the height, width, depth, and distance of the obstacle. Each obstacle has a set of obstacle information, and each set of obstacle information is marked as T1, T2, T3, ..., Tn in sequence. Each set of obstacle information corresponds to a set of vehicle operation information, and each set of vehicle operation information is marked as M1, M2, M3, ..., Mn, and T1 corresponds to M1, T2 corresponds to M2, and Tn corresponds to Mn. The storage module stores Tn and Mn.

[0050] When the car encounters an obstacle for the first time, the obstacle information is marked as T1, and the operation information of the car safely passing through the obstacle is marked as M1, and so on, and the subsequent information is marked.

[0051] In some embodiments, the computing module directly calculates the obstacle information so that the car can pass the obstacle safely.

[0052] The output end of the calculation module is connected to the input end of the control module. After the calculation module calculates the information of the obstacle, the information of the obstacle is Tx, and then sends a control signal to the control module to change the running state of the car so that the car can safely pass the obstacle. The running information of the car is Mx, and x is any value from 1 to n. The storage module can store Tx and Mx.

[0053] In some embodiments, the output end of the first intelligent judgment module is connected to the input end of the control module; in this embodiment, the first intelligent judgment module can judge the obstacle information and issue a control instruction to the control module, thereby enabling the car to pass the obstacle safely.

[0054] When the obstacle is a pothole or a bump on the road surface, the image processing and analysis module analyzes the pothole or bump information to obtain the pothole or bump information Ty1, and the first intelligent judgment module can compare Ty1 with any stored information Tz1 in the storage module;

[0055] When Ty1 is less than or equal to Tz1, the first intelligent judgment module can extract the information of Mz1 corresponding to Tz1 in the storage module, and send Mz1 to the control module, so that the car runs according to Mz1. At this time, the calculation module no longer participates in the operation of the system, thereby speeding up the reaction of the car, so that the car can have more time to change its own operating state and better avoid obstacles.

[0056] When Ty1 is greater than Tz1, Ty1 enters the calculation module, and the calculation module calculates so that the car can pass through the pit safely.

[0057] In some embodiments, the output terminal of the second intelligent determination module is connected to the input terminal of the control module;

[0058] When the obstacle is a roadblock on both sides of the road, the image processing and analysis module analyzes the roadblock information to obtain roadblock information Ty2, and the first intelligent judgment module can compare Ty2 with any stored information Tz2 in the storage module;

[0059] When Ty2 is less than or equal to Tz2, the first intelligent judgment module can extract the information of Mz2 corresponding to Tz2 in the storage module, and send Mz2 to the control module, so that the car runs according to Mz2;

[0060] When Ty2 is greater than Tz2, Ty2 enters the calculation module, and the calculation module calculates so that the car can safely pass the roadblock.

[0061] It also includes an alarm module, which can send out an alarm message when the running information of the vehicle passing through the obstacle is greater than the running information when passing safely.

[0062] It also includes a speaker module, which can emit a warning sound when the car is running.

[0063] The technical features of the above embodiments may be arbitrarily combined (as long as there is no contradiction in the combination of these technical features). To make the description concise, not all possible combinations of the technical features in the above embodiments are described; these embodiments that are not explicitly written should also be considered to be within the scope of this specification.

Claims

1. A multi-agent autonomous decision-making system based on reinforcement learning, characterized in that: It includes an image acquisition module, an image processing and analysis module, a calculation module, a control module, a first intelligent judgment module, a second intelligent judgment module, a vehicle information acquisition module and a storage module; The image acquisition module is used to acquire images in front of the vehicle, and the images in front of the vehicle include road surface images and images on both sides of the road surface; The image processing and analysis module is used to process the collected image to reduce noise in the collected image and analyze obstacle information in the image; The calculation module is used to calculate the image collected by the image acquisition module to ensure that the car can pass safely; The control module is used to control the speed of the car and the direction of the car; The first intelligent judgment module is used to analyze the road surface image in front of the car. When an obstacle appears on the road surface, the first intelligent judgment module can judge the obstacle so that the car can pass the obstacle safely. The second intelligent judgment module is used to analyze obstacles on both sides of the road in front of the car. When obstacles appear on both sides of the road, the second intelligent judgment module can judge the obstacles so that the car can pass safely; The vehicle information collection module is used to collect the vehicle's own data when it safely passes through obstacles; The storage module can store obstacle information and the vehicle's own data information when the calculation module calculates the image collected by the image collection module and the vehicle passes through the obstacle safely.

2. According to the multi-agent autonomous decision-making system based on reinforcement learning in claim 1, it is characterized in that: The obstacles include road potholes, road bumps, obstacles on the road surface, and tunnels on the road surface.

3. The multi-agent autonomous decision-making system based on reinforcement learning according to claim 1, characterized in that: The vehicle's own data includes the vehicle's speed information, the vehicle's wheel size, the vehicle's centrifugal force, and the vehicle's own size.

4. The multi-agent autonomous decision-making system based on reinforcement learning according to claim 1, characterized in that: After the image processing and analysis module processes and analyzes the collected image, the obstacle information is obtained, and the obstacle information includes one or more of the height, width, depth, and distance of the obstacle. Each obstacle has a set of obstacle information, and each set of obstacle information is marked as T1, T2, T3, ..., Tn in sequence. Each set of obstacle information corresponds to a set of vehicle operation information, and each set of vehicle operation information is marked as M1, M2, M3, ..., Mn, and T1 corresponds to M1, T2 corresponds to M2, and Tn corresponds to Mn. The storage module stores Tn and Mn.

5. The multi-agent autonomous decision-making system based on reinforcement learning according to claim 4, characterized in that: The output end of the calculation module is connected to the input end of the control module. After the calculation module calculates the information of the obstacle, the information of the obstacle is Tx, and a control signal is sent to the control module to change the running state of the car so that the car can pass the obstacle safely. The running information of the car is Mx, and x is any value from 1 to n.

6. The multi-agent autonomous decision-making system based on reinforcement learning according to claim 5, characterized in that: The output end of the first intelligent judgment module is connected to the input end of the control module; When the obstacle is a pothole or a bump on the road surface, the image processing and analysis module analyzes the pothole or bump information to obtain the pothole or bump information Ty1, and the first intelligent judgment module can compare Ty1 with any stored information Tz1 in the storage module; When Ty1 is less than or equal to Tz1, the first intelligent judgment module can extract the information of Mz1 corresponding to Tz1 in the storage module, and send Mz1 to the control module, so that the car runs according to Mz1; When Ty1 is greater than Tz1, Ty1 enters the calculation module, and the calculation module calculates so that the car can pass through the pit safely.

7. The multi-agent autonomous decision-making system based on reinforcement learning according to claim 5, characterized in that: The output end of the second intelligent judgment module is connected to the input end of the control module; When the obstacle is a roadblock on both sides of the road, the image processing and analysis module analyzes the roadblock information to obtain roadblock information Ty2, and the first intelligent judgment module can compare Ty2 with any stored information Tz2 in the storage module; When Ty2 is less than or equal to Tz2, the first intelligent judgment module can extract the information of Mz2 corresponding to Tz2 in the storage module, and send Mz2 to the control module, so that the car runs according to Mz2; When Ty2 is greater than Tz2, Ty2 enters the calculation module, and the calculation module calculates so that the car can safely pass the roadblock.

8. The multi-agent autonomous decision-making system based on reinforcement learning according to claim 1, characterized in that: It also includes an alarm module, which can send out an alarm message when the running information of the vehicle passing through the obstacle is greater than the running information when passing safely.

9. The multi-agent autonomous decision-making system based on reinforcement learning according to claim 1, characterized in that: It also includes a speaker module, which can emit a warning sound when the car is running.

Images