Learning system, mobile body autonomous operation system, learning method, and mobile body autonomous operation method
A learning system employing reinforcement learning addresses the challenge of navigating rough terrain by generating a control model that ensures safe and efficient travel routes for construction machines, adapting to terrain changes and obstacles.
Patent Information
- Application Number
- JP2023071324
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-04-25
- Publication Date
- 2024-11-07
AI Technical Summary
Maneuvering construction machines on rough terrain is difficult, and existing autonomous systems lack the ability to select safe and efficient travel routes, posing a risk of accidents such as overturning.
A learning system that uses reinforcement learning to determine travel routes and driving controls based on three-dimensional point cloud data, distance measurements, and reference points, rewarding safe arrival and penalizing overturning or collisions to generate a control model for autonomous driving on uneven terrain.
The system generates a control model enabling mobile objects to safely and efficiently navigate rough terrain, adapting to obstacles and terrain changes, thereby enhancing autonomous driving capabilities.
Smart Images

Figure 2024157158000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a learning system for a moving body, a mobile body autonomous driving system, a learning method, and a mobile body autonomous driving method. [Background technology]
[0002] When performing excavation or construction work at a construction site, a caterpillar-type backhoe is generally used as an off-road vehicle operated by a skilled technician. In recent years, autonomous backhoes equipped with an autonomous driving system that excavates soil accumulated in a soil pit and loads it onto a dump truck have also been used. Such autonomous construction machinery may travel on uneven ground. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2022-145767 A Summary of the Invention [Problem to be solved by the invention]
[0004] However, it is difficult to control a construction machine to travel on rough ground, and if the machine loses balance, it may tip over and cause an accident. When driving a construction machine on rough ground, the choice of route is left to the operator's empirical judgment, and there is room for consideration of a safe and efficient route. In addition, the backhoe's autonomous driving system can perform tasks such as digging up soil and loading it onto dump trucks, but it is desirable for the system to not only perform such tasks but also be able to autonomously navigate uneven terrain on the way to the location where the work will be performed.
[0005] The present invention has been made in consideration of the above circumstances, and its purpose is to provide a learning system, a mobile body autonomous driving system, a learning method, and a mobile body autonomous driving method that are capable of generating a control model enabling a mobile body to drive autonomously on rough terrain. [Means for solving the problem]
[0006] In order to solve the above-mentioned problems, one aspect of the present invention is a learning system that determines the movement path and driving control content of a moving body traveling in an area that includes rough terrain between a starting point and a destination, and has a learning unit that learns the relationship between three-dimensional point cloud data in which the unevenness of the area is measured, measurement data in which the distance from a measurement position in the area to surrounding objects is measured, environmental data representing self-position data that represents the position of the moving body based on a reference point set in the area, and behavioral data that represents the wheel speed and wheel direction of the moving body moving along the surface of the area, and the learning unit is a learning system that performs reinforcement learning in which a high reward is given when the moving body moves from the starting point based on the behavioral data and arrives at the destination, and a low reward is given when the moving body rolls over or collides during the movement.
[0007] Moreover, one aspect of the present invention is a learning method for determining the movement path and driving control content of a moving body traveling in an area including rough terrain between a starting point and a destination, in which a learning unit learns the relationship between environmental data representing three-dimensional point cloud data in which unevenness of the area is measured, measurement data in which the distance from a measurement position in the area to surrounding objects is measured, self-position data representing the position of the moving body based on a reference point set in the area, and behavioral data representing the wheel speed and wheel direction of the moving body moving along the surface of the area, and the learning includes performing reinforcement learning in which a high reward is given when the moving body moves from the starting point based on the behavioral data and arrives at the destination, and a low reward is given when the moving body rolls over or collides during the movement. Effect of the Invention
[0008] As described above, according to the present invention, it is possible to generate a control model that enables autonomous driving of a mobile object on rough terrain. [Brief description of the drawings]
[0009] [Figure 1] 1 is a schematic block diagram showing the configuration of a learning system S according to an embodiment of the present invention. [Diagram 2] 1 is a diagram illustrating the processing flow of the learning system S. [Figure 3A] FIG. 2 is a diagram illustrating an example of three-dimensional point cloud data. [Figure 3B] FIG. 2 is a diagram illustrating an example of three-dimensional point cloud data. [Figure 4] FIG. 13 is a diagram illustrating an example of a height matrix. [Diagram 5] FIG. 13 is a diagram illustrating an example of normalization. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] A learning system according to an embodiment of the present invention will now be described with reference to the drawings. FIG. 1 is a schematic block diagram showing the configuration of a learning system S according to an embodiment of the present invention. The learning system S includes a topography measurement unit 11, a three-dimensional point cloud data acquisition unit 12, a distance measurement unit 13, a distance data acquisition unit 14, a preprocessing unit 15, a simulation unit 16, a learning unit 17, and a memory unit 18. The learning system S may be realized by a single computer, or the functions of the learning system S may be distributed to multiple computers that work together to perform processing. Also, at least some of the functions of the learning system S may be installed on a computer provided on the cloud, such as a cloud server.
[0011] The learning system S determines the travel route and driving control content of a mobile object traveling in an area including rough terrain between a departure point and a destination. The topography measuring unit 11 measures the topography of the area between the starting point and the destination. At least a part of this area includes uneven ground. The area is, for example, a construction site. The starting point and the destination are the starting point from which the moving body moves to perform excavation work and construction work, and the destination is the end point. The moving body is a vehicle used at the construction site, and is, for example, a caterpillar-type backhoe as an uneven terrain traveling vehicle. The moving body may also be a construction machine such as a dump truck. Such a moving body may be operated by a skilled engineer, but has a function capable of autonomous driving and can move by autonomous driving within the construction site. The traveling surface of the uneven ground has unevenness.
[0012] The terrain measurement unit 11 measures the terrain of an area at a construction site. The terrain measurement unit 11 is, for example, a LiDAR (Light Detection And Ranging) device mounted on an unmanned aerial vehicle such as a drone, measures the unevenness of the terrain, and generates three-dimensional point cloud data representing the terrain. The three-dimensional point cloud data acquisition unit 12 acquires the three-dimensional point cloud data from the topography measurement unit 11. Distance measurement unit 13 measures the distance from the moving body to surrounding objects. Distance measurement unit 13 is, for example, a LiDAR, is mounted on the moving body, and measures a measurement area on the front side in the moving direction of the moving body. The distance data acquisition unit 14 acquires the measurement results from the distance measurement unit 13 .
[0013] The pre-processing unit 15 receives the three-dimensional point cloud data acquired by the three-dimensional point cloud data acquiring unit 12 and the distance data acquired by the distance data acquiring unit 14, and performs pre-processing. The simulation unit 16 simulates the behavior of a moving object when it travels on a terrain represented by three-dimensional point cloud data. Here, the simulation unit 16 simulates various behaviors of the moving object, such as movement, tipping, and collision, on rough terrain. The learning unit 17 performs learning using the data that has been preprocessed by the preprocessing unit 15. The learning is reinforcement learning.
[0014] The learning unit 17 learns the relationship between the three-dimensional point cloud data in which the unevenness of the area is measured, the measurement data in which the distance from the measurement position in the area to the surrounding objects is measured, the environment data representing the self-position data representing the position of the mobile object based on the reference point set in the area, and the behavior data representing the rotation speed and the wheel direction of the mobile object moving along the surface of the area. The learning unit 17 performs reinforcement learning to give a high reward when the mobile object moves from the departure point based on the behavior data and arrives at the destination, and gives a low reward when the mobile object rolls over or collides during the movement.
[0015] The storage unit 18 stores various data. For example, the storage unit 18 stores the three-dimensional point cloud data acquired by the three-dimensional point cloud data acquisition unit 12, the distance data acquired by the distance data acquisition unit 14, the result of preprocessing by the preprocessing unit 15, data necessary for the simulation unit 16 to perform a simulation, data necessary for the learning unit 17 to learn, a learned model generated by the learning unit 17 through learning, and the like.
[0016] The memory unit 18 is configured by a storage medium, for example, a hard disk drive (HDD), a flash memory, an electrically erasable programmable read only memory (EEPROM), a random access read / write memory (RAM), a read only memory (ROM), or any combination of these storage media. The storage unit 18 may be, for example, a non-volatile memory.
[0017] In the learning system S, the 3D point cloud data acquisition unit 12, distance data acquisition unit 14, pre-processing unit 15, simulation unit 16, and learning unit 17 may be composed of a processing unit such as a CPU (central processing unit) or a dedicated electronic circuit.
[0018] In the learning system S of this embodiment, when searching for routes and performing autonomous control on rough terrain, the system does not use route search algorithms such as the A* algorithm or autonomous control of driving using conditional branching. Instead, the system uses reinforcement learning to generate routes on rough terrain and repeat trial and error in driving, thereby learning safe and efficient methods of generating routes and controlling driving, and creating a control model.
[0019] FIG. 2 is a diagram for explaining the flow of processing in the learning system S. As shown in FIG. (Step S101) The topography of the area at the construction site is input as data representing the environment. Here, the topography is measured by the topography measurement unit 11, three-dimensional point cloud data is generated, and the three-dimensional point cloud data acquisition unit 12 acquires the three-dimensional point cloud data. 3A and 3B are diagrams showing an example of three-dimensional point cloud data. As shown in FIGS. 3A and 3B, the three-dimensional point cloud data represents the shape of the ground of an uneven terrain. Here, 3D point cloud data may be obtained by measuring actual uneven terrain (Fig. 3A). Also, using multiple types of terrain rather than just one increases the number of trials and allows control even when the terrain changes (Fig. 3B). For example, multiple 3D point cloud data representing uneven terrain different from the acquired 3D point cloud data may be generated. This allows an environment to be obtained in which construction machinery can be placed in random positions in parallel on multiple different uneven terrains to simulate the behavior of the construction machinery.
[0020] Also, here, distance data (angle and distance) to the surface of a surrounding object as viewed from the current position of the construction machine may be obtained by measuring distance data by the distance measuring unit 13 while actually running the construction machine on the terrain where the 3D point cloud data has been measured. Here, distance data is obtained in various directions from the construction machine. In addition, for the terrain represented by the 3D point cloud data generated by a computer as multiple different rough terrains, distance data may be obtained by placing construction machinery in a simulation and measuring the distance using the distance measuring unit 13.
[0021] Next, when data representing the environment, consisting of behavioral information to be controlled during learning, state information for recognizing surrounding information, and reward information for behavior, is input to the learning system S, it learns a control method that maximizes future rewards.
[0022] (S102) Next, the learning system S sets the state of the construction machine by inputting state information. Here, the preprocessing unit 15 divides the plane into a grid for the uneven ground indicated by the three-dimensional point cloud data, and generates a matrix based on the point cloud data at positions corresponding to the grid. FIG. 4 is a diagram illustrating an example of a height matrix. Image 400 is a diagram showing the three-dimensional point cloud data as seen from above, and is expressed in different colors according to the height at positions corresponding to the vertical and horizontal directions. A moving object 401 is placed in image 400. A grid is set on image 400, and multiple points (e.g., point 402, etc.) where the vertical and horizontal directions of the grid intersect are set. The grid interval may be adjusted according to the degree of convergence of learning. Values representing heights are expressed along the vertical and horizontal directions of such an image 400, and a height matrix 410 is generated based on these values. For example, the horizontal and vertical directions of a grid in image 400 are set as rows and columns, and the height values at positions (multiple points) corresponding to the grid are set as matrix values. This makes it possible to generate a height matrix (data in which values representing height are arranged like pixels in an image) that represents the height at each position in the coordinate matrix plane direction. This makes it possible to obtain data that can grasp information about the unevenness of the uneven ground around the construction machine. In addition, since data is thinned out from the 3D point cloud data in a grid pattern, the topographical data can be downsampled, and further, by indicating the horizontal position of the point where the height is measured by the row and column in which the height values are arranged, the information space of two axes, the x-axis and the y-axis, can be omitted, thereby simplifying the state data. Note that the pre-processing unit 15 may generate a matrix in which the x-axis direction of the plane in the 3D point cloud data is the row, the y-axis direction is the column, and the height values are the components. In addition, the pre-processing unit 15 may generate a matrix in which the y-axis direction of the plane in the 3D point cloud data is the row, the x-axis direction is the column, and the height values are the components.
[0023] The pre-processing unit 15 also adds data that enables the construction machine to recognize its own position on the terrain. Here, the pre-processing unit 15 generates a terrain model by adding a rectangular parallelepiped model that inscribes the downsampled terrain, and normalizes the generated terrain model. FIG. 5 is a diagram illustrating an example of normalization. A terrain model is data that represents the position of a terrain using three-dimensional coordinates (x, y, z). The pre-processing unit 15 generates the terrain model C2 by normalizing the terrain model C1 before normalization. For example, this normalization is performed by setting the coordinates of one of the vertices in the terrain model C1 as the origin, and normalizing the coordinates of other vertices by setting them to 0 or 1 according to the position of the coordinates. By normalizing, even if the area size (vertical and horizontal directions) and the height difference (height direction) of the unevenness in the construction site vary widely, they can be easily handled as a learning subject for the learning system S.
[0024] After normalization, the simulation unit 16 generates a self-position vector. Here, the construction machine is placed at a position corresponding to the starting point with respect to the normalized terrain model C2, and a vector toward the current position of the construction machine is generated as the self-position vector, based on the origin of the terrain model C2. When the action step is entered, the construction machine moves, and a new self-position vector is generated in response to the change in the current position of the construction machine. The simulation unit 16 also measures the distance from the construction machine to the surface of the surrounding object. This measurement is performed in a simulation.
[0025] Next, the simulation unit 16 takes in the wheel rotation speed and wheel direction given as the action (control value), and executes a simulation of the moving body moving on rough ground based on the environment prepared in step S101 and the state set in step S102. Here, the simulation unit 16 obtains a vector from the wheel to the center of gravity of the moving body based on the environment and the attitude of the moving body, and also simulates whether or not the moving body will tip over. For example, the simulation unit 16 identifies the position of the moving body in the three-dimensional point cloud data, obtains the slope of the terrain at that position, and obtains the attitude of the moving body (direction and slope of the moving body) when the moving body is placed in a situation with that slope. Then, the simulation unit 16 obtains a vector from the wheel of the moving body to the center of gravity of the moving body based on the attitude, and determines whether or not the combination of the direction and value of the vector exceeds a predetermined reference value, and if it does, it may be determined that the moving body has rolled over. Also, the simulation unit 16 may determine that the moving body cannot run if it has rolled over. This makes it possible to determine whether or not the moving body can travel, based on whether or not it will roll over, depending on the environment in which the moving body is placed and the conditions (the control value given as an action and the travel route determined based on the history of the control values).
[0026] Based on the distance data, the simulation unit 16 determines whether the distance represented by the distance data is equal to or less than a predetermined distance reference value, with the current position of the moving object being used as a reference in the environment set in step S101, thereby determining whether the moving object has collided with an obstacle. The distance reference value may be 0, or may be a value corresponding to a distance that can be regarded as a collision. The simulation unit 16 may be configured to determine that the moving object cannot travel if a collision occurs. This makes it possible to determine whether or not the moving body can travel, based on whether or not it will collide, depending on the environment in which the moving body is placed and the conditions (the control value given as an action and the travel route determined based on the history of the control values).
[0027] (Step S103) The learning unit 17 gives a reward based on the result of the simulation performed by the simulation unit 16. Here, the learning unit 17 gives a positive reward as reward information when the excavation point (destination) is reached, and gives a negative reward when a rollover or collision occurs. Also, the learning unit 17 gives a more negative reward (lower reward) as the elapsed time from the start of movement from the departure point becomes longer. This allows the system to learn that the route, wheel rotation speed, and wheel orientation that will reach the destination are correct, and that the route, wheel rotation speed, and wheel orientation that will result in a rollover or collision are not correct. In addition, the longer the elapsed time from the start of travel from the departure point, the more negative the reward (lower the reward) is given. Therefore, the shorter the travel time from the departure point to the destination (drilling point), the less likely the reward will decrease, making it possible to obtain a route to the destination in a shorter time, thereby improving efficiency in travel.
[0028] (Step S104) The learning unit 17 updates the action plan (control model) based on the reward given in step S103 so as to maximize the reward to be obtained in the future.
[0029] (Step S105) The learning unit 17 provides the rotational speed and direction of the wheels of the vehicle as control values based on the updated control model.
[0030] Thereby, for the environment given in step S101, the simulation unit 16 performs a simulation by taking in the new control values given in step S105 as state data in step S102. Then, the learning unit 17 gives a reward based on the simulation result (step S103).
[0031] By repeatedly executing such processing, it is possible to repeat reinforcement learning in the simulation, which gives rewards (points) according to the results of the actions of the construction machine, based on the relationship between the environmental data representing the environment in which the construction machine is placed (unevenness information of the uneven ground = height matrix, self-position vector, distance to surrounding objects) and the actions taken by the construction machine (wheel rotation speed, wheel direction). This makes it possible to generate a control model in which a route is generated for the construction machine to move more safely through the uneven ground to the excavation site. This control model can also be used as a trained model.
[0032] Furthermore, according to the above-described embodiment, a control model that can handle the occurrence of obstacles and changes in the layout can be obtained.
[0033] Furthermore, according to the above-described embodiment, in addition to the route search method in which the unevenness of uneven terrain is replaced with weights, it is possible to consider combining this with vehicle driving control, such as whether the vehicle can actually pass through without rolling over or colliding.
[0034] In addition, the control model (trained model) obtained by learning using the above-mentioned learning system S may be used to control the operation of construction machinery operating at an actual construction site. This makes it possible to realize an autonomous driving system that autonomously drives a moving object at an actual construction site. In addition, the learning system S and the autonomous driving system may be separate systems, or the learning system S may be incorporated into the autonomous driving system.
[0035] The learning system S in the above-mentioned embodiment may be realized by a computer. In that case, a program for realizing this function may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to realize the function. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into a computer system. The term "computer-readable recording medium" may also include a medium that dynamically holds a program for a short period of time, such as a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, and a medium that holds a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in that case. The above-mentioned program may be a program for realizing a part of the above-mentioned function, or may be a program that can realize the above-mentioned function in combination with a program already recorded in the computer system, or may be a program that is realized using a programmable logic device such as an FPGA (Field Programmable Gate Array).
[0036] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and designs that do not deviate from the gist of the present invention are also included. [Explanation of symbols]
[0037] S... learning system, 11... topography measurement unit, 12... 3D point cloud data acquisition unit, 13... distance measurement unit, 14... distance data acquisition unit, 15... pre-processing unit, 16... simulation unit, 17... learning unit, 18... storage unit
Claims
1. A learning system for determining a travel route and driving control content of a moving object traveling in an area including rough terrain between a departure point and a destination, comprising: environmental data representing three-dimensional point cloud data in which the unevenness of the region is measured, measurement data in which distances from measurement positions in the region to surrounding objects are measured, and self-position data indicative of the position of a moving object based on a reference point set in the region; Action data representing a wheel speed and a wheel direction of a moving object moving along a surface of the area; A learning unit that learns the relationship between The learning unit is A reinforcement learning is performed in which a high reward is given when the moving object moves from the departure point based on the behavioral data and arrives at the destination, and a low reward is given when the moving object rolls over or collides during the movement. Learning system.
2. The learning unit gives a lower reward as the time that has elapsed since the moving object departed from the departure point increases. The learning system according to claim 1 .
3. The method further comprises: dividing a plane into a grid shape for information indicated by the three-dimensional point cloud data; expressing a first direction of the plane as rows, expressing a second direction of the plane perpendicular to the first direction as columns, and generating a matrix having height values as components, based on point cloud data at positions corresponding to the grid. A learning system according to claim 1 or 2.
4. The preprocessing unit performs a process of normalizing a terrain model based on the three-dimensional point cloud data. The learning system according to claim 3 .
5. An autonomous driving system for a mobile object, which controls the mobile object using a trained model trained by the learning system described in claim 1 to drive the mobile object autonomously.
6. A learning method for determining a moving route and driving control content of a moving object traveling in an area including rough terrain between a starting point and a destination, comprising: The learning department, environmental data representing three-dimensional point cloud data in which the unevenness of the region is measured, measurement data in which distances from measurement positions in the region to surrounding objects are measured, and self-position data indicative of the position of a moving object based on a reference point set in the region; Action data representing a wheel speed and a wheel direction of a moving object moving along a surface of the area; Learn the relationship between The learning includes: A reinforcement learning is performed to give a high reward when the moving body moves from the departure point based on the behavior data and arrives at the destination, and to give a low reward when the moving body rolls over or hits a collision during the movement. Learning methods including:
7. The trained model trained by the training method of claim 6 is used to control a moving object to drive autonomously. A method for autonomously driving a mobile object.
Citation Information
Patent Citations
Construction machine, construction machine control device, and construction machine control method
JP2022145767A
