Work vehicle control system and work vehicle control method

Abstract

To control excavation by suppressing influence of reaction forces from a work object.SOLUTION: A determination unit determines the drive amount of a work implement based on work vehicle data including topographical data relating to the topography around a work vehicle, attitude data of the work vehicle, and work implement load data relating to the load on the work implement.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a control system and a control method for a work vehicle. [Background technology]

[0002] Patent Document 1 discloses a work vehicle that causes the cutting edge of a blade to follow a design surface. According to the technology disclosed in Patent Document 1, a control device identifies the position of the work vehicle using a Global Navigation Satellite System (GNSS), and determines a target height for the blade based on the identified position. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2015 / 083469 Summary of the Invention [Problem to be solved by the invention]

[0004] Work vehicles sometimes travel forward while performing work. The work vehicle travels under the influence of a reaction force from the work object. Depending on the geology of the work object, this reaction force may cause the work vehicle to stall or slip. Regardless of whether stall or slip occurs, if the blade cutting edge is forced to follow the design surface faithfully, the work vehicle's speed may decrease, which could reduce work efficiency. An object of the present disclosure is to provide a control system and a control method for a work vehicle that are capable of controlling excavation while suppressing the influence of reaction forces from a work object. [Means for solving the problem]

[0005] According to one aspect of the present disclosure, a control system for a work vehicle is a control system for a work vehicle that includes a vehicle body and a work implement supported on the vehicle body and used to excavate soil and gravel, and includes a determination unit that determines the drive amount of the work implement based on work vehicle data that includes topographical data relating to the topography around the work vehicle, attitude data of the work vehicle, and work implement load data relating to the load on the work implement. [Effects of the Invention]

[0006] According to the above aspect, the control system for the work vehicle can control excavation while suppressing the influence of reaction forces from the work object. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a side view of a work vehicle according to a first embodiment. [Figure 2] FIG. 2 is a diagram showing the internal configuration of a driver's cab according to the first embodiment. [Figure 3] 1 is a schematic diagram showing a power system of a work vehicle according to a first embodiment. [Figure 4] 1 is a diagram showing the configuration of a measurement system and a control device of a work vehicle according to a first embodiment. [Figure 5] FIG. 1 is a diagram illustrating a configuration of a learning device according to a first embodiment. [Figure 6] 4 is a flowchart showing a control model learning process performed by the learning device according to the first embodiment. [Figure 7] 4 is a flowchart showing a position output method according to the first embodiment. [Figure 8] FIG. 6 is a diagram showing the configuration of a control device for a work vehicle according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] First Embodiment DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The following describes the embodiments in detail with reference to the drawings: Fig. 1 is a side view of a work vehicle according to a first embodiment. The work vehicle 100 according to the first embodiment is, for example, a bulldozer. The work vehicle 100 includes a vehicle body 110, a traveling device 120, a work implement 130, and a driver's cab 140. The work vehicle 100 according to the first embodiment has an automatic control function for the work implement 130. When the automatic control function is enabled, when the operator drives the vehicle body, the work vehicle 100 automatically drives the work implement 130 to excavate earth and sand.

[0009] The traveling device 120 is provided on the bottom of the vehicle body 110. The traveling device 120 has a pair of crawlers 121 and sprockets 122. The pair of crawlers 121 are provided on the left and right sides of the vehicle body 110. The crawlers 121 are rotated by the drive of the sprockets 122, causing the work vehicle 100 to travel.

[0010] The work machine 130 is used to excavate and transport excavation targets such as earth and sand. The work machine 130 includes a lift frame 131, a blade 132, a blade lift cylinder 133, a ripper 134, and a ripper lift cylinder 135. The blade 132 is disposed in front of the vehicle body 110. The ripper 134 is disposed in the rear of the vehicle body 110.

[0011] The base end of the lift frame 131 is attached to the side of the vehicle body 110 via a pin extending in the vehicle width direction. The tip end of the lift frame 131 is attached to the back surface of the blade 132 via a ball joint. This supports the blade 132 so that it can move up and down relative to the vehicle body 110. A cutting edge is provided at the lower end of the blade 132. The blade lift cylinder 133 is a hydraulic cylinder. The base end of the blade lift cylinder 133 is attached to the side of the vehicle body 110. The tip end of the blade lift cylinder 133 is attached to the lift frame 131. The blade lift cylinder 133 extends and retracts using hydraulic oil, driving the lift frame 131 and the blade 132 in the raising or lowering direction.

[0012] The base end of the ripper lift cylinder 135 is attached to the vehicle body 110. The tip end of the ripper lift cylinder 135 is rotatably attached to the ripper 134. The ripper 134 is driven in a raising or lowering direction by the extension and contraction of the ripper lift cylinder 135 by hydraulic oil. The ripper 134 may also have a tilt mechanism.

[0013] The operator's cab 140 is a space where an operator rides and operates the work vehicle 100. The operator's cab 140 is provided on top of the vehicle body 110. 2 is a diagram showing the internal configuration of the operator's cab 140 according to the first embodiment. Inside the operator's cab 140, there are provided a seat 141, a console 142, a blade operation lever 143, a ripper operation lever 144, a travel operation lever 145, a brake pedal 146, and a deceleration pedal 147. The brake pedal 146 and the deceleration pedal 147 may be configured as a single pedal.

[0014] An operation panel, instruments, and switches are attached to the console 142. The operator can visually check the console 142 to confirm the status of the work vehicle 100. The operator can enable or disable the automatic control function by operating the operation panel of the console 142. The blade operating lever 143 is operated to set the amount of movement for raising or lowering the blade 132. The blade operating lever 143 receives a lowering operation when tilted forward, and receives a raising operation when tilted backward. The blade operating lever 143 may also receive a tilt operation to rotate the blade 132 about an axis in the front-to-rear direction, and an angle operation to rotate the blade 132 about an axis in the up-and-down direction. The ripper operation lever 144 is operated to set the amount of movement for raising or lowering the ripper 134. The ripper operation lever 144 accepts a lowering operation when tilted forward, and accepts a raising operation when tilted backward. If the ripper 134 has a tilt mechanism, the ripper operation lever 144 may be operated to set the amount of tilt operation of the ripper 134. The travel operation lever 145 is operated to set the traveling direction of the travel device 120. The travel operation lever 145 accepts a forward operation when tilted forward, and accepts a reverse operation when tilted backward. The travel operation lever 145 also accepts a left turn operation when tilted left, and accepts a right turn operation when tilted right. The travel operation lever 145 is also provided with a gear change button for inputting a speed range. The brake pedal 146 is operated to brake the running device 120 . The deceleration pedal 147 is operated to reduce the rotation speed of the running gear 120 .

[0015] 《Power system》 FIG. 3 is a schematic diagram showing the power system of the work vehicle according to the first embodiment. The work vehicle 100 is equipped with an engine 210, a PTO 220 (Power Take Off), a pair of HSTs 230 (Hydro Static Transmissions), a hydraulic pump 250, and a proportional control valve 260.

[0016] The engine 210 is, for example, a diesel engine. The PTO 220 transmits a portion of the driving force of the engine 210 to the hydraulic pump 250. In other words, the PTO 220 distributes the driving force of the engine 210 to the HST 230 and the hydraulic pump 250. The HST 230 changes the speed of the driving force input to the input shaft and outputs it from the output shaft. The HST 230 is equipped with a hydraulic pump that is driven by rotation of the input shaft, and a hydraulic motor that rotates the output shaft. The HST 230 controls the rotation speed of the output shaft by controlling the discharge flow rate of the hydraulic pump. The input shaft of the HST 230 is connected to the PTO 220, and the output shaft is connected to the sprocket 122. In other words, the HST 230 transmits the driving force of the engine 210 distributed by the PTO 220 to the sprocket 122. The output shafts of the pair of HSTs 230 are connected to the left sprocket 122 and the right sprocket 122, respectively. Note that work machine 100 according to other embodiments may be equipped with another power transmission device, such as a torque converter, a transmission, an HMT (Hydraulic Mechanical Transmission), or an electric transmission device, instead of the HST 230. The hydraulic pump 250 is driven by the driving force from the engine 210. The hydraulic oil discharged from the hydraulic pump 250 is supplied to the blade lift cylinder 133 via a proportional control valve 260. The proportional control valve 260 controls the flow rate of the hydraulic oil discharged from the hydraulic pump 250. In addition to the proportional control valve 260, the hydraulic pump 250 may supply the hydraulic oil to other destinations such as a steering clutch (not shown).

[0017] <<Measurement System>> 4 is a diagram showing the configuration of the measurement system and control device of the work vehicle 100 according to the first embodiment. The measurement system of the work vehicle 100 acquires vehicle body data that indicates the state of the work vehicle 100. The work vehicle 100 is equipped with a rotation sensor 310 , an engine tachometer 320 , an oil pressure sensor 330 , an IMU 340 , a blade stroke sensor 350 , and a GNSS sensor 360 .

[0018] A rotation sensor 310 is provided on each of the left and right sprockets 122. The rotation sensor 310 calculates the number of rotations of the sprocket 122.

[0019] The engine tachometer 320 is provided on the engine 210. The engine tachometer 320 measures the number of revolutions of the engine 210.

[0020] The IMU 340 is provided on the vehicle body 110. The IMU 340 measures the tilt angles of the vehicle body 110 in the roll and pitch directions, and the angular displacement in the yaw direction. The vehicle body coordinate system is, for example, an orthogonal coordinate system whose origin is the center of the traveling device 120 and is represented by an X axis extending in the longitudinal direction of the vehicle body, a Y axis extending in the lateral direction of the vehicle body, and a Z axis extending in the vertical direction of the vehicle body. The direction of rotation of the vehicle body 110 about the X axis is defined as the roll direction, the direction of rotation of the vehicle body 110 about the Y axis is defined as the pitch direction, and the direction of rotation of the vehicle body 110 about the Z axis is defined as the yaw direction.

[0021] The blade stroke sensor 350 is provided on the blade lift cylinder 133. The blade stroke sensor 350 measures the stroke amount of the blade lift cylinder 133. The stroke amount measured by the blade stroke sensor 350 can be converted into the position of the cutting edge of the blade 132 in the vehicle body coordinate system. Specifically, the rotation angle of the lift frame 131 is calculated based on the stroke amount of the blade lift cylinder 133. Because the shapes of the lift frame 131 and the blade 132 are known, the position of the cutting edge of the blade 132 can be identified from the rotation angle of the lift frame 131. Note that work vehicles 100 according to other embodiments may detect the rotation angle of the lift frame 131 using other sensors, such as an encoder.

[0022] The GNSS sensor 360 is provided on the vehicle body 110. The GNSS sensor 360 measures the position and orientation of the vehicle body 110 in a global coordinate system based on signals from GNSS satellites. The GNSS sensor 360 may perform RTK (Real Time Kinematic) positioning. In this case, the GNSS sensor 360 communicates with a fixed station provided at the work site and corrects the measurement data. The GNSS sensor 360 outputs the position and orientation as the measurement data.

[0023] Control device 400 The work vehicle 100 is equipped with a control device 400 for controlling the work vehicle 100 . The control device 400 outputs control signals to the fuel injection device of the engine 210, the HST 230, and the proportional control valve 260 in accordance with the amount of operation of each operating device in the operator's cab 140 (console 142, blade operating lever 143, ripper operating lever 144, travel operating lever 145, brake pedal 146, and deceleration pedal 147). The control device 400 also measures the position of the work vehicle 100 based on measurement data from the measurement system, and displays this on the console 142. The control device 400 may autonomously control the power system based on the measured position.

[0024] The control device 400 is a computer that includes a processor 410 , a main memory 430 , a storage 450 , and an interface 470 .

[0025] The storage 450 is a non-transitory tangible storage medium. Examples of the storage 450 include a magnetic disk, a magneto-optical disk, and a semiconductor memory. The storage 450 may be an internal medium directly connected to the bus of the control device 400, or an external medium connected to the control device 400 via the interface 470 or a communication line. The storage 450 stores a control program for controlling the work vehicle 100. The storage 450 also stores a control model 451, which is a machine learning model for calculating the drive amount of the work implement 130 that has been learned in advance, and terrain data 452 that represents the current terrain of the work site. The control model 451 will be described later. Examples of the terrain data 452 include TIN data, which is three-dimensional data that represents the terrain as a set of triangles, and mesh data that stores the elevation of each grid that divides the terrain. The storage 450 stores the terrain data 452, which has been obtained in advance by drone surveying or the like, as initial data for the terrain data 452. The terrain data 452 also includes information on an end line that indicates the end position of excavation. For example, when performing a process of removing excavated soil below a cliff, the end line may follow the cliff edge. In another embodiment, the end line may be determined based on a target terrain rather than the current terrain. In this case, the terrain data 452 may represent the target terrain.

[0026] In other embodiments, the control device 400 may include a custom LSI (Large Scale Integrated Circuit) such as a PLD (Programmable Logic Device) in addition to or instead of the above configuration. Examples of PLDs include PAL (Programmable Array Logic), GAL (Generic Array Logic), CPLD (Complex Programmable Logic Device), and FPGA (Field Programmable Gate Array). In this case, some or all of the functions implemented by the processor 410 may be implemented by the integrated circuit.

[0027] The processor 410 executes a control program to provide a measurement data acquisition unit 411, a terrain update unit 412, a parameter calculation unit 413, a drive amount calculation unit 414, and an output unit 415. Measurement data from the measurement system is input to the processor 410 via an interface 470.

[0028] The measurement data acquisition unit 411 acquires measurement data from the rotation sensor 310, the engine tachometer 320, the oil pressure sensor 330, the IMU 340, the blade stroke sensor 350, and the GNSS sensor 360. That is, the measurement data acquisition unit 411 acquires measurement data on the yaw angle of the vehicle body 110, the roll angle of the vehicle body 110, the pitch angle of the vehicle body 110, the rotation speeds of the left and right sprockets 122, and the stroke amount of the blade lift cylinder 133.

[0029] The terrain update unit 412 updates the terrain data 452 recorded in the storage 450 based on the measurement data of the GNSS sensor 360 acquired by the measurement data acquisition unit 411. Specifically, when the terrain update unit 412 acquires measurement data from the GNSS sensor 360, it identifies the horizontal position and height of the center point of the crawler 121 of the work vehicle 100 from the measurement data, and rewrites the height of the grid point in the terrain data 452 that corresponds to that horizontal position to the identified height. Because the work vehicle 100 travels on the ground after excavation by the work implement 130, the terrain update unit 412 can update the terrain data 452 based on the height of the ground contact surface of the work vehicle 100.

[0030] The parameter calculation unit 413 calculates parameters to be input to the control model 451 based on the measurement data acquired by the measurement data acquisition unit 411 .

[0031] The parameter calculation unit 413 calculates the vehicle speed of the work vehicle 100 based on the measurement data of the GNSS sensor 360. Specifically, the parameter calculation unit 413 calculates the vehicle speed by differentiating the horizontal position displacement indicated by the past measurement data of the GNSS sensor 360.

[0032] The parameter calculation unit 413 calculates the height of the cutting edge of the blade 132 relative to the vehicle body based on measurement data of the stroke amount of the blade lift cylinder 133. Specifically, the parameter calculation unit 413 calculates the height of the cutting edge of the blade 132 in the vehicle body coordinate system from pre-stored dimensional information of the vehicle body 110 and measurement data of the stroke amount of the blade lift cylinder 133. Note that in other embodiments, the height of the cutting edge of the blade 132 may be calculated based on the inclination and acceleration of the blade 132 measured by an IMU provided on the blade 132, instead of the stroke amount of the blade lift cylinder 133.

[0033] The parameter calculation unit 413 calculates the target distance from the start position to the end position of excavation based on the measurement data of the GNSS sensor 360 acquired when automatic control starts. Specifically, the parameter calculation unit 413 calculates the target distance using the following procedure. The parameter calculation unit 413 identifies the horizontal position indicated by the measurement data of the GNSS sensor 360 acquired when automatic control starts as the start position of excavation. The parameter calculation unit 413 identifies the intersection of a straight line extending from the start position of excavation in the direction indicated by the measurement data and the end line indicated by the topographical data 452 as the end position of excavation. The parameter calculation unit 413 calculates the distance between the start position and the end position as the target distance.

[0034] The parameter calculation unit 413 calculates the current distance, which is the distance from the excavation start position to the current position, based on the measurement data of the GNSS sensor 360. Specifically, the parameter calculation unit 413 calculates the distance between the excavation start position and the current horizontal position indicated by the measurement data of the GNSS sensor 360 as the current distance.

[0035] The parameter calculation unit 413 generates relative terrain data that represents the terrain ahead of the work vehicle 100 based on the measurement data of the GNSS sensor 360 and the terrain data 452. The relative terrain data is a numerical sequence that represents the change in height from the current position of the work vehicle 100 to a position a predetermined distance ahead in the direction of travel of the work vehicle 100. The parameter calculation unit 413 identifies the current position and direction of the work vehicle 100 based on the measurement data of the GNSS sensor 360. The parameter calculation unit 413 identifies a cross section of the three-dimensional shape represented by the terrain data 452, cut out by a plane that passes through the current position of the work vehicle 100 and extends in the direction in which the work vehicle 100 is facing. The parameter calculation unit 413 calculates the change in height of the terrain in the direction of travel of the work vehicle 100 relative to the current position by subtracting the height of the work vehicle 100 from the height related to the cross section. The parameter calculation unit 413 can obtain relative topographical data by extracting, from the identified relative height changes, a sequence from the current position of the work vehicle 100 to a position a predetermined distance ahead.

[0036] The parameter calculation unit 413 calculates the volume related to the difference between the topographical data 452 at the start of automatic control and the current topographical data 452 as the excavation volume.

[0037] The drive amount calculation unit 414 calculates the drive amount of the blade 132 by inputting the measurement data acquired by the measurement data acquisition unit 411 and the parameters calculated by the parameter calculation unit 413 into the control model 451. The drive amount calculation unit 414 is an example of a determination unit that determines the drive amount of the work implement.

[0038] The output unit 415 generates an instruction signal for driving the blade 132 by the drive amount calculated by the drive amount calculation unit 414 , and outputs the signal to the proportional control valve 260 .

[0039] Control Model 451 The control model 451 according to the first embodiment receives input of the pitch angle and pitch angular velocity of the vehicle body 110, the attitude (height and inclination) of the blade 132, the vehicle speed, relative terrain data, the target distance, the current distance, the engine rotation speed, the crawler rotation speed, and the excavation amount, and outputs the drive amount of the blade 132. The control model 451 is a machine learning model such as a neural network. The control model 451 may be realized by a reinforcement learning algorithm such as DQN (Deep Q-Network), PPO (Proximal Policy Optimization), A2C (Advantage Actor-Critic), or SAC (Soft Actor-Critic). The control model 451 may also be realized by an algorithm such as Q-learning or SARSA that does not use a neural network.

[0040] Note that the control model 451 according to other embodiments does not necessarily need to use all of the pitch angle and pitch angular velocity of the vehicle body 110, the attitude (height and tilt) of the blade 132, vehicle speed, relative terrain data, target distance, current distance, engine RPM, crawler RPM, and excavation volume to determine the drive amount of the blade 132. As will be described later, the control model 451 may calculate the drive amount of the blade 132 using at least terrain data relating to the terrain around the work vehicle 100, attitude data of the work vehicle 100, and work implement load data relating to the load on the work implement 130. The relative terrain data is an example of terrain data relating to the terrain around the work vehicle 100. The attitude data of the work vehicle 100 includes at least one of vehicle body attitude data relating to the attitude of the vehicle body 110 and work implement attitude data relating to the attitude of the work implement 130. The pitch angle and pitch angular velocity of the vehicle body 110 are each an example of vehicle body attitude data. The attitude of the blade 132 is an example of work implement attitude data relating to the attitude of the work implement 130. The excavation amount, vehicle speed, engine rotation speed, and crawler rotation speed are each an example of work implement load data relating to the load on the work implement 130. The vehicle speed, engine rotation speed, and crawler rotation speed all decrease as the load on the work implement 130 increases. Therefore, these values ​​can be said to represent the load on the work implement 130.

[0041] In particular, the control model 451 can determine the drive amount of the blade 132 with high accuracy by using the pitch angle of the vehicle body 110, the attitude of the blade 132, the vehicle speed, and the relative terrain data as input data.

[0042] Learning Device 500 The control model 451 is learned in advance by a learning device 500. FIG. 5 is a diagram showing the configuration of the learning device 500 according to the first embodiment. The learning device 500 is a computer including a processor 510, a main memory 530, a storage 550, and an interface 570. The storage 550 stores the control model 451 before learning. The processor 410 executes a learning program to provide a learning control unit 511, a simulator 512, an inference unit 513, and an update unit 514.

[0043] The learning control unit 511 performs learning processing for the control model 451 . The simulator 512 simulates the behavior of the work vehicle 100 and the earth and sand. The inference unit 513 calculates the drive amount of the work machine 130 based on the calculation results of the simulator 512 and the control model 451 . The update unit 514 updates the internal parameters of the control model 451 based on the calculation results of the simulator 512 .

[0044] 6 is a flowchart showing the learning process of the control model 451 by the learning device 500 according to the first embodiment. The learning device 500 repeatedly executes a simulation in which the work vehicle 100 performs excavation from the initial position to the end line of the excavation, and updates the control model 451 according to the results of the simulation. In the first embodiment, a series of simulations of excavation by the work vehicle 100 from the initial position to the end line is considered to be one episode.

[0045] First, the learning control unit 511 inputs an instruction to initialize the episode to the simulator 512 (step S1). Upon receiving the initialization instruction, the simulator 512 arranges particles representing soil and sand to form the terrain of the work site and determines the end line of the excavation. Next, the simulator 512 places the work vehicle 100 on the formed terrain. The distance from the initial position of the work vehicle 100 to the end line may be determined randomly.

[0046] Next, the learning control unit 511 acquires data on the pitch angle and pitch angular velocity of the vehicle body 110, the attitude of the blade 132, the vehicle speed, relative terrain data, the target distance, the current distance, the engine rotation speed, the crawler rotation speed, and the excavation volume from the simulator 512 (step S2). All of these data are calculated by the simulator 512 for the simulation.

[0047] The inference unit 513 inputs the data acquired from the simulator 512 into the control model 451 and calculates the drive amount of the work implement 130 (step S3). Note that, depending on the learning algorithm, the learning control unit 511 may occasionally determine the drive amount randomly without using the control model 451. The learning control unit 511 calculates the operation amount of the blade operation lever 143 for driving the work implement 130 at the calculated drive amount, and the operation amount of the travel operation lever 145 for traveling the vehicle body 110 at a predetermined speed, and inputs these to the simulator 512. The simulator 512 simulates the behavior of the work vehicle 100 for one time step in accordance with the input operation amounts (step S4).

[0048] The learning control unit 511 calculates a reward for reinforcement learning based on the state of the work vehicle 100 and the earth and sand one time step later obtained by the simulator 512 (step S5). The reward is expressed, for example, by the sum of the following four reward functions: The first reward function is a function that returns a value obtained by multiplying the excavation amount by a predetermined gain (hyperparameter) when the excavation is completed, and returns zero when the excavation is not completed. The first reward function represents an evaluation of the excavation results. The second reward function is a function that returns the product of the speed, the excavation amount, and the ratio of the current distance to the target distance. The second reward function represents an evaluation of the balance between the speed and the excavation amount. The third reward function is a function that returns a negative value (hyperparameter) when the speed falls below a threshold (hyperparameter), and returns zero when the speed does not fall below the threshold. The third reward function represents a penalty for stalling or slipping of the work vehicle 100. The fourth reward function is a function that returns a negative value (hyperparameter) if the speed remains below the threshold (hyperparameter) for a predetermined time (hyperparameter), and returns zero if the speed does not fall below the threshold. The fourth reward function represents the penalty for the work vehicle 100 becoming stuck. Note that, although the reward according to the first embodiment is expressed by the sum of four reward functions, this is not limiting. For example, in other embodiments, the reward may be expressed by the sum of one or more reward functions selected from the four reward functions, or may be calculated by introducing another reward function, as long as the reward is higher the greater the excavation volume at the end of excavation and lower the vehicle speed during excavation.

[0049] The learning control unit 511 determines whether or not the simulation termination condition (episode termination condition) has been met (step S6). The episode termination condition is that the work vehicle 100 involved in the simulation reaches the end line, or that the time during which the vehicle speed of the work vehicle 100 involved in the simulation falls below the threshold reaches a predetermined time (hyperparameter of the third reward function).

[0050] If the episode termination condition is not met (step S6: NO), the learning device 500 returns the process to step S2 and performs calculations for the next time step. On the other hand, if the episode termination condition is satisfied (step S6: YES), the update unit 514 uses the drive amount calculated in step S3 and the reward calculated in step S5 to update the internal parameters of the control model 451 so that the total reward becomes larger (step S7). Note that the learning device 500 according to another embodiment may update the control model 451 in step S7 during the episode loop (for example, after step S5) rather than after the episode ends.

[0051] The learning control unit 511 determines whether a learning termination condition has been met (step S8). Examples of the learning termination condition include executing a predetermined number of episodes or reaching a predetermined number of episodes that have reached the end line. If the learning termination condition has not been met (step S8: NO), the learning device 500 returns the process to step S1 and starts the next episode. Depending on the learning algorithm, the learning device 500 may start an episode with a state randomly selected from past episodes as the initial state.

[0052] If the learning termination condition is met (step S8: YES), the learning device 500 terminates learning of the control model 451. The learned control model 451 is recorded in storage 450 of the work vehicle 100 and is used for automatic control of the work vehicle 100.

[0053] <<Automatic control of work machine 130>> Next, a method for automatically controlling the work implement 130 by the work vehicle 100 according to the first embodiment will be described. Fig. 7 is a flowchart showing the method for automatically controlling the work implement 130 according to the first embodiment. When the automatic control function is enabled by the operator operating the console 142, the control device 400 starts the automatic control shown in Fig. 7.

[0054] When automatic control is started, the measurement data acquisition unit 411 acquires measurement data from the rotation sensor 310, the engine tachometer 320, the oil pressure sensor 330, the IMU 340, the blade stroke sensor 350, and the GNSS sensor 360 (step S101).

[0055] The parameter calculation unit 413 acquires the topographical data 452 from the storage 450 and stores it in the main memory 430 as topographical data at the start of automatic control (step S102). The parameter calculation unit 413 also identifies the start and end positions of excavation based on the position and orientation indicated by the measurement data of the GNSS sensor 360 acquired by the measurement data acquisition unit 411 and the end line of the topographical data 452, and calculates the target distance (step S103).

[0056] The terrain update unit 412 updates the terrain data 452 recorded in the storage 450 based on the measurement data of the GNSS sensor 360 (step S104).

[0057] The parameter calculation unit 413 calculates the vehicle speed of the work vehicle 100 based on the measurement data of the GNSS sensor 360 (step S105). The parameter calculation unit 413 calculates the height of the cutting edge of the blade 132 relative to the vehicle body based on measurement data of the stroke amount of the blade lift cylinder 133 (step S106). The parameter calculation unit 413 calculates the current distance, which is the distance from the excavation start position to the current position, based on the end position identified in step S103 and the measurement data of the GNSS sensor 360 (step S107). The parameter calculation unit 413 generates relative terrain data that represents the terrain ahead of the work vehicle 100 based on the measurement data of the GNSS sensor 360 and the terrain data 452 updated in step S104 (step S108). The parameter calculation unit 413 calculates the excavation amount based on the topographical data 452 at the start of automatic control stored in the main memory 430 in step S102 and the topographical data 452 updated in step S104 (step S109).

[0058] The drive amount calculation unit 414 calculates the drive amount of the blade 132 (step S110) by inputting the measurement data acquired by the measurement data acquisition unit 411 and the parameters calculated by the parameter calculation unit 413 in steps S103 to S109 into the control model 451. The output unit 415 generates an instruction signal for driving the blade 132 by the drive amount calculated by the drive amount calculation unit 414, and outputs the instruction signal to the proportional control valve 260 (step S111).

[0059] Next, the measurement data acquisition unit 411 acquires measurement data from each of the rotation sensor 310, engine tachometer 320, oil pressure sensor 330, IMU 340, blade stroke sensor 350, and GNSS sensor 360 (step S112). The control device 400 determines whether the work vehicle 100 has reached the end line based on the topographical data 452 updated in step S102 and the measurement data acquired in step S112 (step S113). If the work vehicle 100 has not reached the end line (step S113: NO), the control device 400 returns the process to step S104 and continues automatic control.

[0060] If the work vehicle 100 has reached the end line (step S113: YES), the control device 400 ends the automatic control.

[0061] Actions and Effects In this way, the control device 400 according to the first embodiment determines the amount of drive for the work implement 130 based on data relating to the topography around the work vehicle 100, the attitude of the vehicle body 110, the attitude of the work implement 130, and the load on the work implement 130. The control device 400 according to the first embodiment determines the amount of drive for the work implement 130 in accordance with the load and attitude of the work vehicle 100, rather than control that follows a predetermined design surface, and therefore can control excavation while suppressing the effects of reaction forces from the work object.

[0062] Second Embodiment The work vehicle 100 according to the second embodiment updates the control model 451 based on the results of the automatic control. FIG. 8 is a diagram showing the configuration of a control device 400 of a work vehicle 100 according to the second embodiment.

[0063] The processor 410 of the control device 400 according to the second embodiment further includes a learning unit 416 in addition to the configuration of the first embodiment. The learning unit 416 calculates a reward for an episode from the start to the end of the automatic control shown in Fig. 7, and updates the internal parameters of the control model 451 based on the calculated reward. The method of calculating and updating the reward is the same as the method used by the learning device 500 according to the first embodiment.

[0064] As a result, the control device 400 according to the second embodiment can realize automatic control with an operation suited to the characteristics of the actual work object of the work vehicle 100.

[0065] Other Embodiments Although one embodiment has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design modifications are possible. That is, in other embodiments, the order of the above-described processes may be changed as appropriate. Furthermore, some processes may be executed in parallel. The control device 400 according to the above-described embodiment may be configured by a single computer, or the configuration of the control device 400 may be divided among multiple computers, and the multiple computers may function as the control device 400 by working together. In this case, some of the computers configuring the control device 400 may be mounted inside the work machine, and other computers may be provided outside the work machine. For example, if the work vehicle 100 is remotely operated, some of the configuration of the control device 400 may be provided in a remote driving device. Furthermore, for example, if the work vehicle 100 is driven autonomously, part of the control device 400 may be provided in an external control server.

[0066] Although the work vehicle 100 according to the embodiment described above is a bulldozer, the present invention is not limited to this. For example, the work vehicle 100 according to other embodiments may be a wheel loader, a hydraulic excavator, or another work vehicle that excavates earth and sand.

[0067] Furthermore, although the control model 451 according to the embodiment described above determines the vertical drive amount (lift drive amount) of the blade 132, this is not limiting. For example, the control model 451 according to other embodiments may determine the pitch drive amount and tilt drive amount in addition to the lift drive amount. For example, after a sufficient amount of earth and sand has entered the blade 132, the blade 132 can be pitched back to prevent the earth and sand from spilling out. Also, for example, when the work vehicle 100 is not moving forward in a straight line, the traveling direction can be adjusted by tilting the blade 132. Therefore, the control model 451 according to other embodiments may determine the pitch drive amount and tilt drive amount in addition to the lift drive amount.

[0068] The control model 451 according to the embodiment described above calculates the drive amount of the blade 132, but is not limited to this. For example, the control model 451 according to another embodiment may determine the raising and lowering of the ripper 134. The ripper 134 is an example of a work machine. Furthermore, in another embodiment, if the ripper 134 has a tilt mechanism, the control model 451 may determine the tilt drive amount of the ripper 134. Furthermore, the control model 451 may further generate an engine throttle command to prevent slippage or stalling in addition to the operation signal for the ripper 134. For example, by lowering the engine throttle when there is a possibility of slippage or stalling, the crawler 121 can be engaged with the soil. [Explanation of symbols]

[0069] 100...Work vehicle 110...Body 120...Traveling device 121...Crawler 122...Sprocket 130...Work machine 131...Lift frame 132...Blade 133...Blade lift cylinder 134...Ripper 135...Ripper lift cylinder 140...Operator's cab 141...Seat 142...Console 143...Blade operation lever 144...Ripper operation lever 145...Travel operation lever 146...Brake pedal 147...Deceleration pedal 210...Engine 220...PTO 230...HST 250...Hydraulic pump 260...Proportional control valve 310...Rotation sensor 320...Engine tachometer 330...Hydraulic pressure sensor 340...IMU 350...Blade stroke sensor 360...GNSS sensor 400...Control device 410...Processor 411...Measurement data acquisition unit 412: Terrain update unit 413: Parameter calculation unit 414: Drive amount calculation unit 415: Output unit 416: Learning unit 430: Main memory 450: Storage 451: Control model 452: Terrain data 470: Interface 500: Learning device 510: Processor 511: Learning control unit 512: Simulator 513: Inference unit 514: Update unit 530: Main memory 550: Storage 570: Interface

Claims

1. A control system for a work vehicle including a vehicle body and a work implement supported on the vehicle body and used to excavate earth and sand, a determination unit that determines a drive amount of the work implement based on work vehicle data including topographical data relating to the topography around the work vehicle, attitude data of the work vehicle, and work implement load data relating to the load applied to the work implement; A work vehicle control system comprising:

2. the determination unit determines the drive amount of the work implement by inputting terrain data relating to the terrain around the work vehicle and work vehicle data of the work vehicle into a trained model that has been trained to output a value of the drive amount of the work implement from terrain data and work vehicle data; The work vehicle control system according to claim 1 .

3. The trained model was trained by reinforcement learning based on a reward that is higher the more excavation volume at the end of excavation and lower the vehicle speed during excavation. The work vehicle control system according to claim 2 .

4. The work machine load data includes a value related to the speed of the work vehicle. The control system for a work vehicle according to any one of claims 1 to 3.

5. The work vehicle data is target distance data relating to the distance from the start position to the end position of excavation by the work machine; current distance data relating to the distance from the start position to the position of the work machine; Including, The control system for a work vehicle according to any one of claims 1 to 3.

6. The work vehicle data includes excavation volume data relating to the excavation volume by the work machine. The control system for a work vehicle according to any one of claims 1 to 3.

7. A control method for a work vehicle including a vehicle body and a work implement supported on the vehicle body and used to excavate earth and sand, comprising: determining a drive amount of the work implement based on work vehicle data including topographical data relating to the topography around the work vehicle, attitude data of the work vehicle, and work implement load data relating to the load applied to the work implement; A control method for a work vehicle comprising:

Images