Sensing system, work vehicle, sensing method, and computer program

Abstract

A sensing system according to an embodiment of the present disclosure is for a work vehicle that performs work using a work machine. The sensing system comprises: a sensor that is provided on the work vehicle, senses the environment around the work vehicle, and generates sensor data; and a processing device that, on the basis of the sensor data, detects an object located in a detection area around the work vehicle. The work machine can change the height thereof with respect to the body of the work vehicle, and the processing device changes the pattern of the detection area for detecting the object according to the change in the height of the work machine.

Description

Sensing system, work vehicle, sensing method, and computer program

[0001] The present disclosure relates to a sensing system, a work vehicle, a sensing method, and a computer program.

[0002] Research and development is underway on smart agriculture, which utilizes ICT (Information and Communication Technology) and IoT (Internet of Things) as the next generation of agriculture. Research and development is also underway to automate and unmanned agricultural machinery, such as tractors and harvesters, used in fields. For example, agricultural machinery that performs agricultural work while autonomously driving within fields using positioning systems such as GNSS (Global Navigation Satellite System), which enables precise positioning, is now being put into practical use.

[0003] Patent Literature 1 discloses a harvester that travels autonomously while harvesting crops in a field. The harvester can harvest crops by traveling along a predetermined travel route in the field.

[0004] JP 2018-073399 A

[0005] Technology is also being developed that uses sensors to search the area around a work vehicle such as an agricultural machine and detect obstacles around the work vehicle.

[0006] In obstacle detection processing using a sensor, it is necessary to prevent an object that is not an obstacle from being detected as an obstacle.

[0007] A sensing system according to one embodiment of the present disclosure is a sensing system for a work vehicle that performs work using a work machine, and includes a sensor that is mounted on the work vehicle and senses the environment around the work vehicle to generate sensor data, and a processing device that detects objects located in a detection area around the work vehicle based on the sensor data, wherein the work machine is capable of changing its height relative to the main body of the work vehicle, and the processing device changes the pattern of the detection area in which the objects are detected in accordance with changes in the height of the work machine.

[0008] When the height of a work machine changes, the work machine may enter the sensing area sensed by the sensor. According to an embodiment of the present disclosure, by changing the pattern of the detection area in which objects are detected in response to changes in the height of the work machine, it is possible to prevent object detection processing from being performed on the portion of the sensor data that indicates the work machine. This makes it possible to prevent the work machine from being detected as an obstacle or the like. Furthermore, by reducing the amount of calculation by the processing device, it is possible to increase the object detection processing speed.

[0009] 1 is a diagram illustrating an example of an agricultural machine. FIG. 2 is a block diagram illustrating an example of the configuration of a harvester. FIG. 3 is a top view illustrating an example of a sensing area sensed by a millimeter-wave radar. FIG. 4 is a side view illustrating an example of a sensing area sensed by a millimeter-wave radar. FIG. 5 is a side view illustrating a sensing area and a header displaced upward. FIG. 6 is a flowchart illustrating an example of a process for changing the pattern of a detection area in accordance with a change in the height of the header. FIG. 7 is a top view illustrating an example of a portion to be removed when setting a second detection area. FIG. 8 is a top view illustrating an example of a second detection area. FIG. 9 is a top view illustrating a plurality of types of portions to be removed having different sizes. FIG. 10 is a top view illustrating another example of a portion to be removed when setting a second detection area. FIG. 11 is a top view illustrating another example of a second detection area. FIG. 12 is a diagram illustrating an example of a tractor connected to a work machine. FIG. 13 is a diagram illustrating an example of a construction machine.

[0010] Hereinafter, embodiments of the present disclosure will be described. However, more detailed descriptions than necessary may be omitted. For example, detailed descriptions of well-known matters and redundant descriptions of substantially identical configurations may be omitted. This is to avoid unnecessary redundancy in the following description and to facilitate understanding by those skilled in the art. Note that the inventors provide the accompanying drawings and the following description to enable those skilled in the art to fully understand the present disclosure, and do not intend them to limit the subject matter described in the claims. In the following description, components having the same or similar functions are designated by the same reference numerals. The symbols F, Re, L, R, U, and D in the drawings represent front, rear, left, right, top, and bottom, respectively.

[0011] The following embodiments are merely examples, and the technology of the present disclosure is not limited to the following embodiments. The contents of the following embodiments are merely examples, and various modifications are possible as long as no technical contradiction occurs. Furthermore, one aspect can be combined with another aspect as long as no technical contradiction occurs.

[0012] [Work vehicle] A "work vehicle" according to an embodiment of the present disclosure refers to a mobile agricultural machine or a mobile construction machine used for specific purposes such as agriculture or construction. An "agricultural machine" according to this embodiment refers to a mobile machine used for agricultural purposes. A "construction machine" according to this embodiment refers to a mobile machine used for civil engineering or construction purposes. "Work" includes, for example, agricultural work, civil engineering work, construction work, rubble removal work, snow removal work, etc. The following mainly describes an example in which the work vehicle is an agricultural machine.

[0013] The agricultural machine according to this embodiment may be a mobile agricultural machine that can harvest crops in a field while moving. Examples of the agricultural machine include a harvester, a tractor, and an agricultural mobile robot. In some cases, the entire agricultural machine and an implement attached to or pulled by the agricultural machine, such as a tractor, function as a single "agricultural machine."

[0014] Fig. 1 is a diagram showing an example of an agricultural machine 100 according to this embodiment. Fig. 1 shows a harvester as an example of the agricultural machine 100.

[0015] The harvester 100 may be, for example, a combine harvester. The harvester 100 harvests crops in a field, threshes the harvested crops, stores the harvested crops after threshing, and discharges the harvested crops. The crops in the field may be, but are not limited to, harvestable grains such as rice, wheat, corn, and soybeans.

[0016] The harvester 100 has an automatic driving function. That is, the harvester 100 can travel by the operation of a control device, without manual operation. The control device in this embodiment is provided inside the harvester 100 and can control both the speed and steering of the harvester 100. The harvester 100 may travel automatically not only within a field but also outside the field (e.g., on a road). The harvester 100 is equipped with devices used for positioning or self-location estimation, such as a GNSS unit and a LiDAR sensor. The control device of the harvester 100 causes the harvester 100 to travel automatically based on the position of the harvester 100 and information on a target route.

[0017] The operation of the harvester 100 can be managed by a harvest management system. The harvest management system includes the harvester 100, a terminal device, a management device, and the like.

[0018] The user terminal device is a computer used by a user to remotely monitor the harvester 100. The management device is a computer managed by a business operator that operates the harvest management system. The harvester 100, the user terminal device, and the management device can communicate with each other via a network. The harvest management system may include multiple harvesters 100. The harvest management system may also include other agricultural machinery.

[0019] The management device is a computer that manages agricultural work performed by the harvester 100. The management device may be, for example, a server computer that centrally manages information about a farm field on the cloud and supports agriculture by utilizing data on the cloud. The management device, for example, creates a work plan for the harvester 100 and causes the harvester 100 to perform agricultural work in accordance with the work plan.

[0020] The user terminal device is a computer used by a user who is located away from the harvester 100. The user terminal device can be used to remotely monitor and operate the harvester 100.

[0021] The harvester 100 shown in Fig. 1 includes a vehicle body 101 and a traveling device 102. The traveling device 102 shown in Fig. 1 is a crawler-type traveling device, but may be a traveling device equipped with wheels with tires. A cabin 114 is provided above the vehicle body 101.

[0022] The harvester 100 includes a prime mover (engine) 111 and a transmission 112. Inside a cabin 114, a driver's seat, operation levers, an operation terminal, and a group of switches for operation are provided.

[0023] A work machine 110 that harvests crops in the field is disposed in front of the harvester 100. The work machine 110 includes a header 115. The header 115 includes a reaping device 103 that harvests the crops and a reel 109 that lifts the stalks of the crops. The header 115 is provided in front of the transport device 104. The transport device 104 transports the harvested crops. The reel 109 is disposed above the reaping device 103. The heights of the reaping device 103 and the reel 109 are adjustable. The work machine 110 may include the transport device 104.

[0024] A threshing device 105 and a tank 106 for storing harvested material are arranged side by side in the left-right direction behind the cabin 114. The conveying device 104 is disposed between the reaping device 103 and the threshing device 105. The threshing device 105 threshes the harvested crops. The tank 106 stores the harvested material obtained by threshing grains and the like. A straw waste treatment device 108 is provided behind the threshing device 105. The straw waste treatment device 108 finely cuts the stalks and other parts of the harvested material after grains and other harvested material have been removed, and discharges them outside. The tank 106 is provided with a discharge device 107 for discharging the harvested material from the tank 106.

[0025] The configurations and operations of various devices that perform harvesting operations, such as the reaping device 103, conveying device 104, threshing device 105, tank 106, discharge device 107, straw waste treatment device 108, and reel 109, are well known, so detailed explanations of them will be omitted here.

[0026] The harvester 100 in this embodiment can operate in both a manual operation mode and an automatic operation mode. In the automatic operation mode, the harvester 100 can travel unmanned. Also, in the automatic operation mode, the harvester 100 can travel unmanned while performing an operation to harvest crops in a field.

[0027] The harvester 100 includes a plurality of sensing devices that sense the environment around the harvester 100, and a control device that processes sensor data output from the plurality of sensing devices. The sensing devices may be a millimeter-wave radar 125, a camera 126, a LiDAR sensor 127, and an obstacle sensor 128.

[0028] The millimeter-wave radar 125 illustrated in FIG. 1 is disposed at the front of the harvester 100. Additional millimeter-wave radars 125 may be provided at the sides and / or rear of the harvester 100. The millimeter-wave radar 125 may be a two-dimensional scanning millimeter-wave radar or a three-dimensional scanning millimeter-wave radar. The millimeter-wave radar 125 senses the environment around the harvester 100 and outputs sensor data. The millimeter-wave radar 125 repeatedly outputs sensor data indicating the distance to a measurement point corresponding to an object present in the surrounding environment, the angle of the measurement point, and the speed of the measurement point. The millimeter-wave radar 125 is, for example, but is not limited to, a frequency-modulated continuous wave (FMCW) millimeter-wave radar.

[0029] The sensor data output from the millimeter wave radar 125 is processed by the control device of the harvester 100. The control device can detect objects such as obstacles present around the harvester 100 based on the sensor data.

[0030] The cameras 126 illustrated in Fig. 1 are provided on the front, rear, left, and right sides of the harvester 100. The cameras 126 capture images of the environment around the harvester 100 and generate image data. The images captured by the cameras 126 can be output to a control device mounted on the harvester 100 and transmitted to a user terminal device for remote monitoring. The images can also be used to monitor the harvester 100 during unmanned operation.

[0031] The LiDAR sensors 127 illustrated in FIG. 1 are disposed at the front and rear of the harvester 100. Additional LiDAR sensors 127 may be provided on the sides of the harvester 100. The harvester 100 may include multiple LiDAR sensors disposed at different positions and with different orientations. The LiDAR sensor 127 may be a 3D-LiDAR sensor, but may also be a 2D-LiDAR sensor. The LiDAR sensor 127 senses the environment surrounding the harvester 100 and outputs sensor data. The LiDAR sensor 127 repeatedly outputs sensor data indicating the distance and direction to each measurement point of an object in the surrounding environment, or the three-dimensional or two-dimensional coordinate values ​​of each measurement point. The sensor data output from the LiDAR sensor 127 is processed by a control device of the harvester 100. The control device can estimate the self-position of the harvester 100 by matching the sensor data with an environmental map. The control device can further detect, based on the sensor data, objects such as obstacles present in the surroundings of the harvester 100. The control device can also generate or edit an environmental map using an algorithm such as SLAM (Simultaneous Localization and Mapping).

[0032] The obstacle sensor 128 illustrated in FIG. 1 is provided on the side of the harvester 100. The obstacle sensor 128 may also be located in other locations. For example, the obstacle sensor 128 may be provided on the front and rear of the harvester 100. The obstacle sensor 128 may include, for example, a laser scanner or an ultrasonic sonar. The obstacle sensor 128 is used to detect surrounding obstacles during autonomous driving and to stop or detour the harvester 100. A LiDAR sensor 127 may be used as one of the obstacle sensors 128.

[0033] The harvester 100 includes a positioning device 121 that detects the geographic coordinates of the position of the harvester 100. The positioning device 121 is, for example, a GNSS unit. The GNSS unit 121 includes a GNSS receiver. The GNSS receiver may include an antenna that receives signals from GNSS satellites and a processor that calculates the position of the harvester 100 based on the signals received by the antenna. The GNSS unit 121 receives satellite signals transmitted from multiple GNSS satellites and performs positioning based on the satellite signals. GNSS is a general term for satellite positioning systems such as GPS (Global Positioning System), QZSS (Quasi-Zenith Satellite System, e.g., Michibiki), GLONASS, Galileo, and BeiDou. In this embodiment, the GNSS unit 121 is provided on top of the cabin 114, but may be provided in another location.

[0034] The control device of the harvester 100 may use, for positioning, sensor data acquired by sensing devices such as the camera 126 and / or the LiDAR sensor 127, in addition to the positioning results obtained by the GNSS unit 121. If there are features that function as characteristic points in the environment in which the harvester 100 travels, the position and orientation of the harvester 100 can be estimated with high accuracy based on the data acquired by the camera 126 and / or the LiDAR sensor 127 and an environmental map that is pre-stored in a storage device. By using the data acquired by the camera 126 and / or the LiDAR sensor 127 to correct or complement position data based on satellite signals, the position of the harvester 100 can be identified with higher accuracy.

[0035] The prime mover 111 may be, for example, a diesel engine. An electric motor may be used instead of a diesel engine. The transmission 112 can change the propulsive force and travel speed of the harvester 100 by changing the speed. The transmission 112 can also switch the harvester 100 between forward and reverse travel.

[0036] In a configuration in which the harvester 100 is equipped with a crawler-type traveling device 102, the traveling direction of the harvester 100 can be changed by varying the rotational speeds of the left and right wheels equipped with tracks or by varying the rotational directions of the left and right wheels. In a configuration in which the harvester 100 is equipped with a traveling device equipped with tires and wheels, the harvester 100 is equipped with a power steering device, and the traveling direction of the harvester 100 can be changed by controlling the power steering device to change the turning angle of the steering wheels (also referred to as the "steering angle").

[0037] 1 is capable of being operated with a driver, but may be designed for unmanned operation only. In this case, components required only for manned operation, such as a cabin 114, a steering device, and a driver's seat, may not be provided in the harvester 100. The unmanned harvester 100 can travel autonomously or by remote control by a user.

[0038] 2 is a block diagram showing an example of the configuration of the harvester 100. The harvester 100 can communicate with the above-mentioned user terminal device and management device via a network.

[0039] The harvester 100 illustrated in Fig. 2 includes a GNSS unit 121, an inertial measurement unit (IMU) 122, a millimeter-wave radar 125, a camera 126, a LiDAR sensor 127, an obstacle sensor 128, an operation terminal 131, an operation switch group 132, a drive unit 140, a power transmission mechanism 141, a sensor group 150, a control unit 160, and a communication unit 190. These components are connected to each other via a bus so that they can communicate with each other.

[0040] The GNSS unit 121 includes, for example, a GNSS receiver and an RTK receiver. The sensor group 150 detects various states of the harvester 100. The sensor group 150 includes an operation lever sensor 151, a rotation sensor 152, and a load sensor 156. The control device 160 includes a processor 161, a RAM (Random Access Memory) 162, a ROM (Read Only Memory) 163, a storage device 164, and multiple electronic control units (ECUs) 165 to 167. Figure 2 shows components that are relatively closely related to the automatic driving operation of the harvester 100, and does not illustrate other components.

[0041] The GNSS unit 121 receives satellite signals transmitted from multiple GNSS satellites and generates GNSS data based on the satellite signals. The GNSS data is generated in a predetermined format, such as the NMEA-0183 format. The GNSS data may include, for example, values ​​indicating the identification number, elevation angle, azimuth angle, and reception strength of each satellite from which the satellite signal is received.

[0042] The GNSS unit 121 may perform positioning of the harvester 100 using RTK (Real Time Kinematic)-GNSS. Positioning using RTK-GNSS utilizes satellite signals transmitted from multiple GNSS satellites as well as correction signals transmitted from a reference station. The reference station may be installed near the field where the harvester 100 performs its work (e.g., within 10 km of the harvester 100). The reference station generates correction signals, for example, in RTCM format, based on the satellite signals received from multiple GNSS satellites and transmits them to the GNSS unit 121. The RTK receiver of the GNSS unit 121 includes an antenna and a modem and receives the correction signals transmitted from the reference station. The GNSS unit 121 corrects the positioning results based on the correction signals. Using RTK-GNSS, it is possible to perform positioning with an accuracy of, for example, a few centimeters. Position data including information on latitude, longitude, and altitude is acquired by high-precision positioning using RTK-GNSS. The GNSS unit 121 calculates the position of the harvester 100, for example, at a frequency of about 1 to 10 times per second.

[0043] The positioning method is not limited to RTK-GNSS, and any positioning method (such as interferometric positioning or relative positioning) that can obtain position data with the required accuracy can be used. For example, positioning may be performed using a virtual reference station (VRS) or a differential global positioning system (DGPS). If position data with the required accuracy can be obtained without using a correction signal transmitted from a reference station, the position data may be generated without using a correction signal. In this case, the GNSS unit 121 does not need to be equipped with an RTK receiver.

[0044] Even when RTK-GNSS is used, in places where correction signals from a reference station cannot be obtained (for example, on a road far from a field), the position of the harvester 100 is estimated by other methods without relying on signals from the RTK receiver. For example, the position of the harvester 100 can be estimated by matching data output from the LiDAR sensor 127 and / or the camera 126 with a highly accurate environmental map.

[0045] The IMU 122 may include a three-axis acceleration sensor and a three-axis gyroscope. The IMU 122 may also include a direction sensor such as a three-axis geomagnetic sensor. The IMU 122 functions as a motion sensor and can output signals indicative of various quantities such as the acceleration, velocity, displacement, and attitude of the harvester 100.

[0046] The position data can be supplemented using the output signal of the IMU 122. The IMU 122 can measure the tilt and minute movements of the harvester 100. By using the data acquired by the IMU 122 to supplement the position data based on satellite signals, the positioning performance can be improved.

[0047] In addition to the satellite signals and correction signals described above, the position and orientation of the harvester 100 can be estimated with higher accuracy based on signals output from the IMU 122. The signals output from the IMU 122 can be used to correct or supplement the position calculated based on the satellite signals and correction signals. The IMU 122 outputs signals at a higher frequency than position detection using satellite signals. Using these high-frequency signals, the position and orientation of the harvester 100 can be measured at a higher frequency (e.g., 10 Hz or higher). Instead of the IMU 122, a three-axis acceleration sensor and a three-axis gyroscope may be provided separately. The IMU 122 may be included in the GNSS unit 121.

[0048] The camera 126 is an imaging device that captures images of the environment surrounding the harvester 100. The camera 126 includes an image sensor, such as a charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). The camera 126 may also include an optical system including one or more lenses and a signal processing circuit. The camera 126 captures images of the environment surrounding the harvester 100 while the harvester 100 is traveling and generates image (e.g., video) data. The camera 126 can capture video at a frame rate of, for example, 3 frames per second (fps) or higher. The images generated by the camera 126 can be used, for example, when a remote observer checks the environment surrounding the harvester 100 using a user terminal device. The images generated by the camera 126 may be used for positioning or obstacle detection. Multiple cameras 126 may be installed at different positions on the harvester 100, or a single camera may be installed. A visible camera that generates visible light images and an infrared camera that generates infrared images may be installed separately. Both visible and infrared cameras may be provided to generate images for surveillance, and the infrared camera may also be used to detect obstacles at night.

[0049] The obstacle sensor 128 detects objects present in the vicinity of the harvester 100. The obstacle sensor 128 may include, for example, a laser scanner or an ultrasonic sonar. The obstacle sensor 128 outputs a signal indicating the presence of an obstacle when an object is present closer than a predetermined distance from the obstacle sensor 128. Multiple obstacle sensors 128 may be provided at different positions on the harvester 100. For example, multiple laser scanners and multiple ultrasonic sonars may be arranged at different positions on the harvester 100. Providing multiple obstacle sensors 128 makes it possible to reduce blind spots in monitoring obstacles around the harvester 100.

[0050] The operation lever sensor 151 detects operation of the operation lever by a user in the cabin 114. The output signal of the operation lever sensor 151 is used for operation control by the control device 160. The rotation sensor 152 measures the rotation speed of the axle of the traveling device 102, i.e., the number of rotations per unit time. The rotation sensor 152 may be a sensor that uses, for example, a magnetoresistive element (MR), a Hall element, or an electromagnetic pickup. The rotation sensor 152 outputs a numerical value that indicates, for example, the number of rotations per minute (unit: rpm) of the axle. The rotation sensor 152 is used, for example, to measure the speed of the harvester 100.

[0051] The load sensor 156 is provided at the bottom of the tank 106 and detects the weight of the harvested product in the tank 106. By detecting the weight of the harvested product in the tank 106, the control device 160 can recognize the storage state of the harvested product in the tank 106. A yield sensor and a taste sensor may be provided inside or around the tank 106. The taste sensor outputs data such as the moisture content and protein content of the harvested product as quality data.

[0052] The drive device 140 includes various devices necessary for driving the harvester 100 to travel, such as the prime mover 111 and the transmission 112. The prime mover 111 includes an internal combustion engine such as a diesel engine. The drive device 140 may include an electric motor for traction instead of or in addition to the internal combustion engine.

[0053] The power transmission mechanism 141 transmits the power generated by the prime mover 111 to various devices that perform the harvesting operation. The devices that perform the harvesting operation include the reaping device 103, the transport device 104, the threshing device 105, the tank 106, the discharge device 107, the straw waste treatment device 108, and the reel 109. The harvester 100 may also include a power source (such as an electric motor) that supplies power to at least one of the devices that perform the harvesting operation, separate from the prime mover 111.

[0054] The processor 161 may be, for example, a semiconductor integrated circuit including a central processing unit (CPU). The processor 161 may be realized by a microprocessor or a microcontroller. Alternatively, the processor 161 may be realized by a field programmable gate array (FPGA) equipped with a CPU, a graphics processing unit (GPU), an application specific integrated circuit (ASIC), an application specific standard product (ASSP), or a combination of two or more circuits selected from these circuits. The processor 161 sequentially executes a computer program stored in the ROM 163, which describes a group of instructions for executing at least one process, to achieve the desired process.

[0055] The ROM 163 is, for example, a writable memory (e.g., a PROM), a rewritable memory (e.g., a flash memory), or a read-only memory. The ROM 163 stores a program that controls the operation of the processor 161. The ROM 163 does not need to be a single storage medium, but may be a collection of multiple storage media. Part of the collection of multiple storage media may be removable memory.

[0056] The RAM 162 provides a working area for temporarily loading the control program stored in the ROM 163 at boot time. The RAM 162 does not need to be a single storage medium, but may be a collection of multiple storage media.

[0057] The storage device 164 includes one or more storage media, such as a flash memory or a magnetic disk. The storage device 164 stores various data generated by the GNSS unit 121, the millimeter-wave radar 125, the camera 126, the LiDAR sensor 127, the obstacle sensor 128, the sensor group 150, and the control device 160. The data stored in the storage device 164 may include map data (environmental map) of the environment in which the harvester 100 travels and target route data for autonomous driving. The environmental map includes information on multiple fields in which the harvester 100 will perform agricultural work and the roads in their surrounding areas. The environmental map and target route may be generated by a processor in the management device. The control device 160 may also have a function for generating or editing the environmental map and target route. The control device 160 can edit the environmental map and target route obtained from the management device according to the traveling environment of the harvester 100. The storage device 164 also stores work plan data received by the communication device 190 from the management device.

[0058] The storage device 164 also stores computer programs that cause the processor 161 and the ECUs 165-167 to execute various operations, which will be described later. Such computer programs may be provided to the harvester 100 via a storage medium (e.g., a semiconductor memory or an optical disk) or an electric communication line (e.g., the Internet). Such computer programs may be sold as commercial software.

[0059] The control device 160 includes a plurality of ECUs 165 to 167. The ECU 165 controls the driving speed and turning operation of the harvester 100 by controlling the prime mover 111, the transmission 112, the traveling device 102, etc., which are included in the drive device 140.

[0060] The ECU 165 performs calculations and controls to achieve autonomous driving based on data output from the GNSS unit 121, the millimeter-wave radar 125, the camera 126, the LiDAR sensor 127, the obstacle sensor 128, the sensor group 150, and the processor 161. For example, the ECU 165 identifies the position of the harvester 100 based on data output from at least one of the GNSS unit 121, the camera 126, and the LiDAR sensor 127. Within the field, the ECU 165 may determine the position of the harvester 100 based only on data output from the GNSS unit 121. The ECU 165 may estimate or correct the position of the harvester 100 based on data acquired by the camera 126 and / or the LiDAR sensor 127. By utilizing the data acquired by the camera 126 and / or the LiDAR sensor 127, the accuracy of positioning can be further improved. For example, the ECU 165 may estimate the position of the harvester 100 by matching data output from the LiDAR sensor 127 and / or the camera 126 with an environmental map. During autonomous driving, the ECU 165 performs calculations necessary for the harvester 100 to travel along a target route based on the estimated position of the harvester 100.

[0061] The ECU 166 may determine the movement destination of the harvester 100 based on the work plan stored in the storage device 164, and may determine a target route from the start point to the destination point of the movement of the harvester 100. The ECU 166 may perform processing to detect objects located around the harvester 100 based on data output from the millimeter-wave radar 125, the camera 126, the obstacle sensor 128, and the LiDAR sensor 127.

[0062] The ECU 167 controls the operation of the power transmission mechanism 141 and the like to cause the various devices that perform the harvesting operations described above to perform desired operations.

[0063] Through the operation of these ECUs, the control device 160 realizes automatic driving and crop harvesting operations. During automatic driving, the control device 160 controls the drive device 140 based on the measured or estimated position of the harvester 100 and the target route. In this way, the control device 160 can cause the harvester 100 to travel along the target route.

[0064] The multiple ECUs included in the control device 160 can communicate with each other in accordance with a vehicle bus standard such as CAN (Controller Area Network). Instead of CAN, a faster communication method such as Automotive Ethernet (registered trademark) may be used. In FIG. 2 , each of the ECUs 165 to 167 is shown as an individual block, but the functions of each may be realized by multiple ECUs. An on-board computer that integrates at least some of the functions of the ECUs 165 to 167 may be provided. The control device 160 may include ECUs other than the ECUs 165 to 167, and any number of ECUs may be provided depending on the functions. Each ECU includes a processing circuit including one or more processors. The processor 161 may be integrated with one of the ECUs included in the control device 160.

[0065] The communication device 190 is a device including circuits for communicating with the user terminal device and the management device. The communication device 190 may include an antenna and communication circuits for transmitting and receiving signals over a network between the communication devices of the user terminal device and the management device. The network may include, for example, a cellular mobile communication network such as 3G, 4G, or 5G, and the Internet. The communication device 190 may have a function for communicating with a mobile terminal used by an observer near the harvester 100. Communication with such a mobile terminal may be performed in accordance with any wireless communication standard, such as Wi-Fi (registered trademark), cellular mobile communication such as 3G, 4G, or 5G, or Bluetooth (registered trademark).

[0066] The operation terminal 131 is a terminal through which a user performs operations related to the traveling of the harvester 100 and is also referred to as a virtual terminal (VT). The operation terminal 131 may include a display device such as a touch screen and / or one or more buttons. The display device may be, for example, a liquid crystal display (LCD) or an organic light-emitting diode (OLED) display. By operating the operation terminal 131, a user can perform various operations, such as switching the autonomous driving mode on / off, recording or editing an environmental map, and setting a target route. At least some of these operations can also be realized by operating the operation switch group 132. The operation terminal 131 may be configured to be detachable from the harvester 100. A user located remotely from the harvester 100 may operate the detached operation terminal 131 to control the operation of the harvester 100. Instead of the operation terminal 131, the user may control the operation of the harvester 100 by operating a computer, such as a user terminal device, on which necessary application software is installed.

[0067] [Setting of Detection Area According to Height of Work Machine] Next, a process for setting a detection area according to changes in the height of the work machine will be described.

[0068] As described above, sensing devices such as the millimeter-wave radar 125, the camera 126, the LiDAR sensor 127, and the obstacle sensor 128 sense the environment around the harvester 100 and output sensor data. The processor 161 detects objects located in a detection area around the harvester 100 based on the sensor data. The detection area is an area around the harvester 100 sensed by the sensing device in which object detection is performed. The detection area may be the same size as the sensing area sensed by the sensing device, or may be smaller than the sensing area. The detection area may also be referred to as a search area or a region of interest (ROI).

[0069] In the embodiment exemplified below, the pattern of the detection area in the process of detecting an object using the sensor data output by the millimeter-wave radar 125 is changed in accordance with a change in the height of the work machine 110. Changing the pattern of the detection area means, for example, changing at least one of the shape, size, and relative position of the detection area with respect to the harvester 100.

[0070] The harvester 100 of this embodiment includes a sensing system 10 ( FIG. 2 ) that uses sensor data output by a millimeter-wave radar 125 to detect objects located around the harvester 100. The sensing system 10 includes a control device 160 and the millimeter-wave radar 125.

[0071] The millimeter-wave radar 125 outputs millimeter-wave radio waves and receives reflected waves obtained when the radio waves are reflected by objects in the surrounding environment. The millimeter-wave radar 125 outputs sensor data indicating the distance to a measurement point corresponding to an object in the surrounding environment, the angle of the measurement point, and the speed of the measurement point.

[0072] Fig. 3 is a top view showing an example of a sensing area 200 sensed by the millimeter-wave radar 125. Fig. 4 is a side view showing an example of the sensing area 200 sensed by the millimeter-wave radar 125. The sensing area 200 has a shape that extends forward of the harvester 100 and spreads in all directions. The processor 161 detects an object using, for example, the sensing area 200 as a detection area.

[0073] The height of the work machine 110 of the harvester 100 relative to the vehicle body 101 can be changed. In the harvester 100 of this embodiment, the actuator 116 is operated to rotate the transport device 104 around a rotation shaft 117. By rotating the transport device 104 around the rotation shaft 117, the height of the header 115 arranged in front of the transport device 104 can be changed.

[0074] The rotation shaft 117 is provided with a height sensor 118 for detecting the relative height of the header 115 with respect to the vehicle body 101. The sensor 118 is, for example, an angle sensor, and detects the angle of the conveying device 104 with respect to the vehicle body 101. The height of the header 115 can be calculated based on the angle of the conveying device 104 with respect to the vehicle body 101.

[0075] For example, height information indicating the relationship between the angle of the transport device 104 relative to the vehicle body 101 and the height of the header 115 (e.g., the height of the top of the reel 109) is stored in advance in the ROM 163 or the storage device 164. The processor 161 can calculate the height of the header 115 based on the output signal of the sensor 118 and the height information.

[0076] The height of the header 115 may be changed by a method other than the above. For example, the height may be changed by linearly moving the header 115 up and down.

[0077] The height sensor for detecting the height of the header 115 is not limited to the above-described type, and the height of the header 115 can be detected by providing any sensor at any position.

[0078] 5 is a side view showing the sensing area 200 and the header 115 displaced upward. When the header 115 displaces upward and becomes higher, a portion of the header 115 may enter the sensing area 200, as shown in FIG. 5. In the example shown in FIG. 5, a portion of the reel 109 of the header 115 is within the sensing area 200.

[0079] In this embodiment, in order to prevent the header 115 located within the sensing area 200 from being detected as an obstacle, the pattern of the detection area in which objects are detected is changed according to changes in the height of the header 115.

[0080] FIG. 6 is a flowchart showing an example of a process for changing the pattern of the detection area in response to a change in the height of the header 115.

[0081] The processor 161 detects the height of the header 115 based on the output signal of the height sensor 118 (step S201). The detected height of the header 115 may be a height relative to the vehicle body 101. For example, the height of the header 115 relative to the vehicle body 101 when the header 115 is in the lowest position is set to zero.

[0082] The processor 161 determines whether the height of the detected header 115 is equal to or greater than a predetermined height (step S202). The predetermined height is a height at which a part of the header 115 (for example, a part of the reel 109) falls within the sensing area 200.

[0083] For example, information indicating the value of the predetermined height is stored in advance in the ROM 163 or the storage device 164, and the processor 161 can use this information to determine whether the height of the header 115 is equal to or greater than the predetermined height.

[0084] If the processor 161 determines that the height of the header 115 is less than the predetermined height, it sets a first detection area 201 as the detection area (step S204). The processor 161 sets the first detection area 201 to, for example, the same size as the sensing area 200 sensed by the millimeter-wave radar 125. The first detection area 201 may be smaller than the sensing area 200. The processor 161 performs processing to detect objects in the first detection area 201 using the millimeter-wave radar 125 (step S205).

[0085] If the processor 161 determines in step S202 that the height of the header 115 is equal to or greater than the predetermined height, it sets the second detection area 202 as the detection area (step S203). The processor 161 sets the area of ​​the sensing area 200 excluding the portion of the sensing area 200 in which the header 115 is located as the detection area 202. For example, the processor 161 sets the second detection area 202 as a portion of the sensing area 200 excluding at least a portion of the portion that is a predetermined distance or less from the millimeter-wave radar 125.

[0086] Fig. 7 is a top view showing an example of a portion 210 to be removed in setting the second detection area 202. Fig. 8 is a top view showing an example of the second detection area 202.

[0087] For example, the processor 161 sets the second detection area 202 to a portion of the sensing area 200 excluding a portion 210 that is at a distance from the millimeter-wave radar 125 that is equal to or less than a predetermined distance D1. The predetermined distance D1 is, for example, not less than 2 m and not more than 5 m, but is not limited to this value. The predetermined distance D1 can be set to any length depending on the size of the work machine 110.

[0088] FIG. 8 shows a second detection area 202 that is set by excluding the portion 210 from the sensing area 200 .

[0089] The sensor data output by the millimeter-wave radar 125 includes data on the distance between each measurement point and the millimeter-wave radar 125, data on the angle of each measurement point relative to the millimeter-wave radar 125, etc. The processor 161 can avoid detecting objects in the portion 210 by not using measurement points whose distance from the millimeter-wave radar 125 is equal to or shorter than a predetermined distance D1 for object detection. The processor 161 performs processing to detect objects using the millimeter-wave radar 125 in a second detection area 202, which is the sensing area 200 excluding the portion 210 (step S205).

[0090] When the operation of the harvester 100 is completed, the object detection process is completed (step S206).

[0091] If the height of the header 115 changes, the header 115 may fall within the sensing area 200. According to this embodiment, by changing the pattern of the detection areas 201 and 202 for object detection in accordance with the change in the height of the header 115, it is possible to prevent object detection processing from being performed on the portion of the sensor data that indicates the header 115. This makes it possible to prevent the header 115 from being detected as an obstacle or the like. Furthermore, by reducing the amount of calculation by the processor 161, it is possible to increase the object detection processing speed.

[0092] The size of the predetermined distance D1 may be changed in accordance with the change in the height of the header 115. Fig. 9 is a top view showing a plurality of types of portions 210 having different sizes.

[0093] For example, the processor 161 increases the predetermined distance D1 when the height of the header 115 is relatively high compared to when the height is low. For example, in a configuration in which the header 115 enters a more distant portion of the sensing area 200 as the height of the header 115 increases, the predetermined distance D1 increases as the height of the header 115 increases. This makes it possible to ensure a wide detection area when the header 115 is in a relatively low position while suppressing detection of the header 115 as an obstacle or the like.

[0094] In the examples shown in Figures 7 and 9, the shape of the excluded portion 210 is a substantially fan-shape, but this is not limited to this and other shapes may be used. Figure 10 is a top view showing another example of the excluded portion 210. In the example shown in Figure 10, the shape of the excluded portion 210 is a substantially triangular shape. Figure 11 is a diagram showing an example of the second detection area 202 set by excluding the substantially triangular portion 210 from the sensing area 200. The excluded portion 210 may have any shape depending on the configuration of the harvester 100 and the work machine 110.

[0095] In the description of the above embodiment, a harvester has been given as an example of the agricultural machine 100, but as mentioned above, the agricultural machine 100 is not limited to a harvester. For example, the agricultural machine 100 may be a tractor to which a work machine is connected.

[0096] Figure 12 is a diagram showing an example of a tractor 100 connected to a work machine 110. In the example shown in Figure 12, a front loader is connected to the tractor 100 as an example of the work machine 110. Even in a configuration in which the work machine 110 is connected to the tractor 100, as described above, the pattern of the detection area in which objects are detected using the millimeter-wave radar 125 is changed in accordance with changes in the height of the work machine 110. This makes it possible to prevent the work machine 110 from being detected as an obstacle or the like. Furthermore, by reducing the amount of calculation by the processor 161, the object detection processing speed can be increased.

[0097] As described above, the work vehicle 100 may be a construction machine. FIG. 13 is a diagram showing an example of the construction machine 100 of this embodiment. In the example shown in FIG. 13, the construction machine 100 is a loader. The construction machine 100 may be a wheel loader or a crawler loader. The construction machine 100 may also be a compact track loader (CTL) or a skid steer loader (SSL). The work machine 110 of the construction machine 100 shown in FIG. 13 includes an arm and a bucket. The type of the work machine 110 is not limited to the above and is arbitrary. For example, the work machine 110 may include pallet forks.

[0098] Even in the configuration of a construction machine 100 equipped with a work machine 110, as described above, the pattern of the detection area in which objects are detected using the millimeter wave radar 125 is changed in accordance with changes in the height of the work machine 110. This makes it possible to prevent the work machine 110 from being detected as an obstacle or the like. Furthermore, by reducing the amount of calculation by the processor 161, the object detection processing speed can be increased.

[0099] In the above description of the embodiment, the pattern of the detection area where object detection is performed using the millimeter-wave radar 125 is changed in accordance with changes in the height of the work machine 110, but this is not limiting. The pattern of the detection area where object detection is performed using a sensor other than the millimeter-wave radar 125 may also be changed in accordance with changes in the height of the work machine 110. For example, the pattern of the detection area where object detection is performed using the LiDAR sensor 127 may be changed in accordance with changes in the height of the work machine 110. This makes it possible to suppress detection of the work machine 110 as an obstacle or the like. Furthermore, by reducing the amount of calculation by the processor 161, the object detection processing speed can be increased.

[0100] The sensing system 10 of the present embodiment can also be retrofitted to a work vehicle that does not have these functions. Such a system can be manufactured and sold independently of the work vehicle. The computer program used in such a system can also be manufactured and sold independently of the work vehicle. The computer program can be provided, for example, by being stored on a computer-readable non-transitory storage medium. The computer program can also be provided by downloading via a telecommunications line (for example, the Internet).

[0101] Some or all of the processing performed by the processor 161 in the sensing system 10 may be performed by another device. Such other device may be at least one of the ECUs 165-167 and the signal processing circuit of the millimeter-wave radar 125. In that case, such other device and the processor 161 function as the processing device of the sensing system 10, or such other device functions as the processing device of the sensing system 10. For example, if some of the processing performed by the processor 161 is performed by the signal processing circuit of the millimeter-wave radar 125, the processor 161 and the signal processing circuit of the millimeter-wave radar 125 function as the processing device of the sensing system 10. The control device 160 may function as the processing device of the sensing system 10.

[0102] As described above, the present disclosure includes the sensing system, work vehicle, sensing method, and computer program described below.

[0103] [Item 1] A sensing system for a work vehicle that uses a work machine to perform work, comprising: a sensor that is mounted on the work vehicle and senses the environment around the work vehicle to generate sensor data; and a processing device that detects objects located in a detection area around the work vehicle based on the sensor data, wherein the work machine is capable of changing its height relative to the main body of the work vehicle, and the processing device changes the pattern of the detection area in which the object is detected in accordance with changes in the height of the work machine.

[0104] [Item 2] The sensing system according to Item 1, wherein the processing device sets the detection area to a portion of the sensing area sensed by the sensor, excluding a portion of the sensing area where the work machine enters.

[0105] [Item 3] The sensing system according to Item 1 or 2, wherein when the height of the work machine is equal to or greater than a predetermined height, the processing device sets, as the detection area, a portion of the sensing area sensed by the sensor excluding at least a portion of the portion that is a predetermined distance or less from the sensor.

[0106] [Item 4] The sensing system according to Item 3, wherein the processing device changes the magnitude of the predetermined distance in accordance with a change in the height of the work machine.

[0107] [Item 5] The sensing system according to Item 4, wherein the processing device sets the predetermined distance to be greater when the height of the work machine is high than when the height of the work machine is low.

[0108] [Item 6] The sensing system according to any one of Items 3 to 5, wherein the predetermined distance is 2 m or more and 5 m or less.

[0109] [Item 7] The sensing system according to any one of Items 1 to 6, wherein the processing device acquires data on the height of the work machine based on an output signal of a sensor that detects the height of the work machine relative to the main body of the work vehicle.

[0110] [Item 8] The sensing system according to any one of Items 1 to 7, wherein the sensor is a millimeter wave radar.

[0111] [Item 9] The sensing system according to any one of items 1 to 8, wherein the work vehicle is a harvester, and the work machine includes a header that cuts crops in a field.

[0112] [Item 10] The sensing system according to any one of items 1 to 8, wherein the work vehicle is a tractor, and the work machine includes an implement connected to the tractor.

[0113] [Item 11] The sensing system according to any one of Items 1 to 8, wherein the work vehicle is a mobile construction machine.

[0114] [Item 12] The sensing system according to Item 11, wherein the construction machine is a loader.

[0115] [Item 13] A work vehicle equipped with the sensing system according to any one of items 1 to 12.

[0116] [Item 14] The work vehicle according to Item 13, further comprising: a travel device that causes the work vehicle to travel; and a control device that controls the operation of the travel device and automatically drives the work vehicle.

[0117] [Item 15] A sensing method executed by a computer for detecting objects located in the vicinity of a work vehicle that uses a work machine to perform work, wherein the work machine is capable of changing its height relative to a main body of the work vehicle, and the sensing method includes: detecting objects located in a detection area around the work vehicle based on sensor data obtained by a sensor sensing the environment around the work vehicle; and changing the pattern of the detection area for detecting the objects in accordance with changes in the height of the work machine.

[0118] [Item 16] A computer program that causes a computer to execute a process for detecting objects located in the vicinity of a work vehicle that uses a work machine to perform work, wherein the work machine is capable of changing its height relative to the main body of the work vehicle, and the computer program causes the computer to execute the following: detect objects located in a detection area around the work vehicle based on sensor data obtained by a sensor sensing the environment around the work vehicle; and change the pattern of the detection area for detecting the objects in accordance with changes in the height of the work machine.

[0119] The techniques of the present disclosure are particularly useful in the fields of mobile agricultural machinery and mobile construction machinery.

[0120] 10: Sensing system, 100: Work vehicle (harvester), 101: Vehicle body, 102: Traveling device, 103: Harvesting device, 104: Conveying device, 105: Thresher, 106: Tank, 107: Discharge device, 108: Straw waste treatment device, 109: Reel, 110: Work machine, 111: Prime mover (engine), 112: Transmission, 114: Cabin, 115: Header, 116: Actuator, 117: Rotating shaft, 118: Height sensor, 121: Positioning device (GNSS unit), 122: Inertial measurement unit (IMU), 125: Millimeter wave radar, 126: Camera, 127: LiDAR sensor, 128: Obstacle sensor, 131: Operation terminal, 132: Operation switch group, 140: Drive unit, 141: Power transmission mechanism, 150: Sensor group, 151: Operation lever sensor, 152: Rotation sensor, 156: Load sensor, 160: Control unit, 161: Processor, 162: RAM, 163: ROM, 164: Storage device, 165-167: ECU, 190: Communication device, 200: Sensing range, 201: First detection area, 202: Second detection area, 210: Area where object detection is not performed, D1: Distance

Claims

1. A sensing system for a work vehicle that performs work using a working machine, the sensing system comprising: a sensor provided on the work vehicle that senses the environment around the work vehicle and generates sensor data; and a processing device that detects an object located in a detection area around the work vehicle based on the sensor data, wherein the working machine is capable of changing its height relative to the body of the work vehicle, and the processing device changes the pattern of the detection area for detecting the object in response to a change in the height of the working machine.

2. The sensing system according to claim 1, wherein the processing device sets, as the detection area, a portion of the sensing area sensed by the sensor excluding a portion where the working machine enters the sensing area.

3. The sensing system according to claim 1 or 2, wherein when the height of the working machine is equal to or greater than a predetermined height, the processing device sets, as the detection area, a portion of the sensing area sensed by the sensor excluding at least a part of a portion where the distance from the sensor is equal to or less than a predetermined distance.

4. The sensing system according to claim 3, wherein the processing device changes the magnitude of the predetermined distance in response to a change in the height of the working machine.

5. The sensing system according to claim 4, wherein the processing device increases the predetermined distance when the height of the working machine is high compared to when it is low.

6. The sensing system according to claim 3, wherein the predetermined distance is 2 m or more and 5 m or less.

7. The sensing system according to claim 1 or 2, wherein the processing device acquires data on the height of the working machine based on an output signal of a sensor that detects the height of the working machine relative to the body of the work vehicle.

8. The sensing system according to claim 1 or 2, wherein the sensor is a millimeter-wave radar.

9. The sensing system according to claim 1 or 2, wherein the work vehicle is a harvester, and the working machine includes a header for harvesting crops in a field.

10. The sensing system according to claim 1 or 2, wherein the work vehicle is a tractor, and the working machine includes an implement connected to the tractor.

11. The sensing system according to claim 1 or 2, wherein the work vehicle is a mobile construction machine.

12. The sensing system according to claim 11, wherein the construction machine is a loader.

13. A work vehicle comprising the sensing system according to claim 1 or 2.

14. The work vehicle according to claim 13, further comprising a traveling device that travels the work vehicle, and a control device that controls the operation of the traveling device and automatically operates the work vehicle.

15. A sensing method for detecting an object located around a work vehicle that performs work using a working machine, the computer-executed method comprising: the working machine being capable of changing its height relative to the main body of the work vehicle; the sensing method comprising: detecting an object located in a detection area around the work vehicle based on sensor data obtained by a sensor sensing the environment around the work vehicle; and changing a pattern of the detection area for detecting the object in accordance with a change in the height of the working machine.

16. A computer program for causing a computer to execute a process of detecting an object located around a work vehicle that performs work using a working machine, the working machine being capable of changing its height relative to the main body of the work vehicle, the computer program causing the computer to: detect an object located in a detection area around the work vehicle based on sensor data obtained by a sensor sensing the environment around the work vehicle; and change a pattern of the detection area for detecting the object in accordance with a change in the height of the working machine.

Images