Sensing system, agricultural machine, and sensing device

Abstract

This sensing system comprises a first ranging sensor mounted on a work vehicle, a second ranging sensor mounted on an implement connected to the work vehicle, a marker member located within the sensing range of the first ranging sensor, and a processing device. The processing device estimates relative position and posture of the second ranging sensor with respect to the first ranging sensor on the basis of first sensor data generated by the first ranging sensor sensing an area including the marker member, and outputs data indicating the estimated position and posture.

Description

[Technical Field]

[0001] The present disclosure relates to a sensing system, an agricultural machine, and a sensing device. [Background technology]

[0002] Research and development is underway on smart agriculture, which utilizes ICT (Information and Communication Technology) and IoT (Internet of Things) as the next generation of agriculture. Research and development is also underway to automate and unmanned agricultural machinery used in fields, such as tractors, rice transplanters, and combine harvesters. For example, work vehicles that use positioning systems such as the Global Navigation Satellite System (GNSS), which enables precise positioning, to navigate autonomously within fields and perform agricultural work have been put into practical use.

[0003] Meanwhile, development is also underway for work vehicles that can travel autonomously using distance measurement sensors such as LiDAR (Light Detection and Ranging). For example, Patent Document 1 discloses an example of a work vehicle that uses a LiDAR sensor to recognize crop rows in a field and travels autonomously between the crop rows. Patent Document 2 discloses an example of a work vehicle that can travel autonomously along a target route set in a field while detecting obstacles using a LiDAR sensor. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-154379 [Patent Document 2] Japanese Patent Application Publication No. 2019-170271 Summary of the Invention [Problem to be solved by the invention]

[0005] In a work vehicle that runs autonomously while sensing the surrounding environment using a distance measurement sensor such as a LiDAR sensor, sensing can be hindered by an implement connected to the work vehicle. For example, if an implement is connected to the front of the work vehicle, a blind spot is created within the sensing range in front of the work vehicle, hindering sensing.

[0006] The present disclosure provides a technique for avoiding the implementation from interfering with sensing. [Means for solving the problem]

[0007] A sensing system according to one embodiment of the present disclosure includes a first ranging sensor attached to a work vehicle, a second ranging sensor attached to an implement coupled to the work vehicle, a marker member located within a sensing range of the first ranging sensor, and a processing device. The processing device estimates a relative position and orientation of the second ranging sensor with respect to the first ranging sensor based on first sensor data generated by the first ranging sensor sensing an area including the marker member, and outputs data indicating the estimated position and orientation.

[0008] A general or specific aspect of the present disclosure may be realized by an apparatus, a system, a method, an integrated circuit, a computer program, or a computer-readable non-transitory storage medium, or any combination thereof. The computer-readable storage medium may include a volatile storage medium or a non-volatile storage medium. An apparatus may be composed of multiple devices. When an apparatus is composed of two or more devices, the two or more devices may be located in a single device or may be located separately in two or more separate devices. [Effects of the Invention]

[0009] According to the embodiment of the present disclosure, it is possible to avoid the implementation interfering with sensing. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a side view schematically illustrating an agricultural machine according to an exemplary embodiment of the present disclosure. FIG. [Figure 2] 10A and 10B are diagrams illustrating an example of the configuration of a marker member. [Figure 3] 10A to 10C are diagrams for explaining a process of estimating the relative position and orientation of a second LiDAR sensor with respect to a first LiDAR sensor. [Figure 4] 10A and 10B are diagrams illustrating other examples of marker members. [Figure 5] FIG. 10 is a diagram illustrating another example of a sensing system. [Figure 6] 10A and 10B are diagrams illustrating an example of a marker member including multiple portions of different colors. [Figure 7] FIG. 10 is a diagram illustrating yet another example of a sensing system. [Figure 8] FIG. 1 is a diagram for explaining an overview of a system according to an exemplary embodiment of the present disclosure. [Figure 9] 1 is a side view schematically showing an example of a work vehicle and an implement coupled to the work vehicle. FIG. [Figure 10] FIG. 2 is a block diagram showing an example of the configuration of a work vehicle and an implement. [Figure 11] FIG. 1 is a conceptual diagram showing an example of a work vehicle that performs positioning using RTK-GNSS. [Figure 12] FIG. 1 is a diagram illustrating an example of an environment in which a work vehicle travels. [Figure 13A] FIG. 2 is a diagram schematically illustrating an example of a travel route of a work vehicle. [Figure 13B] FIG. 10 is a diagram schematically illustrating another example of a travel route of a work vehicle. [Figure 14] 4 is a flowchart showing an example of the operation of steering control during automatic driving executed by the control device. [Figure 15A] FIG. 1 is a diagram illustrating an example of a work vehicle traveling along a target route. [Figure 15B] FIG. 10 is a diagram illustrating an example of a work vehicle at a position shifted to the right from the target route. [Figure 15C] FIG. 10 is a diagram illustrating an example of a work vehicle that is shifted to the left from the target route. [Figure 15D] FIG. 10 is a diagram illustrating an example of a work vehicle facing in an inclined direction relative to a target route. [Figure 16] 10 is a flowchart illustrating a specific example of a self-position estimation process. DETAILED DESCRIPTION OF THE INVENTION

[0011] (Definition of terms) In this disclosure, a "work vehicle" refers to a vehicle used to perform work on a work site. A "work site" is any location where work is performed, such as a farm field, forest, or construction site. A "field" is any location where agricultural work is performed, such as an orchard, field, rice paddy, grain farm, or pasture. A work vehicle may be, for example, an agricultural machine such as a tractor, rice transplanter, combine harvester, riding cultivator, or riding brush cutter, or a vehicle used for non-agricultural purposes such as a construction vehicle or snowplow. A work vehicle in this disclosure can be equipped with an implement (also referred to as a "work machine" or "working device") on at least one of its front and rear, depending on the type of work. The act of a work vehicle traveling while performing work is sometimes referred to as "work driving."

[0012] "Agricultural machinery" means machinery used for agricultural purposes. Examples of agricultural machinery include tractors, harvesters, rice transplanters, riding tillers, vegetable transplanters, mowers, seed sowing machines, fertilizer applicators, and agricultural mobile robots. Not only can a work vehicle such as a tractor function alone as "agricultural machinery," but the entire work vehicle and an implement attached to or towed by the work vehicle can also function as one "agricultural machinery." Agricultural machinery performs agricultural tasks on the ground in a field, such as plowing, sowing, pest control, fertilizing, planting crops, or harvesting.

[0013] "Autonomous driving" refers to controlling the driving of a vehicle through the operation of a control device, rather than through manual operation by a driver. During autonomous driving, not only the driving of the vehicle but also the operation of a work task (e.g., the operation of an implement) may be controlled automatically. The driving of a vehicle through autonomous driving is referred to as "autonomous driving." The control device may control at least one of the steering, speed adjustment, and start and stop of driving required for the vehicle's driving. When controlling a work vehicle equipped with an implement, the control device may control operations such as raising and lowering the implement and starting and stopping the implement's operation. Autonomous driving may include not only the vehicle's driving toward a destination along a predetermined route, but also the vehicle's driving to follow a target. An autonomously driving vehicle may drive partially based on user instructions. Furthermore, an autonomously driving vehicle may operate in a manual driving mode, in addition to an autonomous driving mode, in which the vehicle is driven by manual operation by the driver. "Automatic steering" refers to steering a vehicle through the operation of a control device, rather than manually. Part or all of the control device may be external to the vehicle. Communication of control signals, commands, data, etc. may occur between the vehicle and a control device external to the vehicle. An autonomously driven vehicle may travel autonomously while sensing the surrounding environment without a human being involved in controlling the vehicle's travel. A vehicle capable of autonomous travel can travel unmanned. During autonomous travel, obstacles may be detected and obstacle avoidance operations may be performed.

[0014] A "ranging sensor" is a sensor that measures distance, i.e., distance. A ranging sensor may be configured to measure distances to one or more measurement points and output data indicating the distances or data indicating the positions of the measurement points converted from the distances. A ranging sensor may include, for example, a LiDAR sensor, a Time of Flight (ToF) camera, a stereo camera, or any combination thereof. A LiDAR sensor measures distance by emitting light (e.g., infrared light or visible light). A ranging technology such as Time of Flight (ToF) or Frequency Modulated Continuous Wave (FMCW) may be used for ranging.

[0015] An "environment map" is data that uses a specified coordinate system to represent the positions or areas of objects in the environment in which the work vehicle travels. Examples of coordinate systems that define the environment map include not only world coordinate systems, such as geographic coordinate systems fixed relative to the Earth, but also odometry coordinate systems that display poses based on odometry information. The environment map may also include information other than the positions of objects in the environment (e.g., attribute information and other information). The environment map includes various types of maps, such as point cloud maps and grid maps.

[0016] (Embodiment) Hereinafter, embodiments of the present disclosure will be described. However, more detailed descriptions than necessary may be omitted. For example, detailed descriptions of well-known matters and redundant descriptions of substantially identical configurations may be omitted. This is to avoid unnecessary redundancy in the following description and to facilitate understanding by those skilled in the art. Note that the inventors provide the accompanying drawings and the following description to enable those skilled in the art to fully understand the present disclosure, and do not intend for them to limit the subject matter described in the claims. In the following description, components having the same or similar functions are designated by the same reference numerals.

[0017] The following embodiments are examples, and the technology of the present disclosure is not limited to the following embodiments. For example, the numerical values, shapes, materials, steps, step order, display screen layout, etc. shown in the following embodiments are merely examples, and various modifications are possible as long as no technical contradiction occurs. Furthermore, one aspect can be combined with another aspect as long as no technical contradiction occurs.

[0018] FIG. 1 is a side view schematically illustrating an agricultural machine according to an exemplary embodiment of the present disclosure. The agricultural machine illustrated in FIG. 1 includes a work vehicle 100 and an implement 300 coupled to the work vehicle 100. In this embodiment, the work vehicle 100 is a tractor. The work vehicle 100 includes a coupling device 108 at its front. The implement 300 is coupled to the front of the work vehicle 100 via the coupling device 108. The work vehicle 100 may also include a coupling device at its rear for connecting an implement. The rear coupling device and the rear implement are not illustrated in FIG. 1. The implement 300 may be any work machine that performs agricultural work in front of the work vehicle 100, such as a front loader, harvester, sweeper, or sprayer.

[0019] 1 is equipped with a sensing system that senses the environment around work vehicle 100. The sensing system can be used, for example, for obstacle detection or self-localization. Work vehicle 100 can be configured to perform autonomous driving based on the results of self-localization using the sensing system.

[0020] The sensing system in this embodiment includes a first LiDAR sensor 140A, a second LiDAR sensor 140B, a marker member 148, and a processing device 250. The first LiDAR sensor 140A is an example of a first ranging sensor. The second LiDAR sensor 140B is an example of a second ranging sensor. The first LiDAR sensor 140A is attached to the work vehicle 100. In the example shown in FIG. 1, the first LiDAR sensor 140A is attached near the upper front end of the cabin 105 of the work vehicle 100. The second LiDAR sensor 140B is attached to the implement 300. In the example shown in FIG. 1, the second LiDAR sensor 140B is attached near the upper front end of the implement 300. The positions of the first LiDAR sensor 140A and the second LiDAR sensor 140B may be other positions.

[0021] The second LiDAR sensor 140B is externally attached to the implement 300. The second LiDAR sensor 140B can be powered by, for example, a battery or power supplied from the implement 300. The second LiDAR sensor 140B shown in FIG. 1 includes a mounting fixture 149 for mounting to the implement 300. The mounting fixture 149 can include various parts used for mounting to the implement 300, such as magnets, bolts and nuts, screws, or connecting hardware. The mounting fixture 149 makes it easy to mount and remove the second LiDAR sensor 140B to and from the implement 300. B can be mounted at a position on the implement 300 where there are no obstacles between it and the first LiDAR sensor 140A and where there are relatively few blind spots ahead.

[0022] A fixture for fixing the second LiDAR sensor 140B may be provided on the implement 300. The fixture may have a structure that engages with the mounting fixture 149 of the second LiDAR sensor 140B, for example. In this case, the second LiDAR sensor 140B is fixed to the implement 300 by the mounting fixture 149 engaging with the mounting fixture.

[0023] The second LiDAR sensor 140B can be attached to various implements, not limited to a specific implement 300. Therefore, sensing, which will be described later, is possible regardless of the type of implement 300. Such an external second LiDAR sensor 140B can be manufactured or sold separately.

[0024] The marker member 148 is disposed within the sensing range of the first LiDAR sensor 140A. In the example shown in FIG. 1 , the marker member 148 is attached to the second LiDAR sensor 140B. The marker member 148 may be provided at a position on the implement 300 away from the second LiDAR sensor 140B. The positional and orientational relationship between the marker member 148 and the second LiDAR sensor 140B is known, and this information may be recorded in advance in a storage device internal or external to the processing device 250. The marker member 148 may include, for example, one or more reflective portions that have a higher reflectivity of light emitted from the first LiDAR sensor 140A than other portions of the marker member 148.

[0025] When the second LiDAR sensor 140B is provided on an implement 300 in front of the work vehicle 100 as shown in FIG. 1, the first LiDAR sensor 140A is positioned to sense at least the area in front of the work vehicle 100. The first LiDAR sensor 140A generates and outputs first sensor data by sensing an area including the marker member 148. Meanwhile, the second LiDAR sensor 140B generates and outputs second sensor data by sensing an area in front of the implement 300. In FIG. 1, the sensing range 30A of the first LiDAR sensor 140A and the sensing range 30B of the second LiDAR sensor 140B are illustrated as fan shapes with dashed lines. Each of the sensing ranges 30A and 30B extends not only vertically but also horizontally in the traveling direction of the work vehicle 100.

[0026] Each of the first LiDAR sensor 140A and the second LiDAR sensor 140B may be a scanning sensor that generates information indicating the distribution of objects in space by, for example, scanning a laser beam. The LiDAR sensor may be configured to measure the distance to a reflection point located on the surface of an object, for example, using the ToF method. A LiDAR sensor that measures distance using the ToF method may be configured to emit a laser pulse, i.e., a pulsed laser beam, and measure the time it takes for the laser pulse to be reflected by an object in the surrounding environment and return to the LiDAR sensor. The distance measurement method is not limited to the ToF method, and other methods such as the FMCW method may also be used. In the FMCW method, light whose frequency changes linearly over time is emitted, and the distance is calculated based on the beat frequency of the interference light generated by the interference between the emitted light and the reflected light. The coordinates of the reflection point in a coordinate system fixed to the work vehicle 100 are calculated based on the distance and direction to the reflection point. Scanning LiDAR sensors can be classified into two-dimensional LiDAR sensors and three-dimensional LiDAR sensors. A two-dimensional LiDAR sensor may scan an environment by rotating a laser beam within a single plane. On the other hand, a three-dimensional LiDAR sensor may scan an environment by, for example, rotating multiple laser beams along different conical planes. While each of the first LiDAR sensor 140A and the second LiDAR sensor 140B shown in FIG. 1 is a three-dimensional LiDAR sensor, a two-dimensional LiDAR sensor may also be used.

[0027] Each of the first LiDAR sensor 140A and the second LiDAR sensor 140B is not limited to a scan-type sensor, but may be a flash-type sensor that emits light that diffuses over a wide area to acquire information about the distance distribution of objects in space. Scan-type LiDAR sensors use light with higher intensity than flash-type LiDAR sensors, and can therefore acquire distance information from longer distances. On the other hand, flash-type LiDAR sensors have a simple structure and can be manufactured at low cost, making them suitable for applications that do not require strong light.

[0028] In this embodiment, in addition to a first LiDAR sensor 140A attached to the work vehicle 100, a second LiDAR sensor 140B attached to the implement 300 is also provided. If the second LiDAR sensor 140B were not provided, the implement 300 would be located within the sensing range of the first LiDAR sensor 140A, preventing sensing of the area ahead of the implement 300. In particular, if the implement 300 in front of the work vehicle 100 is large, there would be many blind spots within the sensing range of the first LiDAR sensor 140A, making it impossible to acquire sufficient data necessary for obstacle detection or self-localization. To avoid this problem, in this embodiment, an external second LiDAR sensor 140B is attached to the implement 300. Obstacle detection or self-localization is performed based on second sensor data output from the second LiDAR sensor 140B. The first LiDAR sensor 140A can be used to estimate the position and orientation of the second LiDAR sensor 140B. Point cloud data indicating the distribution of features around the work vehicle 100 may be generated by combining the first sensor data output from the first LiDAR sensor 140A and the second sensor data output from the second LiDAR sensor 140B.

[0029] The processing device 250 is a computer that processes data output from the first LiDAR sensor 140A and the second LiDAR sensor 140B. In the example shown in FIG. 1 , the processing device 250 is disposed inside the cabin 105 of the work vehicle 100. The processing device 250 may be implemented as part of a control device that controls the operation of the work vehicle 100. The processing device 250 may be realized by, for example, an electronic control unit (ECU). The processing device 250 is not limited to being disposed inside the cabin 105, and may be disposed in other locations. For example, the processing device 250 may be built into the first LiDAR sensor 140A. The processing device 250 may be realized by multiple computers or multiple circuits. At least some of the functions of the processing device 250 may be realized by an external computer, such as a server, located away from the work vehicle 100.

[0030] The processing device 250 may be communicatively connected to the first LiDAR sensor 140A and the second LiDAR sensor 140B wirelessly (e.g., via Bluetooth (registered trademark) or 4G or 5G communication, etc.) or via a wire (e.g., via CAN communication, etc.). The processing device 250 acquires first sensor data from the first LiDAR sensor 140A and second sensor data from the second LiDAR sensor 140B.

[0031] The processing device 250 estimates the relative position and orientation of the second LiDAR sensor 140B with respect to the first LiDAR sensor 140A based on the first sensor data. For example, the processing device 250 estimates the relative position and orientation of the second LiDAR sensor 140B with respect to the first LiDAR sensor 140A based on the first sensor data and information indicating the positional and orientation relationship between the second LiDAR sensor 140B and the marker member 148. Here, the "position and orientation" is a combination of position and orientation and is also referred to as a "pose." In the following description, the relative position and orientation of the second LiDAR sensor 140B with respect to the first LiDAR sensor 140A may be simply referred to as the "position and orientation of the second LiDAR sensor 140B."

[0032] The positional relationship between the work vehicle 100 and the implement 300 is not fixed, and the position and orientation of the implement 300 relative to the work vehicle 100 may fluctuate due to play in the coupling device 108 or the movement of the implement 300 itself. For example, while the work vehicle 100 is traveling, the implement 300 may swing up and down or left and right relative to the work vehicle 100. Accordingly, the second LiDAR sensor 140B may also swing up and down or left and right relative to the work vehicle 100. In order to use the second sensor data output from the second LiDAR sensor 140B for autonomous driving of the work vehicle 100, it is necessary to convert the second sensor data into data expressed in a coordinate system fixed to the work vehicle 100. This coordinate conversion requires information on the relative position and orientation of the second LiDAR sensor 140B with respect to the first LiDAR sensor 140A (or the work vehicle 100). Therefore, the processing device 250 estimates the relative position and orientation of the second LiDAR sensor 140B with respect to the first LiDAR sensor 140A based on the first sensor data generated by the first LiDAR sensor 140A sensing the area including the marker member 148.

[0033] The processing device 250 can estimate the relative position and orientation of the second LiDAR sensor 140B with respect to the first LiDAR sensor 140A based on the positions and / or shapes of one or more reflective portions of the marker member 148. A specific example of this processing will be described below.

[0034] FIG. 2 is a diagram showing a configuration example of the marker member 148. In this example, the marker member 148 has three reflecting portions 148a, 148b, and 148c. Each of the reflecting portions 148a, 148b, and 148c has a reflecting surface provided with a reflecting material such as a reflective tape that reflects light emitted from the first LiDAR sensor 140A with high reflectivity. The marker member 148 has a reflecting surface provided with a reflective material such as a reflective tape that reflects light emitted from the first LiDAR sensor 140A with high reflectivity. SensorThe marker member 148 may be arranged so that it faces the direction of the first LiDAR sensor 140A. For example, the marker member 148 may be arranged so that the normal direction of the reflective surface of each of the reflective portions 148a, 148b, and 148c is close to the direction toward the first LiDAR sensor 140A. A retroreflective material (e.g., retroreflective tape) may be provided on the reflective surface of each of the reflective portions 148a, 148b, and 148c. By using the retroreflective material, light emitted from the light-emitting portion of the first LiDAR sensor 140A can be strongly reflected toward the light-receiving portion of the first LiDAR sensor 140A. The reflective surface of each of the reflective portions 148a, 148b, and 148c has a two-dimensional extent and may have a size larger than 10 mm × 10 mm, for example. The reflective surface of each of the reflective portions 148a, 148b, and 148c may be larger than 30 mm × 30 mm. The larger the size of the reflecting surface, the easier it is to detect the reflecting portions 148a, 148b, and 148c based on the first sensor data.

[0035] The three reflectors 148a, 148b, and 148c shown in FIG. 2 are supported by three pillars 148e, 148f, and 148g extending in three directions from a central base 148d of the marker member 148. The three pillars 148e, 148f, and 148g are located in a plane perpendicular to the front direction D2 of the second LiDAR sensor 140B. The three pillars 148e, 148f, and 148g extend in directions that are 90 degrees apart within the plane. The pillars 148f and 148g extend in the left-right direction toward the front direction D2 of the second LiDAR sensor 140B, and the pillar 148e extends vertically upward. In the example shown in FIG. 2, the pillars 148e, 148f, and 148g are all the same length. The three reflectors 148a, 148b, and 148c are provided at the tips of the three pillars 148e, 148f, and 148g, respectively. In this manner, the reflectors 148a, 148b, and 148c are provided at positions spaced apart in three directions, left and right, and vertically, from the base 148d. This arrangement makes it easier to estimate the position and orientation of the second LiDAR sensor 140B. Note that the positional relationship between the three reflectors 148a, 148b, and 148c is not limited to the relationship shown in the figure, and can be changed as appropriate.

[0036] The first LiDAR sensor 140A performs a beam scan within an area including the three reflectors 148a, 148b, and 148c to generate first sensor data indicating the distance distribution or position distribution of objects within the area. The processing device 250 identifies the positions of the three reflectors 148a, 148b, and 148c based on the first sensor data. In the example shown in FIG. 2, each of the reflectors 148a, 148b, and 148c has a rectangular shape, but other shapes are also possible. For example, each of the reflectors 148a, 148b, and 148c may have a distinctive shape such as an ellipse, a perfect circle, a triangle, or another polygon, or a star or cross. The shapes of the reflectors 148a, 148b, and 148c may be different from each other. The processing device 250 estimates the position and orientation of the second LiDAR sensor 140B relative to the first LiDAR sensor 140A based on the estimated positions of each of the reflecting portions 148a, 148b, 148c.

[0037] FIG. 3 is a diagram illustrating a process for estimating the relative position and orientation of the second LiDAR sensor 140B with respect to the first LiDAR sensor 140A. In FIG. 3, three light rays directed from the first LiDAR sensor 140A toward the three reflectors 148a, 148b, and 148c are illustrated by solid arrows. In this example, the first LiDAR sensor 140A measures the distance to each reflection point of an object present within an area including the three reflectors 148a, 148b, and 148c by performing a beam scan centered on the forward direction D1. Based on the distance and direction to each reflection point, the first LiDAR sensor 140A can generate point cloud data as first sensor data, which includes information on the three-dimensional coordinate values ​​of each reflection point in a first sensor coordinate system fixed to the first LiDAR sensor 140A. The first sensor coordinate system is also a coordinate system fixed to the work vehicle 100. 3 illustrates mutually orthogonal X, Y, and Z axes that define the first sensor coordinate system. The first sensor data may be, for example, point cloud data that indicates the correspondence between the identification numbers of multiple reflection points at which reflected light is detected and the coordinate values ​​(x, y, z) of each reflection point. The first sensor data may also include information on the brightness of each reflection point.

[0038] The processing device 250 extracts some of the reflection points corresponding to the three reflecting portions 148a, 148b, and 148c from the multiple reflection points indicated by the first sensor data output from the first LiDAR sensor 140A. For example, the processing device 250 may cluster the point cloud indicated by the first sensor data based on the distances between the points, and extract three clusters that are estimated to reflect the actual positional relationship and shape of the reflecting portions 148a, 148b, and 148c as the reflection points corresponding to the reflecting portions 148a, 148b, and 148c. Alternatively, if the first sensor data includes information on the brightness of each reflection point, the processing device 250 may extract, from among the reflection points whose brightness exceeds a threshold, reflection points that constitute three clusters that are close to each other as the reflection points corresponding to the three reflecting portions 148a, 148b, and 148c.

[0039] The processing device 250 determines a representative position for each of the reflecting portions 148a, 148b, and 148c based on the coordinate values ​​of the multiple reflecting points corresponding to the extracted reflecting portions 148a, 148b, and 148c. The representative position for each reflecting portion may be, for example, the average or median value of the positions of the multiple reflecting points corresponding to that reflecting portion. Hereinafter, the representative position for each reflecting portion will be referred to as the "position" of that reflecting portion.

[0040] 3, the relationship between the positions of the three reflectors 148a, 148b, and 148c and the reference position of the second LiDAR sensor 140B (e.g., the position of the beam emission point) and the front direction D2 of the second LiDAR sensor 140B is known, and data indicating this relationship is stored in advance in a storage device. The processing device 250 determines the reference position and the front direction D2 of the second LiDAR sensor 140B in the first sensor coordinate system based on the data indicating this relationship and the positions of the three reflectors 148a, 148b, and 148c determined based on the first sensor data.

[0041] 3, it is assumed that the positions of the three reflectors 148a, 148b, and 148c in the first sensor coordinate system are determined as P1 (x1, y1, z1), P2 (x2, y2, z2), and P3 (x3, y3, z3), respectively. In this case, the processing device 250 can determine the reference position and the front direction D2 of the second LiDAR sensor 140B in the first sensor coordinate system, for example, by the following method.

[0042] The processing device 250 first determines, by calculation, the center of gravity of a triangle formed by the positions P1, P2, and P3 of the three reflectors 148a, 148b, and 148c determined based on the first sensor data, and the normal vector of a plane including the positions P1, P2, and P3. The processing device 250 determines, as the reference position of the second LiDAR sensor 140B, a position displaced a predetermined distance in a predetermined direction (for example, the direction of a vector from point P1 toward the center of gravity). The processing device 250 also determines the direction of the normal vector of the determined plane as the forward direction D2 of the second LiDAR sensor 140B. This allows the processing device 250 to estimate the position and attitude angles (roll, pitch, and yaw) of the second LiDAR sensor 140B. Note that the method for estimating the position and orientation of the second LiDAR sensor 140B depends on the relationship between the relative position and orientation between the second LiDAR sensor 140B and the marker member 148. For example, if there is a deviation between the direction of the normal to the plane including the three points P1, P2, and P3 on the marker member 148 and the front direction D2 of the second LiDAR sensor 140B, the relative position and orientation of the second LiDAR sensor 140B with respect to the first LiDAR sensor 140A is estimated taking into account the deviation.

[0043] The marker member 148 may be provided at a position away from the second LiDAR sensor 140B. The positional relationship between the marker member 148 and the second LiDAR sensor 140B and the size of each reflective portion can be adjusted as appropriate. If the first LiDAR sensor 140A and the second LiDAR sensor 140B are too far apart, it may be difficult to detect the reflective portion. Conversely, if the first LiDAR sensor 140A and the second LiDAR sensor 140B are too close, the reflective portion may fall outside the detection range of the first LiDAR sensor 140A. In such cases, the reflective portions can be properly detected by changing the distance between the marker member 148 and the second LiDAR sensor 140B or the size of each reflective portion.

[0044] FIG. 4 is a diagram illustrating another example of the marker member 148. In this example, the marker member 148 is attached to the implement 300 independently of the second LiDAR sensor 140B. The marker member 148 illustrated in FIG. 4 includes a flat plate 148k supported by two support columns 148j, and three reflecting portions 148a, 148b, and 148c are provided on the flat plate 148k. The reflecting portions 148a, 148b, and 148c may be provided with a reflective material (e.g., a retroreflective material) having a reflectance with respect to light emitted from the first LiDAR sensor 140A higher (e.g., twice or more) than the reflectance of the flat plate 148k and the support columns 148j. The position of the marker member 148 can be changed as appropriate. Even when such a marker member 148 is used, the position and orientation of the second LiDAR sensor 140B can be estimated by processing similar to that of the example illustrated in FIG. 3.

[0045] As described above, in this embodiment, the processing device 250 recognizes the three reflecting portions 148a, 148b, and 148c of the marker member 148 provided near the second LiDAR sensor 140B based on the first sensor data output from the first LiDAR sensor 140A. The processing device 250 estimates the position and orientation of the second LiDAR sensor 140B in a coordinate system fixed to the first LiDAR sensor 140A and the work vehicle 100 based on the positions of the recognized three reflecting portions 148a, 148b, and 148c. The processing device 250 outputs data indicating the estimated position and orientation of the second LiDAR sensor 140B. For example, the processing device 250 may output the data indicating the position and orientation of the second LiDAR sensor 140B to a storage device for storage. Alternatively, the processing device 250 may transmit the data indicating the position and orientation of the second LiDAR sensor 140B to an external computer.

[0046] The processing device 250 may be configured to convert the second sensor data output from the second LiDAR sensor 140B into data expressed in a vehicle coordinate system fixed to the work vehicle 100 based on the estimated position and attitude of the second LiDAR sensor 140B and output the converted data. For example, the processing device 250 may perform coordinate transformation by performing a matrix operation to translate and rotate the coordinate values ​​of each reflection point in the second sensor data based on the estimated position and attitude angle of the second LiDAR sensor 140B. The processing device 250 may generate point cloud data in the vehicle coordinate system using such coordinate transformation. The processing device 250 may generate combined point cloud data using not only the second sensor data but also the first sensor data. For example, the processing device 250 may convert the second sensor data into data expressed in a vehicle coordinate system based on the estimated position and attitude of the second LiDAR sensor 140B, and generate and output point cloud data based on the first sensor data and the converted second sensor data.

[0047] The first LiDAR sensor 140A has been calibrated in advance (for example, before shipping from the factory), and the relationship in position and orientation between the work vehicle 100 and the first LiDAR sensor 140A is known. The processing device 250 can convert the first sensor data expressed in the first sensor coordinate system into data expressed in the vehicle coordinate system based on this known relationship. Alternatively, the processing device 250 may process the first sensor coordinate system as the vehicle coordinate system. In this case, coordinate conversion processing of the first sensor data is not necessary.

[0048] The processing device 250 can generate point cloud data by combining the first sensor data and the second sensor data expressed in the vehicle coordinate system. In this case, the processing device 250 may generate point cloud data by combining data obtained by removing data corresponding to the implement 300, the second LiDAR sensor 140B, and the marker member 148 from the first sensor data with the converted second sensor data. The shapes of the implement 300, the second LiDAR sensor 140B, and the marker member 148 and their positional relationships with the first LiDAR sensor 140A are known. Based on the known positional relationships, the processing device 250 can identify and remove data corresponding to the implement 300, the second LiDAR sensor 140B, and the marker member 148 from the first sensor data. Through such processing, it is possible to generate combined point cloud data from which point clouds corresponding to the implement 300, the second LiDAR sensor 140B, and the marker member 148, which are unnecessary for autonomous driving, have been removed.

[0049] In this embodiment, the processing device 250 itself generates point cloud data based on the first sensor data and the second sensor data, but another computer may instead generate the point cloud data. In that case, the processing device 250 transmits data indicating the estimated position and orientation of the second LiDAR sensor 140B and the second sensor data (and in some cases the first sensor data) to the other computer. The other computer can generate point cloud data indicating the distribution of features around the work vehicle 100 based on the data transmitted from the processing device 250.

[0050] In this embodiment, the second LiDAR sensor 140B transmits the generated second sensor data to the processing device 250, for example, wirelessly. The second LiDAR sensor 140B may downsample the second sensor data as necessary to compress the amount of data before transmitting the second sensor data to the processing device 250. The same applies to the first LiDAR sensor 140A.

[0051] The sensing system in this embodiment may further include a storage device that stores an environmental map. The storage device may be provided inside or outside the processing device 250. The environmental map is data that expresses the positions or areas of objects that exist in the environment in which the work vehicle 100 travels using a predetermined coordinate system. The processing device 250 may be configured to estimate the self-position of the work vehicle 100 by the following processes (S1) to (S4). (S1) The position and orientation of the second LiDAR sensor 140B are estimated by the above-described process based on the first sensor data. (S2) Based on the estimated position and orientation, the second sensor data is converted into data expressed in the vehicle coordinate system. (S3) Based on the converted second sensor data, point cloud data is generated that indicates the distribution of features around the work vehicle 100. At this time, the point cloud data may be generated further based on the first sensor data. (S4) The position and orientation of the work vehicle 100 are estimated by matching the point cloud data with the environmental map.

[0052] The processing device 250 may be configured to estimate the position and attitude of the work vehicle 100 at short time intervals (for example, at least once per second) by repeating the above processes (S1) to (S4) while the work vehicle 100 is traveling. The processing device 250 may send data indicating the estimated position and attitude of the work vehicle 100 to a control device that controls the automatic traveling of the work vehicle 100. The control device may perform steering control and speed control based on the estimated position and attitude of the work vehicle 100 and a predetermined target route. This makes it possible to realize the automatic traveling of the work vehicle 100.

[0053] In the above example, the position and orientation of the second LiDAR sensor 140B are estimated based on the first sensor data output from the first LiDAR sensor 140A. Instead of such a configuration, the position and orientation of the second LiDAR sensor 140B may be estimated using another type of sensing device.

[0054] FIG. 5 is a diagram illustrating another example of a sensing system. In the example illustrated in FIG. 5, a camera 120 is provided instead of the first LiDAR sensor 140A illustrated in FIG. 1. The camera 120 includes an image sensor that outputs image data as first sensor data. In this example, the camera 120 or the image sensor functions as a first distance measurement sensor. In FIG. 5, the shooting range 40 of the camera 120 is illustrated by a dotted line. In this example, the marker member 148 has a characteristic luminance distribution or color distribution. The processing device 250 estimates the relative position and orientation of the LiDAR sensor 140B with respect to the camera 120 based on the luminance distribution or color distribution of the area corresponding to the marker member 148 in the image represented by the first sensor data. For example, if the marker member 148 includes three reflecting portions 148a, 148b, and 148c as shown in FIG. 4, the distances to each of the reflecting portions 148a, 148b, and 148c can be estimated from the sizes of the reflecting portions 148a, 148b, and 148c recognized from the image data. The processing device 250 can estimate the position and attitude angle of the LiDAR sensor 140B as seen from the camera 120 based on the estimated distances to each of the reflecting portions 148a, 148b, and 148c.

[0055] The marker member 148 may be a member including multiple portions of different colors (e.g., three or more portions) instead of the reflective portions 148a, 148b, and 148c having high light reflectivity. FIG. 6 is a diagram showing an example of a marker member 148 including multiple portions of different colors. The marker member 148 shown in FIG. 6 has a color chart having multiple portions of different colors attached to a flat plate 148k. The processing device 250 can estimate the distance to each portion based on the size and positional relationship of the multiple portions of different colors in an image generated by the camera 120 capturing a scene including such a marker member 148. The processing device 250 can estimate the position and orientation of the second LiDAR sensor 140B based on the distance to each portion.

[0056] Fig. 7 is a diagram showing yet another example of a sensing system. In the example shown in Fig. 7, a work vehicle 100 is provided with a first GNSS receiver 110A. An implement 300 is provided with three or more second GNSS receivers 110B and a LiDAR sensor 140B. The work vehicle 100 is not equipped with a LiDAR sensor.

[0057] The first GNSS receiver 110A is provided on top of the cabin 105. The second GNSS receiver 110B is arranged around the LiDAR sensor 140B in the implement 300 at a distance from each other. The GNSS receivers 110A and 110B may be provided at positions different from those shown in FIG. 7. Each of the GNSS receivers 110A and 110B may include an antenna that receives signals from GNSS satellites and a processor that calculates its own position based on the signals received by the antenna. Each of the GNSS receivers 110A and 110B receives satellite signals transmitted from multiple GNSS satellites and performs positioning based on the satellite signals. GNSS is a general term for satellite positioning systems such as GPS (Global Positioning System), QZSS (Quasi-Zenith Satellite System, e.g., Michibiki), GLONASS, Galileo, and BeiDou.

[0058] The processing device 250 may be connected to each of the GNSS receivers 110A and 110B wirelessly or via a wire. The processing device 250 estimates the relative position and attitude of the LiDAR sensor 140B with respect to the first GNSS receiver 110A based on satellite signals received by the first GNSS receiver 110A and satellite signals received by three or more second GNSS receivers 110B. For example, the processing device 250 can calculate the relative position of each second GNSS receiver 110B with respect to the first GNSS receiver 110A by performing a moving baseline calculation with the first GNSS receiver 110A as a reference station (moving base) and the multiple second GNSS receivers 110B as moving stations (rovers). The positional relationship between each second GNSS receiver 110B and the LiDAR sensor 140B is known, and data indicating this positional relationship is recorded in advance in a storage device. The processing device 250 can estimate the position and orientation of the second LiDAR sensor 140B based on the calculated positions of the second GNSS receivers 110B and the known positional relationship between the second GNSS receivers 110B and the LiDAR sensor 140B. Note that the above-described processing performed by the processing device 250 may be performed by a processor in the first GNSS receiver 110A.

[0059] [Example of an autonomous driving work vehicle] Next, a more specific example of a work vehicle 100 that performs autonomous driving while estimating its own position using a sensing system will be described.

[0060] FIG. 8 is a diagram illustrating an overview of a system according to an exemplary embodiment of the present disclosure. The system shown in FIG. 8 includes a work vehicle 100, a terminal device 400, and a management device 600. The terminal device 400 is a computer used by a user to remotely monitor the work vehicle 100. The management device 600 is a computer managed by the business operator that operates the system. The work vehicle 100, the terminal device 400, and the management device 600 can communicate with each other via a network 80. Although FIG. 8 illustrates one work vehicle 100, the system may include multiple work vehicles or other agricultural machinery.

[0061] The work vehicle 100 in this embodiment is a tractor. The work vehicle 100 can be fitted with an implement at either the rear or the front, or both. The work vehicle 100 can travel within a field while performing agricultural work according to the type of implement. The work vehicle 100 may also travel within or outside a field without an implement attached.

[0062] The work vehicle 100 has an automatic driving function. That is, the work vehicle 100 can travel by the operation of a control device, without manual operation. The control device in this embodiment is provided inside the work vehicle 100, and can control both the speed and steering of the work vehicle 100. The work vehicle 100 can travel automatically not only within a field, but also outside the field (for example, on a road).

[0063] The work vehicle 100 is equipped with devices used for positioning or self-location estimation, such as a GNSS receiver and a LiDAR sensor. A control device of the work vehicle 100 causes the work vehicle 100 to travel automatically based on the position of the work vehicle 100 and information about a target route. In addition to controlling the travel of the work vehicle 100, the control device also controls the operation of the implement. This allows the work vehicle 100 to perform agricultural work using the implement while traveling automatically within a field. Furthermore, the work vehicle 100 can automatically travel along roads outside the field (e.g., farm roads or public roads) along a target route. When traveling automatically along roads outside the field, the work vehicle 100 travels while generating a local route along the target route that can avoid obstacles, based on data output from sensing devices such as a camera or LiDAR sensor. Within the field, the work vehicle 100 may travel while generating a local route as described above, or may travel along a target route without generating a local route and stop if an obstacle is detected.

[0064] The management device 600 is a computer that manages agricultural work performed by the work vehicle 100. The management device 600 may be, for example, a server computer that centrally manages information about farm fields on the cloud and supports agriculture by utilizing data on the cloud. The management device 600, for example, creates a work plan for the work vehicle 100 and plans a route for the work vehicle 100 in accordance with the work plan. The management device 600 may also generate and edit an environmental map based on data collected by the work vehicle 100 or other moving objects using a sensing device such as a LiDAR sensor. The management device 600 transmits the generated work plan, target route, and environmental map data to the work vehicle 100. The work vehicle 100 automatically moves and performs agricultural work based on this data.

[0065] Terminal device 400 is a computer used by a user located remotely from work vehicle 100. Terminal device 400 can be used to remotely monitor and operate work vehicle 100. For example, terminal device 400 can display on a display video captured by one or more cameras (imaging devices) equipped on work vehicle 100. The user can view the video to check the situation around work vehicle 100 and send instructions to work vehicle 100 to stop or start.

[0066] 9 is a side view that schematically shows an example of a work vehicle 100 and an implement 300 coupled to the work vehicle 100. The work vehicle 100 in this embodiment can operate in both a manual driving mode and an automatic driving mode. In the automatic driving mode, the work vehicle 100 can travel unmanned.

[0067] The work vehicle 100 shown in Figure 9 includes a vehicle body 101, a prime mover (engine) 102, and a transmission 103. The vehicle body 101 is provided with wheels 104 with tires and a cabin 105. The wheels 104 include a pair of front wheels 104F and a pair of rear wheels 104R. Inside the cabin 105, a driver's seat 107, a steering device 106, an operation terminal 200, and a group of switches for operation are provided. One or both of the front wheels 104F and the rear wheels 104R may be replaced with a plurality of wheels (crawlers) equipped with tracks rather than with tires.

[0068] The work vehicle 100 is equipped with multiple sensing devices that sense the surroundings of the work vehicle 100. In the example of Fig. 9, the sensing devices include a camera 120, a first LiDAR sensor 140A, and multiple obstacle sensors 130. A second LiDAR sensor 140B and a marker member 148 are attached to the implement 300.

[0069] Camera 120 captures images of the environment around work vehicle 100 and generates image data. Multiple cameras 120 may be installed on work vehicle 100. Images acquired by camera 120 may be transmitted to terminal device 400 for remote monitoring. These images may be used to monitor work vehicle 100 during unmanned operation. Camera 120 may also be used to generate images for recognizing surrounding features or obstacles, white lines, signs, or markings when work vehicle 100 travels on roads outside of fields (farm roads or public roads).

[0070] The first LiDAR sensor 140A is disposed on the cabin 105. The second LiDAR sensor 140B is disposed on the implement 300. The first LiDAR sensor 140A and the second LiDAR sensor 140B may be disposed at positions different from those shown in the figure. are respectively While the work vehicle 100 is traveling, first sensor data indicating the distance and direction to each measurement point of an object present in the surrounding environment, or the two-dimensional or three-dimensional coordinate values ​​of each measurement point and second sensor data is output repeatedly.

[0071] The first sensor data output from the first LiDAR sensor 140A and the second sensor data output from the second LiDAR sensor 140B are processed by a processing device in the work vehicle 100. The processing device estimates the relative position and orientation of the second LiDAR sensor 140B with respect to the first LiDAR sensor 140A based on the first sensor data generated by the first LiDAR sensor 140A sensing an area including the marker member 148. The processing device converts the second sensor data into data expressed in a vehicle coordinate system fixed to the work vehicle 100 based on the estimated position and orientation of the second LiDAR sensor 140B. The processing device generates point cloud data in the vehicle coordinate system based on the converted second sensor data or by combining the converted second sensor data with the first sensor data. The processing device can estimate the self-position of the work vehicle 100 by matching the point cloud data with an environmental map. The processing device may also generate or edit an environmental map using an algorithm such as SLAM (Simultaneous Localization and Mapping). The work vehicle 100 and the implement 300 may be equipped with multiple LiDAR sensors arranged in different positions and with different orientations.

[0072] The multiple obstacle sensors 130 shown in FIG. 9 are provided at the front and rear of the cabin 105. The obstacle sensors 130 may also be located in other locations. For example, one or more obstacle sensors 130 may be provided at any position on the side, front, or rear of the vehicle body 101. The obstacle sensors 130 may include, for example, a laser scanner or ultrasonic sonar. The obstacle sensors 130 are used to detect surrounding obstacles during autonomous driving and to stop or detour the work vehicle 100. A LiDAR sensor 140 may be used as one of the obstacle sensors 130.

[0073] The work vehicle 100 further includes a GNSS unit 110. The GNSS unit 110 includes a GNSS receiver. The GNSS receiver may include an antenna that receives signals from GNSS satellites and a processor that calculates the position of the work vehicle 100 based on the signals received by the antenna. The GNSS unit 110 receives satellite signals transmitted from multiple GNSS satellites and performs positioning based on the satellite signals. In this embodiment, the GNSS unit 110 is provided on top of the cabin 105, but it may be provided in another location.

[0074] The GNSS unit 110 may include an inertial measurement unit (IMU). Signals from the IMU can be used to complement position data. The IMU can measure the tilt and minute movements of the work vehicle 100. By complementing position data based on satellite signals with data acquired by the IMU, positioning performance can be improved.

[0075] The control device of the work vehicle 100 may use, in addition to the positioning results from the GNSS unit 110, sensing data acquired by sensing devices such as the camera 120 or LiDAR sensor 140 for positioning. If there are features that function as characteristic points in the environment in which the work vehicle 100 is traveling, such as farm roads, forest roads, public roads, or orchards, the position and orientation of the work vehicle 100 can be estimated with high accuracy based on the data acquired by the camera 120 or LiDAR sensor 140 and an environmental map that has been stored in advance in a storage device. By using the data acquired by the camera 120 or LiDAR sensor 140 to correct or complement position data based on satellite signals, the position of the work vehicle 100 can be determined with higher accuracy.

[0076] The prime mover 102 may be, for example, a diesel engine. An electric motor may be used instead of a diesel engine. The transmission 103 can change the propulsive force and travel speed of the work vehicle 100 by changing gears. The transmission 103 can also switch the work vehicle 100 between forward and reverse travel.

[0077] The steering device 106 includes a steering wheel, a steering shaft connected to the steering wheel, and a power steering device that assists steering by the steering wheel. The front wheels 104F are steerable wheels, and the traveling direction of the work vehicle 100 can be changed by changing the turning angle (also referred to as the "steering angle"). The steering angle of the front wheels 104F can be changed by operating the steering wheel. The power steering device includes a hydraulic device or an electric motor that supplies an assisting force to change the steering angle of the front wheels 104F. When automatic steering is performed, the steering angle is automatically adjusted by the force of the hydraulic device or electric motor under control of a control device arranged inside the work vehicle 100.

[0078] A coupling device 108 is provided at the front of the vehicle body 101. The coupling device 108 may include, for example, a three-point support device (also referred to as a "three-point link" or "three-point hitch"), a PTO (Power Take Off) shaft, a universal joint, and a communication cable. The coupling device 108 allows the implement 300 to be attached to and detached from the work vehicle 100. The coupling device 108 can raise and lower the three-point link using, for example, a hydraulic device, thereby changing the position or attitude of the implement 300. Power can also be sent from the work vehicle 100 to the implement 300 via the universal joint. A coupling device may also be provided at the rear of the vehicle body 101. In this case, an implement can be connected to the rear of the work vehicle 100.

[0079] 10 is a block diagram showing an example configuration of the work vehicle 100 and the implement 300. The work vehicle 100 and the implement 300 can communicate with each other via a communication cable included in the coupling device 108. The work vehicle 100 can communicate with the terminal device 400 and the management device 600 via the network 80.

[0080] In the example of FIG. 10 , the work vehicle 100 includes a GNSS unit 110, a camera 120, an obstacle sensor 130, a first LiDAR sensor 140A, and an operation terminal 200, as well as a group of sensors 150 that detect the operating state of the work vehicle 100, a control system 160, a communication device 190, a group of operation switches 210, and a drive unit 240. These components are communicatively connected to each other via a bus. The GNSS unit 110 includes a GNSS receiver 111, an RTK receiver 112, an inertial measurement unit (IMU) 115, and a processing circuit 116. The group of sensors 150 includes a steering wheel sensor 152, a turning angle sensor 154, and an axle sensor 156. The control system 160 includes a memory device 170 and a control device 180. The control device 180 includes multiple electronic control units (ECUs) 181 to 184. Implement 300 includes a drive device 340, a control device 380, a communication device 390, and a second LiDAR sensor 140B. Note that Fig. 10 shows components that are relatively highly related to the operation of autonomous driving by work vehicle 100, and does not show other components.

[0081] The GNSS receiver 111 in the GNSS unit 110 receives satellite signals transmitted from multiple GNSS satellites and generates GNSS data based on the satellite signals. The GNSS data is generated in a predetermined format, such as the NMEA-0183 format. The GNSS data may include, for example, values ​​indicating the identification number, elevation angle, azimuth angle, and reception strength of each satellite from which a satellite signal is received.

[0082] The GNSS unit 110 shown in FIG. 10 can be configured to perform positioning of the work vehicle 100 using, for example, RTK (Real Time Kinematic)-GNSS. FIG. 11 is a conceptual diagram showing an example of a work vehicle 100 performing positioning using RTK-GNSS. Positioning using RTK-GNSS uses correction signals transmitted from a reference station 60 in addition to satellite signals transmitted from multiple GNSS satellites 50. The reference station 60 can be installed near the field where the work vehicle 100 will be traveling (for example, within 10 km of the work vehicle 100). The reference station 60 generates correction signals, for example in RTCM format, based on the satellite signals received from the multiple GNSS satellites 50 and transmits them to the GNSS unit 110. The RTK receiver 112 includes an antenna and a modem and receives the correction signals transmitted from the reference station 60. The processing circuit 116 of the GNSS unit 110 corrects the positioning results obtained by the GNSS receiver 111 based on the correction signals. By using RTK-GNSS, it is possible to perform positioning with an accuracy of, for example, a few centimeters. Position information including latitude, longitude, and altitude information is obtained through highly accurate positioning using RTK-GNSS. The GNSS unit 110 calculates the position of the work vehicle 100, for example, at a frequency of approximately 1 to 10 times per second.

[0083] The positioning method is not limited to RTK-GNSS, and any positioning method (such as interferometric positioning or relative positioning) that can obtain position information with the required accuracy can be used. For example, positioning may be performed using a Virtual Reference Station (VRS) or a Differential Global Positioning System (DGPS). If position information with the required accuracy can be obtained without using a correction signal transmitted from the reference station 60, the position information may be generated without using a correction signal. In this case, the GNSS unit 110 does not need to be equipped with the RTK receiver 112.

[0084] Even when RTK-GNSS is used, in locations where correction signals from the reference station 60 cannot be obtained (for example, on a road far from a field), the position of the work vehicle 100 is estimated by other methods without relying on signals from the RTK receiver 112. For example, the position of the work vehicle 100 can be estimated by matching data output from the LiDAR sensors 140A, 140B and / or camera 120 with a highly accurate environmental map.

[0085] The GNSS unit 110 in this embodiment further includes an IMU 115. The IMU 115 may include a three-axis acceleration sensor and a three-axis gyroscope. The IMU 115 may also include a direction sensor such as a three-axis geomagnetic sensor. The IMU 115 functions as a motion sensor and can output signals indicating various quantities such as the acceleration, velocity, displacement, and attitude of the work vehicle 100. The processing circuit 116 can estimate the position and orientation of the work vehicle 100 with higher accuracy based on the signals output from the IMU 115 in addition to the satellite signals and correction signals. The signals output from the IMU 115 can be used to correct or complement the position calculated based on the satellite signals and correction signals. The IMU 115 outputs signals at a higher frequency than the GNSS receiver 111. Using these high-frequency signals, the processing circuit 116 can measure the position and orientation of the work vehicle 100 at a higher frequency (e.g., 10 Hz or higher). A three-axis acceleration sensor and a three-axis gyroscope may be provided separately instead of the IMU 115. The IMU 115 may be provided as a device separate from the GNSS unit 110.

[0086] The camera 120 is an imaging device that captures images of the environment surrounding the work vehicle 100. The camera 120 includes an image sensor, such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). The camera 120 may also include an optical system including one or more lenses and a signal processing circuit. The camera 120 captures images of the environment surrounding the work vehicle 100 while the work vehicle 100 is traveling and generates image (e.g., video) data. The camera 120 can capture video at a frame rate of, for example, 3 frames per second (fps) or higher. The images generated by the camera 120 can be used, for example, when a remote observer uses the terminal device 400 to check the environment surrounding the work vehicle 100. The images generated by the camera 120 may be used for positioning or obstacle detection. A single camera 120 may be provided, or multiple cameras 120 may be provided at different positions on the work vehicle 100. A visible light camera that generates a visible light image and an infrared camera that generates an infrared image may be provided separately. Both a visible light camera and an infrared camera may be provided as cameras that generate images for monitoring. The infrared camera may also be used to detect obstacles at night. The camera 120 may be a stereo camera. The stereo camera can acquire a distance image that shows the distance distribution within the shooting range.

[0087] The obstacle sensor 130 detects objects present around the work vehicle 100. The obstacle sensor 130 may include, for example, a laser scanner or an ultrasonic sonar. The obstacle sensor 130 outputs a signal indicating the presence of an obstacle when an object is present closer than a predetermined distance from the obstacle sensor 130. Multiple obstacle sensors 130 may be provided at different positions on the work vehicle 100. For example, multiple laser scanners and multiple ultrasonic sonars may be arranged at different positions on the work vehicle 100. By providing such a large number of obstacle sensors 130, blind spots in monitoring obstacles around the work vehicle 100 can be reduced.

[0088] The steering wheel sensor 152 measures the rotation angle of the steering wheel of the work vehicle 100. The turning angle sensor 154 measures the turning angle of the front wheels 104F, which are the steered wheels. The measurement values ​​from the steering wheel sensor 152 and the turning angle sensor 154 are used for steering control by the control device 180.

[0089] The axle sensor 156 measures the rotational speed of the axle connected to the wheel 104, i.e., the number of rotations per unit time. The axle sensor 156 may be a sensor that uses, for example, a magnetoresistive element (MR), a Hall element, or an electromagnetic pickup. The axle sensor 156 outputs a numerical value that indicates, for example, the number of rotations per minute (unit: rpm) of the axle. The axle sensor 156 is used to measure the speed of the work vehicle 100.

[0090] The drive device 240 includes various devices necessary for the travel of the work vehicle 100 and the driving of the implement 300, such as the prime mover 102, transmission 103, and steering device 106 described above. The prime mover 102 may be equipped with an internal combustion engine such as a diesel engine. The drive device 240 may be equipped with an electric motor for traction instead of or in addition to the internal combustion engine.

[0091] The storage device 170 includes one or more storage media, such as flash memory or a magnetic disk. The storage device 170 stores various data generated by the GNSS unit 110, the camera 120, the obstacle sensor 130, the LiDAR sensor 140, the sensor group 150, and the control device 180. The data stored in the storage device 170 may include map data (environmental map) of the environment in which the work vehicle 100 travels, target route data for autonomous driving, and data indicating the positional relationship between the second LiDAR sensor 140B and the marker member 148. The environmental map includes information on multiple fields in which the work vehicle 100 will perform agricultural work and the roads in their surroundings. The environmental map and target route may be generated by a processor in the management device 600. The control device 180 may have a function for generating or editing the environmental map and target route. The storage device 170 also stores computer programs that cause each ECU in the control device 180 to perform various operations, which will be described later. Such a computer program may be provided to the work vehicle 100 via a storage medium (e.g., a semiconductor memory or an optical disk) or a telecommunications line (e.g., the Internet). Such a computer program may also be sold as commercial software.

[0092] The control device 180 includes a plurality of ECUs, such as an ECU 181 for speed control, an ECU 182 for steering control, an ECU 183 for implement control, and an ECU 184 for automatic driving control.

[0093] The ECU 181 controls the speed of the work vehicle 100 by controlling the prime mover 102 , the transmission 103 , and the brakes included in the drive unit 240 .

[0094] The ECU 182 controls the steering of the work vehicle 100 by controlling the hydraulic device or electric motor included in the steering device 106 based on the measurement value of the steering wheel sensor 152 .

[0095] The ECU 183 controls the operation of the three-point link and PTO shaft included in the coupling device 108, etc., in order to cause the implement 300 to perform a desired operation. The ECU 183 also generates a signal that controls the operation of the implement 300, and transmits the signal from the communication device 190 to the implement 300.

[0096] The ECU 184 functions as the processing device described above. The ECU 184 performs calculations and control to achieve autonomous driving based on data output from the GNSS unit 110, the camera 120, the obstacle sensor 130, the first LiDAR sensor 140A, the sensor group 150, and the second LiDAR sensor 140B. For example, the ECU 184 determines the position of the work vehicle 100 based on data output from the GNSS unit 110, the camera 120, and at least one of the LiDAR sensors 140A and 140B. In an environment where the GNSS unit 110 can receive satellite signals well, the ECU 184 may determine the position of the work vehicle 100 based only on the data output from the GNSS unit 110. Conversely, in an environment where there are objects (e.g., trees, buildings, etc.) in the surrounding area that obstruct the reception of satellite signals, the ECU 184 estimates the position of the work vehicle 100 using the data output from the LiDAR sensors 140A and 140B. For example, ECU 184 estimates the self-position of work vehicle 100 by matching data output from LiDAR sensors 140A and 140B with an environmental map. During autonomous driving, ECU 184 performs calculations necessary for work vehicle 100 to travel along a preset target route based on the estimated position of work vehicle 100. ECU 184 sends a command to ECU 181 to change the speed, and sends a command to ECU 182 to change the steering angle. In response to the command to change the speed, ECU 181 changes the speed of work vehicle 100 by controlling prime mover 102, transmission 103, or brakes. In response to the command to change the steering angle, ECU 182 changes the steering angle by controlling steering device 106.

[0097] Through the operation of these ECUs, control device 180 realizes autonomous driving. During autonomous driving, control device 180 controls drive device 240 based on the measured or estimated position of work vehicle 100 and the target route. In this way, control device 180 causes work vehicle 100 to travel along the target route.

[0098] The multiple ECUs included in the control device 180 can communicate with each other in accordance with a vehicle bus standard such as CAN (Controller Area Network). Instead of CAN, a faster communication method such as Automotive Ethernet (registered trademark) may be used. In FIG. 10, each of the ECUs 181 to 184 is shown as an individual block, but the functions of each of these may be realized by multiple ECUs. An on-board computer that integrates at least some of the functions of the ECUs 181 to 184 may be provided. The control device 180 may include ECUs other than the ECUs 181 to 184, and any number of ECUs may be provided depending on the functions. Each ECU includes one or more processors.

[0099] The communication device 190 is a device including circuits for communicating with the implement 300, the terminal device 400, and the management device 600. The communication device 190 includes circuits for transmitting and receiving signals compliant with an ISOBUS standard, such as ISOBUS-TIM, between the communication device 390 of the implement 300. This allows the implement 300 to perform desired operations and acquire information from the implement 300. The communication device 190 also has a function for wirelessly communicating with the second LiDAR sensor 140B. Communication between the communication device 190 and the second LiDAR sensor 140B may be performed in accordance with any wireless communication standard, such as cellular mobile communication such as Wi-Fi (registered trademark), 3G, 4G, or 5G, or Bluetooth (registered trademark). The communication device 190 may further include an antenna and communication circuits for transmitting and receiving signals via the network 80 between the communication devices of the terminal device 400 and the management device 600. The network 80 may include, for example, a cellular mobile communication network such as 3G, 4G, or 5G, and the Internet.

[0100] The operation terminal 200 is a terminal through which a user performs operations related to the travel of the work vehicle 100 and the operation of the implement 300, and is also referred to as a virtual terminal (VT). The operation terminal 200 may include a display device such as a touch screen and / or one or more buttons. The display device may be, for example, a liquid crystal display or an organic light-emitting diode (OLED) display. By operating the operation terminal 200, a user can perform various operations, such as switching the autonomous driving mode on / off, recording or editing an environmental map, setting a target route, and switching the implement 300 on / off. At least some of these operations can also be achieved by operating the operation switch group 210. The operation terminal 200 may be configured to be detachable from the work vehicle 100. A user located remotely from the work vehicle 100 may operate the detached operation terminal 200 to control the operation of the work vehicle 100. Instead of the operation terminal 200, the user may control the operation of the work vehicle 100 by operating a computer, such as a terminal device 400, on which necessary application software is installed.

[0101] The drive device 340 in the implement 300 shown in FIG. 10 performs the operations required for the implement 300 to perform a predetermined task. The drive device 340 includes devices appropriate for the application of the implement 300, such as a hydraulic device, an electric motor, or a pump. The control device 380 controls the operation of the drive device 340. The control device 380 causes the drive device 340 to perform various operations in response to signals transmitted from the work vehicle 100 via the communication device 390. In addition, a signal appropriate for the state of the implement 300 can also be transmitted from the communication device 390 to the work vehicle 100.

[0102] Next, an example of the operation of automatic traveling by work vehicle 100 will be described.

[0103] FIG. 12 is a diagram schematically illustrating an example of an environment in which work vehicle 100 travels. In this example, work vehicle 100 uses implement 300 to perform predetermined tasks (e.g., mowing, pest control, etc.) while traveling among multiple tree rows 20 in an orchard (e.g., a vineyard). In an orchard, the sky is obstructed by branches and leaves, making automated travel using GNSS difficult. In such an environment, ECU 184 estimates the self-position of work vehicle 100 based on data from first LiDAR sensor 140A and second LiDAR sensor 140B.

[0104] An example of the operation of the work vehicle 100 will be described in detail below. The ECU 184 first generates an environmental map showing the distribution of tree rows 20 in the entire orchard or a section of the orchard using sensor data output from the LiDAR sensors 140A and 140B, and records the map in the storage device 170. The environmental map is generated by repeatedly estimating the vehicle's own position and generating a local map. While the work vehicle 100 is moving, the ECU 184 performs self-position estimation based on the sensor data repeatedly output from the LiDAR sensors 140A and 140B, while detecting tree rows in the environment surrounding the work vehicle 100. The ECU 184 then repeatedly generates a local map showing the distribution of the detected tree rows and records the local map in the storage device 170. The ECU 184 generates the environmental map by piecing together the local maps generated while the work vehicle 100 travels through the entire orchard or a section of the orchard. This environmental map is data that records the distribution of tree rows 20 in the environment in a format that allows them to be distinguished from other objects. Thus, in this embodiment, the tree row 20 can be used as a landmark for SLAM.

[0105] Once the environmental map is generated, the work vehicle 100 becomes capable of autonomous driving. While the work vehicle 100 is traveling, the ECU 184 detects tree rows 20 in the surrounding environment based on sensor data repeatedly output from the LiDAR sensors 140A, 140B, and estimates the position of the work vehicle 100 by matching the detected tree rows 20 with the environmental map. The ECU 184 controls the traveling of the work vehicle 100 according to the estimated position of the work vehicle 100. For example, if the work vehicle 100 deviates from a target route set between two adjacent tree rows, the ECU 184 controls the work vehicle 100 to approach the target route by having the ECU 182 adjust the steering of the work vehicle 100. Such steering control may be performed based not only on the position of the work vehicle 100 but also on its orientation.

[0106] FIG. 13A is a diagram schematically illustrating an example of a travel route for a work vehicle 100. The work vehicle 100 travels, for example, between multiple tree rows 20 in an orchard along a route 25 indicated by an arrow in FIG. 13A. In FIG. 13A, the line segments included in route 25 are depicted as straight lines, but the route actually traveled by the work vehicle 100 may include meandering sections. Here, the multiple tree rows 20 are ordered from end to end as a first tree row 20A, a second tree row 20B, a third tree row 20C, a fourth tree row 20D, and so on. In the example of FIG. 13A, the work vehicle 100 first travels between the first tree row 20A and the second tree row 20B, and when that travel is complete, it turns and travels in the opposite direction between the second tree row 20B and the third tree row 20C. After completing travel between the second row of trees 20B and the third row of trees 20C, the work vehicle 100 turns and travels between the third row of trees 20C and the fourth row of trees 20D. By repeating the same operation, the work vehicle 100 travels to the end of the path 25 between the last two rows of trees. If the distance between adjacent rows of trees is short, the work vehicle 100 may travel between every other row of trees, as shown in FIG. 13B. In this case, after completing travel between the last two rows of trees, the work vehicle 100 may travel between every other row of trees that have not yet been traveled. This type of travel is performed automatically by the work vehicle 100, using the LiDAR sensors 140A and 140B to estimate its own position. Positioning may be performed based on the GNSS signal when the GNSS unit 110 can receive the GNSS signal. For example, when turning around on the path 25 shown in FIGS. 13A and 13B, there are no leaves blocking the GNSS signal, making positioning based on the GNSS signal possible.

[0107] FIG. 14 is a flowchart showing an example of the operation of steering control during automatic driving executed by the control device 180. The control device 180 performs automatic steering by executing the operations of steps S121 to S125 shown in FIG. 14 while the work vehicle 100 is traveling. The speed may be maintained at a preset speed, for example. While the work vehicle 100 is traveling, the control device 180 estimates the self-position of the work vehicle 100 based on data output from the GNSS unit 110 or the LiDAR sensors 140A, 140B (step S121). A specific example of the self-position estimation process will be described later. Next, the control device 180 calculates the deviation between the estimated position of the work vehicle 100 and the target route (step S122). The deviation represents the distance between the position of the work vehicle 100 at that time and the target route. The control device 180 determines whether the deviation of the calculated position exceeds a preset threshold (step S123). If the deviation exceeds the threshold, the control device 180 changes the steering angle by changing the control parameters of the steering device included in the drive device 240 so that the deviation becomes smaller. If the deviation does not exceed the threshold in step S123, the operation of step S124 is skipped. In the following step S125, the control device 180 determines whether or not a command to end the operation has been received. The command to end the operation may be issued, for example, when a user remotely instructs the work vehicle 100 to stop autonomous driving or when the work vehicle 100 reaches its destination. If the command to end the operation has not been issued, the process returns to step S121, and the same operation is performed based on the newly measured position of the work vehicle 100. The control device 180 repeats the operations of steps S121 to S125 until a command to end the operation is issued. The above operations are executed by the ECUs 182 and 184 in the control device 180.

[0108] 14, the control device 180 controls the drive device 240 based only on the deviation between the position of the work vehicle 100 and the target route, but the control may also take into consideration the deviation in heading. For example, when the heading deviation, which is the angular difference between the orientation of the work vehicle 100 estimated by the self-position estimation process described above and the direction of the target route, exceeds a preset threshold, the control device 180 may change the control parameters (for example, steering angle) of the steering device of the drive device 240 in accordance with the deviation.

[0109] An example of steering control by the control device 180 will be described in more detail below with reference to FIGS. 15A to 15D.

[0110] FIG. 15A is a diagram showing an example of a work vehicle 100 traveling along a target route P. FIG. 15B is a diagram showing an example of a work vehicle 100 shifted to the right from the target route P. FIG. 15C is a diagram showing an example of a work vehicle 100 shifted to the left from the target route P. FIG. 15D is a diagram showing an example of a work vehicle 100 facing in an inclined direction with respect to the target route P. In these figures, the pose indicating the position and orientation of the work vehicle 100 measured by the GNSS unit 110 is expressed as r(x, y, θ). (x, y) are coordinates representing the position of the reference point of the work vehicle 100 in the XY coordinate system, which is a two-dimensional coordinate system fixed to the Earth. In the examples shown in FIGS. 15A to 15D, the reference point of the work vehicle 100 is located at the position where the GNSS antenna on the cabin is installed, but the position of the reference point is arbitrary. θ is an angle representing the measured orientation of the work vehicle 100. In the illustrated example, the target path P is parallel to the Y axis, but in general, the target path P is not necessarily parallel to the Y axis.

[0111] As shown in FIG. 15A, if the position and orientation of the work vehicle 100 do not deviate from the target route P, the control device 180 maintains the steering angle and speed of the work vehicle 100 without changing them.

[0112] As shown in Fig. 15B, when the position of work vehicle 100 has shifted to the right from target route P, control device 180 changes the steering angle so that the traveling direction of work vehicle 100 leans to the left and approaches route P. At this time, the speed may also be changed in addition to the steering angle. The magnitude of the steering angle can be adjusted, for example, according to the magnitude of position deviation Δx.

[0113] As shown in Fig. 15C, when the position of the work vehicle 100 has shifted to the left from the target route P, the control device 180 changes the steering angle so that the traveling direction of the work vehicle 100 tilts to the right and approaches the route P. In this case, too, the speed may be changed in addition to the steering angle. The amount of change in the steering angle may be adjusted, for example, according to the magnitude of the position deviation Δx.

[0114] As shown in FIG. 15D, when the position of the work vehicle 100 is not significantly deviated from the target route P but the heading is different from the direction of the target route P, the control device 180 changes the steering angle to reduce the azimuth deviation Δθ. In this case, the speed may also be changed in addition to the steering angle. The magnitude of the steering angle may be adjusted, for example, according to the magnitudes of the position deviation Δx and the azimuth deviation Δθ. For example, the smaller the absolute value of the position deviation Δx, the greater the amount of change in the steering angle according to the azimuth deviation Δθ. When the absolute value of the position deviation Δx is large, the steering angle will be changed significantly to return to the route P, which inevitably increases the absolute value of the azimuth deviation Δθ. Conversely, when the absolute value of the position deviation Δx is small, it is necessary to bring the azimuth deviation Δθ closer to zero. For this reason, it is appropriate to relatively increase the weight of the azimuth deviation Δθ (i.e., the control gain) used to determine the steering angle.

[0115] Control techniques such as PID control or MPC control (model predictive control) can be applied to the steering control and speed control of work vehicle 100. By applying these control techniques, it is possible to smooth the control that brings work vehicle 100 closer to target path P.

[0116] If an obstacle is detected by one or more obstacle sensors 130 while the work vehicle 100 is traveling, the control device 180 will stop the work vehicle 100. At this time, the buzzer 220 may be caused to emit a warning sound or a warning signal may be sent to the terminal device 400. If it is possible to avoid the obstacle, the control device 180 may control the drive device 240 to avoid the obstacle.

[0117] Next, a specific example of the self-position estimation process in step S121 in FIG. 14 will be described.

[0118] Fig. 16 is a flowchart showing a specific example of the process of step S121 in Fig. 14. Here, an example of the process of self-position estimation based on the first sensor data output from the first LiDAR sensor 140A and the second sensor data output from the second LiDAR sensor 140B will be described. In the example of Fig. 16, step S121 includes the processes of steps S141 to S146.

[0119] The ECU 184 (processing device) of the control device 180 first acquires first sensor data and second sensor data (step S141). Next, the ECU 184 identifies the positions of the plurality of reflecting portions of the marker member 148 based on the first sensor data (step S142). The plurality of reflecting portions are portions having a higher optical reflectivity than other portions, such as the reflecting portions 148a, 148b, and 148c illustrated in FIG. 2 or FIG. 4. The ECU 184 estimates the relative position and orientation of the second LiDAR sensor 140B with respect to the first LiDAR sensor 140A based on the positions of the plurality of reflecting portions of the marker member 148 and the known positional relationship between the plurality of reflecting portions and the second LiDAR sensor 140B (step S143). Next, the ECU 184 converts the second sensor data into data in a coordinate system fixed to the work vehicle 100 based on the relative position and orientation of the second LiDAR sensor 140B with respect to the first LiDAR sensor 140A (step S144). Here, the coordinate system fixed to the work vehicle 100 may, for example, match the coordinate system set for the first LiDAR sensor 140A. The ECU 184 generates point cloud data by combining the first sensor data and the converted second sensor data (step S145). At this time, the ECU 184 may remove point data in areas corresponding to the implement 300, the second LiDAR sensor 140B, and the marker member 148 from the first sensor data, and then combine the first sensor data with the second sensor data. This processing prevents data of points unnecessary for self-position estimation from being included in the point cloud data. The ECU 184 may also generate point cloud data based only on the second sensor data that has been subjected to coordinate conversion. Next, ECU 184 matches the generated point cloud data with the environmental map stored in storage device 170, and determines the position and attitude of work vehicle 100 (step S146).

[0120] Through the above operations, the ECU 184 can estimate the self-position of the work vehicle 100. According to the present embodiment, in addition to the first LiDAR sensor 140A provided on the work vehicle 100, the second LiDAR sensor 140B and the marker member 148 are attached to the implement 300. The position and orientation of the second LiDAR sensor 140B can be estimated based on the first sensor data acquired by the first LiDAR 140A sensing the marker member 148. This makes it possible to convert the second sensor data into data in a coordinate system fixed to the work vehicle 100, and use it for self-position estimation. Through these operations, self-position estimation can be performed even when a large implement 300 is attached to the front of the work vehicle 100.

[0121] The above process is not limited to self-localization estimation, but may also be applied to obstacle detection using LiDAR sensors 140A and 140B. In this embodiment, LiDAR sensors 140A and 140B are used, but sensors that measure distance using other methods may also be used. For example, a method similar to the example using the camera 120 shown in FIG. 5 or the multiple GNSS receivers 110A and 110B shown in FIG. 7 may also be applied. Instead of a LiDAR sensor, other types of distance measurement sensors, such as a stereo camera or a ToF camera, may also be used.

[0122] In the above embodiment, an example has been described in which the sensing system is applied to an agricultural work vehicle 100 such as a tractor. The sensing system is not limited to agricultural work vehicles 100, and may also be used in work vehicles used for other purposes, such as civil engineering work, construction work, or snow removal work.

[0123] As described above, the present disclosure includes the systems and methods described in the following sections.

[0124] [Item 1] a first distance measuring sensor attached to the work vehicle; a second distance measuring sensor attached to an implement connected to the work vehicle; a marker member located within a sensing range of the first distance measuring sensor; a processing device; Equipped with The processing device includes: estimating a relative position and orientation of the second distance measuring sensor with respect to the first distance measuring sensor based on first sensor data generated by the first distance measuring sensor sensing an area including the marker member; outputting data indicating the estimated position and orientation; Sensing system.

[0125] [Item 2] Item 1: The sensing system of item 1, wherein the second distance measurement sensor includes a mounting fixture for attaching to the implement and is externally attached to the implement.

[0126] [Item 3] 3. The sensing system of claim 1, wherein the marker member is attached to the second distance measuring sensor.

[0127] [Item 4] A sensing system described in any one of items 1 to 3, wherein the processing device estimates the relative position and orientation of the second ranging sensor with respect to the first ranging sensor based on the first sensor data and information indicating the relationship between the position and orientation of the second ranging sensor and the marker member.

[0128] [Item 5] A sensing system described in any one of items 1 to 4, wherein the processing device converts the second sensor data output from the second ranging sensor based on the estimated position and orientation into data expressed in a coordinate system fixed to the work vehicle and outputs the data.

[0129] [Item 6] the first ranging sensor includes a first LiDAR sensor that outputs the first sensor data; the second ranging sensor includes a second LiDAR sensor that outputs second sensor data; the marker member includes one or more reflective portions having a reflectivity of light emitted from the first LiDAR sensor higher than that of other portions of the marker member; the processing device estimates the relative position and orientation of the second ranging sensor with respect to the first ranging sensor based on the positions and / or shapes of the one or more reflecting portions; 6. The sensing system according to any one of items 1 to 5.

[0130] [Item 7] Item 7. The sensing system of item 6, wherein the processing device converts the second sensor data into data expressed in a coordinate system fixed to the work vehicle based on the estimated position and orientation, and generates and outputs point cloud data based on the first sensor data and the converted second sensor data.

[0131] [Item 8] The sensing system described in item 7, wherein the processing device generates the point cloud data by combining data obtained by removing data from the first sensor data corresponding to the implement and the second ranging sensor with the converted second sensor data.

[0132] [Item 9] the first distance measuring sensor includes an image sensor that outputs image data as the first sensor data; A sensing system described in any one of items 1 to 5, wherein the processing device estimates the relative position and orientation of the second ranging sensor with respect to the first ranging sensor based on the brightness distribution or color distribution of an area corresponding to the marker member in the image represented by the first sensor data.

[0133] [Item 10] further comprising a storage device that stores an environmental map; the processing device converts second sensor data output from the second distance measuring sensor into data expressed in a coordinate system fixed to the work vehicle based on the estimated position and orientation; generating point cloud data indicating a distribution of features around the work vehicle based on the converted second sensor data; 10. The sensing system according to any one of items 1 to 9, wherein the position of the work vehicle is estimated by matching the point cloud data with the environmental map.

[0134] [Item 11] The implement is connected to a front portion of the work vehicle, 11. The sensing system according to any one of items 1 to 10, wherein the first distance measurement sensor senses at least an area ahead of the work vehicle.

[0135] [Item 12] 12. An agricultural machine equipped with the sensing system according to any one of items 1 to 11.

[0136] [Item 13] A sensing device used as the second distance measuring sensor in the sensing system according to any one of items 1 to 11, A mounting tool for mounting to the implement is provided and is attached to the outside of the implement. Sensing device.

[0137] [Item 14] a first sensing device attached to the work vehicle; a second sensing device attached to an implement coupled to the work vehicle; a marker member attached to the implement; a storage device that stores data indicating the relative position and orientation relationship between the second sensing device and the marker member; a processing device; Equipped with the second sensing device includes a LiDAR sensor; The processing device includes: Estimating a relative position and orientation of the LiDAR sensor with respect to the first sensing device based on first sensor data generated by the first sensing device sensing an area including the marker member and the data stored in the storage device; outputting data indicating the estimated position and orientation; Sensing system.

[0138] [Item 15] a first GNSS receiver mounted on the work vehicle; A LiDAR sensor attached to an implement connected to the work vehicle; three or more second GNSS receivers attached to the implement; a processing device; Equipped with The processing device includes: estimating a relative position and attitude of the second sensing device with respect to the first sensing device based on the signal received by the first GNSS receiver and the signals received by the three or more second GNSS receivers; outputting data indicating the estimated position and orientation; Sensing system. [Industrial Applicability]

[0139] The technology of the present disclosure can be applied to a work vehicle that travels while performing work using an implement, for example, a work vehicle such as a tractor that travels autonomously while performing agricultural work with an implement attached to the front or rear. [Explanation of symbols]

[0140] 100...work vehicle, 101...vehicle body, 102...prime mover, 103...transmission, 104...wheel, 105...cabin, 106...steering gear, 107...driver's seat, 108...coupling device, 110...GNSS unit, 110A, 110B, 111...GNSS receiver, 112...RTK receiver, 115...inertial measurement unit, 116...processing circuit, 120...camera, 130...obstacle sensor, 140...LiDAR sensor, 140A...first LiDAR sensor, 140B...second LiDAR sensor iDAR sensor, 148... marker member, 149... mounting fixture, 150... sensor group, 152... steering wheel sensor, 154... turning angle sensor, 156... axle sensor, 160... control system, 170... storage device, 180... control device, 181-184... ECU, 190... communication device, 200... operation terminal, 210... operation switch group, 240... drive device, 250... processing device, 300... implement, 340... drive device, 380... control device, 390... communication device

Claims

1. a first distance measurement sensor attached to the work vehicle; a second distance measuring sensor attached to an implement connected to the work vehicle; a marker member located within a sensing range of the first distance measuring sensor; a processing device; Equipped with The processing device includes: estimating a relative position and orientation of the second distance measuring sensor with respect to the first distance measuring sensor based on first sensor data generated by the first distance measuring sensor sensing an area including the marker member; outputting data indicating the estimated position and orientation; Sensing system.

2. The sensing system according to claim 1 , wherein the second distance measurement sensor includes a mounting fixture for mounting to the implement and is external to the implement.

3. The sensing system according to claim 1 or 2, wherein the marker member is attached to the second distance measuring sensor.

4. The sensing system described in claim 1 or 2, wherein the processing device estimates the relative position and orientation of the second ranging sensor with respect to the first ranging sensor based on the first sensor data and information indicating the relationship between the position and orientation of the second ranging sensor and the marker member.

5. 3. The sensing system according to claim 1, wherein the processing device converts the second sensor data output from the second ranging sensor into data expressed in a coordinate system fixed to the work vehicle based on the estimated position and orientation, and outputs the converted data.

6. the first ranging sensor includes a first LiDAR sensor that outputs the first sensor data; the second ranging sensor includes a second LiDAR sensor that outputs second sensor data; The marker member includes one or more reflective portions having a reflectivity of light emitted from the first LiDAR sensor higher than that of other portions of the marker member, the processing device estimates the relative position and orientation of the second ranging sensor with respect to the first ranging sensor based on the positions and / or shapes of the one or more reflecting portions; The sensing system according to claim 1 or 2.

7. 7. The sensing system according to claim 6, wherein the processing device converts the second sensor data into data expressed in a coordinate system fixed to the work vehicle based on the estimated position and orientation, and generates and outputs point cloud data based on the first sensor data and the converted second sensor data.

8. The sensing system of claim 7, wherein the processing device generates the point cloud data by combining data obtained by removing data from the first sensor data corresponding to the implement and the second ranging sensor with the converted second sensor data.

9. the first distance measurement sensor includes an image sensor that outputs image data as the first sensor data; The sensing system described in claim 1 or 2, wherein the processing device estimates the relative position and orientation of the second ranging sensor with respect to the first ranging sensor based on the brightness distribution or color distribution of the area corresponding to the marker member in the image represented by the first sensor data.

10. further comprising a storage device that stores an environmental map; the processing device converts second sensor data output from the second distance measuring sensor into data expressed in a coordinate system fixed to the work vehicle based on the estimated position and orientation; generating point cloud data indicating a distribution of features around the work vehicle based on the converted second sensor data; The sensing system according to claim 1 or 2, wherein the position of the work vehicle is estimated by matching the point cloud data with the environmental map.

11. The implement is connected to a front portion of the work vehicle, The sensing system according to claim 1 or 2, wherein the first distance measurement sensor senses at least a front area of ​​the work vehicle.

12. An agricultural machine equipped with the sensing system according to claim 1.

Images