Work vehicle system

Abstract

To provide a work vehicle system capable of positioning a work vehicle in a field even if a satellite positioning device is not mounted.SOLUTION: A work vehicle system according to an aspect of an embodiment includes a balloon, an optical sensor, and a control device. The balloon floats above a field that is a work target of the work vehicle. The optical sensor is mounted on a lower portion of the balloon. The control device acquires the image captured by the optical sensor. The work vehicle system picks up an image of the entire field with the balloon by floating one balloon with respect to one field and changing the altitude of the balloon and the viewing angle of the optical sensor.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a work vehicle system. [Background technology]

[0002] BACKGROUND ART Conventionally, an automatic traveling system has been known that includes an image capturing unit that captures images of planted crops and detects abnormal conditions of the planted crops based on the captured images (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-7386 Summary of the Invention [Problem to be solved by the invention]

[0004] There is a demand for a work vehicle system that can determine the position of a work vehicle in a field even if the work vehicle is not equipped with a satellite positioning device. Prior art technology may use a balloon to analyze images of the field condition, but the details are not disclosed.

[0005] The present invention has been made in view of the above, and has an object to provide a work vehicle system that can determine the position of a work vehicle in a field even if the work vehicle is not equipped with a satellite positioning device. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems and achieve the objectives, the system includes a balloon (2) floating above the field (F) that is the work target of the work vehicle (10), an optical sensor (3) mounted on the bottom of the balloon (2), and a control device (20) that acquires images captured by the optical sensor (3). One balloon (2) is levitated for one field (F), and the altitude of the balloon (2) and the field of view angle of the optical sensor (3) are changed, so that the entire field (F) is captured by the balloon (2). [Effects of the Invention]

[0007] According to the work vehicle system of the embodiment, the position of the work vehicle in the field can be determined even if the work vehicle is not equipped with a satellite positioning device. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is an explanatory diagram showing an overview of a work vehicle system according to an embodiment. [Figure 2] FIG. 2 is a block diagram showing an example of a control system of the work vehicle system according to the embodiment. [Figure 3] FIG. 3 is a plan view showing the imaging area. [Figure 4] Figure 4 is an explanatory diagram showing the perimeter of the field and the flight path of the drone. [Figure 5] FIG. 5 is an explanatory diagram showing a straight line guide displayed on the VR goggles. [Figure 6] FIG. 6 is an explanatory diagram showing a configuration in which an environment recognition sensor is provided at the top of the vehicle body. [Figure 7] FIG. 7 is a plan view showing the sensors provided on the tractor. [Figure 8] FIG. 8 is an explanatory diagram showing the process of setting the detection range. [Figure 9] FIG. 9 is an explanatory diagram showing the process of setting the detection range. DETAILED DESCRIPTION OF THE INVENTION

[0009] Below, a work vehicle system according to an embodiment of the present invention will be described in detail with reference to the drawings. Note that the components in the following embodiments include those that can be substituted by a person skilled in the art, or those that are substantially the same, or so-called equivalents. Furthermore, the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the present invention.

[0010] First, the overall configuration of a work vehicle system 1 will be described with reference to Figures 1 and 2. Figure 1 is an explanatory diagram showing an overview of a work vehicle system 1 according to an embodiment. Figure 2 is a block diagram showing an example of a control system for a work vehicle system 1 according to an embodiment. Note that in this application, a tractor 10 will be used as an example of a work vehicle for description.

[0011] The tractor 10, which is a work vehicle, is an agricultural tractor 10 that travels by itself to perform work in a field F or the like. An operator rides in the tractor 10 and performs predetermined work while traveling within the field F. The tractor 10 is equipped with a communication device 11 and can be automatically driven by an automatic driving system.

[0012] As shown in FIGS. 1 and 2, the work vehicle system 1 includes a balloon 2, an optical sensor 3, and a control device 20.

[0013] The balloon 2 is moored to the periphery of the field F and floats in the air above the field F. For example, the balloon 2 is moored to the ground around a corner of the field F via two mooring ropes 4. The mooring ropes 4 are tied to a weight 5 placed on the ground. The balloon 2 is equipped with a communication device 6 that communicates with a tractor 10.

[0014] The optical sensor 3 captures images of the field F. The optical sensor 3 is, for example, an optical camera, 3DLiDAR, or laser sensor. The optical sensor 3 is mounted on the bottom of the balloon 2 so that the imaging direction is downward. For example, the typical performance of the optical sensor 3 mounted on the balloon 2 is such that it can capture images of an area of ​​70 to 75 m square at an altitude of 50 m and a sensor field of view angle of 90°. The balloon 2 described above can be equipped with an optical sensor 3 that is heavier than a drone, which will be described later, and therefore can be equipped with a high-resolution optical sensor 3 to capture images of a wide area of ​​the field F from a higher position.

[0015] The control device 20 includes the balloon 2 and a mobile terminal device (for example, a tablet terminal) that is an information processing device that can be carried into the tractor 10. The control device 20 may have a field map M of the field F where the tractor 10 will work.

[0016] For example, the control device 20 has a balloon-side control device 7 provided on the balloon 2 and a work vehicle-side control device 12 provided on the tractor 10. The balloon-side control device 7 is connected to the optical sensor 3 and acquires images captured by the optical sensor 3. The balloon-side control device 7 is also connected to the work vehicle-side control device 12 via a communication device 6 mounted on the balloon 2 and a communication device 11 mounted on the tractor 10. The balloon-side control device 7 may also transmit images captured by the optical sensor 3 to a management center (not shown) via the communication device 6. By keeping the balloon 2 afloat in the air for a certain period of time, the management center can periodically acquire images captured by the optical sensor 3 and check the growth status of the crops.

[0017] Next, the recognition process for the tractor 10 and its surroundings in the work vehicle system 1 will be described with reference to Figures 2 and 3. Figure 3 is a plan view showing the imaging area R.

[0018] The work vehicle system 1 is able to image one field F with one balloon 2 by changing the altitude of the balloon 2 and the field of view of the optical sensor 3. In this case, the altitude of the balloon 2 and the field of view of the optical sensor 3 are set so that the entire field F can be imaged by the optical sensor 3 even if the balloon 2 is blown away by the wind and moves.

[0019] For example, images captured by the optical sensor 3 are sent to the work vehicle control device 12 via the communication devices 6 and 11. The work vehicle control device 12 recognizes the position of the tractor 10 (host vehicle) based on the sent images, and is capable of automatic driving based on the field map M. The operator of the tractor 10 can operate the tractor 10 while checking the images displayed on the work vehicle control device 12. Alternatively, the images captured by the optical sensor 3 are sent to a management center. The management center may remotely control the tractor 10 based on the sent images.

[0020] The work vehicle system 1 can also capture images of one farm field F using multiple balloons 2. In this case, the balloons 2 are configured to float according to the size of the farm field F in order to capture an image of the entire farm field F that is the target of work. When capturing images using the balloons 2, blurring occurs due to the atmospheric environment, so as shown in Figure 3, the image capturing areas R1 to R4 are configured so that they partially overlap with adjacent image capturing areas R1 to R4. The captured images are stitched together to form an image of one farm field F. For example, an image of an area of ​​1 chobu (approximately 10,000 m) can be captured. 2 ), if the farm field F is square, the four balloons 2 can capture an image of the entire farm field F. Note that FIG. 3 shows imaging regions R1 to R4 when four balloons 2 are floating.

[0021] When an entire field F is imaged using multiple balloons 2, the balloon-side control device 7, for example, compares the locations where each balloon 2 is moored with the imaging areas R1 to R4 and the field map M, and stores correspondence information between the areas on the field map M and the balloons 2. For example, the balloon-side control device 7 stores the correspondence information, assuming that the area imaged by the balloon 2 moored above the imaging area R1 corresponds to the area of ​​the imaging area R1 on the field map M.

[0022] The balloon-side control device 7 recognizes in which area of ​​the field F the tractor 10 is located, based on information from the balloon 2 in which the tractor 10 is recognized in the captured image. The balloon-side control device 7 then reduces the calculation load by limiting the work vehicle position recognition, which is a process for recognizing the position of the tractor 10, and the surroundings recognition process, which recognizes the area around the tractor 10, to the recognized area. The work vehicle position recognition and surroundings recognition processes may be performed by the work vehicle-side control device 12. The work vehicle position recognition and surroundings recognition processes may also be processes that simply display an image of the recognized area on the work vehicle-side control device 12. The operator can confirm the tractor 10 and objects around the tractor 10 by checking the image displayed on the work vehicle-side control device 12.

[0023] In the surrounding recognition process, the balloon-side control device 7 recognizes objects around the tractor 10 (for example, by AI recognition) and classifies the recognized objects (hereinafter referred to as recognized objects). The classification may include, for example, an agricultural machinery class for objects recognized as agricultural machinery, a human class for objects recognized as people, and an animal class for objects recognized as animals. The balloon-side control device 7 converts the object position in the image into position information within the field F by matching the position of the classified recognized object with the boundary of the field F in the captured image and the field map M. The balloon-side control device 7 transmits the position information (including class information) of the recognized object within the field F to the tractor 10, which is operating automatically, via the communication devices 6 and 11.

[0024] When the recognized object is an agricultural machine class, the work vehicle control device 12 treats the position information of the host vehicle (tractor 10) as host vehicle position information of the automatic driving system. Also, when the recognized object is an agricultural machine class, the work vehicle control device 12 treats the position information of the other vehicle as relative position information (e.g., relative distance, relative direction) with respect to the host vehicle in order to avoid a collision with the automatically driven tractor 10 and to make a safe stop. Also, when the recognized object is a person class, the work vehicle control device 12 treats the position information of the person class as relative position information with respect to the host vehicle in order to avoid a collision with the automatically driven host vehicle and to make a safe stop.

[0025] The balloon-side control device 7 classifies the animal class position information into subclasses based on the size of the recognized object (for example, small animals are those with a body length of 30 cm or less, and large animals are those with a body length of 30 cm or more). The work vehicle-side control device 12 treats the animal class position information as relative position information with the autonomously driven vehicle in order to avoid collisions with the vehicle and to perform a safe stop. If the subclass is small animals, the work vehicle-side control device 12 does not perform collision avoidance or a safe stop. In other words, the work vehicle-side control device 12 does not perform collision avoidance or a safe stop of the work vehicle for recognized objects classified as being below a predetermined size.

[0026] Furthermore, the classification includes, for example, a materials class, which is a class recognized as a material, and an other class, which is a class not classified into the above classes.

[0027] If the recognized object is a material class object, the work vehicle control device 12 sets the area (for example, 10 cm) around the material class position information as a no-travel zone and treats it as a non-travelable area on the travel route. If the recognized object is another class object, the work vehicle control device 12 treats the other class position information as relative position information with the vehicle itself in order to avoid collision with the automatically driving tractor 10 and to make a safe stop.

[0028] Next, a work vehicle system 1 using a drone will be described with reference to Figures 4 and 5. Figure 4 is an explanatory diagram showing the perimeter of a field F and the flight path of the drone. Figure 5 is an explanatory diagram showing a straight line guide E1 displayed on VR goggles. Figure 5(a) is an explanatory diagram showing an image seen from the viewpoint of the operator of the work vehicle. Figure 5(b) is an explanatory diagram showing an image in which the straight line guide and the like are combined with Figure 5(a). In Figure 5, the operator is operating a rice transplanter equipped with a center mascot 19 so that it follows the ridge A. In Figure 5(a), a water turbine marker mark T is formed below the water surface S.

[0029] The work vehicle system 1 is equipped with a drone (agricultural drone) used for spraying pesticides, etc. The drone has an optical sensor 3. Before the tractor 10 performs work in the field F, the drone is manually controlled to fly over the field F along a route similar to the travel route of the tractor 10 and capture images of the field F.

[0030] As shown in FIG. 4, for example, the drone makes one revolution along the boundary line L, inside the boundary line L. In the second revolution, the drone flies along the flight trajectory Ft of the first revolution, inside the flight trajectory Ft of the first revolution. The drone completes work on one surface of the field F by repeating revolutions along the inside of the trajectory of the previous revolution from the third revolution onwards in the same manner. This allows the drone to capture images that can be seen from a route similar to the travel route of the tractor 10.

[0031] The captured images are stitched together to form a single image of the field F. The captured images are used as basic data of the field F (such as the boundary line L of the field F and the presence or absence of obstacles within the field F) for the work of the tractor 10. In addition, the drone can automatically fly a route similar to the travel route of the tractor 10 using the basic data.

[0032] Furthermore, as shown in Figure 5, the image captured by the drone may be converted into an image seen from the operator's viewpoint and displayed on the VR goggles. Furthermore, by combining an image of a straight line guide E1, it is possible to facilitate operation by the operator. For example, if the work vehicle is a rice transplanter, a water wheel marker trace T made by a water wheel marker may be used, but in deep water the water wheel marker trace T may not be visible. The VR goggles display the straight line guide E1 to complement the water wheel marker trace T. The VR goggles may also display the remaining amount of fertilizer E2, the current position E3 of the work vehicle obtained using a drone, and seedling shortage information E4.

[0033] Next, the detection range setting of the work vehicle system 1 will be described with reference to Figures 6 to 13. Figure 6 is an explanatory diagram showing a configuration in which an environment recognition sensor 13 is provided at the top of the vehicle body. Figure 7 is a plan view showing the sensor mounted on the tractor 10. Figures 8 and 9 are explanatory diagrams showing the detection range setting process.

[0034] Fig. 9(a) shows a situation in which the tractor 10 is traveling while detecting an obstacle in detection range B. Fig. 9(b) shows a situation in which the tractor 10 erroneously detects the ground as an obstacle at point P1. Fig. 9(c) shows a situation in which the tractor 10 does not erroneously detect the ground as an obstacle at point P2.

[0035] As shown in Figure 6, the tractor 10 is equipped with an environment recognition sensor 13 that detects an obstacle C within a detection range B, and a camera that allows the user to check the surroundings using a tablet or the like. By installing a camera, the tractor 10 can check the surroundings while driving or when starting, improving safety.

[0036] 7, the tractor 10 may be equipped with multiple sensors such as a lidar in addition to a camera. For example, the tractor 10 may be equipped with a front ultrasonic sensor 14, a front 3D LiDAR 15, an omnidirectional camera 16, a GNSS device 17 with a built-in inertial measurement unit (IMU), and a rear 3D LiDAR 18. The safety of the tractor 10 is improved by being equipped with multiple sensors.

[0037] When the tractor 10 is operating autonomously in conditions with a lot of dust, its sensors may detect dust and cause it to repeatedly stop. This can be addressed by combining a camera with sensors (LiDAR, sonar, and millimeter-wave sensors). When the control device 20 detects dust with a sensor, it uses the camera to determine whether dust is occurring. If dust is occurring, the control device 20 ignores the sensor information; if dust is not occurring, it uses the sensor value. Whether dust is occurring can be determined by using AI to learn the conditions around the tractor 10 (conditions above the ground). Learning parameters include "color," "color transparency," and "haze." This method makes it possible to handle situations where false detection occurs due to weather conditions such as fog or rain.

[0038] In this way, the control device 20 can detect obstacles or people using camera information after sensor detection in conditions where sensors are prone to false detection due to dust, etc. This reduces the probability of false detection and reduces temporary stops of the tractor 10, improving productivity.

[0039] Furthermore, as shown in FIG. 8, when the tractor 10 is driven autonomously in a field F with many slopes, the inclination of the vehicle body can cause it to erroneously detect the ground as an obstacle and stop. This can be prevented by combining a camera with sensors (LiDAR, sonar sensor, millimeter-wave sensor), and an inertial measurement unit (IMU). The control device 20 can reduce false detection of the ground by detecting the inclination of the field F using a camera and changing the detection range B based on the angle of the IMU mounted on the tractor 10 and the angle detected by the camera. This can reduce false detection.

[0040] In this way, in situations where the sensor is prone to false detection, such as on slopes, the control device 20 reduces the probability of false detection by using images captured by the camera to determine the detection range B. This reduces the probability of false detection and reduces temporary stops of the tractor 10, thereby improving productivity.

[0041] Furthermore, as shown in FIG. 9, when the tractor 10 is driven automatically in a field F that is close to the neighboring field F, it may mistakenly detect the crops in the neighboring field F as an obstacle and stop (see point P3). False detection can be prevented by using a GNSS device 17 in combination with sensors (LiDAR, sonar sensor, millimeter wave sensor). The control device 20 recognizes the work area (the extent of the field F) using the GNSS device 17, and if the detection range B exceeds the work area, it can change the detection range B (for example, change the detection range B so that it fits within the work area) to reduce false detection.

[0042] In this way, the GNSS information is used to reduce the probability of false detection in situations where there is a possibility of false detection of crops in the neighboring field F. This reduces the probability of false detection and reduces temporary stops of the tractor 10, thereby improving productivity.

[0043] Furthermore, conventionally, tractors 10 with automatic driving functions receive a signal from an external device while turning and change the detection range B. In the detection range setting of the present application, when there is no automatic driving function and only straight-line assist is provided, the detection range B can be changed using the steering angle, which can be configured inexpensively. The control device 20 can set an appropriate detection range B by saving the angle of the steering wheel at the start of turning in advance and switching the detection range B.

[0044] In this way, the control device 20 reduces the probability of false detection by determining the detection range B of the sensor and camera based on the steering angle of the steering wheel. This allows the detection range B to be set inexpensively.

[0045] As described above, the work vehicle system 1 according to the embodiment comprises a balloon 2 floating in the air above the field F that is the work target of the tractor 10, an optical sensor 3 mounted on the bottom of the balloon 2, and a control device 20 that acquires images captured by the optical sensor 3. One balloon 2 is levitated for one field F, and the altitude of the balloon 2 and the field of view angle of the optical sensor 3 are changed to capture an image of the entire field F with the balloon 2.

[0046] According to the work vehicle system 1 configured as described above, it is possible to determine the position of the tractor 10 within the field F even if the tractor 10 is not equipped with a GNSS receiver.

[0047] Furthermore, as described above, in the work vehicle system 1 according to the embodiment, for one field F, multiple balloons 2 are floated according to the size of the field F, and the multiple balloons 2 are configured so that adjacent imaged areas partially overlap, and the control device 20 holds a field map M of the field F and holds correspondence information between the areas on the field map M and the balloons 2, and recognizes in which area on the field map M the tractor 10 is located based on the information of the balloons 2 in which the work vehicle is recognized in the image captured by the optical sensor 3, and performs a work vehicle position recognition process, which is a process of recognizing the position of the tractor 10 only in the recognized area, and a surrounding recognition process, which recognizes the surroundings of the tractor 10.

[0048] According to the work vehicle system 1 configured as above, only images captured from limited balloons 2 are used, and therefore calculation processing is quick.

[0049] Furthermore, as described above, in the work vehicle system 1 according to the embodiment, the control device 20 transmits position information of recognized objects, which are objects recognized in the surrounding recognition processing, to the tractor 10, which is operating autonomously, and classifies the position information when the recognized object is an animal by size of the recognized object, and treats it as relative position information with respect to the tractor 10 in order to avoid collisions and safely stop the tractor 10, and does not perform collision avoidance or safe stopping of the tractor 10 for recognized objects classified as being below a predetermined size.

[0050] With this configuration, analysis is performed from the sky using the optical sensor 3, allowing the tractor 10 to avoid collisions and safely stop at the correct distance intervals. In addition, unnecessary stops of the tractor 10 are reduced, improving work efficiency.

[0051] Further advantages and modifications will readily occur to those skilled in the art. Therefore, the invention in its broader aspects is not limited to the specific details and representative embodiments shown and described above. Accordingly, various modifications may be made without departing from the spirit or scope of the general inventive concept as defined by the appended claims and their equivalents. [Explanation of symbols]

[0052] 1 Work vehicle system 2. Balloons 3 Optical Sensor 20 Control device F field M Field Map R1 imaging area R2 imaging area R3 imaging area R4 imaging area

Claims

1. a balloon floating in the air above a field that is the target of work by the work vehicle; an optical sensor mounted on the lower part of the balloon; a control device for acquiring the image captured by the optical sensor, Levitating one balloon for one of the fields, A work vehicle system that images the entire field using the balloon by changing the altitude of the balloon and the field of view angle of the optical sensor.

2. For one of the farm fields, a plurality of the balloons are floated according to the size of the farm field, The plurality of balloons are configured so that adjacent imaging regions partially overlap each other, The control device a field map of the field is stored, and correspondence information between areas on the field map and the balloons is stored; Recognizing in which area on the field map the work vehicle is located based on information about the balloon in which the work vehicle is recognized within the image captured by the optical sensor; The work vehicle system according to claim 1, wherein a work vehicle position recognition process is performed to recognize the position of the work vehicle only in the recognized area, and a surroundings recognition process is performed to recognize the surroundings of the work vehicle.

3. The control device transmitting position information of a recognized object, which is an object recognized in the surrounding recognition processing, to the work vehicle performing automatic driving; classifying the position information of the recognized object according to the size of the recognized object, and treating it as relative position information with respect to the work vehicle in order to avoid a collision and safely stop the work vehicle; The work vehicle system according to claim 2, wherein collision avoidance and safety stop of the work vehicle are not performed for the recognized objects classified as being equal to or smaller than a predetermined size.

Images