Sensing system and unmanned aircraft

Abstract

This sensing system comprises: a localization device for acquiring position information for an unmanned aircraft; a sensing device for sensing a terrestrial environment and acquiring a remote-sensing image having geographical information that is based on the position information; a processing device for processing a remote-sensing image; and a map data-acquisition means for acquiring map data that includes the geographical information in a map. The processing device estimates an error, with respect the geographical information, between a first remote-sensing image and the map data on the basis of: the geographical information of the first remote-sensing image acquired by the sensing device sensing the terrestrial environment at a first altitude, and the geographical information of a second remote-sensing image acquired by the sensing device sensing the terrestrial environment at a second altitude that is higher than the first altitude; and the geographical information in the map data.

Description

Sensing Systems and Unmanned Aerial Vehicles

[0001] The present disclosure relates to a sensing system and an unmanned aerial vehicle.

[0002] An unmanned aerial vehicle (UAV) is an aircraft that cannot carry a person due to its structure and can fly by remote control or automatic pilot. Rotary-wing unmanned aerial vehicles are unmanned aerial vehicles that obtain lift using propellers, i.e., rotors, that rotate around an axis. Small unmanned aerial vehicles equipped with multiple rotors (multi-rotor UAVs) are also called "drones," "multirotors," or "multicopters," and are widely used for applications such as aerial photography, surveying, logistics, and pesticide spraying.

[0003] Patent Document 1 describes an example of a technology that analyzes field images captured by an unmanned aerial vehicle and calculates a vegetation index for agricultural crops in the field.

[0004] Japanese Patent Application Laid-Open No. 2019-185773

[0005] Map data containing terrestrial geographic information may be used in flight planning for unmanned aerial vehicles. The map data may be, for example, data on electronic maps, aerial photographs, or satellite images. The map data may contain location information (or geographic information). However, the geographic information in the map data is not always accurate, and the latitude and longitude included in the geographic information of the map data may differ from the actual latitude and longitude. Even in such cases, it is desirable to appropriately plan the flight of the unmanned aerial vehicle by utilizing the geographic information in the map data.

[0006] The present disclosure provides a sensing system that can solve such problems and an unmanned aerial vehicle equipped with the sensing system.

[0007] In an exemplary and non-limiting embodiment, the sensing system of the present disclosure is a sensing system used in an unmanned aerial vehicle and sensing a ground environment, comprising: a positioning device that acquires position information of the unmanned aerial vehicle; a sensing device that senses the ground environment and acquires remote sensing images having geographic information based on the position information; a processing device that processes the remote sensing images; and a map data acquisition means that acquires map data including geographic information on a map, wherein the processing device estimates an error in geographic information between the first remote sensing image and the map data based on the geographic information of a first remote sensing image acquired by the sensing device sensing the ground environment at a first altitude, and the geographic information of a second remote sensing image acquired by sensing the ground environment at a second altitude higher than the first altitude, and the geographic information of the map data.

[0008] In an exemplary and non-limiting embodiment, the unmanned aerial vehicle of the present disclosure includes multiple rotors and the above-described sensing system.

[0009] According to an embodiment of the present disclosure, a sensing system is provided that can appropriately plan the flight of an unmanned aircraft by utilizing the geographic information of the map data, even if the geographic information of the map data differs from the actual location information, and an unmanned aircraft equipped with the sensing system is provided.

[0010] 1 is a block diagram schematically showing several examples of rotary drive devices that rotate rotors in an unmanned aerial vehicle with multiple rotors. FIG. 2 is a plan view schematically showing one basic configuration example of an unmanned aerial vehicle with multiple rotors. FIG. 3 is a side view schematically showing one basic configuration example of an unmanned aerial vehicle with multiple rotors. FIG. 4 is a plan view schematically showing another basic configuration example of an unmanned aerial vehicle with multiple rotors. FIG. 5 is a block diagram showing an example basic configuration of a battery-powered multicopter. FIG. 6 is a block diagram showing an example basic configuration of a series hybrid drive multicopter. FIG. 7 is a block diagram showing an example basic configuration of a parallel hybrid drive multicopter. FIG. 8 is a flow diagram showing an example procedure for generating or updating a flight plan. FIG. 9 is a schematic diagram for explaining a first altitude, a second altitude, and a third altitude. FIG. 10 is a schematic diagram showing an example map including a field where a multicopter will perform agricultural work and surrounding fields. FIG. 11 is a schematic diagram showing an example of a first remote sensing image. FIG. 12 is a schematic diagram showing an example of a second remote sensing image. 1 is a schematic diagram showing a state in which the vertices of a field on a map corresponding to the field are shifted relative to the vertices of the field included in a first field image. 2 is a block diagram showing an example of the hardware configuration of a control device. 3 is a schematic diagram showing an example in which a multicopter, an agricultural machine, a server, and a terminal device are connected via a communication network.

[0011] An unmanned aerial vehicle with multiple rotors includes a rotary drive unit that rotates the rotors (hereinafter sometimes referred to as "propellers"). Hereinafter, such an unmanned aerial vehicle will be referred to as a "multicopters."

[0012] There are various configurations of the rotary drive device provided in a multicopter. Fig. 1A is a block diagram schematically illustrating four examples of the rotary drive device 3 in the present disclosure.

[0013] The first rotation drive device 3A shown in FIG. 1A has a plurality of electric motors (hereinafter referred to as "motors") 14 that rotate a plurality of rotors, and a battery 52 that stores power to be supplied to each motor 14. The battery 52 is, for example, a secondary battery such as a polymer lithium-ion battery. Each rotor 2 is connected to the output shaft of the corresponding motor 14 and is rotated by the motor 14. In order to increase the payload and / or flight time, it is necessary to increase the power storage capacity of the battery 52. ​​The power storage capacity of the battery 52 can be increased by increasing the size of the battery 52, but increasing the size of the battery 52 results in an increase in weight.

[0014] The second rotation drive device 3B shown in FIG. 1A includes a power transmission system 23 mechanically connected to the rotor 2 and an internal combustion engine 7a that provides driving force (torque) to the power transmission system 23. The power transmission system 23 includes mechanical components such as gears or belts, and transmits torque from the output shaft of the internal combustion engine 7a to the rotor 2. The internal combustion engine 7a can efficiently generate mechanical energy by burning fuel. Examples of the internal combustion engine 7a include a gasoline engine, a diesel engine, and a hydrogen engine. The number of internal combustion engines 7a included in the rotation drive device 3B is not limited to one.

[0015] The third rotary drive device 3C shown in FIG. 1A includes multiple motors 14, a power buffer 9 that stores power to be supplied to each motor 14, a power generator 8 such as an alternator that generates power, and an internal combustion engine 7a that provides mechanical energy for the power generator 8 to generate electricity. A typical example of the power buffer 9 is a battery such as a secondary battery, but it may also be a capacitor. In the third rotary drive device 3C, even if the power buffer 9 does not have a large storage capacity, the power generator 8 generates power using the driving force (mechanical energy) of the internal combustion engine 7a, thereby enabling an increase in payload and / or flight time. This type of drive is called a "series hybrid drive." The power generator 8 and internal combustion engine 7a in the series hybrid drive are called a "range extender" because they extend the flight distance of the multicopter.

[0016] 1A includes a plurality of motors 14, a power buffer 9 that stores power to be supplied to each motor 14, a power generator 8 such as an alternator that generates power, an internal combustion engine 7a that provides driving force for generating power to the power generator 8, and a power transmission system 23 that transmits the driving force generated by the internal combustion engine 7a to a rotor 2 to rotate the rotor 2. At least one rotor 2 of the plurality of rotors 2 is rotated by the internal combustion engine 7a, and the other rotors 2 are rotated by the motor 14. In the fourth rotary drive device 3D, the mechanical energy generated by the internal combustion engine 7a can also be used to rotate the rotor 2 without being converted into electric power, thereby improving energy utilization efficiency. This type of drive is called a "parallel hybrid drive."

[0017] Fig. 1B is a plan view schematically illustrating one basic configuration example of multicopter 10. The configuration example of Fig. 1B includes the first rotational drive device 3A shown in Fig. 1A as the rotational drive device 3. That is, the rotational drive device 3 (3A) in this example includes a motor 14 and a battery 52. ​​Fig. 1C is a side view schematically illustrating the multicopter.

[0018] 1B and 1C includes a plurality of rotors 2, an airframe 4, and an airframe frame 5 that supports the rotors 2 and the airframe 4. The airframe frame 5 supports the airframe 4 at its center and rotatably supports the plurality of rotors 2 with a plurality of arms 5A extending outward from the center. A motor 14 that rotates the rotor 2 is provided near the tip of each arm 5A.

[0019] 1B, the multicopter 10 is a quad-type multicopter (quadcopter) having four rotors 2. The rotors 2 located on one diagonal line rotate in the same direction (clockwise or counterclockwise), while the rotors 2 located on different diagonal lines rotate in opposite directions.

[0020] The main body 4 includes a control device 4a that controls the operation of devices and components mounted on the multicopter 10, a group of sensors 4b connected to the control device 4a, a communication device 4c connected to the control device 4a, and a battery 52.

[0021] The control device 4 a may include, for example, a flight control device such as a flight controller and a host computer (companion computer). The companion computer can perform advanced arithmetic processing such as image processing, obstacle detection, and obstacle avoidance based on the sensor data acquired by the sensor group 4 b.

[0022] The sensor group 4b may include an acceleration sensor, an angular velocity sensor, a geomagnetic sensor, a barometric pressure sensor, an altitude sensor, a temperature sensor, a flow rate sensor, an imaging device, a laser sensor, an ultrasonic sensor, an obstacle contact sensor, and a Global Navigation Satellite System (GNSS) receiver. The acceleration sensor and the angular velocity sensor may be mounted on the airframe main body 4 as components of an IMU (Inertial Measurement Unit). Examples of the laser sensor may include, for example, a laser range finder used to measure the distance to the ground, and a two-dimensional or three-dimensional LiDAR.

[0023] The communication device 4c may include a wireless communication module for transmitting and receiving signals via an antenna to a transmitter or ground station (Ground Control Station (GCS)) on the ground, a mobile communication module using a cellular communication network, etc. The communication device 4c may receive signals such as control commands transmitted from the ground and transmit sensor data such as image data acquired by the sensor group 4b as telemetry information. The communication device 4c may have a function for communicating between multicopters and a satellite communication function. The control device 4a can be connected to a computer on the cloud via the communication device 4c. Some or all of the functions of the companion computer may be performed by the computer on the cloud.

[0024] The battery 52 is a secondary battery that stores power by charging and supplies power to the motors 14 by discharging. The battery 52 and the multiple motors 14 operate to rotate the multiple rotors 2, making it possible to generate a desired thrust.

[0025] Each of the multiple rotors 2 generally has multiple blades with a fixed pitch angle and generates thrust by rotation. The pitch angle may be variable. The multiple rotors 2 do not all need to have the same diameter (propeller diameter); one or more rotors 2 may have a larger diameter than the other rotors 2. The thrust (static thrust) generated by a rotating rotor 2 is generally proportional to the cube of the rotor 2 diameter. Therefore, when rotors 2 with different diameters are included, the rotor 2 with a relatively larger diameter may be referred to as the "main rotor," and the rotor 2 with a relatively smaller diameter may be referred to as the "sub-rotor." Note that, regardless of the diameter, the configuration of the rotary drive device 3 may include a rotor 2 capable of generating a relatively larger thrust and a rotor 2 with a relatively smaller thrust. In this case, the rotor 2 capable of generating a relatively larger thrust may be referred to as the "main rotor," and the rotor 2 with a relatively smaller thrust may be referred to as the "sub-rotor." For example, the rotor 2 that generates a relatively large thrust per rotation may be referred to as the "main rotor," and the rotor 2 that generates a relatively small thrust per rotation may be referred to as the "sub-rotor." In one example, the main rotor may be positioned more inward than the sub-rotors. In other words, each rotor 2 may be positioned so that the distance from the center of the airframe to the rotation axis of each main rotor is shorter than the distance from the center of the airframe to the rotation axis of each sub-rotor.

[0026] In this example, the rotary drive device 3 includes a plurality of motors 14. As mentioned above, the rotary drive device 3 may include an internal combustion engine 7a.

[0027] 1D is a plan view schematically illustrating an example of the basic configuration of a multicopter 10 including a second rotational drive device 3B as the rotational drive device 3. In the example shown in FIG. 1D, an internal combustion engine 7a is supported by the airframe main body 4. In this example, the driving force generated by the internal combustion engine 7a is transmitted to multiple rotors 2 via multiple power transmission systems 23, causing each rotor 2 to rotate. The control device 4a can change the rotational speed of each rotor 2 by controlling each power transmission system 23.

[0028] In a "parallel hybrid drive" in which some of the multiple rotors 2 are rotated by the internal combustion engine 7a and the other rotors 2 are rotated by the motor 14, the internal combustion engine 7a and the battery 52 are supported on the aircraft body 4. At least one rotor 2 of the multiple rotors 2 is connected to the internal combustion engine 7a via the power transmission system 23, and the other rotors 2 are connected to the motor 14.

[0029] In such a parallel hybrid drive, the diameter of one or more rotors 2 rotated by the internal combustion engine 7a may be larger than the diameter of the other rotors 2 rotated by the motor 14. In other words, the internal combustion engine 7a may be used to rotate the main rotor, and the motor 14 may be used to rotate the sub-rotor. In such a case, the main rotor is mainly used to generate thrust, and the sub-rotor is used to generate thrust and for attitude control. The main rotor may also be called a "booster rotor," and the sub-rotor may also be called an "attitude control rotor."

[0030] In the case of parallel hybrid drive, the internal combustion engine 7a is used for both thrust generation and power generation. By selectively transmitting the driving force (torque) generated by the internal combustion engine to one or both of the rotor and the power generator, it is possible to achieve a good balance between thrust generation and power generation.

[0031] Equipping a multicopter with an internal combustion engine 7a and using the internal combustion engine 7a to generate thrust and / or electricity contributes to an increase in payload and flight time. It is desirable to control the attitude of a multicopter by rotating the propellers with a motor, which has better response characteristics than an internal combustion engine. Therefore, in applications where precise control of the attitude of a multicopter is required, it is desirable to employ a parallel hybrid drive or a series hybrid drive to increase the payload and flight time.

[0032] Increased payload and flight time may further expand the applications of multicopters. For example, in the agricultural field, multicopters are currently being used for spraying pesticides or monitoring crop growth conditions. However, by connecting various ground implements (hereinafter, sometimes simply referred to as "implements") to a multicopter, various agricultural tasks can be performed from the air. Agricultural implements are sometimes called "implements." Examples of implements include sprayers that spray pesticides on crops, mowers, seeders, spreaders, rakes, balers, harvesters, plows, harrows, or rotary tillers. Work vehicles such as tractors are not included in the "implements" of this disclosure.

[0033] In the example shown in FIG. 1C , a work implement 200 is coupled to the multicopter 10. The work implement 200 can spray, for example, pesticides or fertilizers on a field or crops within the field. Increasing the payload and flight time allows for a larger and / or more versatile work implement 200. For example, by changing the work implement 200 coupled to the multicopter 10, a variety of ground tasks (agricultural operations) can be performed, including liquid and granular application of pesticides, fertilization, thinning, weeding, transplanting, direct seeding, and harvesting. The work implement 200 may be equipped with a mechanism such as a robotic hand. In this case, a single work implement 200 can perform a variety of ground tasks. Furthermore, if the work implement 200 has a sufficient space to accommodate the materials, the work implement 200 can also transport agricultural materials or harvested products over a wide area.

[0034] 1C , the multicopter 10 includes a power supply device 76. The power supply device 76 is a device that supplies power to the work machine 200 from a drive energy source, such as the battery 52 or the power generation device 8, included in the multicopter 10. Various functions of the work machine 200 can be performed using this power. The work machine 200 includes actuators such as motors that operate using power obtained from the power supply device 76 of the multicopter 10. The work machine 200 preferably includes a battery that stores power.

[0035] 2A is a block diagram showing an example of the basic configuration of a battery-powered multicopter 10. The battery-powered multicopter 10 includes a plurality of rotors 12, a plurality of motors 14 that rotate the rotors 12, a plurality of ESCs (electric speed controllers) 16 each having a motor drive circuit that drives the motors 14, a battery 52 that supplies power to the corresponding motor 14 via each ESC 16, a control device 4a that controls the plurality of ESCs 16 to control attitude while flying, a sensor group 4b, a communication device 4c, and a power supply device 76 electrically connected to the battery 52. ​​The rotor 12 is an example of a rotor 2. The control device 4a, the sensor group 4b, the communication device 4c, and other devices are connected to each other so that they can communicate with each other, for example, via a controller area network (CAN) bus. For simplicity, the rotor 12, the motor 14, and the ESC 16 are each shown as a single block in Fig. 2A, but there are actually a plurality of rotors 12, motors 14, and ESCs 16. This also applies to Figs. 2B and 2C. The ESC 16 may be included in the control device 4a.

[0036] The control device 4a can receive control commands wirelessly from, for example, a ground station 6 located on the ground via the communication device 4c. The number of ground stations 6 is not limited to one, and they may be distributed across multiple locations. The communication device 4c can also receive control commands wirelessly from a control device of a pilot on the ground. The control device 4a may have a function to automatically or autonomously perform takeoff, flight, obstacle avoidance, and landing operations based on sensor data obtained from the sensor group 4b.

[0037] The control device 4a may be configured to communicate with the work machine 200 connected to the power supply device 76 and to acquire a signal indicating the state of the work machine 200 from the work machine 200. The control device 4a may also provide a signal to the work machine 200 that controls the operation of the work machine 200. Furthermore, the work machine 200 may generate a signal instructing the operation of the multicopter 10 and transmit the signal to the control device 4a. Such communication between the control device 4a and the work machine 200 may be performed wired or wirelessly.

[0038] FIG. 2B is a block diagram showing an example of the basic configuration of a series hybrid drive multicopter 10. Similar to the battery-powered multicopter 10, the series hybrid drive multicopter 10 includes multiple rotors 12, multiple motors 14, multiple ESCs 16, a control device 4a, a sensor group 4b, and a communication device 4c. The illustrated series hybrid drive multicopter 10 further includes an internal combustion engine 7a, a fuel tank 7b for storing fuel for the internal combustion engine 7a, a power generation device 8 driven by the internal combustion engine 7a to generate electric power, a power buffer 9 for temporarily storing the electric power generated by the power generation device 8, and a power supply device 76 electrically connected to the power buffer 9. The power buffer 9 is, for example, a battery such as a secondary battery. The electric power generated by the power generation device 8 is supplied to the motor 14 via the power buffer 9 and the ESC 16. The electric power generated by the power generation device 8 may also be supplied to the work machine 200 via the power supply device 76.

[0039] 2C is a block diagram showing an example of the basic configuration of a parallel hybrid drive multicopter 10. Similar to the series hybrid drive multicopter 10, the parallel hybrid drive multicopter 10 includes multiple rotors 12, multiple motors 14, multiple ESCs 16, a control device 4a, a sensor group 4b, a communication device 4c, an internal combustion engine 7a, a fuel tank 7b, a power generator 8, a power buffer 9, and a power supply device 76. The parallel hybrid drive multicopter 10 further includes a drive train 27 that transmits the driving force of the internal combustion engine 7a, and a rotor 22 that rotates by receiving the driving force of the internal combustion engine 7a from the drive train 27. The rotor 22 is an example of a rotor 2. The number of rotors 22 connected to the drive train 27 and rotating may be one or more.

[0040] In the parallel hybrid drive multicopter 10, the internal combustion engine 7a not only drives the power generation device 8 to generate electricity, but also mechanically transmits energy to the rotor 22 to rotate the rotor 22. On the other hand, in the series hybrid drive multicopter 10, all of the rotors 12 are rotated by the electric power generated by the power generation device 8. For this reason, in the series hybrid drive multicopter 10, if the power generation device 8 is, for example, a fuel cell, the internal combustion engine 7a is not an essential component.

[0041] In one aspect, a multicopter according to an embodiment of the present disclosure includes a sensing system for sensing a ground environment. The sensing system is suitable for use in a multicopter. The sensing system includes a positioning device that acquires position information of the multicopter, a sensing device that senses the ground environment to acquire a remote sensing image, a processing device that processes the remote sensing image to generate a remote sensing image having geographic information based on the position information, and a map data acquisition means that acquires map data including the geographic information on a map.

[0042] The positioning device includes, for example, a GNSS receiver, an RTK (Real Time Kinematic) receiver, a GNSS receiver, and an IMU. The positioning device can perform positioning of the multicopter using RTK-GNSS. By using RTK-GNSS, it is possible to perform positioning with an accuracy of, for example, a few centimeters. Position information including information on the latitude, longitude, and altitude of the measurement point is acquired through high-precision positioning using RTK-GNSS.

[0043] Examples of the sensing device include the aforementioned imaging device (camera) and two-dimensional or three-dimensional LiDAR. An example of the remote sensing image is an aerial image such as an aerial photograph. In this embodiment, the sensing device is an imaging device, and the remote sensing image is a camera image. However, the sensing device may also be LiDAR. In this case, the remote sensing image is an image obtained by visualizing point cloud data from the LiDAR.

[0044] An example of the processing device is one or more semiconductor integrated circuits (e.g., processors). For example, a companion computer included in the control device 4a described above may function as the processing device of the sensing system. Alternatively, a computer on the cloud may function as the processing device of the sensing system. Furthermore, the companion computer and the computer on the cloud may work together to perform the functions of the processing device.

[0045] The processing device in this embodiment is configured to estimate an error in geographic information between the first remote sensing image and the map data based on geographic information of a first remote sensing image acquired by a sensing device sensing the terrestrial environment at a first altitude, geographic information of a second remote sensing image acquired by sensing the terrestrial environment at a second altitude higher than the first altitude, and geographic information of the map data. Details of the operation of the processing device will be described later.

[0046] The processing device in this embodiment can generate metadata including shooting information, for example, in compliance with the Exchangeable Image File Format (Exif) standard. The metadata can include information such as the shooting date and time, location information (geotag), orientation information, image resolution, shutter speed, aperture (F-number), and ISO sensitivity. The processing device can process data of the first or second remote sensing image output from the sensing device, add metadata to the processed data, and generate image data in a file format such as RAW, DNG, TIFF, GeoTIFF, or JPEG.

[0047] An example of the map data acquisition means is the communication device 4c described above. A cloud server may manage map data including geographic information for each point on a map. The map data acquisition means may access the cloud server via a network to acquire the map data and store it in a storage device. The geographic information includes latitude and longitude information for each point. The storage device may include one or more storage media, such as a flash memory or a magnetic disk. For example, when the multicopter is powered on or while the multicopter is flying, the map data acquisition means accesses the cloud server via a network to acquire the map data. Alternatively, the processing device may update the map data acquired and pre-stored in the storage device 83 after a predetermined period of time has elapsed.

[0048] With a sensing system configured in this manner, even if the latitude and longitude included in the geographic information of the map data differ from the actual latitude and longitude, it is possible to appropriately plan a flight for the multicopter based on the geographic information of the map data.

[0049] Below, with reference to Figures 3 to 8, an example of a method for generating or updating a flight plan by correcting the geographic information of map data based on an error in the geographic information between the first remote sensing image and the map data will be described.

[0050] Fig. 3 is a flow diagram showing an example of a procedure for generating or updating a flight plan. Fig. 4 is a schematic diagram for explaining a first altitude A1, a second altitude A2, and a third altitude A3. Fig. 5 is a schematic diagram showing an example of a map including a field where a multicopter will perform agricultural work and surrounding fields.

[0051] In this embodiment, the multicopter 10 flies over fields on the map shown in Fig. 5 and performs agricultural work. In the example shown in Fig. 5, the multicopter 10 flies over field F6 and performs agricultural work. The ground environment on the map may include at least one field. The ground environment on the map shown in Fig. 5 includes fields F1 to F9.

[0052] First, when the power of the multicopter 10 is turned on, the map data acquisition means accesses the cloud server via the network to acquire map data and stores it in the storage device. Once the map data acquisition is complete, the multicopter 10 ascends to a first altitude A1. In the example shown in Figure 4, the first altitude A1 is approximately the same as a third altitude A3, which is the altitude at which the multicopter 10 starts farm work.

[0053] When the multicopter 10 reaches the first altitude A1, the sensing device senses the ground environment at the first altitude and acquires a first remote sensing image (step S10).

[0054] 6 is a schematic diagram showing an example of a first remote sensing image. The first remote sensing image includes a portion of at least one field as a subject. In the example shown in FIG. 6, the first remote sensing image includes portions of fields F1a, F2a, F5a, and F7a, which correspond to fields F1, F2, F5, and F7 on the map, respectively, and further includes the entire field F6a, which corresponds to field F6 on the map.

[0055] In this embodiment, the processing device uses external and internal parameters of the camera (sensing device) to perform coordinate conversion from pixel positions in the image coordinate system to geographic coordinates in a world coordinate system, such as a geographic coordinate system fixed relative to the Earth. Specifically, the processing device associates the position information of the multicopter 10 in the geographic coordinate system acquired by the positioning device at a point at a first altitude A1 with the data of the first remote sensing image acquired by the sensing device at the first altitude, thereby generating a first remote sensing image having geographic coordinates. In this specification, a remote sensing image having such geographic coordinates is referred to as a "field image." In this manner, the processing device converts the first remote sensing image in the image coordinate system into a first field image in the geographic coordinate system.

[0056] The processing device may calculate a three-dimensional translation vector T as an external parameter based on the position information of the multicopter 10 in a geographic coordinate system, and may calculate a 3×3 rotation matrix R based on the attitude information. The attitude information is acquired, for example, by an IMU. The processing device may calculate internal parameters represented by a 3×3 transformation matrix based on the focal length, pixel pitch, and image center of the camera of the sensing device.

[0057] Homography transformations can be used for such coordinate transformations. Homography transformations include the concept of affine transformations. Homography transformations are also called "projection transformations" or "perspective transformations." Applying an affine transformation enables transformations that involve translation, rotation, and scaling, while applying a homography transformation also enables trapezoidal deformation.

[0058] Next, the multicopter 10 ascends further to a second altitude A2, which is higher than the first altitude A1. In this embodiment, the latitude and longitude of the point at the first altitude A1 are equal to the latitude and longitude of the point at the second altitude A2. In other words, the multicopter 10 ascends from the first altitude A1 to the second altitude A2 while maintaining its latitude and longitude. When the multicopter 10 reaches the second altitude A2, the sensing device senses the ground environment at the second altitude A2 to acquire a second remote sensing image (step S20). In this manner, the sensing device acquires first and second remote sensing images at points with the same latitude and longitude. The first remote sensing image acquired at the first altitude A1 has a relatively high resolution, and the second remote sensing image acquired at the second altitude A2 has a relatively low resolution.

[0059] Fig. 7 is a schematic diagram showing an example of a second remote sensing image. The second remote sensing image in the example shown in Fig. 7 includes portions of fields F1b, F2b, F7b, F8b, and F9b corresponding to fields F1, F2, F7, F8, and F9 on the map, respectively, and further includes the entire fields F5b and F6b corresponding to fields F5 and F6 on the map.

[0060] The processing device may associate the position information of the multicopter 10 in the geographic coordinate system acquired by the positioning device at the point at the second altitude A2 with the data of the second remote sensing image acquired by the sensing device at the second altitude A2, similar to the first remote sensing image, to generate a second remote sensing image having geographic coordinates. In other words, the processing device may convert the second remote sensing image in the image coordinate system into a second field image in the geographic coordinate system.

[0061] In this way, the first and second field images in the geographic coordinate system are images based on the first and second remote sensing images in the image coordinate system, respectively, and are strictly different. However, for the sake of simplicity, in this specification, the terms field image and remote sensing image may not be distinguished from each other, and for example, a first field image based on a first remote sensing image may be referred to as a first remote sensing image, and geographic information of the first field image may be referred to as geographic information of the first remote sensing image.

[0062] The processing device then determines a discrepancy between the geographic information of the second remote sensing image (or the second field image) and the geographic information of the map data by matching the second remote sensing image (or the second field image) with the map data, including region-based matching (template matching) and feature-point-based matching (feature point matching).

[0063] The ground environment in the map shown in Figure 5 includes at least one field and features such as a tree 80a, a building 80b, a road 80c, and an intersection 80d. These features can serve as landmarks for matching. For example, the road 80c or the intersection 80d is not included in the first remote sensing image shown in Figure 6, which was captured at a first altitude A1, but is included in the second remote sensing image shown in Figure 7, which was captured at a second altitude A2. In this way, subjects that are difficult to capture at a relatively low altitude can be captured at a relatively high altitude.

[0064] The processing device in this embodiment matches the second remote sensing image (or the second field image) with the map data to identify a second landmark on the map that corresponds to the first landmark included in the second remote sensing image. For example, by matching the second remote sensing image with the map data, the processing device identifies the road 80c or the intersection 80d on the map shown in Figure 5 that corresponds to the road 80c or the intersection 80d included in the second remote sensing image shown in Figure 7.

[0065] The processing device may extract road or intersection regions from the second remote sensing image or the second field image by applying a "segmentation" algorithm, such as semantic segmentation or instance segmentation, to the data of the second field image. For example, the processing device may generate templates corresponding to the extracted road or intersection regions from the extracted road or intersection regions and use the templates to match the data of the second field image with map data. Alternatively, the processing device may determine landmark feature points from the extracted road or intersection regions and perform feature point matching.

[0066] The processing device determines a discrepancy between the geographic information of the second field image and the geographic information of the map data based on the geographic information of the first and second landmarks. In this embodiment, the processing device determines a discrepancy between the geographic information of the road 80c or intersection 80d included in the second field image and the geographic information of the road 80c or intersection 80d in the map data. Specifically, the processing device determines a discrepancy between the two pieces of geographic information by comparing the deviation amounts of the latitude and longitude included in the two pieces of geographic information with a threshold. For example, if the deviation amount of at least one of the latitude and longitude is equal to or greater than the threshold, the processing device determines that there is a discrepancy between the geographic information of the second field image and the geographic information of the map data (YES in step S40). On the other hand, if the deviation amounts of the latitude and longitude are less than the threshold, the processing device determines that there is no discrepancy between the geographic information of the second field image and the geographic information of the map data (NO in step S40).

[0067] The sensing system may further include a control device that controls the flight of the multicopter 10. The aforementioned control device 4a may function as this control device. The control device preferably causes the multicopter 10 to ascend to an altitude at which the processing device can identify a second landmark corresponding to the first landmark. The sensing device may repeatedly sense the ground environment from the first altitude A1 to the second altitude A2 and acquire multiple remote sensing images corresponding to each altitude. In this way, it is possible to acquire sensing images containing subjects that can function as landmarks to be used for matching.

[0068] Next, if the processing device determines that there is a discrepancy between the geographic information of the second field image and the geographic information of the map data, the processing device then matches the first remote sensing image (or the first field image) with the map data to estimate the error in the geographic information between the first remote sensing image and the map data (step S50). Specifically, by matching the first field image with the map data, the processing device identifies fields on the map that correspond to fields included in the first field image, and estimates the error in the geographic information between the two fields.

[0069] For example, an agricultural manager, who is a user, can register a field parcel by identifying a field to be worked on from among the fields included in an aerial photograph displayed on a tablet computer screen. The user then clicks in order on three or more vertices that define the shape of the identified field, creating pins at the vertices, thereby finalizing the field parcel. Information about the finalized parcel is registered in GIS data, for example, as a parcel polygon.

[0070] The processing device can create a template that matches the shape of the field based on information about the field's divisions registered by the user in this manner, for example. Suppose that the user has registered field F6, one of the multiple fields included in the map shown in FIG. 5, as a field to be subjected to agricultural work. In this case, the processing device generates a template that matches the shape of field F6 based on information about the divisions of field F6. The processing device applies template matching using the template generated in this manner to the data of the first field image to identify field F6a included in the first field image that corresponds to field F6 on the map. Alternatively, the processing device may perform feature point matching using, for example, vertices or edges that define the field's shape as feature points. The processing device estimates an error based on the longitude and latitude included in the geographic information of field F6a included in the first field image and field F6 on the corresponding map.

[0071] 8 is a schematic diagram showing a state in which the vertices of field F6 on the map corresponding to field F6a are shifted relative to the vertices of field F6a included in the first field image. If there is an error between the accurate position information acquired by the positioning device and the geographic information of the map data, when the first field image and the map are overlaid as shown in the figure, the vertices of field F6 on the map corresponding to field F6a are shifted relative to the vertices of field F6a included in the first field image.

[0072] The processing device estimates the error in geographic information between the first remote sensing image (or the first field image) and the map data from the amount of deviation in the geographic coordinates of each vertex that defines the shape of the field. In the example shown in FIG. 8 , the shape of field F6a included in the first field image is defined by vertices P1 to P4. The shape of field F6 included in the map data is defined by vertices Q1 to Q4 that correspond to vertices P1 to P4. For example, the processing device calculates the amount of deviation between vertices P1 and Q1, i.e., the amount of deviation between the latitude and longitude of vertex P1 and the latitude and longitude of vertex Q1. The processing device can estimate the error in geographic information between the first field image and the map data from this amount of deviation. Alternatively, the processing device may calculate the amount of deviation in latitude and longitude for each of the four vertices and then calculate their average value. The processing device may estimate the error in geographic information between the first field image and the map data from the average value of these deviation amounts.

[0073] The processing device may generate a flight plan for the multicopter 10 by correcting the geographic information of the map data based on the estimated error (step S60). The flight plan is a plan for flight required to perform agricultural work, such as a flight path. For example, before the multicopter 10 begins flying, the processing device may generate a flight path by correcting the geographic information of the map data based on the estimated error. Specifically, the processing device may correct the geographic information of the map data by adding a deviation amount to each of the latitude and longitude (geographic information) of each point on the map indicated by the map data, and may generate a flight path from the map data having the corrected geographic information. In this way, the geographic information of the map data may be corrected each time the multicopter 10 flies.

[0074] Alternatively, the processing device may appropriately update the flight path of the multicopter 10 by correcting the geographic information of the map data based on the estimated error at an intermediate point during flight of the multicopter 10 along the flight path. The intermediate point during flight may be, for example, a corner of a field or the boundary between the field and its surroundings. After the update, the multicopter 10 can fly along the updated flight path. The generated or updated flight path may be stored in a storage device. Once the correction of the geographic information of the map data is complete, the multicopter 10 may descend from the second altitude A2 to the third altitude A3 and begin flying along the generated or updated flight path, as illustrated in FIG. 4 .

[0075] The flight path generated or updated in this way contains corrected geographic information based on the accurate position information acquired by the positioning device. Therefore, by flying the multicopter 10 along the generated or updated flight path, it becomes possible to appropriately spray pesticides in an area of ​​the multiple fields that is registered as a farm work target, such as field F6.

[0076] If the processing device determines that there is no discrepancy between the geographic information of the second field image and the geographic information of the map data (NO in step S40), it does not correct the geographic information of the map data (step S70).

[0077] As described above, the second remote sensing image, which has a relatively low resolution, includes at least one landmark required for matching as a subject. Therefore, by first performing matching focusing on the landmark, it becomes easier to determine whether there is a discrepancy between the geographic information of the second field image and the geographic information of the map data. Next, depending on the results of the determination using the second remote sensing image, the first remote sensing image, which has a relatively high resolution, is used to estimate the error in the geographic information between the first field image and the map data. This can improve the accuracy of the error in the geographic information estimated from the matching applied to the first field image rather than the second field image.

[0078] The sensing device may acquire a second remote sensing image by sensing the ground environment at a second altitude to include the field and its surroundings. The processing device may analyze the second remote sensing image thus acquired to monitor the conditions around the field. For example, the processing device may monitor whether a natural disaster, such as a landslide or a fire, is occurring in the surrounding environment of the field. Furthermore, the processing device may determine the outline of the field included in the ground environment from the first or second sensing image. In this manner, the first or second sensing image may be used to determine the outline of the field.

[0079] The multicopter 10 or the sensing system may further include a warning device 81, as illustrated in FIG. 4 . Examples of the warning device 81 include a buzzer that emits a warning sound or an optical device such as an LED (Light Emitting Diode) lamp. If the conditions around the monitored field differ from the conditions around the field on a map corresponding to the field, the processing device may cause the warning device to warn of a possible change in the field's surrounding environment. For example, the processing device may detect rubble or soil in the field's surrounding environment by comparing the sensing image with map data. In this case, the processing device determines that a landslide is likely occurring around the field and causes the warning device to issue a warning. This allows the user to be alerted to the change in the field's surrounding environment and to pay attention.

[0080] In an environment where the latitude and longitude included in the geographic information of map data differ from the actual latitude and longitude, an agricultural machine such as a tractor can simultaneously estimate its own position and create a map using, for example, SLAM technology, thereby estimating its position on the map. In contrast, the method for correcting the geographic information of map data according to this embodiment does not require advanced technology such as SLAM, and furthermore, does not require flight for map creation. Using remote sensing images captured by the multicopter itself, the error between the actual information and the geographic information of the map data is appropriately estimated. This type of estimation cannot be achieved by agricultural machines such as tractors. Furthermore, for example, crustal movements may cause the location of an RTK base station to shift or the terrain to deform. Even in such cases, the map data can be reflected in the flight plan by correcting the geographic information based on accurate position information acquired by the positioning device.

[0081] The method for correcting the geographic information of map data according to this embodiment can be used even when the location information acquired by the positioning device contains an error. In this case, the location information acquired by the positioning device can be corrected based on the estimated error. The geographic information of the map data can be used as is without correction.

[0082] The control device 4a in the embodiment of the present disclosure can be realized by a digital computer system programmed to execute the above-described processes.

[0083] 10 is a block diagram showing an example of the hardware configuration of the control device 4a. The control device 4a includes a processor 34, a read-only memory (ROM) 35, a random access memory (RAM) 36, a storage device 37, and a communication I / F 38. These components are interconnected via a bus 39. The bus 39 is, for example, a controller area network (CAN) bus.

[0084] The processor 34 is a device including one or more semiconductor integrated circuits (e.g., processors). The processor is also called a central processing unit (CPU) or a microprocessor. The processor sequentially executes computer programs stored in the ROM 35 to perform the above-described processing. The term "processor" is broadly interpreted as including a field programmable gate array (FPGA), a graphics processor unit (GPU), an application specific integrated circuit (ASIC), or an application specific standard product (ASSP) equipped with a CPU.

[0085] The ROM 35 is, for example, a writable memory (e.g., a PROM), a rewritable memory (e.g., a flash memory), or a read-only memory. The ROM 35 stores a program that controls the operation of the processor. The ROM 35 does not have to be a single recording medium, but may be a collection of multiple recording media. Some of the collection of multiple recording media may be removable memories.

[0086] The RAM 36 provides a working area for temporarily loading the programs stored in the ROM 35 at boot time. The RAM 36 does not have to be a single recording medium, but can be a collection of multiple recording media.

[0087] The communication I / F 38 is an interface for communication between the control device 4a and other electronic components or electronic control units (ECUs). For example, the communication I / F 38 can perform wired communication in accordance with various protocols. The communication I / F 38 may perform wireless communication in accordance with the Bluetooth (registered trademark) standard and / or the Wi-Fi (registered trademark) standard. Both standards include wireless communication standards using frequencies in the 2.4 GHz band.

[0088] The storage device 37 may be, for example, a semiconductor memory, a magnetic storage device, an optical storage device, or a combination thereof. The storage device 37 may store, for example, map data useful for the autonomous flight of the multicopter 10, flight path data, and various sensor data acquired by the multicopter 10 during flight.

[0089] As described above, the control device 4a may include, for example, a flight control device such as a flight controller and a host computer (companion computer). The companion computer may perform various processes necessary to estimate errors in the geographic information, and the results of these processes may be provided to the sensing system. Furthermore, as shown in FIG. 10 , some or all of the functions of electrical components such as the control device 4a mounted on the multicopter 10 may be implemented by one or more servers (computers) 500 or terminal devices (including portable and fixed types) 600 connected to the communication device 4c of the multicopter 10 via a communication network N. An agricultural machine 700 such as a tractor may be connected to such a communication network N, and communication may be performed between the multicopter 10 and the agricultural machine 700. Some of the data used in the processing of the control device 4a and control signals for the multicopter 10 may be provided from the agricultural machine 700 to the multicopter 10 via the communication network N.

[0090] A system providing various functions in the embodiments can also be retrofitted to a multicopter that does not have those functions. Such a system can be manufactured and sold independently of the multicopter. A computer program used in such a system can also be manufactured and sold independently of the multicopter. The computer program can be provided, for example, by being stored in a computer-readable non-transitory storage medium. The computer program can also be provided by downloading via a telecommunications line (e.g., the Internet).

[0091] This specification discloses the solutions described in the following items.

[0092] [Item 1] A sensing system used in an unmanned aerial vehicle that senses a ground environment, comprising: a positioning device that acquires position information of the unmanned aerial vehicle; a sensing device that senses the ground environment and acquires a remote sensing image; a processing device that processes the remote sensing image and generates the remote sensing image having geographic information based on the position information; and a map data acquisition means that acquires map data including geographic information on a map, wherein the processing device is configured to estimate an error in geographic information between the first remote sensing image and the map data based on the geographic information of a first remote sensing image acquired by the sensing device sensing the ground environment at a first altitude, and the geographic information of a second remote sensing image acquired by sensing the ground environment at a second altitude higher than the first altitude, and the geographic information of the map data.

[0093] [Item 2] The sensing system according to Item 1, wherein the processing device generates a flight plan for the unmanned aerial vehicle by correcting geographic information of the map data based on the error.

[0094] [Item 3] The sensing system according to item 1 or 2, wherein the sensing device acquires the first and second remote sensing images at points of the same latitude and longitude.

[0095] [Item 4] The sensing system according to any one of Items 1 to 3, wherein the processing device determines a deviation between geographic information of the second remote sensing image and geographic information of the map data by matching the second remote sensing image with the map data.

[0096] [Item 5] The processing device is configured to: identify a second landmark on the map that corresponds to a first landmark included in the second remote sensing image by matching the second remote sensing image with the map data; and determine a deviation between the geographic information of the second remote sensing image and the geographic information of the map data based on the geographic information of the first and second landmarks.

[0097] [Item 6] The sensing system described in Item 5 includes a control device that controls the flight of the unmanned aerial vehicle, and the control device causes the unmanned aerial vehicle to ascend to an altitude at which the processing device can identify the second landmark that corresponds to the first landmark.

[0098] [Item 7] The sensing system according to Item 5 or 6, wherein the terrestrial environment includes at least one farm field, and when the processing device determines that there is a discrepancy between the geographic information of the second remote sensing image and the geographic information of the map data, the processing device estimates the error by matching the first remote sensing image with the map data.

[0099] [Item 8] The sensing system according to Item 7, wherein, when the processing device determines that there is a discrepancy between the geographic information of the second remote sensing image and the geographic information of the map data, the processing device matches the first remote sensing image with the map data to identify a field on the map that corresponds to the field included in the first remote sensing image, and estimates the error from the longitude and latitude included in the geographic information of the field included in the first remote sensing image and the corresponding field on the map.

[0100] [Item 9] The sensing system described in Item 2, wherein before the unmanned aerial vehicle begins flight, the processing device generates the flight plan by correcting the geographic information of the map data based on the error.

[0101] [Item 10] The sensing system described in Item 2, wherein at a point during the unmanned aerial vehicle's flight along the flight plan, the processing device updates the flight plan of the unmanned aerial vehicle by correcting the geographic information of the map data based on the error, and after the update, the unmanned aerial vehicle flies along the updated flight plan.

[0102] [Item 11] The sensing system according to Item 1, wherein the sensing device senses the terrestrial environment including the farm field and the surrounding area of ​​the farm field at the second altitude to acquire the second remote sensing image, and the processing device analyzes the second remote sensing image to monitor the condition of the surrounding area of ​​the farm field.

[0103] [Item 12] The sensing system according to Item 11, further comprising a warning device, wherein the processing device causes the warning device to warn that a change may be occurring in the surrounding environment of the field when the condition of the surrounding area of ​​the field being monitored differs from the condition of the surrounding area of ​​the field on the map corresponding to the surrounding area of ​​the field.

[0104] [Item 13] The sensing system according to Item 1, wherein the processing device determines the contour of a field included in the ground environment from the first or second sensed image.

[0105] [Item 14] An unmanned aerial vehicle comprising: a plurality of rotors; and the sensing system according to any one of items 1 to 13.

[0106] The unmanned aerial vehicle disclosed herein can be widely used not only for aerial photography, surveying, logistics, and pesticide spraying, but also for ground work related to agricultural work, transporting harvested products and agricultural materials, and the like.

[0107] 2: Rotor (propeller), 3: Rotation drive device, 4: Airframe body, 4a: Control device, 4b: Sensor group, 4c: Communication device, 5: Airframe frame, 6: Ground station, 7a: Internal combustion engine, 7b: Fuel tank, 8: Power generation device, 9: Power buffer, 10: Multicopter, 12, 22: Rotor, 14: Motor, 16: ESC, 76: Power supply device, 200: Work machine

Claims

1. A sensing system for use in an unmanned aerial vehicle that senses a ground environment, comprising: a positioning device that acquires position information of the unmanned aerial vehicle; a sensing device that senses the terrestrial environment and acquires a remote sensing image; a processing device that processes the remote sensing image and generates the remote sensing image having geographic information based on the location information; a map data acquisition means for acquiring map data including geographic information on a map; Equipped with a processing device configured to estimate an error in geographic information between the first remote sensing image and the map data based on geographic information of a first remote sensing image acquired by the sensing device sensing the terrestrial environment at a first altitude, geographic information of a second remote sensing image acquired by sensing the terrestrial environment at a second altitude higher than the first altitude, and geographic information of the map data.

2. The sensing system of claim 1 , wherein the processing device generates a flight plan for the unmanned aerial vehicle by correcting geographic information of the map data based on the error.

3. The sensing system according to claim 1 or 2, wherein the sensing device acquires the first and second remote sensing images at points of the same latitude and longitude.

4. 3. The sensing system according to claim 1, wherein the processing device determines a deviation between geographic information of the second remote sensing image and geographic information of the map data by matching the second remote sensing image with the map data.

5. The processing device includes: by matching the second remote sensing image with the map data, identifying a second landmark on the map that corresponds to a first landmark included in the second remote sensing image; The sensing system according to claim 4 , further comprising: determining a deviation between the geographic information of the second remote sensing image and the geographic information of the map data based on the geographic information of the first and second landmarks.

6. a control device for controlling the flight of the unmanned aerial vehicle; The sensing system of claim 5 , wherein the control device causes the unmanned aerial vehicle to ascend to an altitude at which the processing device can identify the second landmark that corresponds to the first landmark.

7. the terrestrial environment comprises at least one field; 6. The sensing system according to claim 5, wherein, when the processing device determines that there is a discrepancy between the geographic information of the second remote sensing image and the geographic information of the map data, the processing device estimates the error by matching the first remote sensing image with the map data.

8. When the processing device determines that there is a discrepancy between the geographic information of the second remote sensing image and the geographic information of the map data, by matching the first remote sensing image with the map data, identifying a field on the map that corresponds to a field included in the first remote sensing image; The sensing system according to claim 7 , wherein the error is estimated from the longitude and latitude included in the geographic information of the field included in the first remote sensing image and the corresponding field on the map.

9. The sensing system of claim 2 , wherein the processing unit generates the flight plan by correcting geographic information of the map data based on the error before the unmanned aerial vehicle begins flight.

10. At an intermediate point during the flight of the unmanned aerial vehicle along the flight plan, the processing device updates the flight plan of the unmanned aerial vehicle by correcting the geographic information of the map data based on the error; The sensing system of claim 2 , wherein after updating, the unmanned aerial vehicle flies according to the updated flight plan.

11. the sensing device senses the terrestrial environment including the farm field and the surroundings of the farm field at the second altitude to acquire the second remote sensing image; The sensing system of claim 1 , wherein the processing device analyzes the second remote sensing image to monitor conditions around the field.

12. Equipped with a warning device, The sensing system of claim 11, wherein the processing device causes the warning device to warn that a change may be occurring in the surrounding environment of the field when the condition of the surrounding area of the field being monitored differs from the condition of the surrounding area of the field on the map corresponding to the surrounding area of the field.

13. The sensing system according to claim 1 , wherein the processing device determines an outline of a field included in the ground environment from the first or second sensed image.

14. A plurality of rotors; The sensing system according to claim 1 or 2; An unmanned aerial vehicle comprising: