Aircraft control device and aircraft control method

Abstract

To provide a flying object control device and a flying object control method that achieve high-precision self-position estimation with a simple configuration.SOLUTION: A flying object control device according to an embodiment comprises: a first storage unit that stores three-dimensional 3D map information consisting of point group data; a first photographing unit that generates an omnidirectional image by photographing all directions of a flying object; a first self-position estimation unit that generates first estimated position information indicating an absolute position of the flying object based on the omnidirectional image and the 3D map information; a second photographing unit that generates a moving image by photographing a predetermined direction from the flying object; a second self-position estimation unit that generates second estimated position information indicating a relative position of the flying object on the basis of a predetermined position based on the moving image; and a position monitoring unit that generates self-estimated position information indicating an estimated position of the flying object based on at least one of the first estimated position information and the second estimated position information.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] An embodiment of the present invention relates to an aircraft control device and an aircraft control method that realize autonomous flight of an aircraft, for example. [Background technology]

[0002] Currently, inspection work at plant facilities such as power plants and substations is carried out by people making rounds. However, due to the declining birthrate and aging population, the aging of inspectors and the resulting labor shortages are becoming more apparent. To solve these issues, there is a demand for unmanned inspections using robots and other tools.

[0003] It is expected that in the future, inspections using drones and other flying objects will become a reality for plant facilities that require inspections of high places or multiple floors. A method has been proposed for drone flight control that uses multiple sensors to create an environmental map to be referenced by autonomous patrols. This method requires a large amount of data processing, as many maps must be synthesized to create the environmental map. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2018-55695 Summary of the Invention [Problem to be solved by the invention]

[0005] The flying object control device and flying object control method according to the embodiment aim to provide an flying object control device and flying object control method that achieve highly accurate self-position estimation with a simple configuration. [Means for solving the problem]

[0006] The aircraft control device of the embodiment includes a first memory unit that stores three-dimensional 3D map information consisting of point cloud data, a first photographing unit that photographs the aircraft in all directions to generate an omnidirectional image, a first self-position estimation unit that generates first estimated position information indicating the absolute position of the aircraft based on the omnidirectional image and the 3D map information, a second photographing unit that generates moving images photographed in a predetermined direction from the aircraft, a second self-position estimation unit that generates second estimated position information indicating the relative position of the aircraft with respect to a predetermined position based on the moving image, and a position monitoring unit that generates self-estimated position information indicating the estimated position of the aircraft based on at least one of the first estimated position information and the second estimated position information.

[0007] An embodiment of an aircraft control method includes storing three-dimensional 3D map information consisting of point cloud data in a first storage unit that stores the data, capturing images of the aircraft in all directions to generate an omnidirectional image, generating first estimated position information indicating the absolute position of the aircraft based on the omnidirectional image and the 3D map information, generating moving images captured in a predetermined direction from the aircraft, generating second estimated position information indicating the relative position of the aircraft with respect to the predetermined position based on the moving image, and generating self-estimated position information indicating the estimated position of the aircraft based on at least one of the first estimated position information and the second estimated position information; It is characterized by: [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 2 is a conceptual diagram showing the state of an aircraft according to the first embodiment. [Figure 2] 1 is a block diagram showing the configuration of an aircraft control device according to a first embodiment. [Figure 3] FIG. 3 is a transition diagram showing an example of a motion pattern of the flying object according to the first embodiment. [Figure 4] 4 is a flowchart showing the operation of the aircraft control device according to the first embodiment. [Figure 5] FIG. 10 is a block diagram showing the configuration of an aircraft control device according to a second embodiment. [Figure 6] FIG. 10 is a transition diagram showing an example of a motion pattern of the flying object according to the second embodiment. [Figure 7] 10 is a flowchart showing the operation of an aircraft control device according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] (Configuration of the first embodiment) The configuration of an air vehicle FV according to a first embodiment will be described below with reference to Figures 1 and 2. As shown in Figure 1, the air vehicle FV of this embodiment is configured to be capable of autonomous flight control, like a drone. The air vehicle FV includes an air vehicle control device 1 that enables its autonomous flight, a first image capture unit 10 and a second image capture unit 20 as optical sensors, a motor 70, and a propeller P driven by the motor 70.

[0010] The flying vehicle FV of the embodiment is equipped with multiple image capturing units as optical sensors to enable flight in areas where plant facilities exist, inside plant facility buildings, etc. The flying vehicle control device 1 estimates the self-position of the flying vehicle FV based on image information acquired by the first image capturing unit 10 and the second image capturing unit 20, and realizes autonomous three-dimensional flight.

[0011] The first image capturing unit 10 captures images in all directions around the flying object FV. The first image capturing unit 10 is a camera capable of acquiring image information in all directions around the flying object FV, such as a 360-degree camera. The first image capturing unit 10 may be realized by a plurality of cameras capable of capturing images in all directions. The second image capturing unit 20 is a camera capable of acquiring stereoscopic image information in a specific direction from the flying object FV, such as a stereo camera.

[0012] The motor 70 is driven by a control signal from the aircraft control device 1. The motor 70 is connected to a propeller P. The aircraft FV has at least two or more combinations of the motor 70 and the propeller P.

[0013] 2, the aircraft control device 1 of the embodiment is connected to a first image capturing unit 10, a second image capturing unit 20, and a motor 70. The aircraft control device 1 has a first self-position estimating unit 15, a second self-position estimating unit 25, a position monitoring unit 30, a memory unit 40, a flight control unit 50, a mechanism control unit 60, and a transceiver unit 80.

[0014] The aircraft control device 1 of the embodiment includes a storage unit 40 that stores three-dimensional 3D map information. The 3D map information uses point cloud data to three-dimensionally represent obstacles and terrain in the area in which the aircraft FV will fly. By using 3D map information made up of point cloud data, it is possible to represent the surrounding terrain and structures in three dimensions while keeping the data size small.

[0015] The 3D map information is generated, for example, by actually flying an aircraft FV, extracting feature points such as terrain and structures from omnidirectional image information acquired by the first image capture unit 10, and converting the extracted feature points into point cloud data. The point cloud data in the 3D map information associates three-dimensional position information with each point, and can be used as a three-dimensional map. The 3D map information may be generated by providing the first image capture unit 10 with a point cloud data conversion function, and storing converted point cloud data in the storage unit 40, or by storing point cloud data acquired by another aircraft in the storage unit 40.

[0016] The generation of 3D map information is not limited to that based on omnidirectional image information actually captured by the first photographing unit 10 or the like. Point cloud data may also be generated based on topographical maps, precision drawing data, etc. The omnidirectional image information for generating the point cloud data may be acquired by a 360-degree camera in the first photographing unit 10, or may be generated by combining image information acquired by multiple cameras.

[0017] The storage unit 40 may store route information such as a flight path when the flying object FV flies autonomously. The route information includes parameters necessary for autonomous flight, such as a flight path, flight time, and flight time.

[0018] The first self-position estimation unit 15 is a calculation block that estimates the position of the flying object FV based on the 3D map information stored in the memory unit 40 and omnidirectional image information acquired by the first photographing unit 10 while the flying object FV is in operation. The first self-position estimation unit 15 generates omnidirectional information consisting of point cloud data based on the image information acquired by the first photographing unit 10. The omnidirectional information is image information acquired in all directions from the flying object FV.

[0019] The first self-position estimation unit 15 compares the generated omnidirectional information with the 3D map information stored in the storage unit 40 to calculate the position of the flying object FV in the area indicated by the 3D map information (first estimated position). The first estimated position is position information calculated by comparing the 3D map information generated in advance with the omnidirectional information based on image information actually acquired by the first imaging unit 10 of the flying object FV. Note that the generation of the omnidirectional information may be performed by the first imaging unit 10 instead of the first self-position estimation unit 15. In this case, the first self-position estimation unit 15 compares the omnidirectional information received from the first imaging unit 10 with the 3D map information stored in the storage unit 40.

[0020] The second self-position estimation unit 25 is a calculation block that estimates the position of the flying object FV based on image information acquired by the second image capture unit 20 while the flying object FV is in operation. The second self-position estimation unit 25 generates a movement trajectory of the flying object FV based on the image information acquired by the second image capture unit 20. The movement trajectory is relative position information from the starting position of the flying object FV. That is, before the flying object FV starts to operate, position information of the starting point or the like is provided to the second self-position estimation unit 25 as a reference point, and the position of the flying object FV (second estimated position) is calculated using the relative position of the flying object FV calculated based on the image information acquired by the second image capture unit 20. The second estimated position is position information calculated based on the position information of the starting point and the relative position based on the image information actually acquired by the second image capture unit 20 of the flying object FV. The calculation of the second estimated position by the second self-position estimation unit 25 can be processed faster than the calculation of the first estimated position by the first self-position estimation unit 15.

[0021] The position monitoring unit 30 is a calculation block that monitors the first estimated position generated by the first self-position estimation unit 15 and the second estimated position generated by the second self-position estimation unit 25, and determines the position of the flying object FV (its own estimated position) based on at least one of the first estimated position and the second estimated position. A possible method for the position monitoring unit 30 to determine its own estimated position is to calculate the likelihood of each of the first estimated position and the second estimated position, and determine the estimated position with the greatest likelihood as its own estimated position. Alternatively, the position monitoring unit 30 may determine whether the first estimated position and the second estimated position have been correctly output from the first self-position estimation unit 15 and the second self-position estimation unit 25, respectively (e.g., whether output data exists), and select the correctly output estimated position as its own estimated position.

[0022] Because the first and second estimated positions are both calculated from the results of processing image information acquired by the first and second image capture units 10 and 20, there is a possibility that the self-estimated position may be lost due to disturbances while the aircraft FV is flying. Therefore, the aircraft control device 1 of the embodiment generates a final self-estimated position by combining the first estimated position derived from image information acquired by the first image capture unit 10 and the second estimated position derived from image information acquired by the second image capture unit 20, thereby enabling reliable self-position estimation. In other words, the first estimated position is an absolute position based on 3D map information, and the second estimated position is a relative position from a reference point. The aircraft control device 1 of the embodiment generates estimated positions using multiple different methods, thereby reducing accidents caused by disturbances.

[0023] The flight control unit 50 is a calculation block that performs autonomous control of the air vehicle FV based on the self-estimated position determined by the position monitoring unit 30. The flight control unit 50 reads route information from the memory unit 40 and performs flight control according to the route information using the self-estimated position. Flight control involves control of the flight direction and altitude of the air vehicle FV, and is achieved, for example, by controlling the rotation speed of the motor 70. The mechanism control unit 60 is a mechanical component that controls the mechanical parts that realize the flight of the air vehicle FV, such as the motor 70. When the air vehicle FV flies under the control of only the propeller P, such as a so-called drone, the mechanism control unit 60 generates and supplies a drive voltage to the motor 70 under control from the flight control unit 50. If the air vehicle FV has wings, the mechanism control unit 60 may include mechanical components such as an actuator or a servo motor.

[0024] The transceiver 80 is a wireless interface that communicates with a slave unit (not shown). When the flying body control device 1 of the embodiment performs manual flight control rather than autonomous flight, the transceiver 80 receives a control signal and sends a command directly to the mechanism control unit 60. The mechanism control unit 60 controls the drive of the motor 70 and the like based on the control signal sent from the transceiver 80. In other words, the flying body FV of the embodiment is configured to be capable of not only autonomous flight control but also wireless manual flight control.

[0025] (Operation of the flying vehicle FV of the first embodiment) The operation of the flying object FV according to the first embodiment will be described with reference to Figures 3 and 4. As shown in Figure 3, the flying object FV has four states: "standby," "takeoff," "automatic patrol" or "remote control," and "landing." In the standby state, the user records 3D map information, route information, starting point information, etc. in the memory unit 40. The flight control unit 50 performs takeoff operations based on the route information stored in the memory unit 40.

[0026] When the takeoff operation is normal and the autonomous flight mode is in (normal: a), the first image capture unit 10 acquires omnidirectional image information of the flying object FV, and the first self-position estimation unit 15 generates a first estimated position based on the omnidirectional image information. Meanwhile, the second image capture unit 20 generates stereoscopic image information of a specific direction (for example, forward) of the flying object FV, and the second self-position estimation unit 25 generates a second estimated position based on the stereoscopic image information. The position monitoring unit 30 generates an estimated self-position based on the first estimated position and the second estimated position.

[0027] 4 shows an example of a position estimation operation by the position monitoring unit 30. If the position monitoring unit 30 cannot receive the first estimated position (Yes in S100) and cannot receive the second estimated position either (Yes in S110), it cannot generate its own estimated position (S120). Therefore, the flight control unit 50 controls the flying object FV by manual remote control rather than automatic patrol (normal: t in FIG. 3). If the transceiver unit 80 cannot receive a control signal, the flight control unit 50 interrupts remote control and ends processing.

[0028] If the position monitoring unit 30 cannot receive the first estimated position (Yes in S100) but can receive the second estimated position (No in S110), the position monitoring unit 30 generates its own estimated position based on the second estimated position (S130). Since the flight control unit 50 is capable of automatic patrol based on the self-estimated position, it continues autonomous flight based on the route information and the self-estimated position.

[0029] If the position monitoring unit 30 receives the first estimated position (No in S100) but cannot receive the second estimated position (Yes in S140), the position monitoring unit 30 generates an estimated position based on the first estimated position (S150). The flight control unit 50 continues autonomous flight based on the route information and the estimated position, since it is capable of automatic patrol based on the estimated position.

[0030] If the position monitoring unit 30 receives the first estimated position (No in S100) and also receives the second estimated position (No in S140), the position monitoring unit 30 generates an estimated position based on the first estimated position and the second estimated position (S160). The flight control unit 50 is capable of automatic patrol based on the estimated position, and therefore continues autonomous flight based on the route information and the estimated position.

[0031] The flight control unit 50 flies around predetermined waypoints based on the route information and then lands at the landing site.

[0032] According to the aircraft control device of the embodiment, the aircraft position is estimated by combining a plurality of means, for example, a means for acquiring the absolute position of the aircraft and a means for acquiring the relative position of the aircraft, so that the possibility of the aircraft position becoming unstable can be significantly reduced. Furthermore, according to the aircraft control device of the embodiment, the aircraft position is estimated based on optical image information, so that autonomous control of the aircraft can be realized even inside a building where radio waves such as GNSS cannot be used.

[0033] (Configuration of the second embodiment) The configuration of an aircraft FVa according to the second embodiment will be described below with reference to Figures 1 and 5. As shown in Figure 5, the aircraft control device 2 in the aircraft FVa of this embodiment is equipped with a distance sensor 90 and a third self-position estimation unit 95 in addition to the configuration of the aircraft FV according to the first embodiment. In the following description, common elements are denoted by common reference numerals, and duplicated explanations will be omitted.

[0034] The distance sensor 90 is a sensor, such as a two-dimensional laser sensor or ultrasonic sensor, that can measure the distance to surrounding obstacles and acquire distance information in a planar area. While the first imaging unit 10 and the second imaging unit 20 acquire optical image information, the distance sensor 90 acquires distance information. Although the distance sensor 90 can only acquire information in a planar direction (the xy plane in FIG. 1 ), it can achieve a higher processing speed than the first imaging unit 10 and the second imaging unit 20.

[0035] In this embodiment, the memory unit 40 stores 2D map information in addition to the 3D map information and route information. The 2D map information is a two-dimensional representation of obstacles and terrain using point cloud data for the area in which the flying vehicle FV will fly. The 2D map information is generated by actually flying the flying vehicle FV and converting distance information about the entire surrounding area in the planar direction acquired by the distance sensor 90 into point cloud data.

[0036] The third self-position estimation unit 95 is a calculation block that estimates the planar position of the aircraft FV based on the 2D map information stored in the memory unit 40 and the planar distance information acquired by the distance sensor 90 while the aircraft FV is in operation. The third self-position estimation unit 95 compares the planar distance information of the entire periphery acquired by the distance sensor 90 with the 2D map information stored in the memory unit 40, thereby calculating the planar position (third estimated position) of the aircraft FV in the area indicated by the 2D map information. The third estimated position is position information calculated by comparing the 2D map information generated in advance with the planar distance information of the entire periphery based on the distance information actually acquired by the distance sensor 90 of the aircraft FV.

[0037] The third estimated position calculated based on the distance information acquired by the distance sensor 90 is information in a planar direction (x and y directions), and therefore remains merely interpolated information of the first estimated position and the second estimated position. Therefore, the third self-position estimation unit 95 may calculate the third estimated position by further using z-direction parameters of the first estimated position and the second estimated position. This makes it possible to use the third estimated position together with the first estimated position and the second estimated position as estimated information of the three-dimensional self-position.

[0038] (Operation of the flying object FVa of the second embodiment) The operation of the flying object FVa according to the second embodiment will be described with reference to Figures 6 and 7. As shown in Figure 6, the flying object FVa has four states: "standby," "takeoff," "automatic patrol," "automatic patrol (LiDAR)," "remote control," and "landing." In the standby state, the user records 3D map information, route information, starting point information, etc. in the storage unit 40. The flight control unit 50 performs takeoff operations based on the route information stored in the storage unit 40.

[0039] When the takeoff operation is normal and the autonomous flight mode is in (normal: a), the first image capture unit 10 acquires image information of the flying object FVa in all directions, and the first self-position estimation unit 15 generates a first estimated position. Meanwhile, the second image capture unit 20 generates stereoscopic image information of the flying object FVa in a specific direction (for example, forward), and the second self-position estimation unit 25 generates a second estimated position. The position monitoring unit 30 generates an estimated position based on the first estimated position and the second estimated position. Furthermore, the distance sensor 90 acquires distance information of the entire surroundings of the flying object FVa in the planar direction, and the third self-position estimation unit 95 generates a third estimated position.

[0040] 7 shows an example of a position estimation operation by the position monitoring unit 30. If the position monitoring unit 30 cannot receive the first estimated position (Yes in S200) and cannot receive the second estimated position either (Yes in S210), the position monitoring unit 30 generates an estimated position based on the third estimated position (S220). Since the flight control unit 50 is capable of automatic patrol based on the estimated position, it continues autonomous flight based on the route information and the estimated position.

[0041] If the position monitoring unit 30 cannot receive the first estimated position (Yes in S200) but can receive the second estimated position (No in S210), the position monitoring unit 30 generates its own estimated position based on the second estimated position (S230). Since the flight control unit 50 is capable of automatic patrol based on the own estimated position, it continues autonomous flight based on the route information and the own estimated position.

[0042] If the position monitoring unit 30 receives the first estimated position (No in S200) but cannot receive the second estimated position (Yes in S240), the position monitoring unit 30 generates an estimated position based on the first estimated position (S250). The flight control unit 50 continues autonomous flight based on the route information and the estimated position, since it is capable of automatic patrol based on the estimated position.

[0043] If the position monitoring unit 30 receives the first estimated position (No in S200) and also receives the second estimated position (No in S240), the position monitoring unit 30 generates an estimated position based on the first estimated position and the second estimated position (S260). The flight control unit 50 is capable of automatic patrol based on the estimated position, and therefore continues autonomous flight based on the route information and the estimated position.

[0044] The flight control unit 50 patrols predetermined waypoints based on the route information, and then lands at the landing site. According to the aircraft control device 2 of this embodiment, automatic patrol is possible through a sequence of "automatic patrol" based on the first estimated position and / or the second estimated position, and "automatic patrol (LiDAR)" based on the third estimated position, depending on whether the estimated positions of the first and second photographing units 10 and 20 are calculated successfully. This reduces the processing load during normal operation when the first and second estimated positions are used, and enables autonomous flight control that is less likely to become uncontrollable even in situations where the first and second estimated positions of the optical system cannot be obtained.

[0045] In this embodiment, if a fault occurs in the position estimation using the first estimated position and the second estimated position, autonomous control using the third estimated position is possible until the fault is resolved. Note that, since the distance sensor 90 can measure the distance to surrounding structures, if a fault occurs in the position estimation using the first estimated position and the second estimated position, the flight control unit 50 may fly at a position that maintains a certain distance from surrounding structures and obstacles without using the third estimated position until the fault is resolved.

[0046] While several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments may be embodied in various other forms, and various omissions, substitutions, and modifications may be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as set forth in the claims. [Explanation of symbols]

[0047] FV...flying vehicle, 1...Flying vehicle control device, 10...first imaging unit, 15...first self-position estimation unit, 20... second imaging unit, 25... second self-position estimation unit, 30...Position monitoring section, 40...Storage section, 50...flight control unit, 60...mechanism control unit, 70...Motor, 80...Transmitter / receiver unit.

Claims

1. An aircraft control device for controlling an aircraft, a first storage unit that stores three-dimensional 3D map information made up of point cloud data; a first image capturing unit that captures images of the aircraft in all directions to generate an omnidirectional image; a first self-position estimation unit that generates first estimated position information indicating an absolute position of the aircraft based on the omnidirectional image and the 3D map information; a second image capturing unit configured to capture a moving image in a predetermined direction from the aircraft; a second self-position estimation unit that generates second estimated position information indicating a relative position of the flying object with respect to a predetermined position based on the moving image; a position monitoring unit that generates self-estimated position information indicating an estimated position of the aircraft based on at least one of the first estimated position information and the second estimated position information; Equipped with The position monitoring unit If the first estimated position information cannot be received but the second estimated position information can be received, the self-estimated position information is generated based on the second estimated position information; generating the self-estimated location information based on the first estimated location information when the first estimated location information can be received but the second estimated location information cannot be received; generating the self-estimated position information based on the first estimated position information and the second estimated position information when the first estimated position information and the second estimated position information are received; Aircraft control device.

2. The aircraft control device according to claim 1, characterized in that the position monitoring unit calculates the likelihood of each of the first estimated position information and the second estimated position information, and generates the self-estimated position information based on whichever has the higher likelihood.

3. a second storage unit that stores two-dimensional 2D map information made up of point cloud data; a distance sensor that captures images of the aircraft in all directions and generates distance information to surrounding obstacles; a third self-position estimator that generates third estimated position information indicating a position in a planar direction around the aircraft based on the distance information and the 2D map information, the location monitoring unit generates the self-estimated location information based on the third estimated location information when the first estimated location information and the second estimated location information cannot be received.

2. The aircraft control device according to claim 1,

4. A method for controlling an aircraft, comprising: storing the point cloud data in a first storage unit that stores three-dimensional 3D map information; capturing an omnidirectional image of the aircraft to generate an omnidirectional image; generating first estimated position information indicating an absolute position of the aircraft based on the omnidirectional image and the 3D map information; generating a moving image captured in a predetermined direction from the aircraft; generating second estimated position information indicating a relative position of the aircraft with respect to a predetermined position based on the moving image; If the first estimated position information is not received but the second estimated position information is received, generating self-estimated position information indicating an estimated position of the aircraft based on the second estimated position information; generating the self-estimated location information based on the first estimated location information when the first estimated location information is received but the second estimated location information is not received; An aircraft control method characterized by generating the self-estimated position information based on the first estimated position information and the second estimated position information when the first estimated position information and the second estimated position information are received.

Images