Method for controlling an aircraft
A flying object controls flight routes and images structures to provide an efficient bridge inspection method, overcoming the limitations of traditional methods by eliminating the need for additional infrastructure and accessing hard-to-reach areas.
Patent Information
- Application Number
- JP2019109965
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2019-06-12
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2039-06-12
AI Technical Summary
Existing bridge inspection methods require additional structures like rails, which are time-consuming and costly, and struggle with inspecting areas without roads, such as rivers or ravines.
A method involving a flying object that moves to a structure, sets a flight route based on structure information, and images the structure, allowing for efficient inspection without physical infrastructure.
Enables efficient bridge inspection by automating visual work and facilitating access to difficult-to-reach areas, improving inspection efficiency and reducing costs.
Smart Images

Figure 0007591859000001 
Figure 0007591859000002 
Figure 0007591859000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a method for controlling an aircraft. [Background technology]
[0002] Previously, maintenance and inspection of railway bridges was carried out by erecting scaffolding on the bridge and having workers walk across it and visually inspect it, which took time and also had an impact on traffic on the line.
[0003] In particular, visual inspection of the underside of a bridge is considered difficult due to the presence of roads and rivers, and systems to assist in such inspections include a system in which rails are installed under the bridge and a robotic camera runs along the rails to photograph the underside of the bridge (Patent Document 1), and a system in which an inspection vehicle with multiple cameras installed on its roof runs along the road under the bridge, photographs the underside of the bridge, and maps the photographs on a drawing (Patent Document 2). [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2016-079615 A [Patent Document 2] JP 2017-106170 A Summary of the Invention [Problem to be solved by the invention]
[0005] However, while the systems disclosed in the patent documents can provide support for bridge inspection, the system disclosed in Patent Document 1 requires the installation of additional structures such as rails, which is time-consuming and costly. Also, the system disclosed in Patent Document 2 makes inspection difficult when there is no road under the bridge, for example, when the terrain is a river or a canyon.
[0006] Therefore, an object of the present invention is to solve such problems and provide a more efficient bridge inspection method. [Means for solving the problem]
[0007] One embodiment of the present invention is a method for controlling an aircraft, comprising the steps of controlling the aircraft to move to a structure, referencing information related to the structure and setting a flight route, and capturing an image of a portion of the structure. Effect of the Invention
[0008] According to the present invention, a more efficient bridge inspection method can be provided. [Brief description of the drawings]
[0009] [Figure 1] FIG. 1 is a schematic diagram of an inspection method using an aircraft 10 according to a first embodiment. [Diagram 2] FIG. 2 is a functional block diagram of the aircraft 10 according to the first embodiment. [Diagram 3] FIG. 3 is a functional block diagram showing control related to inspection of an aircraft according to the first embodiment. [Figure 4] FIG. 4 is a flow chart showing the control of the aircraft system according to the first embodiment. [Diagram 5] FIG. 5 shows an example in which mapping is performed on drawing data of a bridge in the first embodiment. [Figure 6] FIG. 6 is a flow chart showing control related to inspection of an aircraft according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] The contents of the embodiments of the present invention will be described below. The flying object according to the embodiment of the present invention has the following configuration. [Item 1] A method for controlling an aircraft, comprising: Controlling the aircraft to move to the structure; A step of setting a flight route by referring to information related to the structure; imaging a portion of the structure; A method for controlling an aircraft, comprising: [Item 2] A method for controlling an aircraft according to claim 1, The method for controlling an aircraft further includes a step of transmitting the captured image. [Item 3] A method for controlling an aircraft according to claim 1, The method for controlling an aircraft further includes a step of checking for the presence or absence of damage based on the captured image. [Item 3] A method for controlling an aircraft according to claim 1, A method for controlling an aircraft, comprising the steps of: referring to drawing information of the structure; and mapping the captured image onto the drawing information. [Item 4] A method for controlling an aircraft according to claim 3, comprising: A method for controlling an aircraft, characterized in that, when damage is confirmed in the confirmation step, drawing information of the structure is referenced and the location of the damage is mapped onto the drawing information.
[0011] (First embodiment) Hereinafter, an aircraft according to a first embodiment of the present invention will be described with reference to the drawings. Fig. 1 is a schematic view of a bridge inspection method using an aircraft 10 according to the first embodiment.
[0012] As shown in Fig. 1, the aircraft 10 can fly around a structure (e.g., a bridge 1) and capture images of each part of the bridge 1 with a camera. Since the aircraft 10 has no movement restrictions, it can fly over any part of the bridge, including the upper and lower surfaces, and can fly as far into the bridge as possible given the size of the aircraft. In addition, although not shown, the captured images can be transmitted to an external device such as a management terminal via a network.
[0013] FIG. 2 is a functional block diagram of the flying object 10 according to the first embodiment. First, the flight controller 21 can have one or more processors such as a programmable processor (for example, a central processing unit (CPU)). The camera 5 is mounted on the aircraft via a gimbal, and can rotate, for example, in the vertical direction relative to the aircraft by the gimbal. It is preferable that the camera can rotate in three axial directions (pitch angle, roll angle, yaw angle) relative to the aircraft. In addition, when it is difficult to capture an image in a certain direction as viewed from the aircraft 10, such as above or below the aircraft 10, due to the relationship between the camera and the aircraft, for example, multiple cameras can be provided on the upper and lower surfaces of the aircraft 10. In addition, by using a stereo camera as the camera, the depth and distance of the captured image can be measured in subsequent image processing.
[0014] Flight controller 21 also has and has access to memory 22. Memory 22 stores logic, code, and / or program instructions that are executable by the flight controller to perform one or more steps.
[0015] The memory 22 may include, for example, a separable medium such as an SD card or a random access memory (RAM) or an external storage device. Data acquired from the camera or sensors may be directly transmitted to and stored in the memory. For example, still image and video data captured by a camera or the like is recorded in an internal memory or an external memory. The camera is installed on the aircraft via a gimbal.
[0016] The flight controller 21 includes a control module configured to control the state of the air vehicle. For example, the control module may be configured to control six degrees of freedom (translational motion x, y, and z, and rotational motion θ x , θ y and θ zThe control module controls the propulsion mechanism (motor, etc.) of the aircraft via the ESC to adjust the spatial arrangement, speed, and / or acceleration of the aircraft having the ESC. The motor rotates the propeller to generate lift for the aircraft. The control module can control one or more of the states of the onboard parts and sensors.
[0017] The flight controller 21 can communicate with a transceiver 24 configured to transmit and / or receive data from one or more external devices (e.g., a transceiver 23, a terminal, a display device, or other remote control). The transceiver 23 can use any suitable communication means, such as wired or wireless communication.
[0018] For example, the transceiver 24 may utilize one or more of a local area network (LAN), a wide area network (WAN), infrared, radio, WiFi, a point-to-point (P2P) network, a telecommunications network, cloud communications, and the like.
[0019] The transceiver unit 24 can transmit and / or receive one or more of the following: data acquired by sensors, processing results generated by a flight controller, predetermined control data, user commands from a terminal or a remote controller, etc.
[0020] The sensors 25 according to this embodiment may include an inertial sensor (acceleration sensor, gyro sensor), a GPS sensor, a proximity sensor (e.g., LIDAR), or a vision / image sensor (e.g., camera).
[0021] Furthermore, the inspection unit 31 stores information required for inspection, such as image data captured by the camera 5, in the memory 32, and performs the bridge inspection. As another example, all or part of the functions of the inspection unit 31 may be provided in a management terminal or the like outside the aircraft (not shown).
[0022] Next, a method for controlling an aircraft in this embodiment will be described with reference to Fig. 3 and Fig. 4. Fig. 3 is a functional block diagram of aircraft control related to bridge inspection by an aircraft in this embodiment. Fig. 4 is a flow chart of the same control of an aircraft system including an aircraft and a management terminal in this embodiment.
[0023] In this embodiment, the control method for an aircraft to inspect a bridge can be realized by executing a control program in the CPU in the flight controller 21 and / or the CPU in the inspection unit 31. As another example, all or part of the functions of the flight controller 21 and / or the inspection unit 31 can be provided in an external device 28 (e.g., a server or a management terminal), and the aircraft 10 and the external device 28 can cooperate to execute necessary processing. In this embodiment, the aircraft 10 cooperates with an external device 28 such as a management terminal to execute bridge inspection.
[0024] First, in FIG. 3, the bridge information acquisition unit 51 refers to the position information of the bridge stored in the bridge information storage unit 61 of the memory 22, for example, and confirms the position of the object of the survey. For the position information confirmed by referring to the bridge information, the current position information acquisition unit 52 acquires current position information from the GPS sensor 54, the acceleration sensor 55, etc., and the flight control unit 53 controls the motor 26 to move the flying object 10 to the inspection position that is the destination (S401). The acquired current position information is stored in the current position information storage unit 62 in the memory 22. Note that this step can be replaced by manually operating the flying object, and in cases where the GPS does not work, it is also possible to replace it by controlling the flying object to fly around the bridge by recognizing the bridge using image recognition technology without relying on the position information. Also, in this step, the flight route can be set for each inspection item by referring to information on the bridge structure and each part to be inspected, which is included in the bridge information or the drawing information 83 stored in the memory 32 of the inspection unit 31. For example, image recognition technology can be used to recognize the underside of a bridge and control the drone to fly along a specified route.
[0025] Next, when the flying object 10 is positioned at a portion of the bridge to be inspected, the bridge inspection unit 71 of the inspection unit 31 can, for example, refer to the drawing information 83 to check whether the target railroad track has already been inspected. In this example, an example is assumed in which the inspected positions are mapped on the drawing, but other formats are acceptable as long as they can be understood as position information. In addition, the flying object 10 can inquire about this information to an external device 28 such as a management terminal via the transmitting / receiving unit 24 and the network. In another example, the flying object 10 can understand the inspected and uninspected positions in advance, set the routing for the uninspected positions in advance, and fly. In yet another example, the flying object 10 can turn off the power of the camera at the inspected positions. This can save battery resources.
[0026] Next, the imaging control unit 74 of the inspection unit 31 controls the camera 5 to capture an image of the target bridge portion (S402). When capturing an image, an area camera such as a digital camera can be used to capture an image instead of visual inspection. Also, when capturing an image at night, an infrared camera can be used or an image captured using LED lighting can be captured. As described above, by using a stereo camera, the depth and distance of the captured image can be recognized in a later image processing step from multiple parallax images. The captured image can be stored in the memory 32.
[0027] Next, the notification generating unit 72 of the inspection unit 31 transmits a notification to the external device 28 such as a management terminal via the transmitting / receiving unit 24 (S403). The notification contents may include location information, bridge part information, images, etc. The notification process may be performed periodically by accumulating images, but is preferably performed simultaneously with imaging.
[0028] The external device 28 such as a management terminal receives the image transmitted from the flying object 10 (S404). Next, the external device 28 such as a management terminal inspects the bridge based on the received image (S405). For the inspection, for example, an image processed by an image processing unit (not shown) of the management terminal using a known method such as binarization of the image or detection of the feature amount of edges and lines can be used. Alternatively, the presence or absence of damage can be confirmed by comparing the image-processed inspection image of the bridge part with a normal image captured under normal conditions and image-processed. In addition to the presence or absence of damage, the size of the damage can be measured using a known method such as triangulation based on a parallax image obtained by a stereo camera. In addition, damage can be recognized by machine learning images of normal conditions of each part of the bridge to be inspected, extracting damage discrimination rules from the feature amounts of edges and corners as learning data, and comparing the feature amounts of the image captured by the unmanned aerial vehicle 10 with the learning data. In addition, if damage is determined to be present, a score can be calculated based on the size and feature amount of the damage described above, and the score can be used to prioritize the determination of whether or not repairs are required later.
[0029] Next, the external device 28 such as a management terminal maps the images and damaged areas, etc., on the drawing data of each part of the bridge based on the images received in S404 and the inspection results obtained in S405. The images are corrected to fit the format of the specified drawing data, and the damaged areas can be displayed with icons representing the damage or with special color schemes so that the inspector can easily recognize them later. Also, depending on the degree of damage (size, score obtained from feature amount), the size and color of the markers indicating the damaged areas can be changed. Figure 5 shows an example of mapping on the drawing data. A plurality of captured images are arranged, and an icon indicating the damaged area is shown on each image. The size of the icon varies depending on the size of the damage, and when one icon is selected, details of the damage and an enlarged image are displayed. Also, the parts that could not be photographed because the aircraft was unable to fly can be displayed so that they can be identified on the drawing data. This allows the inspector to visually inspect the relevant parts.
[0030] In this way, by using an aircraft to inspect bridges, not only can the conventional visual inspection work be automated, but also inspection of areas that were difficult to inspect using known inspection devices can be easily performed, providing an efficient bridge inspection method.
[0031] (Second embodiment) 5 shows a flowchart of the control of the aircraft related to the bridge inspection of the aircraft according to the second embodiment. The feature of the aircraft in this embodiment is that the inspection that was performed by the management terminal in the first embodiment is performed by the aircraft itself.
[0032] The structure and functional blocks of the aircraft 20 of this embodiment can be the same as those of the aircraft 10 disclosed in the first embodiment shown in FIG. 2, except where specifically mentioned.
[0033] The step (S601) of flying over a structure (e.g., a bridge) in Fig. 6 and the step (S602) of capturing an image of a predetermined portion of the bridge to be inspected are the same as those in the method of the first embodiment shown in Fig. 4, and therefore a description thereof will be omitted. Also, the control block diagram of the aircraft corresponding to the control method of this embodiment can be the same as that shown in Fig. 3, except for matters that are specifically mentioned, and therefore a description thereof will be omitted.
[0034] In FIG. 6, the bridge inspection unit 71 of the inspection unit 31 of the aircraft inspects the bridge based on the captured image (S603). For the inspection, for example, an image processed by the image processing unit 73 using a known method such as image binarization processing, detection of edge and corner feature amounts, etc. can be used. Alternatively, the presence or absence of an abnormality can be confirmed by comparing an image-processed inspection image 81 of a predetermined part of the bridge with a normal image 82 captured under normal circumstances and image-processed. Alternatively, the presence or absence of damage can be confirmed by comparing an image-processed inspection image of the bridge part with a normal image captured under normal circumstances and image-processed. In addition to the presence or absence of damage, it is also possible to measure the size of the damage using a known method such as triangulation based on a parallax image obtained by a stereo camera. Also, machine learning is performed on normal images of each part of the bridge to be inspected, and damage discrimination rules are extracted as learning data from the feature amounts of edges and corners, and damage can be recognized by comparing the feature amounts of the image captured by the unmanned aerial vehicle 10 with the learning data. In addition, if damage is determined, a score can be calculated based on the size and characteristics of the damage described above, and this can be used to prioritize the damage and determine whether or not it needs to be repaired.
[0035] Next, the bridge inspection unit 71 checks whether there is any abnormality in the track to be inspected as a result of the inspection (S604). Based on the image captured in S602 and the inspection result obtained in S603, the image and damaged parts are mapped on the drawing data of each part of the bridge. The image is corrected to fit the format of the specified drawing data, and the damaged parts can be displayed with icons representing the damage or with special color schemes so that the inspector can easily recognize them later. In addition, the size and color of markers indicating the damaged parts can be changed depending on the degree of damage (size, score obtained from feature amount). In addition, parts that could not be captured because the aircraft was unable to fly can be displayed so that they can be identified on the drawing data. This allows the inspector to visually inspect the parts.
[0036] In this embodiment, the notification generating unit 72 can transmit the drawing data generated in S604 to an external device 28 such as a management terminal via the network through the transmitting / receiving unit 24 (S605). In addition, together with the drawing data or separately from the drawing data, the notification generating unit 72 can determine the presence or absence of damage, and if damage is found, can transmit to the external device 28 a notification indicating that damage has been found, location information of the bridge where the damage was found, location information of the damage on the bridge, the extent of the damage, and information such as images and drawing data.
[0037] In this manner, in this embodiment, the aircraft can be responsible for all inspections, making it possible to further improve work efficiency.
[0038] The aircraft of the present invention is expected to be used as an industrial aircraft for the investigation, measurement, and observation of structures such as bridges, mainly in the railway business. The aircraft of the present invention can also be used in the aircraft-related industry, such as multicopters and drones, and furthermore, the present invention can contribute to improving the safety of these aircraft and the flight of aircraft.
[0039] The above-described embodiment is merely an example for facilitating understanding of the present invention, and is not intended to limit the present invention. The present invention can be modified or improved without departing from the spirit of the present invention, and it goes without saying that the present invention includes equivalents thereof. [Explanation of symbols]
[0040] 1 Bridges 5. Camera 10 Flying Objects 21 Flight Controller 22 Memory 23 Sensors 24 Transmitter / receiver 25 Transmitter / Receiver 26 Motor 28 External device 31 Inspection Department 32 Memory 51 Bridge Information Acquisition Department 52 Current location information acquisition unit 53 Flight Control Unit 54 GPS sensor 55 Acceleration Sensor 61 Bridge information storage unit 62 Current location information storage section 71 Bridge Inspection Department 72 Notification generator 73 Image Processing Section 74 Imaging control section 81 Inspection images 82 Normal Images 83 Drawing Information
Claims
1. A method for controlling an aircraft, comprising: Controlling the aircraft to move to the structure; obtaining drawing information including a drawing onto which the surveyed locations are mapped; A step of setting a route for an unsurveyed location by referring to the map information; imaging a portion of the structure; a step of checking whether or not there is damage based on the captured image, and when damage is confirmed, referring to drawing information of the structure and mapping the position of the damage onto the drawing information; A method for controlling an aircraft, comprising:
2. 2. A method for controlling an aircraft according to claim 1, comprising: The method for controlling an aircraft further includes a step of transmitting the captured image.
3. 2. A method for controlling an aircraft according to claim 1, comprising: A method for controlling an aircraft, comprising the steps of: referring to drawing information of the structure; and mapping the captured image onto the drawing information.
Citation Information
Patent Citations
Bridge inspection robot system
JP2016079615A
Controller, control method and computer program
JP2016082441A
Bridge inspection support system and bridge inspection support method
JP2017106170A
Pilotless flight device control system, pilotless flight device controlling method, and image projector
JP2017174159A
Indoor space inspection method
JP2018133010A
