Processing system, aircraft, processing method and program

Abstract

To more accurately control movement of a flight vehicle.SOLUTION: A processing system 4 generates posture data on posture of a flight vehicle 1.The flight vehicle 1 comprises a camera 2 and a sensor 3. The camera 2 generates image data by imaging a peripheral environment of the flight vehicle 1. The sensor 3 detects movement of the flight vehicle 1 and generates detection data. The processing system 4 comprises a first acquisition unit 41, a second acquisition unit 42, and a processing unit 43. The first acquisition unit 41 acquires the image data generated by the camera 2. The second acquisition unit 42 acquires the detection data generated by the sensor 3. The processing unit 43 generates posture data using the image data and the detection data.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates generally to a processing system, an air vehicle, a processing method, and a program, and more particularly to a processing system, an air vehicle, a processing method, and a program that generate attitude data related to the attitude of the air vehicle. [Background technology]

[0002] The aircraft attitude angle detection device (processing system) described in Patent Document 1 includes a stereo camera consisting of a pair of cameras, a stereo processing unit, and an attitude angle calculation unit. The stereo camera is mounted on the aircraft and captures stereo images of the scenery below. The stereo processing unit processes a pair of images captured by the stereo cameras to calculate distance information for the scenery below. The attitude angle calculation unit determines the shape of the surface below as a plane based on distance information from multiple measurement points, and calculates the attitude angle of the aircraft from the inclination of the plane with respect to the aircraft. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2002-188917 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the attitude angle detection device (processing system) described in Patent Document 1, the period in which the attitude angle (movement) of the aircraft is calculated is determined depending on the period (imaging period) in which the camera generates captured images (image data). Therefore, the movement of the aircraft cannot be calculated in a period shorter than the imaging period. Therefore, there are cases in which the movement of the aircraft cannot be controlled with high accuracy.

[0005] The present disclosure aims to provide a processing system, an aircraft, a processing method, and a program that can control the movement of the aircraft with greater precision. [Means for solving the problem]

[0006] A processing system according to one aspect of the present disclosure generates attitude data relating to the attitude of an air vehicle. The air vehicle includes a camera and a sensor. The camera captures an image of a surrounding environment of the air vehicle to generate image data. The sensor detects a movement of the air vehicle to generate detection data. The processing system includes a first acquisition unit, a second acquisition unit, and a processing unit. The first acquisition unit acquires the image data generated by the camera. The second acquisition unit acquires the detection data generated by the sensor. The processing unit generates the attitude data using the image data and the detection data. The processing unit resets an integral error of the detection data when the second error is greater than the first error. The first error is an error of first attitude data related to the attitude of the aircraft, and the first attitude data is obtained using the image data. The second error is an integral error of second attitude data related to the attitude of the aircraft, and the second attitude data is obtained using the detection data. A processing system according to an embodiment of the present disclosure generates attitude data related to the attitude of an aircraft. The aircraft includes a camera and a sensor. The camera captures an image of the surrounding environment of the aircraft to generate image data. The sensor detects the movement of the aircraft to generate detection data. The processing system includes a first acquisition unit, a second acquisition unit, and a processing unit. The first acquisition unit acquires the image data generated by the camera. The second acquisition unit acquires the detection data generated by the sensor. The processing unit generates the attitude data using the image data and the detection data. When the second error is greater than the first error, the processing unit resets a zero point error of the detection data. The first error is an error of first attitude data related to the attitude of the aircraft, and the first attitude data is obtained using the image data. The second error is an integral error of second attitude data related to the attitude of the aircraft, and the second attitude data is obtained using the detection data.

[0007] An aircraft according to one aspect of the present disclosure comprises the processing system, the camera, the sensor, a device body equipped with the processing system, and a power unit that flies the device body using a power source.

[0008] A processing method according to one aspect of the present disclosure is a processing method for generating attitude data relating to the attitude of an aircraft. The aircraft includes a camera and a sensor. The camera captures an image of a surrounding environment of the aircraft to generate image data. The sensor detects a movement of the aircraft to generate detection data. The processing method includes a first acquisition step, a second acquisition step, and a generation step. an integral error reset step; In the first obtaining step, the image data generated by the camera is obtained. In the second obtaining step, the detection data generated by the sensor is obtained. In the generating step, the posture data is generated using the image data and the detection data. In the integral error reset step, if the second error is greater than the first error, an integral error of the detection data is reset. The first error is an error of first attitude data related to the attitude of the flying object, and the first attitude data is obtained using the image data. The second error is an integral error of second attitude data related to the attitude of the flying object, and the second attitude data is obtained using the detection data. A processing method according to one aspect of the present disclosure is a processing method for generating attitude data related to the attitude of an air vehicle. The air vehicle includes a camera and a sensor. The camera captures an image of a surrounding environment of the air vehicle to generate image data. The sensor detects a movement of the air vehicle to generate detection data. The processing method includes a first acquisition step, a second acquisition step, a generation step, and a zero point error reset step. In the first acquisition step, the image data generated by the camera is acquired. In the second acquisition step, the detection data generated by the sensor is acquired. In the generation step, the attitude data is generated using the image data and the detection data. In the zero point error reset step, if the second error is greater than the first error, a zero point error of the detection data is reset. The first error is an error of first attitude data related to the attitude of the air vehicle, and the first attitude data is obtained using the image data. The second error is an integral error of second attitude data related to the attitude of the air vehicle, and the second attitude data is obtained using the detection data.

[0009] A program according to one embodiment of the present disclosure is a program for causing one or more processors to execute the processing method. Effect of the Invention

[0010] The present disclosure has the advantage of being able to control the movement of an aircraft with greater precision. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram of an aircraft according to one embodiment. [Diagram 2] FIG. 2 is a schematic bottom view of the flying vehicle. [Diagram 3] FIG. 3 is a plan view diagrammatically illustrating the above flying object. [Figure 4] FIG. 4 is a flowchart showing the processing performed in the aircraft. [Diagram 5] FIG. 5 is a graph showing detection errors in the above flying object. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] The processing system, the flying object, the processing method, and the program according to the embodiment will be described below with reference to the drawings. However, the embodiment described below is merely one of various embodiments of the present disclosure. The embodiment described below can be modified in various ways depending on the design, etc., as long as the object of the present disclosure can be achieved. In addition, each figure described in the embodiment described below is a schematic diagram, and the size and thickness ratio of each component in the figure do not necessarily reflect the actual dimensional ratio.

[0013] (overview) First, an overview of the processing system 4 and the aircraft 1 will be described with reference to FIG.

[0014] The processing system 4 of this embodiment generates attitude data related to the attitude of the aircraft 1. The aircraft 1 includes a camera 2 and a sensor 3. The camera 2 captures an image of the surrounding environment of the aircraft 1 to generate image data. The sensor 3 detects the movement of the aircraft 1 to generate detection data. The processing system 4 includes a first acquisition unit 41, a second acquisition unit 42, and a processing unit 43. The first acquisition unit 41 acquires the image data generated by the camera 2. The second acquisition unit 42 acquires the detection data generated by the sensor 3. The processing unit 43 generates attitude data using the image data and the detection data.

[0015] In general, the period in which the camera 2 generates image data is longer than the period in which the sensor 3 that detects the movement of the flying object 1 generates (updates) detection data. Therefore, with the above configuration, the frequency of updating the attitude data can be increased compared to when the attitude data is generated using only image data. Also, the accuracy of the attitude data can be improved compared to when the attitude data is generated using only detection data. Therefore, the movement of the flying object 1 can be controlled with higher accuracy.

[0016] The aircraft 1 also includes a processing system 4, a camera 2, a sensor 3, a device body 61 (see FIG. 2), and a power unit 5. The device body 61 is equipped with the processing system 4. The power unit 5 uses a power source to fly the device body 61. The power source is, for example, electricity.

[0017] (detail) (1) Overall structure The configuration of the aircraft 1 will be described in more detail below.

[0018] The aircraft 1 is a drone (aerial drone) that can fly by remote control or automatic piloting but is structurally incapable of carrying a human. A drone is a type of unmanned aerial vehicle. A drone is also a type of multicopter with three or more propellers 52. A drone has the ability to fly autonomously. A drone controls the attitude of the aircraft (device body 61) by controlling the rotation speed of each of the three or more propellers 52. Also, the direction of movement of the aircraft changes according to the change in the attitude of the aircraft.

[0019] As shown in Fig. 1, the aircraft 1 includes a camera 2, a sensor 3, a processing system 4, a power unit 5, a memory unit 11, a power supply unit 12, and a communication unit 13. The power unit 5 has a plurality of motors 51 (four in this embodiment; only one is shown in Fig. 1) and a plurality of propellers 52 (four in Fig. 2). As shown in Figs. 2 and 3, the aircraft 1 further includes a device main body 61, a plurality of arms 62 (four in Fig. 2), and a plurality of skids 63 (four in this embodiment; only two are shown in Fig. 3).

[0020] (2) Structural parts of the aircraft 2 and 3, the roll axis, pitch axis, and yaw axis of the device main body 61 are illustrated as the X-axis, Y-axis, and Z-axis. The arrows representing the X-axis, Y-axis, and Z-axis are merely shown for the purpose of explanation and do not have any substance.

[0021] The device body 61 is formed, for example, in a rectangular parallelepiped shape. Four arms 62 extend from the device body 61. A propeller 52 is attached to the tip of each arm 62. In the following, in order to distinguish between the four propellers 52, the four propellers 52 may be referred to as propellers 52A, 52B, 52C, and 52D, respectively. The four propellers 52A, 52B, 52C, and 52D are arranged in this order in the circumferential direction surrounding the device body 61 (around the yaw axis (Z axis)).

[0022] Of the four propellers 52, two propellers 52A and 52C rotate in a first direction, and the remaining two propellers 52B and 52D rotate in a second direction opposite to the first direction. Two propellers 52 located diagonally to each other rotate in the same direction.

[0023] The four propellers 52 correspond one-to-one to the four motors 51. Each of the four propellers 52 rotates due to the driving force of the corresponding motor 51, generating a thrust. The attitude and acceleration of the device body 61 change according to the thrust of each of the four propellers 52. For example, by making the rotation speed of the two propellers 52C, 52D provided in the front half of the aircraft 1 smaller than the rotation speed of the two propellers 52A, 52B provided in the rear half of the aircraft 1, the aircraft 1 tilts forward, generating a forward thrust in the aircraft 1 and moving the aircraft 1 forward.

[0024] Furthermore, the torque acting on device body 61 from four propellers 52 is determined by the rotation speed of each of the four propellers 52. For example, suppose that the rotation speed of two diagonally positioned propellers 52A, 52C is made smaller or larger than the rotation speed of the remaining two propellers 52B, 52D. Then, a torque equivalent to the difference between the torque of the two propellers 52A, 52C and the torque of the two propellers 52B, 52D acts on device body 61, causing device body 61 to rotate about the yaw axis (Z axis).

[0025] The sensor 3 is built into the device body 61. As shown in Fig. 2, the sensor 3 is disposed in a central region including a center 610 (geometric center) of the device body 61 when viewed from the yaw axis direction of the aircraft 1.

[0026] A part of the camera 2 is exposed from a bottom surface 611 of the device body 61. The camera 2 is disposed in a region including a center 610 of the device body 61 when viewed from the yaw axis direction of the flying object 1. When viewed from the yaw axis direction of the flying object 1, at least a part of the camera 2 and the sensor 3 overlap.

[0027] The multiple skids 63 protrude from the multiple arms 62. When the aircraft 1 lands, the multiple skids 63 come into contact with a landing surface such as the ground. In this way, the multiple skids 63 support the device main body 61.

[0028] (3) Camera The camera 2 has a two-dimensional image sensor, such as a charge coupled device (CCD) image sensor or a complementary metal-oxide semiconductor (CMOS) image sensor.

[0029] The camera 2 captures an image of the surrounding environment of the aircraft 1. As a result, the camera 2 generates image data. The camera 2 of this embodiment captures an image of the environment below the aircraft 1. More specifically, the camera 2 captures an image of the environment including a reference plane such as the ground below the aircraft 1. The camera 2 generates image data at a predetermined time interval.

[0030] In this embodiment, as an example, the camera 2 is a stereo camera. A stereo camera is a two-lens camera. That is, the camera 2 has two lenses. The camera 2 generates two pieces of image data based on the light incident on each of the two lenses.

[0031] In addition, even if a camera 2 including two monocular cameras is used instead of the camera 2 that is a stereo camera, the camera 2 can generate two pieces of image data based on the light that is respectively incident on the two lenses.

[0032] (4) Sensor The sensor 3 detects the movement of the flying object 1. As a result, the sensor 3 generates detection data. As shown in FIG. 1, the sensor 3 includes an acceleration sensor 31 and a gyro sensor 32. The acceleration sensor 31 detects acceleration in three axial directions. The gyro sensor 32 detects angular velocity in the three axial directions. In other words, the detection data includes information about the acceleration of the flying object 1 and information about the angular velocity of the flying object 1. The three axial directions of the acceleration sensor 31 and the three axial directions of the gyro sensor 32 may be the same or different. In this embodiment, the three axial directions of the acceleration sensor 31 and the three axial directions of the gyro sensor 32 are the X-axis, Y-axis, and Z-axis directions.

[0033] The sensor 3 generates detection data at a predetermined time interval. Note that the first time interval at which the acceleration sensor 31 generates detection data and the second time interval at which the gyro sensor 32 generates detection data may be different from or the same as each other.

[0034] The time interval at which the sensor 3 generates the detection data is shorter than the time interval at which the camera 2 generates the image data. That is, both the first time interval and the second time interval are shorter than the time interval at which the camera 2 generates the image data.

[0035] The sensor 3 of this embodiment includes an IMU (Inertial Measurement Unit). The IMU includes an acceleration sensor 31 and a gyro sensor 32 integrated into one package.

[0036] Moreover, the sensor 3 preferably further includes, for example, a geomagnetic sensor, a Global Navigation Satellite System (GNSS) sensor, and an air pressure sensor. The geomagnetic sensor detects the orientation of the aircraft 1. The GNSS sensor detects the current position of the aircraft 1. An example of a GNSS sensor is a Global Positioning System (GPS) sensor. The air pressure sensor detects the air pressure at the current position of the aircraft 1.

[0037] (5) Storage section The storage unit 11 is, for example, a Read Only Memory (ROM), a Random Access Memory (RAM), or an Electrically Erasable Programmable Read-Only Memory (EEPROM). The storage unit 11 stores information related to the control of the aircraft 1. The storage unit 11 stores, for example, identification information of the aircraft 1, identification information of a device that communicates with the aircraft 1, map data of the area in which the aircraft 1 flies, and setting parameters of the aircraft 1.

[0038] (6) Power supply section The power source of the flying object 1 is electricity. The power supply unit 12 supplies the power as the power source. That is, the power supply unit 12 supplies the power for driving the motor 51 of the power unit 5. The power supply unit 12 also supplies the power for operating the electric circuits and electronic components of the flying object 1, such as the processing unit 43, the camera 2, the sensor 3, and the communication unit 13.

[0039] The power supply unit 12 includes a battery and a power conversion circuit. The battery may be a primary battery or a secondary battery. The power conversion circuit converts the output voltage of the battery to a desired voltage and outputs the converted voltage.

[0040] (7) Communications Department The communication unit 13 includes, for example, an antenna and a communication circuit. The communication unit 13 has a function of performing wireless communication using radio waves as a medium. The communication unit 13 receives, for example, a command signal indicating the destination of the flying object 1 from an operation terminal. The processing unit 43 controls the movement of the flying object 1 based on the command signal.

[0041] (8) Power plant The power plant 5 includes a plurality of motors 51 and a plurality of propellers 52 that correspond one-to-one to the motors 51. Each motor 51 rotates a corresponding propeller 52. The operation of the power plant 5 is controlled by the processing unit 43. That is, the processing unit 43 controls the power plant 5 to control the rotation of each of the propellers 52, thereby making it possible to move the flying object 1 in all directions.

[0042] (9) First acquisition part The first acquisition unit 41 of the processing system 4 acquires the image data generated by the camera 2 from the camera 2. The first acquisition unit 41 transfers the image data to the processing unit 43.

[0043] The first acquisition unit 41 may perform image processing on the image data and transmit the image data after the image processing to the processing unit 43. The image processing may also be performed by the processing unit 43. Examples of the image processing include a process of compressing the image data, a process of enlarging or reducing an image shown in the image data, a process of cropping an image, a process of correcting image distortion, and a process of removing noise from the image data.

[0044] (10)Second acquisition part The second acquisition unit 42 acquires, from the sensor 3, the detection data generated by the sensor 3. The second acquisition unit 42 transfers the detection data to the processing unit 43.

[0045] (11) Processing section The processing unit 43 is configured by a computer system having one or more processors and a memory. At least a part of the functions of the processing unit 43 is realized by the processor of the computer system executing a program recorded in the memory of the computer system. The program may be recorded in the memory, or may be provided through a telecommunication line such as the Internet, or may be recorded on a non-transitory recording medium such as a memory card and provided.

[0046] The processing unit 43 includes a data processing unit 431 and an instruction unit 432. These merely indicate functions realized by the processing unit 43, and do not necessarily indicate a substantial configuration.

[0047] The data processing unit 431 can generate attitude data related to the attitude of the aircraft 1 using at least one of the image data and the detection data. The data processing unit 431 can also generate position data related to the position of the aircraft 1 using at least one of the image data and the detection data. The instruction unit 432 generates an instruction signal that controls the operation of the power plant 5 based on the attitude data and position data generated by the data processing unit 431. This allows the instruction unit 432 to set the attitude of the aircraft 1 to a desired attitude and move the aircraft 1 to a desired position.

[0048] (11.1) Processing to generate attitude and position data using image data First, an example of the process in which the data processing unit 431 generates posture data and position data using image data will be described. The process of generating posture data and position data using image data as described below is described in, for example, Patent Document 1, JP2020-056627A, and "Development of self-position detection technology without using GPS (Yamane Tomoyuki) (https: / / tiit.or.jp / userfiles / Reports%20of%20the%20Tottori%20institute%20of%20Industrial%20Tecnology%20No22%202019%20-%20p13-17.pdf)", so only an outline will be described below.

[0049] It is assumed that the aircraft 1 flies near a planar area, which is an area consisting of substantially horizontal surfaces, and the camera 2 captures an image of the planar area.

[0050] The data processing unit 431 generates distance distribution data based on the image data. The distance distribution data is data indicating the distribution of distances between the camera 2 and the subject in the image shown in the image data.

[0051] Camera 2 generates two image data based on the light incident on the two lenses. Camera 2 generates two image data captured at the same time for each predetermined imaging period. Data processor 431 divides the image shown in the two image data into multiple blocks. Each block may include a single pixel or multiple pixels.

[0052] Since there is a parallax between the two lenses of the camera 2, a pixel shift due to the parallax occurs between the two image data. The data processing unit 431 calculates the distance from the camera 2 to the subject of each block based on the magnitude of the pixel shift. The distance calculated in this way is called the distance value of the block. A collection of information on the distance values ​​of each block is the above-mentioned distance distribution data.

[0053] The memory unit 11 stores map data of the area in which the aircraft 1 flies. The map data includes information on the shape of the edges of planar areas, which are areas consisting of substantially horizontal surfaces, and position information of the planar areas. The data processing unit 431 extracts the edges of the planar areas from the image data based on the distance distribution data. When the aircraft 1 is tilted with respect to the normal direction of the planar areas, the data of the planar areas in the distance distribution data becomes data of a plane tilted with respect to the aircraft 1. Therefore, the data processing unit 431 can obtain the tilt of the aircraft 1 with respect to the planar areas based on the distance distribution data. That is, the data processing unit 431 can obtain the rotation angles of the aircraft 1 around the roll axis and the pitch axis.

[0054] Furthermore, the data processing unit 431 can determine the orientation of the aircraft 1 with respect to the planar area (the rotation angle around the yaw axis) based on the shape of the edge of the planar area.

[0055] The rotation angles about the roll axis, pitch axis, and yaw axis obtained in this manner correspond to attitude data that represents the attitude of the aircraft 1. In other words, the data processing unit 431 can generate attitude data using image data.

[0056] Furthermore, the data processing unit 431 compares information such as the distance distribution data and attitude data representing the attitude of the aircraft 1 with the map data. As a result, the data processing unit 431 can obtain the positional relationship between the aircraft 1 and the planar area and generate position data representing the position of the aircraft 1.

[0057] (11.2) Processing to generate attitude and position data using detection data Next, an example of a process in which the data processing unit 431 generates attitude data and position data using detection data will be described.

[0058] The detection data includes information on the acceleration in three axial directions detected by the acceleration sensor 31 and information on the angular velocity in three axial directions detected by the gyro sensor 32. The data processing unit 431 applies a Kalman filter to this information. In this embodiment, the "detection data" processed by the data processing unit 431 refers to the detection data after being applied with the Kalman filter.

[0059] Attitude data of the aircraft 1 at an initial position (for example, when the aircraft 1 lands) is stored in the memory unit 11. The data processing unit 431 obtains the amount of change in the attitude of the aircraft 1 based on the detection data at each predetermined time interval. The data processing unit 431 generates current attitude data of the aircraft 1 by adding an integral value of the amount of change in the attitude of the aircraft 1 to the attitude data of the aircraft 1 at the initial position.

[0060] The memory unit 11 also stores position data of the aircraft 1 at an initial position (for example, when the aircraft 1 lands, or a position determined by GNSS at a certain point in time). The data processing unit 431 determines the amount of movement of the aircraft 1 based on the detection data at each predetermined time interval. The amount of movement is the amount of movement in each of the three axial directions. The data processing unit 431 generates current position data of the aircraft 1 by adding an integral value of the amount of movement of the aircraft 1 to the position data of the aircraft 1 at the initial position.

[0061] (11.3) Processing to generate attitude and position data using image data and detection data The data processing unit 431 can generate attitude data and position data using both the image data and the detection data. That is, the processing unit 43 can generate position data regarding the position of the aircraft 1 using the image data and the detection data.

[0062] An example of a process in which the data processing unit 431 generates posture data and position data using image data and detection data will be described below with reference to Fig. 4. Note that the flowchart shown in Fig. 4 is merely an example, and the order of the processes may be changed as appropriate, and processes may be added or omitted as appropriate.

[0063] In some of the flows in the flowchart, the data processing unit 431 generates attitude data and position data. The attitude data and position data are successively updated. The instruction unit 432 controls the movement of the aircraft 1 based on the most recently generated attitude data and position data.

[0064] The camera 2 generates image data at a predetermined time interval (imaging cycle). That is, the image data is updated at each imaging cycle. The sensor 3 generates detection data at a predetermined time interval (detection cycle). That is, the detection data is updated at each detection cycle. The imaging cycle is longer than the detection cycle. The detection data is updated multiple times between updates of the image data. As an example, the imaging cycle is about 1 / 20 [second] and the detection cycle is about 1 / 1000 [second].

[0065] Here, the start point of the flow is the time when the image data generated by the camera 2 is updated (step ST1). As described in section (11.1), the data processing unit 431 generates attitude data and position data using the image data (step ST2).

[0066] Thereafter, when the detection data is updated (step ST3: Yes), the data processing unit 431 generates attitude data and position data using the image data and the detection data (step ST4). More specifically, after the first acquisition unit 41 acquires the image data and the second acquisition unit 42 acquires the detection data, the data processing unit 431 (i.e., the processing unit 43) generates, as attitude data, information on an attitude tilted by the amount of change obtained using the detection data from the attitude of the aircraft 1 obtained using the image data. That is, after the first acquisition unit 41 acquires the image data in step ST1, the second acquisition unit 42 acquires the detection data in step ST3. Then, in step ST4, the data processing unit 431 generates, as attitude data, information on an attitude tilted by the amount of change obtained using the detection data from the attitude of the aircraft 1 obtained using the image data in step ST2. The attitude data is, for example, a vector quantity including three components consisting of rotation angles around the roll axis, pitch axis, and yaw axis.

[0067] For example, the rotation angle (vector quantity) of the aircraft 1 obtained in step ST2 is θ1. The angular velocity (vector quantity) of the aircraft 1 obtained using the detection data in step ST4 is ω1. Furthermore, the time from when the image data is acquired (updated) to when the detection data is updated is Δt1. In step ST4, the data processing unit 431 obtains the rotation angle θ2 of the aircraft 1 using [Mathematical Expression 1]. [Equation 1] θ2 = θ1 + ω1 × Δt1 After the first acquisition unit 41 acquires the image data, the second acquisition unit 42 updates the detection data multiple times until the first acquisition unit 41 updates the image data. Each time the detection data is updated, the data processing unit 431 obtains the rotation angle θ2 of the flying object 1 by the following [Equation 2]. That is, the data processing unit 431 obtains the rotation angle θ2 by integrating the angular velocity of the flying object 1 over time. [Equation 2] θ2 = θ1 + Σωi × Δti If N is the number of times the detection data has been updated since the most recent image data was acquired (N is a natural number equal to or greater than 1), then i=1, 2, 3, ....N. ωi is the angular velocity (vector quantity) of the aircraft 1 calculated at the time of the i-th update of the detection data. When i ≥ 2, Δti is the time from the (i-1)-th update of the detection data to the i-th update.

[0068] In addition, the rotation angle θ1 is updated every time the image data is updated. In other words, when the image data is updated in step ST5 described later, the data processing unit 431 uses the rotation angle of the aircraft 1 calculated using the updated image data as the above-mentioned rotation angle θ1 to calculate the rotation angle θ2.

[0069] In step ST4, position data is also generated. After the first acquisition unit 41 acquires image data, and the second acquisition unit 42 acquires detection data, the data processing unit 431 (i.e., the processing unit 43) generates, as position data, information on a position moved by the amount of movement calculated using the detection data from the position of the aircraft 1 calculated using the image data. That is, after the first acquisition unit 41 acquires image data in step ST1, the second acquisition unit 42 acquires detection data in step ST3. Then, in step ST4, the data processing unit 431 generates, as position data, information on a position moved by the amount of movement calculated using the detection data from the position of the aircraft 1 calculated using the image data in step ST2. The position data is, for example, information on the coordinates (vector quantity) of the aircraft 1 in a three-dimensional coordinate space set on the ground.

[0070] For example, the coordinate (vector quantity) of the aircraft 1 obtained in step ST2 is set to X1. The velocity (vector quantity) of the aircraft 1 obtained using the detection data in step ST4 is set to v1. Furthermore, the time from when the image data is acquired (updated) to when the detection data is updated is set to Δt1. In step ST4, the data processing unit 431 obtains the coordinate X2 of the aircraft 1 by [Mathematical formula 3]. [Number 3] X2 = X1 + v1 × Δt1 After the first acquisition unit 41 acquires the image data, the second acquisition unit 42 updates the detection data multiple times until the first acquisition unit 41 updates the image data. Each time the detection data is updated, the data processing unit 431 obtains the coordinate X2 of the flying object 1 by the following [Equation 4]. That is, the data processing unit 431 obtains the coordinate X2 by integrating the speed of the flying object 1 over time. [Number 4] X2 = X1 + Σvi × Δti vi is the velocity (vector quantity) of the aircraft 1 determined at the time of the i-th update of the detection data.

[0071] In addition, the coordinate X1 is updated every time the image data is updated. In other words, when the image data is updated in step ST5 described later, the data processing unit 431 uses the coordinates of the aircraft 1 obtained in step ST8 described later using the updated image data as the above-mentioned coordinate X1 to obtain the coordinate X2. However, if the determination in step ST7 described later is "No", the coordinate X1 is not updated.

[0072] Next, the process from step ST5 onwards will be described.

[0073] When the image data is updated (step ST5: Yes), the data processing unit 431 generates attitude data using the image data (step ST6) as described in section (11.1). That is, the data processing unit 431 obtains the rotation angles of the aircraft 1 about the roll axis, pitch axis, and yaw axis as the attitude data.

[0074] After step ST6, the data processing unit 431 compares the attitude angle error based on the image data with the attitude angle error based on the detection data (step ST7). The attitude angle is the inclination of the yaw axis (Z axis) with respect to the vertical direction. The attitude angle error is the difference between the attitude angle indicated in the detection data and the actual attitude angle. The attitude angle error is an example of an error in the attitude of the aircraft 1. First, the attitude angle error will be described with reference to FIG. 5.

[0075] Regarding the attitude angle of the aircraft 1 obtained using image data, the attitude angle error 701 (first attitude angle error) is a value independent of time. On the other hand, the attitude angle of the aircraft 1 obtained using detection data is obtained by integrating the detected detection data (angular velocity) over time, so the attitude angle error 702 (second attitude angle error) of the attitude angle of the aircraft 1 obtained using the detection data accumulates as time passes. For example, if the detection error of the detection data (angular velocity) is constant regardless of time, the attitude angle error 702 increases in proportion to time as shown in FIG. 5. In reality, the detection error of the detection data (angular velocity) may fluctuate with time. The time Tm1 can be estimated by previously obtaining the change in the detection error (attitude angle error 702) of the detection data (angular velocity) and the detection error (attitude angle error 701) of the image data by measurement or the like. The time Tm1 is the time required from when the sensor 3 starts detecting the angular velocity until the attitude angle error 702 becomes equal to or greater than the attitude angle error 701. The time Tm1 is stored in the storage unit 11.

[0076] When generating attitude data and position data using only detection data, errors may accumulate due to time integration as described above, but by using image data in combination, the accuracy of the attitude data and position data can be improved.

[0077] In step ST7 in FIG. 4, if the attitude angle error 701 is smaller than the attitude angle error 702 (step ST7: Yes), as described in section (11.1), the data processing unit 431 generates position data using the image data (step ST8).

[0078] In short, the data processing unit 431 continues the process of creating position data using image data and detection data (step ST4) while the error of the sensor 3 (attitude angle error 702) does not exceed the error of the camera 2 (attitude angle error 701). On the other hand, when the error of the sensor 3 (attitude angle error 702) exceeds the error of the camera 2 (attitude angle error 701), the data processing unit 431 temporarily executes the process of creating position data using image data without using detection data (step ST8).

[0079] After step ST8 (i.e., when the attitude angle error 701 is smaller than the attitude angle error 702), the data processing unit 431 calculates the zero point error E1 of the angular velocity of the detection data (step ST9). The data processing unit 431 calculates the zero point error E1 based on the image data and the detection data. The zero point error E1 is a vector quantity. The unit of the zero point error E1 is [deg / s]. The zero point error E1 is calculated by [Equation 5]. [Equation 5] E1 = ("angle obtained using image data" - "integral value of angular velocity detected by sensor 3") / integral time The first term of the numerator is calculated using the image data most recently acquired in step ST5. That is, the first term of the numerator is the angle (rotation angle) of the aircraft 1 calculated in step ST6. The second term of the numerator is the rotation angle θ2 of the aircraft 1 calculated by [Equation 2]. That is, the second term of the numerator is the rotation angle θ2 calculated based on the image data acquired immediately before the most recently acquired image data and the detection data. The "integral time" in the denominator is "ΣΔti" (where i = 1, 2, 3, ... N).

[0080] After calculating the zero point error E1 of the angular velocity of the detection data, the data processing unit 431 resets the zero point error E1. Furthermore, the data processing unit 431 resets the integral error of the angular velocity of the detection data (step ST10). This makes it possible to improve the accuracy of the detection data.

[0081] The process of resetting the zero point error E1 is a process of subtracting the zero point error E1 from the offset of the angular velocity of the detection data. If the zero point error E1 is a negative value, the offset will be reduced by the process of resetting the zero point error E1. For example, from an offset of -10 [deg / s], -7 [deg / s] is subtracted as the zero point error E1, and the offset becomes -3 [deg / s].

[0082] The integral error is, for example, the attitude angle error 702 (see FIG. 5) and is calculated from the elapsed time. The process of resetting the integral error of the angular velocity of the detection data is a process of subtracting the integral error from the offset of the angular velocity of the detection data.

[0083] In this way, the data processing unit 431 (i.e., processing unit 43) resets the integral error of the detection data when the second error is greater than the first error. The first error is an error (attitude angle error) of first attitude data related to the attitude of the aircraft 1. The first attitude data is obtained using image data. The second error is an integral error (attitude angle error) of second attitude data related to the attitude of the aircraft 1. The second attitude data is obtained using detection data.

[0084] Furthermore, when the second error is greater than the first error, the data processing unit 431 (ie, the processing unit 43) resets the zero point error E1 of the detection data.

[0085] Next, step ST11 will be described. The data processing unit 431 periodically determines whether or not to continue controlling the operation of the aircraft 1 based on the command signal received by the communication unit 13 (step ST11). For example, when the aircraft 1 lands at a point designated by the command signal, the data processing unit 431 ends the control.

[0086] After step ST10, if the control is to be continued (step ST11: Yes), the process returns to step ST3.

[0087] (12) Processing method and program The same functions as those of the processing system 4 may be embodied as a processing method, a (computer) program, or a non-transitory recording medium on which a program is recorded.

[0088] A processing method according to one embodiment is a processing method for generating attitude data relating to the attitude of an aircraft 1. The aircraft 1 includes a camera 2 and a sensor 3. The camera 2 captures an image of the surrounding environment of the aircraft 1 to generate image data. The sensor 3 detects the movement of the aircraft 1 to generate detection data. The processing method has a first acquisition step, a second acquisition step, and a generation step. In the first acquisition step, image data generated by the camera 2 is acquired. In the second acquisition step, detection data generated by the sensor 3 is acquired. In the generation step, attitude data is generated using the image data and the detection data.

[0089] A program according to one embodiment is a program for causing one or more processors to execute the above processing method. The program may be recorded in a non-transitory computer-readable recording medium.

[0090] The processing system 4 in the present disclosure includes a computer system. The computer system is mainly composed of a processor and a memory as hardware. At least a part of the functions of the processing system 4 in the present disclosure is realized by the processor executing a program recorded in the memory of the computer system. The program may be pre-recorded in the memory of the computer system, provided through an electric communication line, or provided by recording it in a non-transitory recording medium such as a memory card, an optical disk, or a hard disk drive that can be read by the computer system. The processor of the computer system is composed of one or more electronic circuits including a semiconductor integrated circuit (IC) or a large-scale integrated circuit (LSI). The integrated circuits such as IC or LSI referred to here are called by different names depending on the degree of integration, and include integrated circuits called system LSI, VLSI (Very Large Scale Integration), or ULSI (Ultra Large Scale Integration). Furthermore, a field-programmable gate array (FPGA) that is programmed after the manufacture of the LSI, or a logic device that can reconfigure the connection relationship inside the LSI or the circuit partition inside the LSI, can also be adopted as a processor. The electronic circuits may be integrated in one chip or distributed among multiple chips. The chips may be integrated in one device or distributed among multiple devices. The computer system referred to here includes a microcontroller having one or more processors and one or more memories. Thus, the microcontroller is also composed of one or more electronic circuits including a semiconductor integrated circuit or a large-scale integrated circuit.

[0091] Furthermore, it is not essential for the processing system 4 that multiple functions in the processing system 4 are integrated into one device, and the components of the processing system 4 may be distributed across multiple devices. Furthermore, at least a portion of the functions of the processing system 4, for example, at least a portion of the functions of the data processing unit 431, may be realized by cloud (cloud computing) or the like.

[0092] (Modification of the embodiment) Modifications of the embodiment will be listed below. The following modifications may be implemented in appropriate combination.

[0093] The aircraft 1 is not limited to a drone (aerial drone) and may be, for example, a radio-controlled aircraft or a helicopter.

[0094] The number of propellers 52 and the number of motors 51 included in the aircraft 1 are not limited to four. The number of propellers 52 and the number of motors 51 included in the aircraft 1 may be, for example, two, three, six, or eight.

[0095] The camera 2 is not limited to a stereo camera, and may be configured to include one or more monocular cameras.

[0096] The processing system 4 may be mounted on a separate device, rather than on the device main body 61. In other words, the processing system 4 may acquire image data and detection data from an aircraft 1 that does not include the processing system 4, and generate attitude data and position data.

[0097] The data processing unit 431 (that is, the processing unit 43) may reset the integral error of the detection data every time the first acquisition unit 41 acquires image data.

[0098] The data processing unit 431 (that is, the processing unit 43) may reset the zero point error E1 of the detection data every time the first acquisition unit 41 acquires image data.

[0099] (summary) The above-described embodiments and the like disclose the following aspects.

[0100] A processing system (4) according to a first aspect generates attitude data relating to the attitude of an aircraft (1). The aircraft (1) includes a camera (2) and a sensor (3). The camera (2) captures an image of the surrounding environment of the aircraft (1) to generate image data. The sensor (3) detects the movement of the aircraft (1) to generate detection data. The processing system (4) includes a first acquisition unit (41), a second acquisition unit (42), and a processing unit (43). The first acquisition unit (41) acquires the image data generated by the camera (2). The second acquisition unit (42) acquires the detection data generated by the sensor (3). The processing unit (43) generates attitude data using the image data and the detection data.

[0101] In general, the cycle in which the camera (2) generates image data is longer than the cycle in which the sensor (3) that detects the movement of the flying object (1) generates (updates) detection data. Therefore, with the above configuration, the frequency of updating the attitude data can be increased compared to when the attitude data is generated using only image data. Also, the accuracy of the attitude data can be improved compared to when the attitude data is generated using only detection data. Therefore, the movement of the flying object (1) can be controlled with higher accuracy.

[0102] In addition, in the processing system (4) according to the second aspect, in the first aspect, after the first acquisition unit (41) acquires image data and the second acquisition unit (42) acquires detection data, the processing unit (43) generates, as attitude data, information on an attitude tilted by an amount of change determined using the detection data from the attitude of the aircraft (1) determined using the image data.

[0103] According to the above configuration, during the period from when the camera (2) generates the first image data to when it generates the subsequent second image data, posture data can be generated using both the first image data and the detection data.

[0104] In the processing system (4) according to the third aspect, in the first or second aspect, the processing unit (43) generates position data regarding the position of the aircraft (1) using the image data and the detection data.

[0105] According to the above configuration, the frequency of updating the position data can be increased compared to when the position data is generated using only image data. Also, the accuracy of the position data can be improved compared to when the position data is generated using only detection data. Therefore, the movement of the flying object (1) can be controlled with higher accuracy.

[0106] In addition, in the processing system (4) according to the fourth aspect, in the third aspect, after the first acquisition unit (41) acquires image data and the second acquisition unit (42) acquires detection data, the processing unit (43) generates, as position data, information on a position moved from the position of the aircraft (1) determined using the image data by an amount of movement determined using the detection data.

[0107] According to the above configuration, during the period from when the camera (2) generates the first image data to when it generates the subsequent second image data, position data can be generated using both the first image data and the detection data.

[0108] In addition, in the processing system (4) according to a fifth aspect, in any one of the first to fourth aspects, the processing unit (43) resets an integral error of the detection data when the second error is larger than the first error. The first error is an error in first attitude data related to the attitude of the flying object (1). The first attitude data is obtained using image data. The second error is an integral error in second attitude data related to the attitude of the flying object (1). The second attitude data is obtained using the detection data.

[0109] According to the above configuration, the accuracy of the detection data can be improved by resetting the integral error.

[0110] In addition, in the processing system (4) according to a sixth aspect, in any one of the first to fifth aspects, the processing unit (43) resets a zero point error of the detection data when the second error is larger than the first error. The first error is an error of first attitude data related to the attitude of the aircraft (1). The first attitude data is obtained using image data. The second error is an integral error of second attitude data related to the attitude of the aircraft (1). The second attitude data is obtained using the detection data.

[0111] According to the above configuration, the zero point error is reset, thereby making it possible to improve the accuracy of the detection data.

[0112] In addition, in a processing system (4) according to a seventh aspect, in any one of the first to sixth aspects, the sensor (3) includes an acceleration sensor (31) that detects acceleration in three axial directions, and a gyro sensor (32) that detects angular velocity in three axial directions.

[0113] According to the above configuration, the sensor (3) can detect the inclination and movement of the flying object (1) in any direction.

[0114] The configurations other than those of the first embodiment are not essential to the processing system (4) and can be omitted as appropriate.

[0115] In addition, the aircraft (1) of the eighth aspect includes a processing system (4) of any one of the first to seventh aspects, a camera (2), a sensor (3), an apparatus main body (61) equipped with the processing system (4), and a power unit (5) that flies the apparatus main body (61) using a power source.

[0116] According to the above configuration, an aircraft (1) having an integrated processing system (4) can be provided.

[0117] In the flying object (1) according to the ninth aspect, the sensor (3) is disposed in the central region of the eighth aspect. The central region includes the center (610) of the device body (61) when viewed from the yaw axis direction of the flying object (1).

[0118] According to the above configuration, the movement of the flying object (1) can be controlled with greater precision.

[0119] The configurations other than those of the eighth embodiment are not essential to the processing system (4) and may be omitted as appropriate.

[0120] A processing method according to a tenth aspect is a processing method for generating attitude data relating to the attitude of an aircraft (1). The aircraft (1) includes a camera (2) and a sensor (3). The camera (2) captures an image of the surrounding environment of the aircraft (1) to generate image data. The sensor (3) detects the movement of the aircraft (1) to generate detection data. The processing method includes a first acquisition step, a second acquisition step, and a generation step. In the first acquisition step, the image data generated by the camera (2) is acquired. In the second acquisition step, the detection data generated by the sensor (3) is acquired. In the generation step, the attitude data is generated using the image data and the detection data.

[0121] According to the above configuration, the movement of the flying object (1) can be controlled with greater precision.

[0122] A program according to an eleventh aspect is a program for causing one or more processors to execute the processing method according to the tenth aspect.

[0123] According to the above configuration, the movement of the flying object (1) can be controlled with greater precision.

[0124] Not limited to the above-mentioned aspects, various configurations (including modified examples) of the processing system (4) according to the embodiment can be embodied as a processing method and a program. [Explanation of symbols]

[0125] 1. Flying object 2 Camera 3 Sensors 4 Processing System 5 Power plant 31 Acceleration Sensor 32 Gyro sensor 41 First acquisition part 42 Second acquisition part 43 Processing section 61 Device body 610 center

Claims

1. 1. A processing system for generating attitude data relating to an attitude of an air vehicle, comprising: The flying object includes a camera that captures an image of a surrounding environment of the flying object and generates image data, and a sensor that detects a movement of the flying object and generates detection data; The processing system includes: A first acquisition unit that acquires the image data generated by the camera; A second acquisition unit that acquires the detection data generated by the sensor; a processing unit for generating the posture data using the image data and the detection data, The processing unit resets an integral error of the detection data when the second error is greater than the first error, the first error is an error of first attitude data relating to an attitude of the aircraft, the first attitude data being calculated using the image data; the second error is an integral error of second attitude data relating to the attitude of the aircraft, and the second attitude data is obtained using the detection data. Processing system.

2. A processing system for generating attitude data relating to the attitude of an aircraft, comprising: The flying object includes a camera that captures an image of a surrounding environment of the flying object and generates image data, and a sensor that detects a movement of the flying object and generates detection data; The processing system includes: A first acquisition unit that acquires the image data generated by the camera; A second acquisition unit that acquires the detection data generated by the sensor; a processing unit for generating the posture data using the image data and the detection data, The processing unit resets a zero point error of the detection data when the second error is greater than the first error, the first error is an error of first attitude data relating to an attitude of the aircraft, the first attitude data being calculated using the image data; the second error is an integral error of second attitude data relating to the attitude of the aircraft, and the second attitude data is obtained using the detection data. Processing system.

3. When the first acquisition unit acquires the image data and the second acquisition unit acquires the detection data, the processing unit generates, as the attitude data, information on an attitude of the flying object tilted by an amount of change determined using the detection data from an attitude of the flying object determined using the image data.

3. The processing system according to claim 1 or 2.

4. The processing unit generates position data regarding a position of the flying object using the image data and the detection data. The processing system according to any one of claims 1 to 3.

5. When the first acquisition unit acquires the image data and the second acquisition unit acquires the detection data, the processing unit generates, as the position data, information on a position of the flying object that has moved from the position of the flying object determined using the image data by an amount of movement determined using the detection data. The processing system of claim 4 .

6. The sensor includes an acceleration sensor that detects acceleration in three axial directions and a gyro sensor that detects angular velocity in three axial directions. The processing system according to any one of claims 1 to 5.

7. A processing system according to any one of claims 1 to 6, The camera; The sensor; An apparatus main body having the processing system mounted thereon; A power unit that uses a power source to fly the device body. Flying vehicle.

8. The sensor is disposed in a central region including the center of the device body when viewed from the direction of the yaw axis of the flying object. The flying vehicle according to claim 7.

9. A processing method for generating attitude data relating to the attitude of an aircraft, comprising: The flying object includes a camera that captures an image of a surrounding environment of the flying object and generates image data, and a sensor that detects a movement of the flying object and generates detection data; The processing method includes: a first acquisition step of acquiring the image data generated by the camera; a second acquisition step of acquiring the detection data generated by the sensor; generating the posture data using the image data and the detection data; an integral error reset step of resetting an integral error of the detection data when the second error is greater than the first error; the first error is an error of first attitude data relating to an attitude of the aircraft, the first attitude data being calculated using the image data; the second error is an integral error of second attitude data relating to the attitude of the aircraft, and the second attitude data is obtained using the detection data. Processing methods.

10. 1. A processing method for generating attitude data relating to an attitude of an aircraft, comprising: The flying object includes a camera that captures an image of a surrounding environment of the flying object and generates image data, and a sensor that detects a movement of the flying object and generates detection data; The processing method includes: a first acquisition step of acquiring the image data generated by the camera; a second acquisition step of acquiring the detection data generated by the sensor; generating the posture data using the image data and the detection data; a zero point error reset step of resetting a zero point error of the detection data when the second error is greater than the first error, the first error is an error of first attitude data relating to an attitude of the aircraft, the first attitude data being calculated using the image data; the second error is an integral error of second attitude data relating to the attitude of the aircraft, and the second attitude data is obtained using the detection data. Processing methods.

11. For causing one or more processors to execute the processing method according to claim 9 or claim 10, program.

Images