Information processing device and information processing method

Abstract

This information processing device comprises: a mission information acquisition unit; an image processing unit; and a display control unit. The mission information acquisition unit acquires mission information pertaining to an automatic flight position. The image processing unit superimposes the automatic flight position on a plurality of viewpoint images obtained from different viewpoints. The display control unit presents, as verification images for verifying the automatic flight position, the individual viewpoint images on each of which the automatic flight position is superimposed.

Description

Information processing device and information processing method

[0001] The present invention relates to an information processing device and an information processing method.

[0002] It is known that autonomously moving aircraft such as drones are used to carry out missions such as aerial photography. Users can design the automatic flight position of the aircraft using publicly available map information. For example, when performing aerial photography, the automatic flight position is set around the object to be photographed (such as a building, bridge, or temple). The automatic flight position may be a flight route or a photography point (a point in space from which an object is photographed from a specific distance and angle).

[0003] JP 2023-519641 A International Publication No. 2019 / 077682

[0004] If there is map information with accurate positioning and complete information, including obstacles, in advance, the purpose of autonomous flight can be achieved simply by executing a pre-created autonomous flight mission on-site. However, if the position recorded on the map information is different from the actual position, or if there are obstacles such as standing trees that are not on the map information, it is necessary to verify whether the mission is safe to execute before or during the autonomous flight, and in some cases edit the mission.

[0005] Therefore, the present disclosure proposes an information processing device and an information processing method that support mission verification.

[0006] According to the present disclosure, there is provided an information processing device having a mission information acquisition unit that acquires mission information related to an automatic flight position, an image processing unit that superimposes the automatic flight position on a plurality of viewpoint images each having a different viewpoint, and a display control unit that presents each viewpoint image on which the automatic flight position is superimposed as a verification image for verifying the automatic flight position. Also, according to the present disclosure, there is provided an information processing method in which information processing of the information processing device is executed by a computer.

[0007] FIG. 1 is a diagram illustrating the planning of an automatic flight position using commercially available map information. FIG. 2 is a diagram illustrating a method for verifying a flight route using a verification image. FIG. 3 is a diagram illustrating a method for correcting a flight route using a verification image. FIG. 4 is a diagram illustrating an example of the configuration of a flight control system. FIG. 5 is a diagram illustrating an example of the flow of verification work. FIG. 6 is a diagram illustrating a modified example of the method for verifying a flight route. FIG. 7 is a diagram illustrating a modified example of the method for verifying a flight route. FIG. 8 is a diagram illustrating verification and correction of a flight route using sensor information. FIG. 9 is a diagram illustrating an example of the configuration of a flight control system. FIG. 10 is a diagram illustrating an example of the flow of verification work.

[0008] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and redundant description will be omitted.

[0009] The description will be given in the following order: [1. First embodiment] [1-1. Planning of automatic flight position using commercially available map information] [1-2. Verification and correction of automatic flight position using verification images] [1-3. Example of system configuration of flight control system] [1-4. Verification work flow] [1-5. Modification] [1-6. Effects] [2. Second embodiment] [2-1. Verification and correction of automatic flight position using sensor information] [2-2. Example of system configuration of flight control system] [2-3. Verification work flow]

[0010] [1. First embodiment] [1-1. Planning of automatic flight position using commercially available map information] Fig. 1 is a diagram showing planning of automatic flight position using commercially available map information. In the following description, a drone 10 is used as an example of an aircraft that performs automatic flight.

[0011] The automatic flight position FP is planned based on a commercially available map MP created using a Global Navigation Satellite System (GNSS) or the like. The user sets the automatic flight position FP on the map MP. The automatic flight position FP may be a flight route connecting two points or a flight point represented by a point (e.g., a point in space for photographing an object from a specific distance and angle). In the example of FIG. 1, a flight route TR from a start point ST to an end point ED is shown as the automatic flight position FP.

[0012] The flight route TR is set around the target TG. The target TG is an object that is the target of the automatic flight. In the example of FIG. 1 , the purpose (mission) of the automatic flight is to take an aerial photograph of the target TG. Examples of the target TG include man-made objects such as buildings, bridges, and temples, as well as natural objects such as islands, waterfalls, and rock walls. In the example of FIG. 1 , a high-rise building is shown as the target TG. The flight route TR is set as a route that flies from one end of the wall of the target TG to the other end at a certain distance from the wall.

[0013] [1-2. Verification and Correction of Automatic Flight Position Using Verification Image] FIG. 2 is a diagram illustrating a method for verifying the automatic flight position FP (flight route TR) using a verification image VI.

[0014] In the present disclosure, to facilitate verification of the flight route TR, the flight route TR is displayed from multiple viewpoints. The flight route TR is superimposed on a viewpoint image IM depicting a target TG. An image obtained by superimposing the flight route TR on the viewpoint image IM is presented as a verification image VI. The verification image VI is used by a user to verify the safety of the flight route TR.

[0015] In the example of Fig. 2, a viewpoint image IM of the target TG viewed from a direction perpendicular to the flight route TR is acquired as a first viewpoint image IM1. A viewpoint image IM of the target TG viewed from a direction parallel to the flight route TR is acquired as a second viewpoint image IM2. In the example of Fig. 2, the first viewpoint image IM1 is an image of the wall surface of the target TG captured by the drone 10. The second viewpoint image IM2 is an image of the target TG captured from a position where the user's tablet 20 looks up at the flight route TR.

[0016] A verification image VI obtained by superimposing the flight route TR on the first viewpoint image IM1 is presented as a first verification image VI1 ("View 1"). A verification image VI obtained by superimposing the flight route TR on the second viewpoint image IM2 is presented as a second verification image VI2 ("View 2"). By displaying the flight route TR from multiple viewpoints, the flight route TR can be grasped three-dimensionally. As a result, the positional relationship between the flight route TR and the target TG becomes easier to understand, making it easier to verify the safety of the flight route TR.

[0017] FIG. 3 is a diagram illustrating a method for correcting the flight route TR using the verification image VI.

[0018] The user US can perform adjustment operations to correct the flight route TR on the display screen of the verification image VI. The adjustment operations can be performed using an input device such as a touch panel or a mouse. The results of the adjustment operations performed on one verification image VI are reflected in other verification images VI.

[0019] 2, the viewpoint ST and the end point ED of the flight route TR are positioned at positions that are shifted from their planned positions in a direction parallel to the flight route TR (depth direction) and in a direction perpendicular to the flight route TR (left / right direction). If the drone 10 flies based on the pre-planned flight route TR, it will collide with the target TG.

[0020] Such deviations are caused by errors between commercially available map information used in the pre-planning and the position information used during mission execution (when estimating the drone's 10's position). Specifically, commercially available map information is created based on GNSS or the like. Meanwhile, the drone 10 uses Real Time Kinematic (RTK) to estimate its position. RTK outputs position information using position information obtained from GNSS and corrected position information from a unique reference station installed on the ground. Therefore, higher position accuracy can be achieved than with commercially available map information.

[0021] The verification image VI shows an error in the positional relationship between the flight route TR and the background object due to an error in the positional information of the background object included in the viewpoint image IM and an error in the positional information of the flight route TR. The user US can recognize such an error based on the verification image VI. The user US can appropriately adjust the flight route TR while viewing the verification image VI.

[0022] In the example of FIG. 3 , an adjustment operation is performed on the second verification image VI2 ("View 2"). In the second verification image VI2, the flight route TR is adjusted left and right. The user US uses a touch operation to move the viewpoint ST and end point ED of the flight route TR to positions that do not interfere with the target TG. The flight route TR displayed in the first verification image VI1 ("View 1") is adjusted in conjunction with the adjustment operation on the second verification image VI2.

[0023] As a result of the adjustment operation, the positions of the start point ST and end point ED of the flight route TR are appropriately adjusted to positions that do not interfere with the target TG on the second verification image VI2. However, the adjustment results in the depth direction cannot be fully recognized on the second verification image VI2. Looking at the first verification image VI1, it can be seen that the positions of the start point ST and end point ED are still shifted in the depth direction. The user US can perform adjustment operations on the first verification image VI1 and readjust the depth positions of the start point ST and end point ED. The adjustments made to the first verification image VI1 are also reflected in the second verification image VI2. The user US makes sufficient adjustments in both the left-right direction and the depth direction while checking both the first verification image VI1 and the second verification image VI2.

[0024] [1-3. System Configuration Example of Flight Control System] FIG. 4 is a diagram showing an example of the configuration of the flight control system 1.

[0025] The flight control system 1 includes a drone 10 and a tablet 20. The drone 10 performs automatic flight based on a flight plan and executes a given mission. In this example, the mission assigned to the drone 10 is to take an aerial photograph of a target TG along a flight route TR. The tablet 20 functions as an information processing device that supports the user US in planning the flight of the drone 10. The tablet 20 presents the above-described verification image VI before or during the execution of the mission, prompting the user US to confirm the flight route TR and make appropriate corrections.

[0026] For example, the drone 10 has a drone position acquisition unit 11 and a drone camera image acquisition unit 12. The tablet 20 has a tablet camera image acquisition unit 21, a tablet position acquisition unit 22, a user operation input unit 23, a mission information acquisition unit 24, an image processing unit 25, a display control unit 28, and an image display unit 29.

[0027] The drone position acquisition unit 11 acquires the position of the drone 10 detected using RTK as the drone position. The drone camera image acquisition unit 12 acquires an image taken by the camera of the drone 10 (an FPV (First Person View) image of the drone 10) as a drone camera image. The drone camera image can be used as a viewpoint image IM for verifying the flight route TR.

[0028] The tablet camera image acquisition unit 21 acquires an image captured by the camera of the tablet 20 as a tablet camera image. The tablet camera image can be used as a viewpoint image IM for verifying the flight route TR. The tablet position acquisition unit 22 acquires the position of the tablet 20 detected using RTK as the tablet position. The user operation input unit 23 accepts operations by the user US on the tablet 20. The user operation input unit 23 includes a touch panel, a mouse, a keyboard, etc.

[0029] The mission information acquisition unit 24 acquires mission information related to the flight route TR. For example, the mission information includes the waypoints along which the drone 10 flies, the position of the flight route TR, the position of the planar area over which the drone 10 flies, the position of the target TG and the position of the area photographed by the camera of the drone 10, etc. The mission information is input by the user US via the user operation input unit 23.

[0030] The image processing unit 25 superimposes the flight route TR on a plurality of viewpoint images IM, each having a different viewpoint. In the example of FIG. 2, a first viewpoint image IM1 and a second viewpoint image IM2 are used as the viewpoint images IM. The image processing unit 25 has a first mission superimposed image processing unit 26 and a second mission superimposed image processing unit 27. The first mission superimposed image processing unit 26 superimposes the flight route TR on the first viewpoint image IM1 to generate a first verification image VI1. The second mission superimposed image processing unit 27 superimposes the flight route TR on the second viewpoint image IM2 to generate a second verification image VI2. High-precision position information acquired using RTK is used for the superimposition process.

[0031] In the example of Fig. 2, the viewpoint image IM includes an image of the target TG captured from a specific viewpoint. However, the viewpoint image IM may also include a map image such as a planar map or a 3D map, or an image reproducing the target TG from a specific viewpoint based on sensor information (e.g., information from an obstacle sensor using infrared or ultrasonic waves). In the example of Fig. 2, a drone camera image and a tablet camera image are used as the viewpoint image IM, but an image captured with a handheld transmitter such as a smartphone may also be used as the viewpoint image IM.

[0032] The display control unit 28 controls the image display unit 29 to present the individual viewpoint images IM (first viewpoint image IM1, second viewpoint image IM2) on which the flight route TR is superimposed as verification images VI (first verification image VI1, second verification image VI2) for verifying the flight route TR. The image display unit 29 includes a display such as an LCD (Liquid Crystal Display) or an OLED (Organic Light Emitting Diode).

[0033] The user operation input unit 23 can accept an adjustment operation of the flight route TR performed by the user US on the verification image VI. The mission information acquisition unit 24 adjusts the flight route TR based on the adjustment operation. The display control unit 28 reflects the adjustment of the flight route TR performed individually based on each adjustment operation on all other verification images VI.

[0034] [1-4. Verification Work Flow] FIG. 5 is a diagram showing an example of the verification work flow.

[0035] The first mission superimposed image processing unit 26 acquires a drone camera image (FPV image of the drone 10) and the drone position from the drone 10 (step S1). The second mission superimposed image processing unit 27 acquires a tablet camera image and the tablet position (step S2). The first mission superimposed image processing unit 26 and the second mission superimposed image processing unit 27 acquire mission information (step S3). The drone camera image is used as a first viewpoint image IM1. The tablet camera image is used as a second viewpoint image IM2.

[0036] The image processing unit 25 acquires the flight route TR from the mission information. The first mission superimposition image processing unit 26 superimposes the flight route TR on the first viewpoint image IM1 to generate a first verification image VI1. The second mission superimposition image processing unit 27 superimposes the flight route TR on the second viewpoint image IM2 to generate a second verification image VI2 (step S4). The display control unit 28 presents the first verification image VI1 and the second verification image VI2 to the user US via the image display unit 29 (step S5).

[0037] The mission information acquisition unit 24 determines whether the user US has performed an adjustment operation via the user operation input unit 23 (step S6). If an adjustment operation has been performed (step S6: Yes), the mission information acquisition unit 24 updates the mission information (step S8) and returns to step S1. If no adjustment operation has been performed (step S6: No), the mission information acquisition unit 24 causes the drone 10 to perform automatic flight based on the acquired mission information (step S7).

[0038] [1-5. Modifications] FIGS. 6 to 8 are diagrams illustrating modifications of the verification method for the flight route TR.

[0039] In the above-described embodiment, the automated flight mission was planned and verified using drone camera images and tablet camera images. In this modified example, the automated flight mission is planned and verified using a planar map FM and drone camera images. In the example of Figure 6, the target TG is a bridge BR. Using the planar map FM, a mission is planned to move up and down between the main girder and the water surface at a position a certain distance away from the bridge pier.

[0040] In the example of Figure 7, a planar map FM showing a bridge BR from above is acquired as a first viewpoint image IM1. A flight route TR is superimposed on the first viewpoint image IM1 to generate a first verification image VI1 ("View 1"). A second viewpoint image IM2 is acquired as a drone camera image of the bridge BR taken from the side. A second verification image VI2 ("View 2") is generated by superimposing the flight route TR on the second viewpoint image IM2. By checking the mission from multiple viewpoints, the accurate positional relationship between the bridge BR and the flight route TR can be determined.

[0041] For example, in the first verification image VI1, the flight route TR is located at the planned position (below the main girder). However, in the second verification image VI2, the end point ED of the flight route TR is located at a position deviated from the plan (a position interfering with the main girder). By verifying the mission using multiple verification images VI, deviations from the plan that cannot be identified using only one verification image VI become clear.

[0042] The user US can adjust the flight route TR based on the second verification image VI2. In the example of FIG. 8, an adjustment operation is performed on the second verification image VI2, and the end point ED of the flight route TR is moved below the main beam. The adjustment results for the second verification image VI2 are automatically reflected in the first verification image VI1. Although the adjustment results in the depth direction cannot be seen on the second verification image VI2, the adjustment results in the depth direction can be recognized by checking the first verification image VI1. In the example of FIG. 8, the position of the end point ED has been incorrectly adjusted horizontally. The user US can perform an adjustment operation on the first verification image VI1 to readjust the depth position of the end point ED.

[0043] [1-6. Effects] The tablet 20 has a mission information acquisition unit 24, an image processing unit 25, and a display control unit 28. The mission information acquisition unit 24 acquires mission information related to the automatic flight position FP. The image processing unit 25 superimposes the automatic flight position FP on a plurality of viewpoint images IM, each having a different viewpoint. The display control unit 28 presents each viewpoint image IM on which the automatic flight position FP is superimposed as a verification image VI for verifying the automatic flight position FP. In the information processing method disclosed herein, the processing of the tablet 20 is executed by a computer.

[0044] This configuration allows the automatic flight position FP to be verified from multiple viewpoints, making it easier to verify the safety of the mission.

[0045] The tablet 20 has a user operation input unit 23. The user operation input unit 23 accepts an adjustment operation of the automatic flight position FP performed by the user US on the verification image VI.

[0046] According to this configuration, the automatic flight position FP can be adjusted based on the verification results.

[0047] The mission information acquisition unit 24 adjusts the automatic flight position FP based on the adjustment operation. The display control unit 28 reflects the adjustment of the automatic flight position FP individually performed based on each adjustment operation in all other verification images VI.

[0048] With this configuration, the automatic flight positions FP of all verification images VI are adjusted in conjunction with one another with a single adjustment operation, allowing the adjustment results to be grasped three-dimensionally, enabling highly accurate adjustment.

[0049] The viewpoint image IM includes an image of an object (target TG) related to the purpose of automatic flight taken from a specific viewpoint, or an image of an object reproduced from a specific viewpoint based on sensor information.

[0050] This configuration allows direct comparison of the pre-planned automatic flight position FP with the actual position of the object.

[0051] The verification image VI shows an error in the positional relationship between the automatic flight position FP and the background object due to an error in the positional information of the background object contained in the viewpoint image IM and an error in the positional information of the automatic flight position FP.

[0052] According to this configuration, the deviation of the automatic flight position FP caused by the error between the map information used in the advance plan and the position information used when executing the mission can be presented to the user US, and appropriate action can be taken.

[0053] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0054] [2. Second Embodiment] [2-1. Verification and Correction of Automatic Flight Position Using Sensor Information] In the first embodiment, the flight route TR was verified based on an image (drone camera image, tablet camera image) of the target TG taken from a specific viewpoint. However, the verification method is not limited to this. The flight route TR may also be verified based on an image of the target TG reproduced from a specific viewpoint based on sensor information. Figure 9 is a diagram illustrating verification and correction of the flight route TR using sensor information.

[0055] In the example of Figure 9, the target TG is a bridge BR. A mission is planned to move up and down between the main girder and the water surface at a position a certain distance from the bridge pier. In the example of Figure 9, a planar map FM showing the bridge BR from above is acquired as a first viewpoint image IM1. A first verification image VI1 ("View 1") is generated by superimposing a flight route TR on the first viewpoint image IM1. A second viewpoint image IM2 is acquired as an image of the space below the bridge BR reproduced from sensor information. A second verification image VI2 ("View 2") is generated by superimposing the flight route TR on the second viewpoint image IM2. The sensors include, for example, obstacle sensors using infrared or ultrasonic waves.

[0056] The above configuration also allows the mission to be confirmed from multiple viewpoints. The accurate positional relationship between the bridge BR and the flight route TR can be grasped, allowing for verification of an appropriate flight route TR.

[0057] 10 is a diagram showing an example of the configuration of the flight control system 2 of this embodiment. The following description will focus on the differences from the flight control system 1 of the first embodiment shown in FIG.

[0058] The drone 30 has a drone 3D obstacle information acquisition unit 31 instead of the drone camera image acquisition unit 12. The drone 3D obstacle information acquisition unit 31 acquires information measured by an obstacle sensor or the like as sensor information. The first mission superimposed image processing unit 26 acquires information (obstacle information) regarding the position and shape of objects around the drone 30 based on the sensor information and the drone position. In the example of Figure 9, 3D information of the space (flight space) below the main girder where the drone 30 is scheduled to fly is acquired as obstacle information.

[0059] The first-mission superimposed image processing unit 26 generates an image that reproduces the flight space below the main girder from a specific viewpoint (the side of the bridge BR in the example of FIG. 9 ) based on the obstacle information, and acquires the generated image as a first viewpoint image IM1. The first-mission superimposed image processing unit 26 superimposes the flight route TR on the first viewpoint image IM1 to generate a first verification image VI1.

[0060] The tablet 40 has a map information acquisition unit 41 instead of the tablet camera image acquisition unit 21 and the tablet position acquisition unit 22. The map information acquisition unit 41 acquires a planar map FM of the bridge BR having highly accurate position information. The second mission superimposed image processing unit 27 generates a second viewpoint image IM2 using the planar map FM of the bridge BR. The second mission superimposed image processing unit 27 superimposes the flight route TR on the second viewpoint image IM2 to generate a second verification image VI2. The second mission superimposed image processing unit 27 may also superimpose an image of the flight space reproduced based on obstacle information on the second viewpoint image IM2 (see FIG. 9 ).

[0061] 2-3. Verification Work Flow FIG. 11 is a diagram showing an example of the verification work flow.

[0062] The first mission superimposed image processing unit 26 acquires sensor information and drone position from the drone 30 (step S11). The second mission superimposed image processing unit 27 acquires map information related to a planar map FM of the bridge BR (step S12). The first mission superimposed image processing unit 26 and the second mission superimposed image processing unit 27 acquire mission information (step S13). The first mission superimposed image processing unit 26 recreates the flight space below the main girder from a specific viewpoint based on the sensor information and the drone position, and acquires the recreated image as a first viewpoint image IM1. The planar map FM of the bridge BR is used as a second viewpoint image IM2.

[0063] The image processing unit 25 acquires the flight route TR from the mission information. The first mission superimposition image processing unit 26 superimposes the flight route TR on the first viewpoint image IM1 to generate a first verification image VI1. The second mission superimposition image processing unit 27 superimposes the flight route TR on the second viewpoint image IM2 to generate a second verification image VI2 (step S14). The display control unit 28 presents the first verification image VI1 and the second verification image VI2 to the user US via the image display unit 29 (step S15).

[0064] [Additional Notes] The present technology may also be configured as follows. (1) An information processing device having: a mission information acquisition unit that acquires mission information related to an automatic flight position; an image processing unit that superimposes the automatic flight position on a plurality of viewpoint images, each having a different viewpoint; and a display control unit that presents each viewpoint image on which the automatic flight position is superimposed as a verification image for verifying the automatic flight position. (2) The information processing device according to (1), further having a user operation input unit that accepts an adjustment operation of the automatic flight position performed by a user on the verification image. (3) The information processing device according to (2), wherein the mission information acquisition unit adjusts the automatic flight position based on the adjustment operation, and the display control unit reflects the adjustment of the automatic flight position performed individually based on each of the adjustment operations in all other verification images. (4) The information processing device according to any one of (1) to (3), wherein the viewpoint image includes an image of an object related to the purpose of automatic flight taken from a specific viewpoint, or an image of the object reproduced from the specific viewpoint based on sensor information. (5) The information processing device according to any one of (1) to (4), wherein the verification image indicates an error in the positional relationship between the automatic flight position and the background object due to an error in position information of the background object included in the viewpoint image and an error in position information of the automatic flight position. (6) An information processing method executed by a computer, comprising: acquiring mission information related to the automatic flight position; superimposing the automatic flight position on a plurality of viewpoint images each having a different viewpoint; and presenting each viewpoint image on which the automatic flight position is superimposed as a verification image for verifying the automatic flight position.

[0065] 20, 40 Tablet (information processing device) 23 User operation input unit 24 Mission information acquisition unit 25 Image processing unit 28 Display control unit FP Automatic flight position IM Viewpoint image VI Verification image

Claims

1. An information processing device having: a mission information acquisition unit that acquires mission information related to an automatic flight position; an image processing unit that superimposes the automatic flight position on a plurality of viewpoint images each having a different viewpoint; and a display control unit that presents each viewpoint image on which the automatic flight position is superimposed as a verification image for verifying the automatic flight position.

2. An information processing device as described in claim 1, further comprising a user operation input unit that accepts adjustment operations of the automatic flight position performed by a user on the verification image.

3. The information processing device described in claim 2, wherein the mission information acquisition unit adjusts the automatic flight position based on the adjustment operation, and the display control unit reflects the adjustment of the automatic flight position made individually based on each of the adjustment operations in all other verification images.

4. The information processing device according to claim 1, wherein the viewpoint image includes an image of an object related to the purpose of automatic flight taken from a specific viewpoint, or an image of the object reproduced from the specific viewpoint based on sensor information.

5. The information processing device described in claim 1, wherein the verification image shows an error in the positional relationship between the automatic flight position and the background object due to an error in the positional information of the background object contained in the viewpoint image and an error in the positional information of the automatic flight position.

6. An information processing method executed by a computer, comprising: acquiring mission information regarding an automatic flight position; superimposing the automatic flight position on a plurality of viewpoint images, each having a different viewpoint; and presenting each viewpoint image on which the automatic flight position is superimposed as a verification image for verifying the automatic flight position.

Images