Control equipment, control methods, unmanned aerial vehicles, information processing equipment, information processing methods and programs

Abstract

This invention relates to control devices, control methods, unmanned aerial vehicles (UAVs), information processing devices, information processing methods, and programs configured to reflect user intentions and reduce damage in the event of a malfunction in an unmanned aerial vehicle (UAV). A first aspect of the invention involves a control device that controls the movement of an UAV during a crash based on control commands generated from an image taken by the UAV showing the crash location. The invention can be applied to devices controlling unmanned aerial vehicles (UAVs) where movement is controlled to deviate from the crash location in the event of a malfunction in the machine itself.

Description

Technical Field

[0001] This technology specifically relates to control devices, control methods, unmanned aerial vehicles, information processing devices, information processing methods, and programs that can reflect the user's intentions and reduce damage in the event of a malfunction in the unmanned aerial vehicle. Background Technology

[0002] In recent years, small, unmanned aerial vehicles (UAVs) that can be remotely controlled have attracted much attention. If an UAV malfunctions during flight and becomes uncontrollable, it can crash and cause a major accident. Therefore, various technologies have been proposed to reduce damage in the event of a malfunction.

[0003] For example, PTL 1 proposes a technique for estimating the fall radius and detecting people based on images obtained by taking pictures of the lower part of the aircraft, and for controlling the aircraft so that the position of the person and the fall radius do not overlap.

[0004] [List of Citations]

[0005] [Patent Literature]

[0006] [PTL 1]

[0007] WO 2017 / 033976 Summary of the Invention

[0008] [Technical Issues]

[0009] In the technology disclosed in PTL 1, it is necessary to predetermine the detection target. Furthermore, since the control of the aircraft is delegated to self-sustaining control, it is impossible to reflect the values ​​and intentions of the user operating the unmanned aerial vehicle as needed.

[0010] This technology is designed specifically for this situation and reflects the user's intent, enabling the reduction of damage in the event of a malfunction in an unmanned aerial vehicle.

[0011] [Problem Solution]

[0012] The control device according to a first aspect of the present technology includes a control unit that controls the movement of the unmanned aerial vehicle during the crash based on control commands generated in response to an image taken by the unmanned aerial vehicle showing the crash location.

[0013] The unmanned aerial vehicle according to a second aspect of the present technology includes: an imaging unit that captures images of the surrounding environment; and a control unit that controls the movement of the unmanned aerial vehicle during a crash based on control commands generated from images captured by the imaging unit that indicate the crash location.

[0014] The information processing apparatus according to a third aspect of the present technology includes: a display control unit that displays an image taken by an unmanned aerial vehicle (UAV) showing the crash location; a generation unit that generates control commands for controlling the movement of the UAV based on the image; and a transmission unit that transmits the control commands to the UAV.

[0015] In a first aspect of this technology, the movement of the unmanned aerial vehicle during the crash is controlled according to control commands generated in response to an image taken by the unmanned aerial vehicle showing the crash location.

[0016] In a second aspect of this technology, the surrounding environment is photographed, and the movement of the unmanned aerial vehicle during the crash is controlled based on control commands generated from the images showing the crash location.

[0017] In a third aspect of this technology, an image taken by an unmanned aerial vehicle (UAV) showing the crash location is displayed, control commands for controlling the movement of the UAV are generated based on the image, and the control commands are transmitted to the UAV. Attached Figure Description

[0018] Figure 1 This is a diagram illustrating an example configuration of a control system according to an embodiment of the present technology.

[0019] Figure 2 It is a diagram showing the state of the fall.

[0020] Figure 3 This is a diagram showing an example of a composite image.

[0021] Figure 4 This is a diagram illustrating an example of user interaction.

[0022] Figure 5 This is a diagram illustrating the relationship between images taken by unmanned aerial vehicles and composite images displayed on smartphones.

[0023] Figure 6 This is a diagram showing another configuration example of the controller.

[0024] Figure 7 This is a block diagram illustrating an example configuration of an unmanned aerial vehicle.

[0025] Figure 8 This is a diagram illustrating an example of a method for estimating the crash location.

[0026] Figure 9 This is a diagram showing an example of an image used to generate a synthetic image.

[0027] Figure 10This is a diagram illustrating an example of a method for synthesizing captured images.

[0028] Figure 11 This is a diagram illustrating an example of the orientation in the composite image coordinate system as represented by control commands.

[0029] Figure 12 This is a diagram illustrating an example of controlling the movement of an unmanned aerial vehicle that has received control commands.

[0030] Figure 13 This is a diagram showing an example of the motor's rotation direction in fall mode.

[0031] Figure 14 This is a diagram showing an example of controller configuration.

[0032] Figure 15 This is a flowchart illustrating a crash damage mitigation process for unmanned aerial vehicles.

[0033] Figure 16 This is a flowchart illustrating the composite image display processing of the controller.

[0034] Figure 17 This is a diagram illustrating an example of a user's actions to specify the fall location.

[0035] Figure 18 This is a diagram illustrating an example of the movement of an unmanned aerial vehicle that has received control commands.

[0036] Figure 19 This is a block diagram illustrating a configuration example of an unmanned aerial vehicle that receives control commands to specify the crash location.

[0037] Figure 20 This is an image showing an example of objects near the crash site of an unmanned aerial vehicle.

[0038] Figure 21 This is a diagram showing an example of a composite image in which object detection results are synthesized.

[0039] Figure 22 This is a block diagram illustrating an example configuration of an unmanned aerial vehicle that detects objects appearing in a synthetic image.

[0040] Figure 23 This is a flowchart illustrating a crash damage mitigation process performed by an unmanned aerial vehicle.

[0041] Figure 24 This is a diagram illustrating an example of actions used to avoid collisions.

[0042] Figure 25 This is a diagram showing an example of a composite image illustrating the direction of movement according to a plan.

[0043] Figure 26 This is a diagram illustrating an example of a user's instruction to move in a direction different from the planned direction.

[0044] Figure 27 This is a diagram illustrating an example of the movement of an unmanned aerial vehicle.

[0045] Figure 28 This is a block diagram illustrating an example configuration of an unmanned aerial vehicle with autonomous avoidance capabilities.

[0046] Figure 29 This is a flowchart illustrating a crash damage mitigation process performed by an unmanned aerial vehicle.

[0047] Figure 30 This is a diagram showing an example of a smartphone configuration.

[0048] Figure 31 This is a block diagram illustrating an example configuration of computer hardware. Detailed Implementation

[0049] Embodiments of this technology will be described below. Note that the descriptions will be given in the following order.

[0050] 1. An example of the first control of an unmanned aerial vehicle

[0051] 2. Configuration of each device

[0052] 3. Operation of each device

[0053] 4. An example of the second control system for unmanned aerial vehicles

[0054] 5. An example of third control for unmanned aerial vehicles

[0055] 6. An example of the fourth control system for unmanned aerial vehicles

[0056] 7. Changes

[0057] <An Example of First Control for an Unmanned Aerial Vehicle>

[0058] Figure 1 This is a diagram illustrating an example configuration of a control system according to an embodiment of the present technology.

[0059] Figure 1 The control system includes an unmanned aerial vehicle 1 and a controller 2.

[0060] Unmanned aerial vehicle 1 is a so-called drone, and moves (flies) based on signals from controller 2. Unmanned aerial vehicle 1 can be an autonomously moving aircraft. Figure 1In the scene, unmanned aerial vehicle 1 moves above house O1, and car O2 is parked next to house O1.

[0061] Unmanned aerial vehicle 1 is an aircraft equipped with a camera. Images taken by unmanned aerial vehicle 1 while it is moving are transmitted to controller 2 via wireless communication, as indicated by the dashed arrow.

[0062] Images transmitted from the unmanned aerial vehicle 1 can be moving or still images. Images can also be transmitted via wired communication instead of wireless communication.

[0063] Controller 2, acting as the transmitter of unmanned aerial vehicle 1, receives images transmitted from unmanned aerial vehicle 1 and displays them on smartphone 3. Figure 1 In the example, controller 2 is a device that uses the display of a smartphone 3 attached to the housing of controller 2 as a display of the destination of images taken by the camera of unmanned aerial vehicle 1.

[0064] In this way, even when the drone 1 is moving away from the user, the user can operate the controller 2 and control the drone 1 while viewing the image displayed on the smartphone 3.

[0065] Incidentally, accidents such as propeller detachment or motor failure may occur. In such cases, unmanned aerial vehicle 1 will crash.

[0066] exist Figure 1 The control system continues imaging even as the drone 1 crashes, and presents the images from the crash to the user. The user can control the movement of the drone 1 by viewing the images during the crash.

[0067] Figure 2 It is a diagram showing the state of the fall.

[0068] When a part of the aircraft malfunctions, the unmanned aircraft 1 crashes while spinning, for example, as indicated by the dotted arrow A1.

[0069] During the crash, the unmanned aerial vehicle 1 combines information indicating the crash location with images captured by the camera and transmits the composite image to the controller 2.

[0070] The smartphone 3 attached to the controller 2 displays a composite image transmitted from the unmanned aerial vehicle 1.

[0071] Because information indicating the crash location is displayed, the user can move the drone 1 in a direction that avoids house O1 and car O2 while viewing a composite image displayed on smartphone 3. Figure 2In the example, unmanned aerial vehicle 1 avoids house O1 and car O2 as indicated by solid arrow A3, in response to an operation performed by the user, such as moving unmanned aerial vehicle 1 in the direction indicated by the empty arrow A2.

[0072] Figure 3 This is a diagram showing an example of a composite image.

[0073] like Figure 3 As shown, the display 3A of the smartphone 3 shows a composite image of the crash location of the unmanned aerial vehicle 1. The crash location of the unmanned aerial vehicle 1 is an estimated location that takes into account surrounding environmental information such as wind speed.

[0074] For example, the crash location of unmanned aerial vehicle 1 is represented by crash location image P. Figure 3 In the example, the fall location image P is an image in which a cross is arranged in a circle. The center of the cross that makes up the fall location image P represents the fall location. The fall location can be represented by a point or a region.

[0075] exist Figure 3 In the example, the crash site is on house O1. Car O2 is to the right of house O1. If drone 1 crashes like this, it will collide with house O1.

[0076] Because the crash location and objects such as house O1 are displayed, users can indicate the direction of movement of UAV 1 while viewing the composite image, either in a direction that avoids the object that UAV 1 is expected to collide with, or in a direction that minimizes damage even if UAV 1 collides with an object. Figure 3 In the example, the left direction in the composite image is considered to be the direction to avoid objects or reduce damage.

[0077] Figure 4 This is a diagram illustrating an example of user interaction.

[0078] By viewing the display of the composite image, such as Figure 4 As shown, the user tilts the joystick of controller 2 to the left and indicates the left direction (i.e., the direction avoiding house O1 and car O2) in the composite image as the direction of movement. Figure 4 In the composite image shown, the shaded arrows indicate the direction in which the unmanned aerial vehicle 1 is expected to move, rather than the image superimposed on the composite image.

[0079] The images taken by the unmanned aerial vehicle 1 during its crash are images whose imaging range changes rapidly. Therefore, it is difficult for a user to determine the possible crash location of the unmanned aerial vehicle 1 or the object that the unmanned aerial vehicle 1 may collide with simply by viewing the captured images.

[0080] By displaying a composite image showing the crash location P, the user can allow the unmanned aerial vehicle 1 to crash in a location where the damage is considered to be minimized, based on the user's values ​​and the circumstances at the time of the crash.

[0081] In addition, it may be possible to avoid collisions with objects that are difficult to detect, such as those detected by object detection systems.

[0082] During the crash of UAV 1, the user can refer to the orientation in the composite image to specify the direction in which UAV 1 is moved. Based on the attitude of UAV 1 at that time, the user-specified orientation is converted into an orientation in the coordinate system of UAV 1, and the direction of movement is controlled.

[0083] Figure 5 This is a diagram showing the relationship between images taken by the unmanned aerial vehicle 1 and composite images displayed on the smartphone 3.

[0084] like Figure 5 As shown on the left, when a malfunction occurs in the aircraft, the unmanned aerial vehicle 1 crashes while spinning. Figure 5 The status of unmanned aerial vehicle 1 at each time point from T1 to T7 is shown. The dotted arrow A11, pointing directly downwards, indicates the direction of fall when there is no user intervention.

[0085] The blank triangle shown on the bottom surface of the unmanned aerial vehicle 1 represents the field of view of the camera provided on the unmanned aerial vehicle 1. For example, at time T1, the direction of the camera's field of view is directly downward.

[0086] In this case, as indicated by the tip of arrow #1, the camera's field of view overlaps entirely with the display area of ​​the composite image.

[0087] Figure 5 The rectangle F shown on the right represents the display area of ​​the composite image (the area displayed on monitor 3A). The display area of ​​the composite image is set such that, for example, the fall location is centered. The shaded area represents the range of the camera's field of view.

[0088] At time T1, the composite image is displayed as is using the images taken by the unmanned aerial vehicle 1.

[0089] At time T2, the camera's field of view points diagonally downwards to the right. In this case, as indicated by the tip of arrow #2, the camera's field of view partially overlaps with the display area of ​​the composite image.

[0090] At time T2, the composite image is displayed using the portion of the image captured by UAV 1 that overlaps with the display range of the composite image. Within the entire composite image, areas outside the range displayed using the image captured at time T2 are displayed using images captured at, for example, times prior to time T2.

[0091] At time T3, the camera's field of view is diagonally upward to the right. In this case, as indicated by the tip of arrow #3, the camera's field of view does not overlap with the display area of ​​the composite image.

[0092] At time T3, the composite image is displayed using images taken at times prior to time T3.

[0093] If the camera's field of view does not match the display range of the composite image, the composite image may not be displayed.

[0094] The viewpoints of the cameras at times T4 and T5 do not overlap with the display range of the composite image, as indicated by the tips of arrows #4 and #5, as in the viewpoint of the camera at time T3.

[0095] At times T4 and T5, the composite image is displayed using images taken before times T4 and T5, respectively.

[0096] The viewpoint of the time T6 camera partially overlaps with the display range of the composite image, as indicated by the tip of arrow #6.

[0097] At time T6, the composite image is displayed using the portion of the image captured by UAV 1 that overlaps with the display area of ​​the composite image.

[0098] The field of view of the Time T7 camera overlaps with the display area of ​​the composite image, as indicated by the tip of arrow #7.

[0099] At time T7, the composite image is displayed as is using images taken by unmanned aerial vehicle 1.

[0100] In this way, the display 3A of the smartphone 3 viewed by the user continues to show only the area including the crash location from the images taken during the fall. In this way, even when the drone 1 is falling while rotating, the user can indicate the direction of movement of the drone 1 while viewing a composite image showing the crash location.

[0101] In the above text, we assume that controller 2 is the controller in which smartphone 3 is attached, but it can be another type of controller.

[0102] Figure 6 This is a diagram showing another configuration example for controller 2.

[0103] Figure 6 The controller 2 shown in Figure A is a controller in which a display 2A is provided in the housing. The user can steer the unmanned aerial vehicle 1 by viewing a composite image displayed on the display 2A, etc. For example, the direction of movement is indicated by the direction the user tilts the joystick, and the amount of movement is indicated by the amount of tilting the joystick.

[0104] like Figure 6 As shown in B, the smartphone 3 itself can be used as a transmitter. The user can steer the unmanned aerial vehicle 1 by performing operations such as swipes by viewing a composite image displayed on a monitor 3A, etc.

[0105] <Configuration for each device>

[0106] Configuration of Unmanned Aerial Vehicle 1

[0107] Figure 7 This is a block diagram showing an example configuration of unmanned aerial vehicle 1.

[0108] like Figure 7 As shown, the unmanned aerial vehicle 1 includes a sensor 11 and an information processing unit 12.

[0109] Sensor 11 includes a wind speed sensor 21, an imaging sensor 22, a position sensor 23, and an IMU (inertial measurement unit) 24. The imaging sensor 22 is provided in a camera mounted on the unmanned aerial vehicle 1.

[0110] The wind speed sensor 21 detects and outputs a wind speed vector that includes wind direction and wind speed (wind volume).

[0111] The imaging sensor 22 is configured with an image sensor, a stereo camera, a ToF (Time of Flight) sensor, and a LiDER (Light Detection and Ranging, Laser Imaging Detection and Ranging).

[0112] The image sensor constituting the imaging sensor 22 images the surrounding environment and outputs image data.

[0113] Additionally, for example, the stereo camera constituting the imaging sensor 22 calculates the distance to each object captured in the image based on the image obtained through imaging, and outputs distance information. The distance to the object can be detected by a ToF sensor or the like.

[0114] The position sensor 23 is configured with a GPS (Global Positioning System) sensor, a barometer, etc. The position sensor 23 receives radio waves from satellites, performs positioning, and outputs the position information of the unmanned aerial vehicle 1.

[0115] IMU 24 includes an accelerometer, a gyroscope, and a magnetometer. IMU 24 measures the speed, acceleration, and magnetic field strength of the unmanned aerial vehicle 1 and outputs these as IMU information.

[0116] The information processing unit 12 includes a wind speed vector acquisition unit 31, an image acquisition unit 32, a position information acquisition unit 33, an IMU information acquisition unit 34, an internal state acquisition unit 35, a self-position and motion estimation unit 36, a crash location estimation unit 37, a fault and crash determination unit 38, an image synthesis unit 39, a data transmission unit 40, a data receiving unit 41, and an aircraft control unit 42.

[0117] The wind speed vector acquisition unit 31 acquires the wind speed vector output from the wind speed sensor 21 and outputs the wind speed information representing the acquired wind speed vector to the fall location estimation unit 37.

[0118] The image acquisition unit 32 acquires image data and distance information output from the imaging sensor 22 and outputs it to its own position and motion estimation unit 36. The image acquisition unit 32 then outputs the image data to the image synthesis unit 39.

[0119] The position information acquisition unit 33 acquires the position information output from the position sensor 23 and outputs it to its own position and motion estimation unit 36.

[0120] IMU information acquisition unit 34 acquires the IMU information output from IMU 24 and outputs it to its own position and motion estimation unit 36.

[0121] The internal status acquisition unit 35 acquires the output values ​​of the current monitor, voltage monitor, encoder, etc. of the unmanned aerial vehicle 1 as information indicating the internal status, and outputs the output values ​​to the fault and crash determination unit 38.

[0122] The self-position and motion estimation unit 36 ​​calculates the flight state of the unmanned aerial vehicle 1 based on image data and distance information supplied by the image acquisition unit 32, position information supplied by the position information acquisition unit 33, and IMU information supplied by the IMU information acquisition unit 34. The flight state includes the position, attitude, velocity, angular velocity, acceleration, and angular acceleration of the unmanned aerial vehicle 1.

[0123] The self-position and motion estimation unit 36 ​​estimates the inertial force and gravity applied to the unmanned aerial vehicle 1 based on the flight state.

[0124] The self-position and motion estimation unit 36 ​​outputs information representing the flight state and the inertial forces and gravity applied to the unmanned aerial vehicle 1 as the self-position and motion estimation result. The self-position and motion estimation result output from the self-position and motion estimation unit 36 ​​is supplied to the crash location estimation unit 37, the fault and crash determination unit 38, the image synthesis unit 39, and the aircraft control unit 42.

[0125] The crash location estimation unit 37 estimates the crash location of the crashing unmanned aerial vehicle 1 based on the wind speed information provided by the wind speed vector acquisition unit 31 and the self-position and motion estimation results provided by the self-position and motion estimation unit 36.

[0126] Figure 8 This is a diagram illustrating an example of a method for estimating the crash location.

[0127] like Figure 8 The blank arrow in the diagram indicates that an inertial force is applied to the unmanned aerial vehicle 1, which malfunctioned during flight, in the direction corresponding to the direction of movement to that point, and a downward gravitational force is also applied. Additionally, a wind force corresponding to the wind direction is applied. The inertial force, gravity, and wind force applied to the unmanned aerial vehicle are collectively referred to as external forces.

[0128] The crash location estimation unit 37 estimates the crash location based on flight status and external forces. Figure 8 In this context, position C1, which is offset from the location directly below where the malfunction occurred, is estimated as the fall location.

[0129] The crash location estimation unit 37 sets a region with a predetermined shape centered on the estimated crash location as the estimated crash location range. The estimated crash location range is set as a region that gradually narrows as the unmanned aerial vehicle 1 approaches the ground.

[0130] Return to Figure 7 As described above, the fall location estimation unit 37 outputs the fall location estimation result, representing the estimated fall location and the estimated fall location range, to the image synthesis unit 39.

[0131] The fault and crash determination unit 38 determines the fault or crash of the unmanned aerial vehicle 1 based on the information representing the internal state supplied by the internal state acquisition unit 35 and the self-position and motion estimation results supplied by the self-position and motion estimation unit 36.

[0132] Specifically, the fault and crash determination unit 38 uses internal state data to diagnose faults, such as motor malfunctions, propeller damage, and foreign object entanglement in the unmanned aerial vehicle 1. The fault and crash determination unit 38 diagnoses faults based on the deviation between the actual internal state and the internal state assumed in the absence of a fault.

[0133] In this way, the fault and crash determination unit 38 detects that a fault hindering movement has occurred and the unmanned aerial vehicle 1 cannot move as expected and begins to crash. For fault and crash determination, rule-based determination can be performed, or a model obtained through machine learning can be used.

[0134] The fault and crash determination unit 38 outputs fault and crash determination information to the image synthesis unit 39 and the aircraft control unit 42. The fault and crash determination information includes, for example, information indicating whether a fault has occurred and information indicating the faulty part of the unmanned aerial vehicle 1.

[0135] The image synthesis unit 39 generates a composite image by combining an image of the fall location with an image of the fall location captured therein. To generate the composite image, image data supplied by the image acquisition unit 32, self-position and motion estimation results supplied by the self-position and motion estimation unit 36, fall location estimation results supplied by the fall location estimation unit 37, and fault and fall determination information supplied by the fault and fall determination unit 38 are used.

[0136] Figure 9 This is a diagram showing an example of an image used to generate a synthetic image.

[0137] Figure 9 The state of unmanned aerial vehicle 1 at each time point from T1 to T3 is shown. Here, it is assumed that unmanned aerial vehicle 1 is equipped with a camera. The range indicated by the dashed line represents the range of the camera's field of view at each time.

[0138] Images were captured at times T1 to T3 with the camera pointing downwards, diagonally downwards to the left, and further to the left than at time T2, respectively, and images with characteristics derived from... Figure 9 Images P11 to P13 show the perspective of the trapezoidal indicator.

[0139] For example, at time T3, a composite image is generated based on images P11 to P13 taken in this manner.

[0140] exist Figure 9 In the example, at time T3, the estimated crash location range is represented by an ellipse. Among the captured images P11 to P13, the area captured at time T3, which is the estimated crash location range and the region directly below UAV 1, is used to generate the composite image.

[0141] Figure 10 This is a diagram illustrating an example of a method for synthesizing captured images.

[0142] When viewing the images P11 to P13 taken from directly above, the viewpoint of each image is represented as follows: Figure 10The shape shown. Figure 10 The shape of each image is shown when viewed from directly above relative to a plane representing the ground.

[0143] In the image synthesis unit 39, images P11 to P13 are captured by projecting and transforming the position and attitude of the unmanned aerial vehicle 1 and the internal and external parameters of the camera relative to a plane representing the ground.

[0144] In the image synthesis unit 39, the images P11 to P13 captured after projection transformation are stitched together, and a range including the position corresponding to the position directly below time T3 and the estimated fall position range is cut out. Figure 10 In the example, the area of ​​the image indicated by the dashed line is cut out as the cut-out image P21.

[0145] In the image synthesis unit 39, a composite image is generated by synthesizing a fall position image representing the estimated fall position range and a position corresponding to the position directly below the fall position onto the cropped image P21. The composite image generated in this way is output from the image synthesis unit 39 to... Figure 7 Data transmission unit 40.

[0146] When multiple cameras are provided in the unmanned aerial vehicle 1, images captured by overlapping estimated crash location ranges and viewpoints are used to generate synthetic images.

[0147] The data transmission unit 40 transmits the composite image supplied by the image synthesis unit 39 to the controller 2. The composite image transmitted by the data transmission unit 40 is displayed on the display 3A of the smartphone 3 and is used to indicate the direction of movement of the unmanned aerial vehicle 1.

[0148] The data receiving unit 41 receives control commands from the controller 2 that represent user operations, and outputs the control commands to the aircraft control unit 42. For example, the control command indicates a direction given by a user viewing a composite image.

[0149] The aircraft control unit 42 determines whether a fault has occurred based on fault and crash determination information supplied from the fault and crash determination unit 38, and sets the operating mode.

[0150] The unmanned aerial vehicle 1 has two operating modes: flight mode and crash mode. Flight mode is the operating mode set when no malfunction occurs, while crash mode is the operating mode set when a malfunction has occurred.

[0151] The aircraft control unit 42 controls the movement of the unmanned aircraft 1 in response to control commands supplied from the data receiving unit 41.

[0152] When the operating mode is flight mode, the aircraft control unit 42 controls the position and attitude in the aircraft coordinate system.

[0153] The aircraft coordinate system represents the coordinate system within the unmanned aerial vehicle 1. When the operating mode is flight mode, the user of the operating controller 2 performs operations on the aircraft coordinate system to control the movement of the unmanned aerial vehicle 1.

[0154] On the other hand, when the operating mode is the fall mode, the aircraft control unit 42 performs control considering the location of the fault based on the self-position and motion estimation results supplied from the self-position and motion estimation unit 36 ​​and the fault and fall determination information supplied from the fault and fall determination unit 38.

[0155] In this scenario, the aircraft control unit 42 converts the direction of the user's command, represented by the composite image coordinate system, into the direction of the aircraft coordinate system and executes the control. Since the direction of the user's command, represented by the control command, is the direction for viewing the composite image, it is represented as the direction of the composite image coordinate system, which is the coordinate system within the composite image.

[0156] That is, when the operation mode is the fall mode, the user of the operation controller 2 performs operations on the composite image coordinate system to control the movement of the unmanned aerial vehicle 1.

[0157] Figure 11 This is a diagram illustrating an example of the orientation in the composite image coordinate system as represented by control commands.

[0158] When an object to be avoided is present within the estimated crash location area, a control command instructing the unmanned aerial vehicle (UAV) to move to the left relative to the composite image is transmitted to the UAV 1 via an operation such as tilting the joystick of controller 2 to the left. At this time, as... Figure 11 Arrow A12 indicates that a control command is transmitted to unmanned aerial vehicle 1, instructing it to move to the left in the composite image coordinate system.

[0159] By indicating movement in the depth direction in the composite image, it is possible to implement control to accelerate the fall.

[0160] Figure 12 This is a diagram illustrating an example of controlling the movement of an unmanned aerial vehicle 1 that has received control commands.

[0161] When indicated on the composite image as reference Figure 11 When the aircraft moves to the left as described, the aircraft control unit 42 controls the aircraft to ensure that the actual crash location is as follows: Figure 12 To the left of the estimated crash location range indicated by the middle arrow A13.

[0162] Specifically, the aircraft control unit 42 uses a predetermined transformation matrix to convert the user-specified direction into an orientation in the aircraft coordinate system at time T3 and controls the aircraft.

[0163] When it is impossible to move in any direction due to motor failure or other reasons, the aircraft control unit 42 takes into account the position and attitude of the aircraft and controls the rotation direction of the operable motors so that thrust is generated in the direction specified by the user.

[0164] Figure 13 This is a diagram showing an example of the motor's rotation direction in fall mode.

[0165] Figure 13 The image shows the state of unmanned aerial vehicle 1 as it falls while rotating at each time point from T1 to T9. Additionally, as indicated by the blank arrows, it is assumed that the user viewing the composite image has specified a leftward movement.

[0166] When there is only one operable motor, the aircraft control unit 42 rotates the motor in the direction in which the dot product of the thrust vector generated when the motor is rotated and the direction vector pointing to the direction specified by the user becomes positive.

[0167] exist Figure 13 In this example, suppose that, of two propeller motors, referring to the correct vertical direction at time T1, the left propeller motor malfunctions, and only the right propeller motor is operable. The solid arrow shown near the right propeller motor represents the thrust vector generated by the motor's forward rotation. The dotted arrow represents the thrust vector generated by the motor's reverse rotation. Here, forward rotation refers to rotation in the direction that generates buoyancy under normal conditions.

[0168] At time T1 when the upper surface of the aircraft is pointing straight upwards, the inner product of the thrust vector generated by the rotating motor and the direction vector facing the direction specified by the user becomes 0, so the aircraft control unit 42 does not rotate the motor.

[0169] On the other hand, between times T2 and T4, when the upper surface of the aircraft faces left, the aircraft control unit 42 rotates the motor in the positive direction to generate a thrust vector, the dot product of which is positive with a direction vector pointing in the direction specified by the user. In this way, the unmanned aircraft 1 will fall while moving to the left.

[0170] At time T5, when the upper surface of the aircraft is pointing straight down, the inner product of the thrust vector generated by the rotating motor and the direction vector facing the direction specified by the user becomes 0, so the aircraft control unit 42 does not rotate the motor.

[0171] Between times T6 and T8, when the upper surface of the aircraft faces to the right, the aircraft control unit 42 rotates the motor in the opposite direction to generate a thrust vector whose inner product with the direction vector pointing in the user-specified direction is positive. In this way, the unmanned aerial vehicle 1 will fall while moving to the left.

[0172] At time T9, when the upper surface of the aircraft is pointing straight upwards, the inner product of the thrust vector generated by the rotating motor and the direction vector pointing in the direction specified by the user is 0, so the aircraft control unit 42 does not rotate the motor.

[0173] As described above, the aircraft control unit 42 can shift the crash position to the left by controlling the rotation direction of the operable motors to generate a leftward thrust specified by the user.

[0174] • Controller 2 configuration

[0175] Figure 14 This is a diagram illustrating an example configuration of controller 2. A smartphone 3 is connected to controller 2 via wired or wireless communication.

[0176] like Figure 14 As shown, the controller 2 includes an information processing unit 51 and an input unit 52.

[0177] The information processing unit 51 includes a data receiving unit 61, a data display control unit 62, an input acquisition unit 63, and a data transmission unit 64.

[0178] The data receiving unit 61 receives the composite image transmitted from the unmanned aerial vehicle 1 and outputs it to the data display control unit 62.

[0179] The data display control unit 62 outputs the composite image supplied by the data receiving unit 61 to the display 3A of the smartphone 3 and displays it.

[0180] The input acquisition unit 63 acquires the instruction information output from the input unit 52 and outputs the instruction information to the data transmission unit 64. The instruction information represents the direction and amount of movement specified by the user.

[0181] The data transmission unit 64 transmits the instruction information supplied by the input acquisition unit 63 as control commands to the unmanned aerial vehicle 1.

[0182] The input unit 52 is configured with a joystick, touch panel, etc. The input unit 52 detects the user's operation and outputs instruction information based on the detected user operation.

[0183] <Operation of each device>

[0184] The operation of each device with the above configuration will be described here.

[0185] Operation of Unmanned Aerial Vehicle 1

[0186] First, refer to Figure 15 The flowchart describes the crash damage mitigation process for unmanned aerial vehicle 1.

[0187] For example, when unmanned aerial vehicle 1 begins flight, it begins Figure 15 The crash damage mitigation process. The unmanned aerial vehicle 1 was in flight mode at the start of the process.

[0188] In step S1, the information processing unit 12 acquires sensor data supplied by the sensor 11. Specifically, the wind speed vector acquisition unit 31, the image acquisition unit 32, the location information acquisition unit 33, and the IMU information acquisition unit 34 acquire wind speed information, image data, distance information, location information, and IMU information, respectively.

[0189] In step S2, the self-position and motion estimation unit 36 ​​estimates its own position and motion estimation results based on image data, distance information, position information and IMU information.

[0190] In step S3, the fault and crash determination unit 38 determines whether the unmanned aerial vehicle 1 has malfunctioned or crashed based on information indicating its internal state and its own position and motion estimation results.

[0191] If it is determined in step S3 that the unmanned aerial vehicle 1 has malfunctioned or crashed, then the process proceeds to step S4.

[0192] In step S4, the crash location estimation unit 37 estimates the crash location of the unmanned aerial vehicle 1 during the crash based on wind speed information and its own position and motion estimation results.

[0193] In step S5, the image synthesis unit 39 generates a synthesized image by combining the image of the fall location with an image showing the fall location, based on the image data, its own position and motion estimation results, the fall location estimation results, and the fault and fall determination information.

[0194] In step S6, the aircraft control unit 42 sets the operating mode to fall mode based on the fault and fall determination information.

[0195] In step S7, the data transmission unit 40 transmits the composite image to the controller 2.

[0196] If the composite image is transmitted to the controller 2 in step S7, or if it is determined in step S3 that the unmanned aerial vehicle 1 has neither malfunctioned nor crashed, then the process proceeds to step S8.

[0197] In step S8, the data receiving unit 41 determines whether a control command has been received from the controller 2.

[0198] If it is determined in step S8 that no control command has been received, the process returns to step S1 and subsequent processing is performed.

[0199] On the other hand, if it is determined in step S8 that a control command has been received, the process proceeds to step S9. The control command is supplied from the data receiving unit 41 to the aircraft control unit 42.

[0200] In step S9, the aircraft control unit 42 determines whether the operating mode is a fall mode.

[0201] If it is determined in step S9 that the operating mode is fall mode, then the process proceeds to step S10.

[0202] In step S10, the aircraft control unit 42 converts the direction represented by the control command from the direction in the composite image coordinate system to the direction in the aircraft coordinate system based on the composite image.

[0203] In step S11, the aircraft control unit 42 controls the motors of the unmanned aerial vehicle 1 based on its own position and motion estimation results and fault and crash determination information, taking into account the fault location, and moves the unmanned aerial vehicle 1 in the desired direction corresponding to the control command.

[0204] On the other hand, if it is determined in step S9 that the operating mode is not the fall mode, then the process proceeds to step S12.

[0205] In step S12, the aircraft control unit 42 controls the motors of the unmanned aerial vehicle 1 and moves the unmanned aerial vehicle 1 in the desired direction corresponding to the control command.

[0206] The above process is repeated during the flight or crash of the unmanned aerial vehicle 1.

[0207] • Operation of controller 2

[0208] Next, we will refer to Figure 16 The flowchart describes the composite image display processing of controller 2.

[0209] In step S21, the data receiving unit 61 of the controller 2 receives the composite image transmitted from the unmanned aerial vehicle 1.

[0210] In step S22, the data display control unit 62 outputs the synthesized image to the display 3A of the smartphone 3 and displays it.

[0211] In step S23, the input unit 52 receives the user's operation and generates instruction information.

[0212] In step S24, the input acquisition unit 63 acquires instruction information.

[0213] In step S25, the data transmission unit 64 transmits the instruction information as control commands to the unmanned aerial vehicle 1.

[0214] Through the above processing, users can allow the unmanned aerial vehicle 1 to crash to a location where damage is considered to be reduced during the crash, based on their values ​​and the circumstances at the time of the crash.

[0215] <An example of second control for unmanned aerial vehicles>

[0216] Users can specify the crash location instead of the crash direction. In this case, the drone 1 is controlled to crash at the user-specified location.

[0217] Figure 17 This is a diagram illustrating an example of a user's actions to specify the fall location.

[0218] like Figure 17 As shown, the user specifies the fall location by touching the display 3A, which has a touch panel. Figure 17 In the text, the locations indicated by the blank cross, where there is no house O1 or car O2, are specified by the user.

[0219] In this scenario, the smartphone 3, acting as a transmitter, transmits control commands to the unmanned aerial vehicle 1, indicating a location specified by the user.

[0220] Figure 18 This is a diagram illustrating an example of the movement of an unmanned aerial vehicle 1 that has received control commands.

[0221] Figure 18 The position C11 indicated by the blank cross on the left represents the actual three-dimensional position in space corresponding to the position on the composite image specified by the user.

[0222] Upon receiving the control command, the unmanned aerial vehicle 1 controlled itself to crash at position C11, as indicated by the dotted line.

[0223] Figure 19 This is a block diagram illustrating a configuration example of an unmanned aerial vehicle that receives control commands to specify the crash location.

[0224] In addition to the connection between the crash location estimation unit 37 and the aircraft control unit 42, Figure 19 Configuration and reference of the unmanned aerial vehicle 1 shown Figure 7 The configurations described are identical. Duplicate explanations will be omitted as appropriate.

[0225] Figure 19 The aircraft control unit 42 is supplied with the same information as the crash location estimation result supplied from the crash location estimation unit 37 to the image synthesis unit 39.

[0226] The aircraft control unit 42 calculates the crash position in three-dimensional space specified by the user based on the control command for specifying the crash position supplied from the data receiving unit 61. The aircraft control unit 42 performs feedback control based on the difference between the crash position specified by the user and the crash position estimation result, and moves the unmanned aerial vehicle 1 to the crash position specified by the user.

[0227] As described above, the user can allow the drone 1 to crash at a designated location based on their values ​​and circumstances. For example, if the user expects the drone 1 to crash between the objects shown in the composite image, they can allow the drone 1 to crash at that desired location.

[0228] <An example of third control for unmanned aerial vehicles>

[0229] Object detection results, which detect objects appearing in a synthetic image, can be combined with the synthetic image to provide information to assist the user.

[0230] Figure 20 This is a diagram showing an example of objects near the crash site of unmanned aerial vehicle 1.

[0231] exist Figure 20 In the example, the drone 1 crashes on the house O1. Car O2 is parked next to house O1, and a person O3 is standing next to car O2.

[0232] The unmanned aerial vehicle 1 detects objects in an image showing the crash location, specifically houses O1, cars O2, and people O3. It detects specific objects such as houses, cars, and people.

[0233] The unmanned aerial vehicle 1 generates a composite image in which information fragments representing house O1, car O2 and person O3 are synthesized, and transmits the composite image to controller 2 to display it on smartphone 3.

[0234] Figure 21 This is a diagram showing an example of a composite image in which object detection results are synthesized.

[0235] like Figure 21 As shown, the composite image is displayed on the display 3A of the smartphone 3. In the composite image, house O1, car O2, and person O3 are captured side by side.

[0236] exist Figure 21In the synthesized image, the rectangular information R1 in which an L-shaped line is synthesized is displayed as surrounding the house O1. The rectangular information R1 represents the area where the house is detected on the synthesized image. Above the rectangular information R1, the character information "House" indicating the presence of a house is displayed.

[0237] Similarly, the rectangular information R2 is displayed as surrounding the car O2. The rectangular information R2 represents the area where the car is detected on the synthesized image. Above the rectangular information R2, the character information "Car" indicating the presence of a car is displayed.

[0238] The rectangular information R3 is displayed as surrounding the person O3. The rectangular information R3 represents the area where the person is detected on the synthesized image. Above the rectangular information R3, the character information "Person" representing a person is displayed.

[0239] The user can identify the type of object that the unmanned aerial vehicle 1 may collide with by viewing the rectangular information and character information displayed on the synthesized image.

[0240] Information indicating the recommended moving direction for the user can be displayed on the synthesized image based on the object detection result and the estimated fall range.

[0241] Figure 22 It is a block diagram showing a configuration example of the unmanned aerial vehicle 1 that detects objects appearing in the synthesized image.

[0242] In addition to providing the object detection unit 101, Figure 22 the configuration of the unmanned aerial vehicle 1 shown in Figure 7 is the same as the configuration described in the reference. Repeated explanations will be appropriately omitted.

[0243] The image synthesis unit 39 outputs the image in which the fall position is photographed to the object detection unit 101. For example, the object detection unit 101 is supplied with the image in which the fall position is photographed, which is generated by the image synthesis unit 39, as described in the reference Figure 10 described.

[0244] The image synthesis unit 39 generates a synthesized image by synthesizing the rectangular information and character information together with the fall position image and the image showing the fall position based on the object detection result supplied from the object detection unit 101.

[0245] The object detection unit 101 detects objects on the image supplied from the image synthesis unit 39. The object detection unit 101 outputs the object detection result to the image synthesis unit 39.

[0246] Here, referring to the Figure 23 flowchart, the fall damage mitigation process performed by the unmanned aerial vehicle 1 having the above configuration will be described.

[0247] The processing in steps S51 to S56 is respectively related to Figure 15 The processing of steps S1 to S6 is the same. That is, when the unmanned aerial vehicle 1 malfunctions, an image showing the crash location is synthesized, and the operating mode of the unmanned aerial vehicle 1 is set to crash mode.

[0248] In step S57, the object detection unit 101 detects the object on the image of the fall location it captured.

[0249] In step S58, the image synthesis unit 39 generates a synthesized image based on the object detection results by combining the rectangle information and character information together with the image showing the fall location and the image showing the fall location.

[0250] The processing in steps S59 to S64 is respectively related to Figure 15 Steps S7 to S12 are processed in the same way. That is, the synthesized image is transmitted to the controller 2, and the movement of the unmanned aerial vehicle 1 is controlled according to the user's operation.

[0251] As described above, the user can identify the type of object that the unmanned aerial vehicle 1 may collide with, and can allow the unmanned aerial vehicle 1 to crash into a location where the damage is considered to be reduced, based on the user's values ​​and the circumstances at the time of the crash.

[0252] <An example of fourth control for unmanned aerial vehicles>

[0253] Action plans can be formulated for the unmanned aerial vehicle 1 based on object detection results, and the unmanned aerial vehicle 1 can be autonomously controlled to move according to the planned actions. For example, actions are planned to avoid collisions with objects in synthetic images.

[0254] Figure 24 This is a diagram illustrating an example of actions used to avoid collisions.

[0255] like Figure 24 As shown, when the location on house O1 is the crash site, the unmanned aerial vehicle 1 detects house O1, car O2, and person O3 based on an image showing the crash site and plans the actions necessary to avoid collisions with these objects. For example, as indicated by the shaded arrows, it plans actions that would allow the aircraft to crash in front of house O1, car O2, and person O3 and performs autonomous avoidance maneuvers.

[0256] Figure 25 This is a diagram showing an example of a composite image illustrating the direction of movement according to a plan.

[0257] like Figure 25 As shown in the image, arrows indicating the planned direction of movement are displayed in the composite image. Figure 25 Other displays and references to the composite image shown Figure 21 The description of the composite image is displayed the same.

[0258] If the planned actions of the unmanned aerial vehicle 1 do not align with the user's values ​​and circumstances, the user who has viewed the composite image can instruct it to move in a direction different from the planned direction, such as... Figure 26 As shown in the image.

[0259] exist Figure 26 In the example, the direction of autonomous obstacle avoidance is represented by the downward direction indicated by the shaded arrow, while the user indicates to move to the left. The control command representing the user-indicated direction is transmitted to the unmanned aerial vehicle 1.

[0260] Figure 27 This is a diagram illustrating an example of the movement of unmanned aerial vehicle 1.

[0261] When the user specifies a direction different from the autonomous avoidance direction, such as reference Figure 26 As described, the unmanned aerial vehicle 1 prioritizes user commands, such as... Figure 27 The blank arrow in the diagram indicates the direction and controls the aircraft to move in the direction specified by the user.

[0262] In this way, users can intervene in the autonomous obstacle avoidance of the unmanned aerial vehicle 1.

[0263] Instead of simply prioritizing user instructions, the direction of autonomous avoidance can be combined with the direction indicated by the user to plan new actions.

[0264] To make it easier for users to determine the direction of movement, information indicating the direction in which the unmanned aerial vehicle 1 can be easily moved can be displayed. For example, wind direction and wind speed can be displayed as information indicating the direction in which the unmanned aerial vehicle 1 can be easily moved.

[0265] It can display the time until the unmanned aerial vehicle 1 crashes.

[0266] Figure 28 This is a block diagram illustrating a configuration example of an unmanned aerial vehicle 1 with autonomous obstacle avoidance capabilities.

[0267] In addition to providing an avoidance action generation unit 111, Figure 28 Configuration and reference of the unmanned aerial vehicle 1 shown Figure 22 The configurations described are identical. Duplicate explanations will be omitted as appropriate.

[0268] The aircraft control unit 42 controls the unmanned aerial vehicle 1 according to the action plan for autonomous avoidance supplied from the avoidance action generation unit 111. When a control command representing a user instruction is received, the aircraft control unit 42 controls the unmanned aerial vehicle 1 with the user's operation taking priority, as described above.

[0269] The object detection unit 101 calculates the three-dimensional position of the detected object in an image of its fall location. Distance information to the ground and distance information to the object are appropriately used to calculate the three-dimensional position. The object detection unit 101 outputs information representing the object's three-dimensional position and the object detection result to the avoidance action generation unit 111.

[0270] The self-position and motion estimation results are supplied from the self-position and motion estimation unit 36 ​​to the avoidance action generation unit 111. In addition, the avoidance action generation unit 111 is supplied with the fall position estimation results from the fall position estimation unit 37 and the fault and fall determination information from the fault and fall determination unit 38.

[0271] The avoidance action generation unit 111 plans the actions necessary to avoid a collision with an object shown in the synthetic image based on information supplied from each unit. Information representing the actions planned by the avoidance action generation unit 111 is supplied to the aircraft control unit 42.

[0272] When a moving object is shown in a synthetic image, the position of the moving object when the unmanned aerial vehicle 1 collides with the ground or an object can be predicted by the avoidance action generation unit 111. In this case, the predicted position of the moving object is used to plan actions.

[0273] Here, for reference Figure 29 The flowchart will describe the crash damage mitigation procedures performed by the unmanned aerial vehicle 1 with the above configuration.

[0274] The processing steps S101 to S109 are respectively related to Figure 23 The processing steps S51 to S59 are the same. That is, the operating mode of the unmanned aerial vehicle 1 is set to fall mode, and the composite image is transmitted to the controller 2.

[0275] In step S110, the data receiving unit 41 determines whether a control command has been received from the controller 2.

[0276] If it is determined in step S110 that no control command has been received, then the process proceeds to step S111.

[0277] In step S111, the aircraft control unit 42 determines whether the operating mode is a fall mode.

[0278] If it is determined in step S111 that the operating mode is not the fall mode, then the process returns to step S101 and subsequent processing is performed.

[0279] On the other hand, if it is determined in step S111 that the operating mode is a fall mode, then the process proceeds to step S112.

[0280] In step S112, the avoidance action generation unit 111 plans the action for autonomous avoidance. After the action for autonomous avoidance is planned, in step S115, the movement direction of the unmanned aerial vehicle 1 is controlled according to the planned action.

[0281] On the other hand, if it is determined in step S110 that a control command has been received, then the process proceeds to step S113.

[0282] The processing in steps S113 to S116 is respectively related to Figure 23 Steps S61 to S64 are processed in the same way. That is, it is determined whether the operating mode is fall mode, and based on the determination result, the movement of the unmanned aerial vehicle 1 is controlled according to the user's operation.

[0283] As described above, the unmanned aerial vehicle 1 can autonomously take actions to avoid collisions with the objects shown in the synthetic image.

[0284] <Change>

[0285] System Configuration

[0286] Although the sensor 11 and the information processing unit 12 are located in the unmanned aerial vehicle 1 ( Figure 7 However, some of the functions of the information processing unit 12 can be implemented in any device, such as the controller 2 or the smartphone 3.

[0287] Figure 30 This is a diagram showing an example configuration of a smartphone 3.

[0288] like Figure 30 As shown, the information processing unit 151 is implemented in the smartphone 3. Besides the display unit 161, input acquisition unit 162, control command generation unit 163, and data transmission unit 164, Figure 30 Configuration and reference of the information processing unit 151 shown Figure 7 The configuration of the information processing unit 12 of the described unmanned aerial vehicle 1 is the same. Repetitive explanations will be omitted where appropriate.

[0289] The information processing unit 151 acquires sensor data, including captured images and internal status, from the sensors 11 installed in the unmanned aerial vehicle 1 and various devices.

[0290] Display unit 161 causes display 3A to display the composite image supplied from image synthesis unit 39.

[0291] The input acquisition unit 162 outputs instruction information to the control command generation unit 163, which represents the content of the operation performed by the user on the display 3A with a touch panel.

[0292] The control command generation unit 163 is supplied with self-position and motion estimation results from the self-position and motion estimation unit 36 ​​and fault and fall determination information from the fault and fall determination unit 38. Additionally, a composite image is supplied from the image synthesis unit 39 to the control command generation unit 163.

[0293] The control command generation unit 163 determines whether a fault exists based on fault and crash determination information and sets the operating mode of the smartphone 3. If a fault has occurred in the unmanned aerial vehicle 1, the operation mode is set to crash mode, and if no fault has occurred, the operation mode is set to flight mode.

[0294] The control command generation unit 163 generates a control command indicating the direction specified by the user based on the instruction information supplied by the input acquisition unit 162. When the operating mode is flight mode, the instruction information supplied by the input acquisition unit 162 is used as is.

[0295] On the other hand, when the operating mode is the fall mode, the control command generation unit 163 converts the instruction information about the coordinate system of the composite image supplied by the input acquisition unit 162 into the aircraft coordinate system based on the composite image to generate control commands.

[0296] The control command generation unit 163 outputs its own position and motion estimation results, fault and fall determination information, and control commands to the data transmission unit 164.

[0297] The data transmission unit 164 transmits the self-position and motion estimation results, fault and crash determination information and control commands supplied by the control command generation unit 163 to the unmanned aerial vehicle 1.

[0298] As mentioned above, Figure 7 Part of the configuration of the unmanned aerial vehicle 1 shown can be provided on the smartphone 3.

[0299] • Examples of computers

[0300] The above series of processes can be performed by hardware or software. When the series of processes are performed by software, the program constituting the software is installed from a program recording medium onto a computer with embedded dedicated hardware, a general-purpose personal computer, or the like.

[0301] Figure 31This is a block diagram illustrating an example of the hardware configuration of a computer that executes the program to perform the series of processes described above.

[0302] The central processing unit (CPU) 1001, read-only memory (ROM) 1002 and random access memory (RAM) 1003 are interconnected via bus 1004.

[0303] The input / output interface 1005 is also connected to the bus 1004. An input unit 1006, including a keyboard and mouse, and an output unit 1007, including a display and speakers, are connected to the input / output interface 1005. A storage unit 1008, including a hard disk or non-volatile memory, a communication unit 1009, including a network interface, and a driver 1010, driving the removable medium 1011, are connected to the input / output interface 1005.

[0304] In a computer with this configuration, for example, the CPU 1001 loads a program stored in the storage unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and executes the program to perform the series of processes described above.

[0305] The program executed by the CPU 1001 is recorded on, for example, a removable medium 1011, or provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting to be installed in the storage unit 1008.

[0306] A program executed by a computer may be a program that performs processing sequentially in the process described in this specification, or it may be a program that performs processing at necessary time intervals (such as in parallel or when invoked).

[0307] In this specification, a system is a collection of multiple components (devices, modules (components), etc.), and all components may or may not be located in the same housing. Therefore, multiple devices housed in different housings and connected via a network, and a device in which multiple modules are housed in one housing, are both systems.

[0308] Furthermore, the beneficial effects described in this specification are merely exemplary and not limiting, and other beneficial effects may be obtained.

[0309] The embodiments of this technology are not limited to the above embodiments, and various changes can be made without departing from the spirit of this technology.

[0310] For example, this technology can be configured as cloud computing, where one function is shared and processed jointly by multiple devices via a network.

[0311] In addition, the corresponding steps described in the above flowchart can be performed by one device or by multiple devices in a shared manner.

[0312] Furthermore, in cases where a single step includes multiple processes, those processes can be executed by a single device or by multiple devices in a shared manner.

[0313] <Example of configuration combinations>

[0314] This technology can be configured as follows.

[0315] (1) A control device comprising: a control unit that controls the movement of an unmanned aerial vehicle during a crash based on control commands generated in response to an image taken by the unmanned aerial vehicle showing the crash location.

[0316] (2) The control device according to (1) further includes: a synthesis unit that synthesizes information representing the crash location with an image taken by an imaging device installed on the unmanned aerial vehicle and generates a synthesized image for use by the user.

[0317] (3) The control device according to (2) further includes: an estimation unit that estimates the flight state of the unmanned aerial vehicle based on sensor data output by sensors installed on the unmanned aerial vehicle; and a crash position estimation unit that estimates the crash position based on the flight state and the external force applied to the unmanned aerial vehicle.

[0318] (4) The control device according to (3) further includes: a determining unit that detects the crash of the unmanned aerial vehicle based on at least one of the flight state and the internal state of the unmanned aerial vehicle.

[0319] (5) According to the control device of (3) or (4), wherein the synthesis unit generates a composite image by projecting and converting the image captured by the imaging device relative to a plane representing the ground based on the flight state and parameters of the imaging device, and by combining information representing the crash location with the image obtained by projection and conversion.

[0320] (6) The control device according to any one of (2) to (5), wherein the synthesis unit generates a composite image including information indicating the crash location and information indicating the location directly below the unmanned aerial vehicle.

[0321] (7) The control device according to any one of (1) to (6), wherein the control unit controls the movement of the unmanned aerial vehicle according to the direction or position on an image specified by the user.

[0322] (8) According to the control device described in (7), the control unit converts the direction specified by the user into the direction in the coordinate system of the unmanned aerial vehicle and controls the movement of the unmanned aerial vehicle.

[0323] (9) The control device according to (7), wherein the control unit controls the movement of the unmanned aerial vehicle based on the difference between the position specified by the user and the crash position.

[0324] (10) The control device according to any one of (2) to (9) further includes: a detection unit that detects an object appearing in an image captured by an imaging device, wherein the synthesis unit synthesizes object information representing the object detected by the detection unit into a synthesized image.

[0325] (11) The control device according to (10) further includes: an action planning unit that plans actions of the unmanned aerial vehicle to avoid contact with objects detected by the detection unit, wherein the control unit controls the movement of the unmanned aerial vehicle based on the planned actions and user operations.

[0326] (12) The control device according to (11), wherein the control unit takes into account the user’s operation to control the movement of the unmanned aerial vehicle.

[0327] (13) A control method comprising: allowing a control device to control the movement of an unmanned aerial vehicle during a crash based on a control command generated in response to an image taken by the unmanned aerial vehicle showing the crash location.

[0328] (14) A program for causing a computer to perform the following control: controlling the movement of an unmanned aerial vehicle during a crash based on control commands generated in response to an image taken by the unmanned aerial vehicle showing the crash location.

[0329] (15) An unmanned aerial vehicle, comprising: an imaging unit that captures images of the surrounding environment; and a control unit that controls the movement of the unmanned aerial vehicle during a crash based on control commands generated from images captured by the imaging unit that indicate the crash location.

[0330] (16) An information processing device, comprising: a display control unit that displays an image taken by an unmanned aerial vehicle (UAV) showing the crash location; a generation unit that generates control commands for controlling the movement of the UAV based on the image; and a transmission unit that transmits the control commands to the UAV.

[0331] (17) An information processing method for allowing an information processing device to perform: displaying an image taken by an unmanned aerial vehicle showing the crash location; generating a control command for controlling the movement of the unmanned aerial vehicle based on the image; and transmitting the control command to the unmanned aerial vehicle.

[0332] (18) A program for causing a computer to perform: displaying an image taken by an unmanned aerial vehicle showing the crash location; generating control commands for controlling the movement of the unmanned aerial vehicle in response to the image; and transmitting the control commands to the unmanned aerial vehicle.

[0333] [List of reference numerals]

[0334] 1. Unmanned aerial vehicle

[0335] 2 Controllers

[0336] 3 Smartphone

[0337] 11 Sensors

[0338] 12 Information Processing Units

[0339] 31 Wind speed vector acquisition unit

[0340] 32 Image Acquisition Unit

[0341] 33 Location Information Acquisition Unit

[0342] 34 IMU Information Acquisition Unit

[0343] 35 Internal Information Acquisition Unit

[0344] 36 Self-position and motion estimation units

[0345] 37. Fall location estimation unit

[0346] 38 Fault and Fall Determination Unit

[0347] 39 Image Synthesis Unit

[0348] 40 Data Transmission Units

[0349] 41 Data Receiving Unit

[0350] 42. Aircraft Control Unit

[0351] 101 Object Detection Unit

[0352] 111 Avoidance Action Generation Unit

[0353] 151 Information Processing Unit

[0354] 161 display units

[0355] 162 Input Acquisition Unit

[0356] 163 Control Command Generation Unit

[0357] 164 Data Transmission Units

Claims

1. A control device, comprising: The control unit controls the movement of the unmanned aerial vehicle during the crash based on control commands generated from images taken by the unmanned aerial vehicle showing the crash location; as well as The compositing unit combines information indicating the crash location with images captured by imaging equipment mounted on the unmanned aerial vehicle, generating a composite image used for user interaction. The crash location is centered within the display area of ​​the composite image. The control unit controls the movement of the unmanned aerial vehicle based on the orientation or position on the synthesized image specified by the user, and The control unit will transform the coordinate system of the synthesized image (either the direction or position specified by the user) to the coordinate system of the unmanned aerial vehicle (UAV), and control the movement of the UAV based on the transformed direction or position. The image includes a first image captured at a first time and a second image captured at a second time. The viewing angle of the first image overlaps entirely with the display area of ​​the composite image, and the viewing angle of the second image partially overlaps with the display area of ​​the composite image. The control unit immediately displays the synthesized image using the first image as is, and The control unit uses the portion of the second image that overlaps with the display range of the composite image at the second time to display the composite image, and uses an image taken at a time prior to the second time to display the range outside the range displayed using the second image throughout the composite image.

2. The control device according to claim 1, further comprising: An estimation unit that estimates the flight state of an unmanned aerial vehicle (UAV) based on sensor data output from sensors mounted on the UAV. as well as The crash location estimation unit estimates the crash location based on the flight state and the external forces applied to the unmanned aerial vehicle.

3. The control device according to claim 2, further comprising: A determining unit detects the crash of the unmanned aerial vehicle (UAV) based on at least one of the flight state and the internal state of the UAV.

4. The control device according to claim 2, wherein, The synthesis unit generates a composite image by projecting and transforming the image captured by the imaging device relative to a plane representing the ground based on the flight state and the parameters of the imaging device, and by combining the information representing the crash location with the image obtained through projection and transformation.

5. The control device according to claim 4, wherein, The synthesis unit generates a composite image that includes information indicating the crash location and information indicating the location directly below the unmanned aerial vehicle.

6. The control device according to claim 1, wherein, The control unit controls the movement of the unmanned aerial vehicle based on the difference between the user-specified location and the crash location.

7. The control device according to claim 1, further comprising: The detection unit detects objects appearing in an image captured by the imaging device, wherein... The synthesis unit synthesizes object information representing objects detected by the detection unit into a synthetic image.

8. The control device according to claim 7, further comprising: The action planning unit plans actions for the unmanned aerial vehicle to avoid contact with objects detected by the detection unit, wherein... The control unit controls the movement of the unmanned aerial vehicle based on planned actions and user input.

9. The control device according to claim 8, wherein, The control unit prioritizes user input to control the movement of the unmanned aerial vehicle.

10. A control method, comprising: The control equipment can control the movement of the unmanned aerial vehicle during the crash based on control commands generated in response to images taken by the unmanned aerial vehicle that show the crash location; as well as Information indicating the crash location is combined with images captured by imaging equipment mounted on the unmanned aerial vehicle to generate a composite image used for user interaction, wherein the crash location is centered within the display area of ​​the composite image. Among them, the movement of the unmanned aerial vehicle is controlled according to the direction or position on the synthetic image specified by the user, and The user will transform the coordinate system of the synthesized image from one of the directions and positions specified by the user to the coordinate system of the unmanned aerial vehicle (UAV), and control the movement of the UAV based on the transformed direction or position. The image includes a first image captured at a first time and a second image captured at a second time. The viewing angle of the first image overlaps entirely with the display area of ​​the composite image, and the viewing angle of the second image partially overlaps with the display area of ​​the composite image. The synthesized image is displayed as is using the first image immediately, and The composite image is displayed using the portion of the second image that overlaps with the display range of the composite image at a second time, and within the entire composite image, the range outside the range displayed using the second image is displayed using an image taken at a time prior to the second time.

11. A computer program product storing a program for causing a computer to perform the following controls: The movement of the unmanned aerial vehicle (UAV) during the crash is controlled based on control commands generated from images taken by the UAV showing the crash location; and Information indicating the crash location is combined with images taken by imaging equipment mounted on the unmanned aerial vehicle to generate a composite image used for user operations. The fall location is at the center of the display area of ​​the composite image. Among them, the movement of the unmanned aerial vehicle is controlled according to the direction or position on the synthetic image specified by the user, and The user will transform the coordinate system of the synthesized image from one of the directions and positions specified by the user to the coordinate system of the unmanned aerial vehicle (UAV), and control the movement of the UAV based on the transformed direction or position. The image includes a first image captured at a first time and a second image captured at a second time. The viewing angle of the first image overlaps entirely with the display area of ​​the composite image, and the viewing angle of the second image partially overlaps with the display area of ​​the composite image. The synthesized image is displayed as is using the first image immediately, and The composite image is displayed using the portion of the second image that overlaps with the display range of the composite image at a second time, and within the entire composite image, the range outside the range displayed using the second image is displayed using an image taken at a time prior to the second time.

12. An unmanned aerial vehicle, comprising: Imaging unit, which captures images of the surrounding environment; A control unit that controls the movement of the unmanned aerial vehicle during the descent based on control commands generated from images taken by an imaging unit showing the crash location; and The compositing unit combines information indicating the crash location with images captured by imaging equipment mounted on the unmanned aerial vehicle, generating a composite image used for user interaction. The crash location is centered within the display area of ​​the composite image. The control unit controls the movement of the unmanned aerial vehicle based on the orientation or position on the synthesized image specified by the user, and The control unit will transform the coordinate system of the synthesized image (either the direction or position specified by the user) to the coordinate system of the unmanned aerial vehicle (UAV), and control the movement of the UAV based on the transformed direction or position. The image includes a first image captured at a first time and a second image captured at a second time. The viewing angle of the first image overlaps entirely with the display area of ​​the composite image, and the viewing angle of the second image partially overlaps with the display area of ​​the composite image. The control unit immediately displays the synthesized image using the first image as is, and The control unit uses the portion of the second image that overlaps with the display range of the composite image at the second time to display the composite image, and uses an image taken at a time prior to the second time to display the range outside the range displayed using the second image throughout the composite image.

Images