Behavior controller and behavior control method
The behavior control device enhances drone flight accuracy by dynamically switching between different self-position estimation systems, addressing the limitations of conventional methods to achieve precise navigation and obstacle avoidance.
Patent Information
- Application Number
- JP2024151434
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2019-03-06
- Filing Date
- 2024-09-03
- Publication Date
- 2024-11-21
- Estimated Expiration
- 2040-02-21
AI Technical Summary
Conventional drones face challenges in performing accurate mission flights due to the limitations of existing self-position estimation systems, which either offer wide coverage but low accuracy or high accuracy but limited range, leading to difficulties in precise flight control.
A behavior control device and method that selectively uses multiple self-position estimation systems, such as vision-based and GPS-based systems, to enhance accuracy and adapt to varying flight conditions by switching between them based on indices like estimation accuracy and state identification.
Enables more appropriate behavioral control of drones by ensuring accurate positioning and flight control, allowing for precise navigation and obstacle avoidance, even in varying environments.
Smart Images

Figure 2024163227000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to a behavior control device and a behavior control method, and more particularly to a behavior control device and a behavior control method that are capable of realizing more appropriate behavior control. [Background technology]
[0002] Conventionally, moving objects such as unmanned aerial vehicles (UAVs), commonly known as drones, can fly, for example, in response to user operation, as well as according to a mission of passing through pre-specified waypoints (hereinafter referred to as mission flight).
[0003] To realize such a mission flight, the drone needs to be equipped with a self-location estimation system for estimating its own position. For example, the self-location estimation system mainly uses a method of estimating its own position using a GPS (Global Positioning System) sensor or a method of estimating its own position using a vision sensor, but various other methods can also be used.
[0004] For example, a self-location estimation system using a GPS sensor has the advantage of being able to use GPS over a wide range, but has the disadvantage of low accuracy in estimating the self-location (for example, an error of about a few meters).Furthermore, a self-location estimation system using a vision sensor has the advantage of high accuracy in estimating the self-location (for example, an error of about a few centimeters), but has the disadvantage that the range in which vision can be effectively used is narrow and it cannot be used for uniform appearances or distant views.
[0005] Furthermore, Patent Document 1 discloses a moving body that can set the route or range of movement more accurately by acquiring multiple pieces of position information and setting information on the range of movement. [Prior art documents] [Patent documents]
[0006] [Patent Document 1] Patent No. 6289750 Summary of the Invention [Problem to be solved by the invention]
[0007] However, in conventional drones, for example, the same self-location estimation system is configured to be used throughout the entire mission flight, so it is difficult to perform appropriate flight control according to the situation while changing the estimation accuracy when estimating the self-location. For example, it has not been possible to perform a mission flight in which the drone flies while estimating the self-location with high accuracy at one waypoint and flies at high speed even if the accuracy is low at another waypoint.
[0008] The present disclosure has been made in consideration of such circumstances, and makes it possible to realize more appropriate behavioral control. [Means for solving the problem]
[0009] A behavior control device of one aspect of the present disclosure includes a state identification unit that selects a self-location estimation unit to be used to estimate the position of a moving body from among a plurality of self-location estimation units with different estimation accuracy based on an index, and determines the state of the moving body, and a behavior control unit that controls the behavior of the moving body in accordance with an action plan created by a specified behavior task based on the state of the moving body, and the self-location estimation unit includes a vision-based self-location estimation unit that uses information obtained from an image sensor, and a Lidar-based self-location estimation unit that uses information obtained from a Lidar sensor.
[0010] A behavior control method of one aspect of the present disclosure includes a behavior control device that controls the behavior of a moving body, selecting a self-location estimation unit to be used to estimate the position of the moving body from among a plurality of self-location estimation units with different estimation accuracy based on an index, determining a state of the moving body, and controlling the behavior of the moving body in accordance with a behavior plan created by a specified behavior task based on the state of the moving body, wherein the self-location estimation unit includes a vision-based self-location estimation unit that uses information obtained from an image sensor, and a Lidar-based self-location estimation unit that uses information obtained from a Lidar sensor.
[0011] In one aspect of the present disclosure, a behavior control device that controls the behavior of a moving body selects a self-location estimation unit to be used for estimating the position of the moving body from among a plurality of self-location estimation units with different estimation accuracy based on an index, obtains a state of the moving body, and controls the behavior of the moving body according to an action plan created by a specified action task based on the state of the moving body. The self-location estimation unit includes a vision-based self-location estimation unit that uses information acquired from an image sensor, and a Lidar-based self-location estimation unit that uses information acquired from a Lidar sensor. [Brief description of the drawings]
[0012] [Figure 1] FIG. 2 is a diagram illustrating a mission flight. [Diagram 2] 1 is a block diagram showing a configuration example of a first embodiment of a mission flight control device to which the present technology is applied; [Diagram 3] FIG. 2 is a diagram illustrating an example of the configuration of an EKF processing unit. [Figure 4] FIG. 2 is a diagram showing an example of a mission flight setting screen. [Diagram 5] FIG. 11 is a diagram illustrating switching of the self-location estimation system. [Figure 6] 4 is a flowchart illustrating a mission flight control process. [Figure 7] 13 is a flowchart illustrating an EKF process. [Figure 8] FIG. 1 illustrates an example of a user interface visualizing altitude and speed ranges. [Figure 9] FIG. 13 is a diagram illustrating an example of a user interface that visualizes estimation accuracy. [Figure 10] FIG. 2 is a diagram illustrating the effective range of altitude and speed for each self-location estimation system. [Figure 11] FIG. 13 is a diagram showing an example of a user interface that visualizes that vision-based self-localization is not selectable. [Figure 12] 13 is a flowchart illustrating a waypoint setting process for automatically selecting a waypoint WP with priority given to high accuracy. [Figure 13] 13 is a flowchart illustrating a waypoint setting process for automatically selecting a waypoint WP with priority given to speed. [Figure 14] 11 is a flowchart illustrating a waypoint setting process for automatically selecting an altitude for each waypoint WP in accordance with a speed and a self-position estimation system designation. [Figure 15] FIG. 2 is a diagram for explaining alignment in the mission flight control device. [Figure 16] FIG. 13 illustrates a method for customizing a mission flight to avoid obstacles based on the location of the obstacles. [Figure 17] FIG. 13 is a diagram illustrating fine-tuning of a waypoint using an obstacle map. [Figure 18] 1 is a block diagram showing a configuration example of a second embodiment of a mission flight control device to which the present technology is applied. FIG. [Figure 19] A flowchart illustrating a process for customizing a mission flight is shown. [Figure 20] 13 is a diagram showing an example of a user interface that visualizes the distance measurement accuracy of an obstacle. FIG. [Figure 21] 1 is a block diagram showing an example of the configuration of an embodiment of a computer to which the present technology is applied. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0013] Hereinafter, specific embodiments to which the present technology is applied will be described in detail with reference to the drawings.
[0014] <About the mission flight> Referring to Figure 1, the drone mission flight will be described.
[0015] For example, when six waypoints WP1 to WP6 are specified as shown in Fig. 1, an action plan is created for a route that passes through waypoints WP2 to WP5 in order from waypoint WP1, which is the start position, to waypoint WP6, which is the goal position. In this case, the drone 1 can execute a mission flight that flies along the route shown by the two-dot chain line in Fig. 1.
[0016] In this embodiment, a mission flight using the waypoint WP as an indicator for the drone 1 to pass through will be described. However, in addition to using the waypoint WP, for example, an action plan may be created by specifying an action task of the drone 1 (such as taking an image or transporting an item). That is, the waypoint WP may be specified according to the content of the action task, not limited to being specified by the user. The position of the waypoint WP may be specified based on, for example, position information or area information acquired according to GPS when the drone 1 performs a pre-flight.
[0017] <First Configuration Example of Mission Flight Control Device> FIG. 2 is a block diagram showing a configuration example of a first embodiment of a mission flight control device to which the present technology is applied.
[0018] 2, the mission flight control device 11 includes a personal computer 12, an action plan creation unit 13, an image sensor 14, an inertial measurement unit 15, a vision-based self-position estimation unit 16, a GPS-based self-position estimation unit 17, a flight control unit 18, and a drive unit 19. The flight control unit 18 includes an EKF (Extended Kalman Filter) processing unit 21, a navigator 22, and a drive control unit 23.
[0019] For example, the mission flight control device 11 can be configured so that the drone 1 in Figure 1 has an image sensor 14, an inertial measurement unit 15, a vision-based self-position estimation unit 16, a GPS-based self-position estimation unit 17, a flight control unit 18, and a drive unit 19 built in.
[0020] The personal computer 12 has a display unit that displays a mission flight setting screen 41 as shown in Fig. 4 described later, and an operation unit such as a mouse and a keyboard, and is used by the user to specify a waypoint WP. The personal computer 12 then supplies the waypoint WP specified by the user to the action plan creation unit 13 and also to the EKF processing unit 21 of the flight control unit 18.
[0021] The path plan creation unit 13 creates a path plan for flying the drone 1 so as to pass through the waypoints WP supplied from the personal computer 12, and supplies the path plan to the navigator 22 of the flight control unit 18.
[0022] The image sensor 14 has a solid-state imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor. The image sensor 14 supplies the vision-based self-position estimation unit 16 with image data of an image obtained by capturing an image of the surroundings of the drone 1 with the solid-state imaging element.
[0023] The inertial measurement device 15 is, for example, an IMU (Inertial Measurement Unit) composed of a three-axis gyro sensor, etc. The inertial measurement device 15 supplies the acceleration and angular velocity measured in association with the flight of the drone 1 as inertial measurement data (IMU data) to the vision-based self-position estimation unit 16, and also supplies the same to the EKF processing unit 21 of the flight control unit 18.
[0024] The vision-based self-position estimation unit 16 performs self-position estimation (visual inertial odometry) to estimate the position of the drone 1 using image data supplied from the image sensor 14 and inertial measurement data supplied from the inertial measurement unit 15. For example, the vision-based self-position estimation unit 16 can use SLAM (Simultaneous Localization and Mapping). The vision-based self-position estimation unit 16 then supplies the pose and velocity (Pose / Velocity) obtained as a result of performing self-position estimation based on the image data to the EKF processing unit 21 of the flight control unit 18 as vision-based observation information.
[0025] The GPS-based self-location estimation unit 17 uses, for example, a GPS to perform self-location estimation to estimate the position of the drone 1. Then, the GPS-based self-location estimation unit 17 supplies the attitude (Pose in NED) on the NED (North-East-Down) coordinate system obtained as a result of performing self-location estimation based on the GPS to the EKF processing unit 21 of the flight control unit 18 as GPS-based observation information.
[0026] The flight control unit 18 controls the drive unit 19 so that the drone 1 flies in accordance with the action plan supplied from the action plan creation unit 13, and performs flight control of the drone 1.
[0027] The drive unit 19 is composed of, for example, a motor that drives multiple rotors, and causes the drone 1 to fly by the thrust generated by those rotors.
[0028] The EKF processing unit 21 performs EKF processing to obtain the state (position and attitude) of the drone 1 based on the inertial measurement data supplied from the inertial measurement device 15, the vision-based observation information supplied from the vision-based self-position estimation unit 16, and the GPS-based observation information supplied from the GPS-based self-position estimation unit 17. Then, the EKF processing unit 21 supplies state information indicating the state of the drone 1 as the EKF processing result obtained as a result of performing the EKF processing to the navigator 22 and the drive control unit 23. Note that a detailed configuration of the EKF processing unit 21 will be described later with reference to FIG. 2.
[0029] Based on the status information supplied from the EKF processing unit 21, the navigator 22 gives the drive control unit 23 instructions necessary for flying along the route from the current state of the drone 1 so that the drone 1 flies in accordance with the action plan supplied from the action plan creation unit 13.
[0030] Based on the status information supplied from the EKF processing unit 21, the drive control unit 23 performs drive control (e.g., position control and attitude control) on the drive unit 19 to fly the drone 1 in accordance with instructions from the navigator 22.
[0031] The mission flight control device 11 is configured as described above, and for example, when a user specifies a waypoint WP using the personal computer 12, the user can specify a self-location estimation system for each waypoint WP. That is, in the mission flight control device 11, the vision-based self-location estimation unit 16 or the GPS-based self-location estimation unit 17 is specified as the self-location estimation system for each waypoint WP, and flight control is performed using either one of the observation information. Note that, in the mission flight control device 11, the observation information of both the vision-based self-location estimation unit 16 and the GPS-based self-location estimation unit 17 may be combined for each waypoint WP, and flight control may be performed by selectively using, for example, multiple observation information.
[0032] Therefore, when the vision-based self-location estimation unit 16 is specified as the self-location estimation system at the nearest waypoint WP, the EKF processing unit 21 selects the vision-based observation information and performs EKF processing. On the other hand, when the GPS-based self-location estimation unit 17 is specified as the self-location estimation system at the nearest waypoint WP, the EKF processing unit 21 selects the GPS-based observation information and performs EKF processing.
[0033] FIG. 3 is a block diagram showing an example of the configuration of the EKF processing unit 21.
[0034] As shown in FIG. 3, the EKF processing unit 21 is configured to include an EKF state prediction unit 31, a nearest waypoint selection unit 32, an observation information selection unit 33, and an EKF state update unit .
[0035] The EKF state prediction unit 31 is supplied with state information indicating the latest state of the drone 1, which is periodically output from the EKF processing unit 21, from the EKF state update unit 34. The EKF state prediction unit 31 then predicts the state of the drone 1 at the timing of next outputting the state information from the latest state information, based on the inertial measurement data (IMU data) supplied from the inertial measurement unit 15 of FIG. 2. The EKF state prediction unit 31 then supplies predicted state information indicating the predicted state of the drone 1 to the nearest waypoint selection unit 32 and the EKF state update unit 34. For example, the predicted state information is a state variable of the EKF, such as the position and attitude of the drone 1, and the bias value of the sensor of the inertial measurement unit 15.
[0036] The nearest waypoint selection unit 32 selects the nearest waypoint that is closest to the position of the drone 1 based on the predicted state information supplied from the EKF state prediction unit 31 from among the multiple waypoints WP supplied from the personal computer 12. For example, the nearest waypoint selection unit 32 selects the waypoint WP with the shortest Euclidean distance in the NED coordinate system as the nearest waypoint. Then, the nearest waypoint selection unit 32 supplies the observation information selection unit 33 with designation information indicating which of the vision-based self-location estimation unit 16 and the GPS-based self-location estimation unit 17 is designated as the self-location estimation system for the nearest waypoint.
[0037] The observation information selection unit 33 selects either the vision-based observation information (Pose / Velocity) supplied from the vision-based self-location estimation unit 16 or the GPS-based observation information (Pose in NED) supplied from the GPS-based self-location estimation unit 17 in accordance with the designation information supplied from the nearest waypoint selection unit 32. For example, when the vision-based self-location estimation unit 16 is designated as the self-location estimation system at the nearest waypoint, the observation information selection unit 33 selects the vision-based observation information and supplies it to the EKF state update unit 34. On the other hand, when the GPS-based self-location estimation unit 17 is designated as the self-location estimation system at the nearest waypoint, the observation information selection unit 33 selects the GPS-based observation information and supplies it to the EKF state update unit 34.
[0038] The EKF state update unit 34 updates the predicted state information supplied from the EKF state prediction unit 31 based on the observation information selected by the observation information selection unit 33, and obtains the state of the drone 1 based on the latest observation information. For example, the EKF state update unit 34 updates the state of the drone 1 predicted by the EKF state prediction unit 31 to be more accurate according to the estimated position indicated in the observation information. Then, the EKF state update unit 34 outputs state information indicating the state of the drone 1 as the EKF processing result in the EKF processing unit 21, and supplies it to the EKF state prediction unit 31.
[0039] As described above, the EKF processing unit 21 can select the self-position estimation system at the nearest waypoint, and can determine the state of the drone 1 with an estimation accuracy according to the situation at each waypoint WP.
[0040] Therefore, the mission flight control device 11 can perform a mission flight of the drone 1 by performing appropriate flight control according to the situation while changing the estimation accuracy when estimating the drone's own position for each waypoint WP.
[0041] In addition to the configuration in which any of the self-location estimation systems is designated for the nearest waypoint as in this embodiment, for example, a configuration in which any of the self-location estimation systems is designated according to a position on a map instead of the waypoint WP may be adopted. Also, a configuration in which any of the self-location estimation systems is designated according to an area on a map may be adopted.
[0042] In this embodiment, the vision-based self-location estimation unit 16 and the GPS-based self-location estimation unit 17 are used as the self-location estimation system, but the configuration is not limited to using these two. For example, the mission flight control device 11 may be configured to use an altitude sensor, a geomagnetic sensor, or other self-location estimation systems (UWB (Ultra Wide Band), wifi SLAM, OptiTrack, short-distance wireless communication-based, etc.) and switch between them.
[0043] FIG. 4 is a diagram showing an example of a mission flight setting screen.
[0044] For example, the mission flight setting screen 41 is used to input various instructions required for performing a mission flight using a GUI (Graphical User Interface) of the personal computer 12. As shown in the figure, the mission flight setting screen 41 is made up of a map display area 42 and an instruction GUI display area 43.
[0045] In the map display area 42, for example, a map of the area in which the drone 1 is to fly is displayed. Then, when the user designates the waypoints WP1 to WP6 using the map displayed in the map display area 42, the waypoints WP1 to WP6 are displayed so as to be superimposed on the map.
[0046] At this time, the waypoints WP1 to WP6 are displayed so as to visualize the self-localization system designated for each of them. That is, in the example shown in Fig. 4, the waypoints WP1 to WP3 and WP6 are represented by solid lines, and it is visualized that the vision-based self-localization unit 16 is designated as the self-localization system. Also, the waypoints WP4 and WP5 are represented by dashed lines, and it is visualized that the GPS-based self-localization unit 17 is designated as the self-localization system.
[0047] In addition, the map display area 42 also displays the routes that sequentially connect the waypoints WP1 to WP6 in a manner that visualizes the self-location estimation system used when the drone 1 flies each route.
[0048] 4, the route connecting waypoints WP1 to WP3, for which the vision-based self-location estimation unit 16 is specified as the self-location estimation system, is shown by a solid line, visualizing the use of the vision-based self-location estimation unit 16 when the drone 1 flies along that route. Similarly, the route connecting waypoints WP4 and WP5, for which the GPS-based self-location estimation unit 17 is specified as the self-location estimation system, is shown by a dashed line, visualizing the use of the GPS-based self-location estimation unit 17 when the drone 1 flies along that route.
[0049] Furthermore, for the route connecting waypoints WP3 and WP4, the fact that the vision-based self-location estimation unit 16 is switched to the GPS-based self-location estimation unit 17 when the drone 1 flies along that route is visualized by changing from a solid line to a dashed line. Similarly, for the route connecting waypoints WP5 and WP6, the fact that the GPS-based self-location estimation unit 17 is switched to the vision-based self-location estimation unit 16 when the drone 1 flies along that route is visualized by changing from a dashed line to a solid line.
[0050] In the instruction GUI display area 43, for example, as described later with reference to FIG. 5, a GUI used for instructing a self-localization system for each waypoint WP is displayed.
[0051] Switching of the self-location estimation system designated for the waypoint WP will be described with reference to Fig. 5. Fig. 5 shows an example in which the self-location estimation system designated for the waypoint WP4 is switched from the GPS-based self-location estimation unit 17 to the Vision-based self-location estimation unit 16.
[0052] 5, when a waypoint WP4 for switching the self-location estimation system is specified in the map display area 42, a check box for specifying the self-location estimation system is displayed in the instruction GUI display area 43. That is, a check box for specifying the GPS-based self-location estimation unit 17 and a check box for specifying the Vision-based self-location estimation unit 16 are displayed.
[0053] At this time, a check mark is displayed in the checkbox for specifying the GPS-based self-location estimation unit 17, visualizing that the GPS-based self-location estimation unit 17 has been specified for the waypoint WP4.
[0054] Then, when the user operates the checkbox for specifying the vision-based self-localization estimation unit 16, a check mark is displayed in the checkbox for specifying the vision-based self-localization estimation unit 16, as shown in the lower part of Fig. 5. This changes the display of the waypoint WP4 in the map display area 42 from a dashed line to a solid line. Furthermore, the route connecting the waypoints WP3 and WP4 is displayed with a solid line, and the route connecting the waypoints WP4 and WP5 is displayed by changing from a dashed line to a solid line.
[0055] For example, when it is desired to fly the drone 1 with higher accuracy in a place where the drone 1 has been flown with the estimation accuracy of the GPS-based self-location estimation unit 17, an operation for switching the setting to the vision-based self-location estimation unit 16 is performed. In this way, the user can specify a self-location estimation system based on the estimation accuracy required when passing through the waypoint WP.
[0056] <Mission flight control processing> The mission flight control process executed by the mission flight control device 11 will be described with reference to the flowchart shown in Fig. 6. Note that information required for performing a mission flight, such as information on a map and topography of the area in which the drone 1 is to be flown, is assumed to have been acquired in advance by flying with low accuracy.
[0057] In step S11, the user uses the personal computer 12 to specify a waypoint WP on the map displayed in the map display area 42 of the mission flight setting screen 41 shown in Fig. 4. In response to this, the personal computer 12 supplies the action plan creation unit 13 with the waypoint WP specified by the user.
[0058] In step S12, the user uses the personal computer 12 to specify the vision-based self-location estimation unit 16 or the GPS-based self-location estimation unit 17 as the self-location estimation system for each waypoint WP specified in step S11. In response to this, the personal computer 12 supplies the self-location estimation system for each waypoint WP specified by the user to the EKF processing unit 21 of the flight control unit 18.
[0059] In step S13, the behavior plan creation unit 13 creates a behavior plan for the drone 1 to pass through the waypoint WP specified in step S11, and supplies it to the navigator 22 of the flight control unit 18.
[0060] In step S14, when the user instructs the drone 1 to start flying, the flight control unit 18 performs flight control in response to the instruction. As a result, the drone 1 flies toward, for example, waypoint WP1, which is the start position shown in FIG. 1, and starts flying according to the action plan created in step S13.
[0061] In step S15, the vision-based self-location estimation unit 16 estimates the position of the drone 1 based on the image data supplied from the image sensor 14 and the inertial measurement data supplied from the inertial measurement unit 15, and acquires vision-based observation information. In addition, the GPS-based self-location estimation unit 17 estimates the position of the drone 1 using a GPS, and acquires GPS-based observation information.
[0062] In step S16, the EKF processing unit 21 performs EKF processing to determine the state of the drone 1 based on the inertial measurement data supplied from the inertial measurement device 15, the vision-based observation information supplied from the vision-based self-position estimation unit 16, and the GPS-based observation information supplied from the GPS-based self-position estimation unit 17. The EKF processing will be described later with reference to FIG.
[0063] In step S17, the flight control unit 18 performs flight control to fly the drone 1 along the route. That is, the navigator 22 instructs the drive control unit 23 based on the status information supplied from the EKF processing unit 21 so that the drone 1 flies according to the action plan created by the action plan creation unit 13 in step S13. Then, the drive control unit 23 performs drive control on the drive unit 19 based on the status information supplied from the EKF processing unit 21 so that the drone 1 flies according to the instructions from the navigator 22.
[0064] In step S18, the navigator 22 determines, for example, whether the drone 1 has flown to the waypoint WP6, which is the goal position shown in FIG.
[0065] In step S18, if the navigator 22 determines that the drone 1 has not flown to the goal position, the process returns to step S15, and the same process is repeated thereafter. On the other hand, in step S18, if the navigator 22 determines that the drone 1 has flown to the goal position, the process proceeds to step S19.
[0066] In step S19, the flight control unit 18 performs flight control so that the drone 1 performs a predetermined operation associated with the end of flight, such as hovering, and then the mission flight control process is terminated.
[0067] FIG. 7 is a flowchart illustrating the EKF processing performed in step S16 of FIG.
[0068] In step S21, the EKF state prediction unit 31 predicts the state (position and attitude) of the drone 1 based on the inertial measurement data supplied from the inertial measurement unit 15 of FIG.
[0069] In step S22, the nearest waypoint selection unit 32 selects the nearest waypoint that is closest to the position of the drone 1 predicted by the EKF state prediction unit 31 in step S21. Then, the nearest waypoint selection unit 32 supplies the observation information selection unit 33 with designation information indicating the self-position estimation system designated for the nearest waypoint.
[0070] In step S23, the observation information selection unit 33 selects observation information in accordance with the designation information supplied from the nearest waypoint selection unit 32 in step S22, and supplies the selected observation information to the EKF state update unit 34. For example, when the vision-based self-location estimation unit 16 is designated as the self-location estimation system for the nearest waypoint, the observation information selection unit 33 selects the vision-based observation information, and when the GPS-based self-location estimation unit 17 is designated, the observation information selection unit 33 selects the GPS-based observation information.
[0071] In step S24, the EKF state update unit 34 updates the state of the drone 1 predicted by the EKF state prediction unit 31 in step S21 based on the observation information supplied from the observation information selection unit 33 in step S23. Then, after outputting state information indicating the state of the drone 1, the EKF processing is terminated.
[0072] Through the above-described mission flight control processing and EKF processing, the mission flight control device 11 can select a self-position estimation system for each waypoint WP and realize appropriate flight control of the drone 1 with each estimation accuracy.
[0073] <Visualization of information required to specify a self-localization system> The visualization of information required for specifying a self-location estimation system will be described with reference to FIGS.
[0074] FIG. 8 shows an example of a user interface that visualizes altitude and speed ranges.
[0075] For example, as shown in Fig. 8, when the vision-based self-location estimation unit 16 is specified as the self-location estimation system at waypoint WP3, the effective ranges of altitude and speed are visualized and presented. That is, when using the vision-based self-location estimation unit 16, the altitude and speed at which the self-location can be estimated with a certain degree of accuracy or higher are determined, and in the example shown in Fig. 8, the ranges above those altitudes and speeds are grayed out.
[0076] This allows the user to decide whether or not to designate the vision-based self-location estimation unit 16 as the self-location estimation system by referring to the effective ranges of altitude and speed.
[0077] FIG. 9 shows an example of a user interface that visualizes the estimation accuracy.
[0078] For example, as shown in Fig. 9, an error ellipse representing an error range based on the accuracy of estimating the self-location is presented as a dashed line circle. That is, the vision-based self-location estimation unit 16 can estimate the self-location with high accuracy, and small circles representing a small error range are displayed at regular intervals on a route using the vision-based self-location estimation unit 16. On the other hand, the GPS-based self-location estimation unit 17 estimates the self-location with low accuracy, and large circles representing a large error range are displayed at regular intervals on a route using the GPS-based self-location estimation unit 17.
[0079] For example, the circle representing the error range can be specified by a constant based on the specifications of the sensors used by the vision-based self-location estimation unit 16 and the GPS-based self-location estimation unit 17, or can be specified by the user.
[0080] This allows the user to easily grasp the estimation accuracy of the self-location estimation system used by the drone 1 along the route on the map, and can check, for example, the margin between the drone and obstacles.
[0081] The visualization of the self-location estimation system that cannot be selected will be described with reference to FIG. 10 and FIG.
[0082] For example, as shown in FIG. 10, the effective range for the altitude and speed of drone 1 is determined for each self-localization system.
[0083] That is, as shown in A of FIG. 10, in the vision-based self-location estimation unit 16 using SLAM using a downward stereo camera in the drone 1, as shown in B of FIG. 10, the range in which the SLAM vision is effective is obtained. In other words, the altitude and speed that are below the stereo parallax effective limit and above the frame (time) common view limit and the stereo common view limit are the range in which the SLAM vision is effective. Then, at altitudes higher than the stereo parallax effective limit and at altitudes lower than the frame (time) common view limit, only the inertial measurement unit 15 is used. Furthermore, at altitudes lower than the stereo common view limit, the inertial measurement unit 15 and the proximity ranging sensor are used.
[0084] For example, drone 1 has a speed of X [m / s 2 When flying at , the effective altitude range in which the vision-based self-position estimation unit 16 can be used is determined to be from H1, which is the common field of view limit between frames (times), to altitude H2, which is the effective limit of stereo parallax.
[0085] Therefore, based on the altitude and speed specified at the waypoint WP, the mission flight control device 11 can determine whether or not the altitude and speed are within the effective range of the vision-based self-location estimation unit 16. Then, for example, if it is determined that the specified altitude and speed are not within the effective range of the vision-based self-location estimation unit 16, that is, if they are outside the effective range of the vision-based self-location estimation unit 16, selection of the vision-based self-location estimation unit 16 is automatically disabled.
[0086] 11, when the specified altitude and speed are outside the effective range of the vision-based self-location estimation unit 16, the check box for selecting the vision-based self-location estimation unit 16 is grayed out in the instruction GUI display area 43. That is, it is visualized that the vision-based self-location estimation unit 16 is not selectable.
[0087] This allows the user to recognize the self-location estimation systems that can be selected based on altitude and speed, and makes it easy to select a self-location estimation system that suits the situation.
[0088] <Indicator-based automatic setting process> 12 to 14, an automatic setting process for automatically setting the self-position estimation system, altitude, and the like for each waypoint WP based on the priority items will be described.
[0089] 12 is a flowchart illustrating a waypoint setting process for automatically selecting a waypoint WP with a high degree of accuracy as a priority. For example, the waypoint setting process shown in FIG. 12 can be executed in place of step S12 in FIG.
[0090] For example, after the process of step S11 in FIG. 6, in step S31, the user uses the personal computer 12 to specify an altitude and a speed for each waypoint WP specified in step S11.
[0091] In step S32, the user uses the personal computer 12 to instruct the start of processing for automatically setting the self-location estimation system for each waypoint WP with priority given to accuracy.
[0092] In step S33, for example, the personal computer 12 sets the self-location estimation system to be set in order from the waypoint WP which is the start position, and provisionally sets the self-location estimation system with the highest accuracy for the target waypoint WP.
[0093] In step S34, the personal computer 12 determines whether or not the altitude and speed specified in step S31 are within a valid range in which the currently provisionally set self-location estimation system can be used.
[0094] If the personal computer 12 determines in step S34 that the altitude and speed are not within the valid range, that is, if the altitude and speed are outside the valid range, the process proceeds to step S35.
[0095] In step S35, the personal computer 12 provisionally sets a self-location estimation system with the next highest accuracy to the currently provisionally set self-location estimation system. After that, the process returns to step S34, and the same processes are repeated thereafter.
[0096] On the other hand, if it is determined in step S34 that the altitude and speed are within the valid range, the process proceeds to step S36.
[0097] In step S36, the personal computer 12 sets the currently provisionally set self-location estimation system for the target waypoint WP.
[0098] In step S37, the personal computer 12 determines whether or not the self-location estimation system has been set for all of the waypoints WP designated by the user in step S11 of FIG.
[0099] In step S37, if the personal computer 12 determines that the self-location estimation system has not been set for all the waypoints WP, the process returns to step S33 for the next waypoint WP. Then, the same process is repeated thereafter.
[0100] On the other hand, if the personal computer 12 determines in step S37 that the self-location estimation systems have been set for all the waypoints WP, the waypoint setting process ends and the process proceeds to step S13 in FIG.
[0101] By using the above-described waypoint setting process, the mission flight control device 11 can automatically set the self-position estimation system for each waypoint WP with a priority on high accuracy, thereby realizing appropriate flight control of the drone 1 so as to fly with higher accuracy.
[0102] 13 is a flowchart illustrating a waypoint setting process for automatically selecting a waypoint WP with priority given to speed. For example, the waypoint setting process shown in FIG. 13 can be executed in place of step S12 in FIG.
[0103] For example, after the process of step S11 in FIG. 6, in step S41, the user uses the personal computer 12 to specify an altitude and a speed for each waypoint WP specified in step S11.
[0104] In step S42, the user uses the personal computer 12 to instruct the start of processing for automatically setting the self-location estimation system for each waypoint WP with priority given to speed.
[0105] In step S43, for example, the personal computer 12 sets the self-location estimation system to be set in order from the waypoint WP which is the start position, and provisionally sets the fastest self-location estimation system for the target waypoint WP.
[0106] In step S44, the personal computer 12 determines whether or not the altitude and speed specified in step S41 are within a valid range in which the currently provisionally set self-location estimation system can be used.
[0107] If the personal computer 12 determines in step S44 that the altitude and speed are not within the valid range, that is, if the altitude and speed are outside the valid range, the process proceeds to step S45.
[0108] In step S45, the personal computer 12 provisionally sets the self-location estimation system with the next highest speed to the currently provisionally set self-location estimation system. After that, the process returns to step S44, and the same processes are repeated thereafter.
[0109] On the other hand, if it is determined in step S44 that the altitude and speed are within the valid range, the process proceeds to step S46.
[0110] In step S46, the personal computer 12 sets the currently provisionally set self-location estimation system for the target waypoint WP.
[0111] In step S47, the personal computer 12 determines whether or not the self-location estimation system has been set for all of the waypoints WP designated by the user in step S11 of FIG.
[0112] In step S47, if the personal computer 12 determines that the self-location estimation system has not been set for all the waypoints WP, the process returns to step S43 for the next waypoint WP. Then, the same process is repeated thereafter.
[0113] On the other hand, if the personal computer 12 determines in step S47 that the self-location estimation systems have been set for all the waypoints WP, the waypoint setting process ends and the process proceeds to step S13 in FIG.
[0114] By using the above-described waypoint setting process, the mission flight control device 11 can automatically set the self-position estimation system with speed priority for each waypoint WP, thereby realizing appropriate flight control of the drone 1 so as to fly at a higher speed.
[0115] 14 is a flowchart illustrating a waypoint setting process for automatically selecting the altitude for each waypoint WP in accordance with the speed and the designation of the self-location estimation system. For example, the waypoint setting process shown in FIG. 12 can be executed in place of step S12 in FIG.
[0116] For example, after the process of step S11 in FIG. 6, in step S51, the user uses the personal computer 12 to specify a self-location estimation system and a speed for each waypoint WP specified in step S11.
[0117] In step S52, the user uses the personal computer 12 to instruct the start of processing for automatically setting the altitude for each waypoint WP.
[0118] In step S53, for example, the personal computer 12 sets targets for setting the self-location estimation system in order starting from the waypoint WP that is the start position, and calculates the lowest altitude at which the self-location estimation system is valid for the target waypoint WP.
[0119] In step S54, the personal computer 12 determines whether or not the altitude calculated in step S53 is within the valid range when the self-location estimation system set for the target waypoint WP is used.
[0120] If the personal computer 12 determines in step S54 that the altitude is not within the valid range, that is, if the altitude is outside the valid range, the process proceeds to step S55.
[0121] In step S55, an error message is displayed indicating that the self-localization system and speed specified for the target waypoint WP cannot be specified, and the waypoint setting process is terminated. In other words, in this case, the process of automatically selecting the altitude for each waypoint WP is stopped.
[0122] On the other hand, if the personal computer 12 determines in step S54 that the altitude is within the valid range, the process proceeds to step S56.
[0123] In step S56, the personal computer 12 sets the altitude calculated in step S53 for the target waypoint WP.
[0124] In step S57, the personal computer 12 determines whether or not the altitudes have been set for all of the waypoints WP designated by the user in step S11 of FIG.
[0125] If the personal computer 12 determines in step S57 that the altitudes have not been set for all the waypoints WP, the process returns to step S53 for the next waypoint WP, and the same process is repeated thereafter.
[0126] On the other hand, if the personal computer 12 determines in step S57 that the altitudes have been set for all of the waypoints WP, the waypoint setting process ends and the process proceeds to step S13 in FIG.
[0127] By performing the above-described waypoint setting process, the mission flight control device 11 can realize appropriate flight control of the drone 1 by automatically setting the altitude for each waypoint WP and flying at a lower altitude.
[0128] <How to display the location on the map when using only the vision> A method of displaying a position on a map when only a vision is used will be described with reference to FIG.
[0129] For example, in order to display the position of drone 1 on the map displayed in the map display area 42 in Fig. 4, it is necessary to align the actual position of drone 1 with the position on the map. As a method for performing this alignment, the following two methods are proposed.
[0130] The first method is a method in which a user inputs data to the personal computer 12 from outside, as indicated by the dashed arrow pointing to the personal computer 12 in Fig. 15. That is, by inputting data indicating the latitude, longitude, and azimuth angle of the position where the drone 1 takes off into the personal computer 12, the user can align the data on a map.
[0131] As a second method, a method is used in which data is input from the GPS-based self-location estimation unit 17 to the personal computer 12, as indicated by the dashed arrow pointing from the GPS-based self-location estimation unit 17 to the personal computer 12 in Fig. 15. That is, by inputting GPS-based observation information output from the GPS-based self-location estimation unit 17 mounted on the drone 1 to the personal computer 12, data indicating the latitude, longitude, and azimuth angle of the position where the drone 1 takes off can be aligned on a map.
[0132] <Second Configuration Example of Mission Flight Control Device> A second configuration example of the mission flight control device will be described with reference to FIGS.
[0133] For example, the mission flight controller 11 can customize the mission flight to avoid obstacles based on the location of the obstacles.
[0134] First, as shown in the upper part of Fig. 16, a waypoint WP is specified on the map displayed in the map display area 42 within a range that does not hit any obstacles. Then, by flying the drone 1 along a route that follows the waypoint WP, a viewpoint image and an obstacle map as shown in the middle part of Fig. 16 are acquired.
[0135] The viewpoint image is, for example, an image captured in the flight direction by a camera mounted on the drone 1. The obstacle map is, for example, a map (occupancy map) that three-dimensionally represents obstacles based on distance measurement by a vision sensor mounted on the drone 1, and an example of the obstacle map corresponding to the viewpoint image is shown in the middle part of Fig. 16.
[0136] The user can then refer to the obstacle map to fine-tune the position of waypoint WP, for example, customizing the mission flight path to move waypoint WP5 closer to the obstacle while avoiding it, as shown in the lower part of Figure 16.
[0137] At this time, for example, as shown in Fig. 17, the mission flight control device 11 can display the waypoint WP and the route by superimposing them on an obstacle map that is expressed three-dimensionally. By using such an obstacle map, it is possible to fine-tune the waypoint WP in more detail, for example, by referring to the altitude of the obstacle. Alternatively, a map that uses contour lines to represent obstacles at the altitude of the waypoint WP that is the target of fine adjustment may be used.
[0138] Fig. 18 is a block diagram showing a configuration example of a second embodiment of the mission flight control device. In the mission flight control device 11A shown in Fig. 18, components common to the mission flight control device 11 in Fig. 2 are given the same reference numerals, and detailed description thereof will be omitted.
[0139] In other words, the mission flight control device 11A is similar to the mission flight control device 11 in Figure 2 in that it is configured with a personal computer 12, an action plan creation unit 13, an image sensor 14, an inertial measurement unit 15, a vision-based self-position estimation unit 16, a GPS-based self-position estimation unit 17, a flight control unit 18, and a drive unit 19.
[0140] 2 in that the mission flight control device 11A includes an obstacle map creation unit 20.
[0141] The obstacle map creation unit 20 is supplied with image data from the image sensor 14, and also with vision-based observation information (pose / velocity) from the vision-based self-position estimation unit 16. Then, the obstacle map creation unit 20 creates an obstacle map as shown in FIG. 16 based on the image data and the vision-based observation information, and supplies the obstacle map to the personal computer 12.
[0142] This allows the user to fine-tune the waypoints WP using obstacle maps such as those shown in Figures 16 and 17 to customize the mission flight to avoid obstacles.
[0143] FIG. 19 shows a flow chart illustrating the process of customizing a mission flight.
[0144] In step S61, the user uses the personal computer 12 to specify a waypoint WP on the map displayed in the map display area 42 of the mission flight setting screen 41 within a range where the drone 1 will not hit any obstacles.
[0145] In step S62, when the user instructs the drone 1 to begin flying, the flight control unit 18 performs flight control in response to the instruction, and the drone 1 begins an obstacle map creation flight to create an obstacle map.
[0146] In step S63, the obstacle map creation unit 20 performs distance measurement based on the image data supplied from the image sensor 14 and the vision-based observation information supplied from the vision-based self-position estimation unit 16 during the obstacle map creation flight of the drone 1. Then, the obstacle map creation unit 20 creates an obstacle map that indicates the range in which obstacles exist on the route based on the waypoint WP specified in step S61.
[0147] In step S64, the flight control unit 18 causes the drone 1 to wait at the starting position after passing through all the routes based on the waypoints WP specified in step S61 and completing the obstacle map creation flight.
[0148] In step S65, the personal computer 12 displays on the display unit the obstacle map created by the obstacle map creation unit 20. At this time, as shown in Fig. 16, the obstacle map may be displayed by superimposing an obstacle zone on the map displayed in the map display area 42, or the obstacle map may be displayed three-dimensionally as seen from the viewpoint of the drone 1, as shown in Fig. 17.
[0149] In step S66, the user uses the personal computer 12 to fine-tune the position of the waypoint WP by referring to the obstacle map.
[0150] Thereafter, when the user finishes fine-tuning the waypoint WP, the behavior plan creation unit 13 updates the behavior plan of the drone 1, and thereafter, processing similar to that described from step S14 onwards in Figure 6 is performed, and flight control is performed based on the fine-tuned waypoint WP.
[0151] With reference to FIG. 20, visualization of the distance measurement accuracy of an obstacle will be described.
[0152] As shown in A of FIG. 20, in the map display area 42, an obstacle zone including a distance measurement error can be displayed superimposed on the map in accordance with the distance measurement error when creating the obstacle map.
[0153] Furthermore, as shown in B of Figure 20, in the map display area 42, as in Figure 9 described above, an error ellipse representing the error range based on the estimation accuracy of the self-location estimation system can be displayed superimposed on a map showing an obstacle zone including ranging errors.
[0154] For example, when measuring the distance to an obstacle using a stereo camera, it is possible to calculate the distance measurement error.
[0155] That is, the distance measurement error ΔZc [m] can be calculated according to the following formula (1) using the distance measurement depth Zc [m], the minimum disparity unit Δd [pixel], the baseline distance b [m], and the focus distance f [pixel].
[0156]
number
[0157] For example, based on formula (1), the greater the distance measurement depth Zc, which indicates the distance to an obstacle, the greater the distance measurement error ΔZc.
[0158] <Application of this technology> This technology can be applied to flight control other than that described above, in which the drone 1 passes through the waypoint WP.
[0159] For example, when the drone 1 is configured with an image sensor, a GPS sensor, and a downward-facing ToF (Time of Flight) sensor, this can be applied to switching between sensors to be used. For example, a ToF sensor can measure in places without texture or in dark places, and has the advantage of being more accurate than an image sensor, but has the disadvantage that it can only measure the distance that the self-emitting near-infrared light can reach, and consumes high power due to the self-emitting light.
[0160] Therefore, when drone 1 is in landing mode, the downward-facing ToF sensor is turned on and the ToF sensor measures the shape of the surrounding area to search for a flat place where it can land. At this time, the ToF sensor can measure the shape even in places with little texture, so it can search for a flat place more effectively than the image sensor.
[0161] In contrast, the downward ToF sensor is turned off when the drone 1 takes off or during a mission flight. In other words, because the characteristics of the ToF sensor mean that it can only measure short distances from the ToF sensor, the drone 1 can reduce power consumption by turning off the ToF sensor.
[0162] Furthermore, the present technology can be applied to, for example, autonomous robot vacuum cleaners.
[0163] For example, a robot vacuum cleaner is configured with an image sensor and a ToF sensor, and starts cleaning from a power supply port. In this case, when the robot vacuum cleaner is in a cleaning mode to perform cleaning, the ToF sensor is turned on, and the robot vacuum cleaner uses the ToF sensor and the image sensor to measure its own position and nearby obstacles with high accuracy. At the same time, the robot vacuum cleaner creates a map from the power supply port.
[0164] In contrast, when the voltage of the robot vacuum cleaner drops and the robot vacuum cleaner enters the power supply port return mode, the ToF sensor is turned off. In this case, since the robot vacuum cleaner holds the map from the power supply port, additional obstacle detection by the ToF sensor is not necessary, and the robot vacuum cleaner can detect its own position and return to the power supply port using only the image sensor. In addition, the robot vacuum cleaner can reduce power consumption by turning off the ToF sensor. Then, after the robot vacuum cleaner moves close to the power supply port, the ToF sensor is turned on for highly accurate alignment with the power supply port.
[0165] In addition, in this technology, the sensor specified according to the task can be applied to active sensors other than the ToF sensor, such as Lidar and Sonar, in the same manner as the ToF sensor. In addition, the technology can be applied to robots that perform autonomous driving, such as factory transport robots, in addition to robot vacuum cleaners.
[0166] Furthermore, this technology can be applied to, for example, driving control in an autonomous vehicle, in addition to flight control of the drone 1. For example, when an autonomous vehicle travels on a general road, the GPS-based self-location estimation unit 17 is used, whereas when the autonomous vehicle parks in a parking lot, the vision-based self-location estimation unit 16 is used, or a self-location estimation unit using active sensors such as various radars and ToF sensors can be used.
[0167] <Computer configuration example> Next, the above-mentioned series of processes (information processing method) can be performed by hardware or software. When the series of processes is performed by software, a program constituting the software is installed in a general-purpose computer or the like.
[0168] FIG. 21 is a block diagram showing an example of the configuration of an embodiment of a computer in which a program for executing the series of processes described above is installed.
[0169] The program can be recorded in advance on the hard disk 105 or the ROM 103 as a recording medium built into the computer.
[0170] Alternatively, the program can be stored (recorded) in a removable recording medium 111 driven by the drive 109. Such a removable recording medium 111 can be provided as a so-called package software. Here, examples of the removable recording medium 111 include a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a magnetic disk, and a semiconductor memory.
[0171] The program can be installed in the computer from the removable recording medium 111 as described above, or can be downloaded to the computer via a communication network or a broadcasting network and installed in the built-in hard disk 105. That is, the program can be transferred to the computer wirelessly from a download site via an artificial satellite for digital satellite broadcasting, or transferred to the computer by wire via a network such as a LAN (Local Area Network) or the Internet.
[0172] The computer includes a built-in CPU (Central Processing Unit) 102 , to which an input / output interface 110 is connected via a bus 101 .
[0173] When a command is input by a user operating an input unit 107 via an input / output interface 110, the CPU 102 executes a program stored in a ROM (Read Only Memory) 103 in accordance with the command. Alternatively, the CPU 102 loads a program stored in a hard disk 105 into a RAM (Random Access Memory) 104 and executes the program.
[0174] As a result, CPU 102 performs processing according to the above-mentioned flowchart or processing performed by the configuration of the above-mentioned block diagram. CPU 102 then outputs the processing result from output unit 106 via input / output interface 110, transmits the result from communication unit 108, and further records the result on hard disk 105, as necessary.
[0175] The input unit 107 includes a keyboard, a mouse, a microphone, etc. The output unit 106 includes an LCD (Liquid Crystal Display), a speaker, etc.
[0176] Here, in this specification, the processing performed by a computer according to a program does not necessarily have to be performed in chronological order according to the order described in the flowchart. In other words, the processing performed by a computer according to a program also includes processing executed in parallel or individually (for example, parallel processing or processing by objects).
[0177] The program may be processed by one computer (processor), or may be distributed among multiple computers. Furthermore, the program may be transferred to a remote computer for execution.
[0178] Furthermore, in this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device in which multiple modules are housed in a single housing, are both systems.
[0179] Also, for example, the configuration described above as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, the configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Also, it is of course possible to add configurations other than those described above to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the system as a whole are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).
[0180] Furthermore, for example, the present technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.
[0181] Furthermore, for example, the above-mentioned program can be executed in any device, as long as the device has the necessary functions (functional blocks, etc.) and can obtain the necessary information.
[0182] Also, for example, each step described in the above flow chart can be executed by one device, or can be shared and executed by multiple devices. Furthermore, when one step includes multiple processes, the multiple processes included in the one step can be executed by one device, or can be shared and executed by multiple devices. In other words, multiple processes included in one step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as one step.
[0183] In addition, the processing of the steps of the program executed by the computer may be executed in chronological order according to the order described in this specification, or may be executed in parallel, or individually at a necessary timing such as when a call is made. In other words, the processing of each step may be executed in an order different from the order described above, as long as no contradiction occurs. Furthermore, the processing of the steps of the program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.
[0184] In addition, the present technologies described in this specification can be implemented independently and individually, as long as there is no contradiction. Of course, any of the present technologies can be implemented in combination. For example, a part or all of the present technology described in any of the embodiments can be implemented in combination with a part or all of the present technology described in another embodiment. In addition, any of the present technologies described above can be implemented in combination with other technologies not described above.
[0185] <Examples of configuration combinations> The present technology can also be configured as follows. (1) a state specifying unit that selects a self-location estimating unit to be used for estimating a position of the moving object from among a plurality of self-location estimating units having different estimation accuracies based on an index, and determines a state of the moving object; a behavior control unit that controls the behavior of the moving object according to a previously created behavior plan based on the state of the moving object; A behavior control device comprising: (2) The index is location information or area information obtained from a GPS (Global Positioning System). The behavior control device according to (1) above. (3) The state specifying unit is a nearest neighbor index selection unit that selects the index that is the nearest neighbor in accordance with the behavior of the moving object from among the indices when the moving object behaves; an observation information selection unit that selects, from among a plurality of pieces of observation information output as a result of each of the plurality of self-location estimation units estimating the position of the moving object, the observation information output from the self-location estimation unit that is designated based on the index selected by the nearest index selection unit; a state update unit that obtains a latest state of the moving object based on the observation information selected by the observation information selection unit; have The behavior control device according to (1) or (2) above. (4) a designation unit that designates the self-position estimation unit selected by the state identification unit for each of the indicators; The behavior control device according to any one of (1) to (3) above, further comprising: (5) The designation unit designates the self-location estimation unit for each of the predetermined indices in accordance with a priority that is to be prioritized when the self-location estimation unit estimates the self-location of the moving body for each of the predetermined indices. The behavior control device according to (4) above. (6) When the priority is estimation accuracy, the designation unit designates the self-location estimation unit that can obtain the highest estimation accuracy within an effective range of altitude and speed in which the self-location estimation unit can function effectively. The behavior control device according to (5) above. (7) When the priority is the moving speed, the designation unit designates the self-location estimation unit that can obtain the fastest moving speed within an effective range of altitude and speed in which the self-location estimation unit can function effectively. The behavior control device according to (5) above. (8) The designation unit, when the self-location estimation unit and the speed of the moving body in a predetermined index are designated, designates the lowest altitude at which the designated self-location estimation unit functions effectively for each of the predetermined indexes. The behavior control device according to (4) above. (9) In the designation unit, a plurality of the indices are designated by a user who instructs the creation of the action plan, and the self-location estimation unit is designated for each of the indices. The behavior control device according to (4) above. (10) a display unit that visualizes the self-location estimation unit designated for each of the indicators by the designation unit, and visualizes and displays the self-location estimation unit used for each route following the indicators. The behavior control device according to (9) above, further comprising: (11) The display unit displays an effective range of altitude and speed of the moving body in which a specific self-location estimation unit can function effectively based on the indicator designated by the self-location estimation unit, and visually displays that anything outside the effective range cannot be designated. The behavior control device according to (10) above. (12) The display unit visualizes and displays the estimation accuracy of the self-location estimation unit used on the route along each route. The behavior control device according to (10) above. (13) In response to the altitude and speed of the moving body being specified, the display unit visually displays that the self-location estimation unit cannot be specified because the self-location estimation unit cannot function effectively at the specified altitude and speed. The behavior control device according to (10) above. (14) an obstacle map creation unit that creates an obstacle map showing an area where obstacles exist near the route based on distance measurements performed while the moving object moves according to the action plan; Further equipped with the display unit displays the obstacle map by superimposing it on a map showing an area in which the moving object is to be moved; The designation unit allows the user to finely adjust and redesignate the position of the marker on the map. The behavior control device according to (10) above. (15) The display unit displays an error range for a range in which the obstacle exists, superimposed on the map, according to a distance measurement error of distance measurement when creating the obstacle map. The behavior control device according to (14) above. (16) The display unit displays an error ellipse, which indicates an error range based on the estimation accuracy of the self-position estimation unit, on the map by superimposing the error ellipse on the map. The behavior control device according to (15) above. (17) A behavior control device that controls the behavior of a moving object, selecting a self-location estimation unit to be used for estimating the position of the moving body from among a plurality of self-location estimation units having different estimation accuracies based on an index, and obtaining a state of the moving body; Controlling the behavior of the moving object according to a previously created behavior plan based on the state of the moving object; A behavior control method comprising: (18) A computer of a behavior control device that controls the behavior of a moving object, selecting a self-location estimation unit to be used for estimating the position of the moving body from among a plurality of self-location estimation units having different estimation accuracies based on an index, and obtaining a state of the moving body; Controlling the behavior of the moving object according to a previously created behavior plan based on the state of the moving object; A program for executing information processing including the above.
[0186] In addition, the present embodiment is not limited to the above-described embodiment, and various modifications are possible without departing from the scope of the present disclosure. In addition, the effects described in this specification are merely examples and are not limiting, and other effects may be obtained. [Explanation of symbols]
[0187] 11 Mission Flight Control Unit, 12 Personal Computer, 13 Action Plan Creation Unit, 14 Image Sensor, 15 Inertial Measurement Unit, 16 Vision-Based Self-Localization Unit, 17 GPS-Based Self-Localization Unit, 18 Flight Control Unit, 19 Driving Unit, 20 Obstacle Map Creation Unit, 21 EKF Processing Unit, 22 Navigator, 23 Driving Control Unit, 31 EKF State Prediction Unit, 32 Nearest Waypoint Selection Unit, 33 Observation Information Selection Unit, 34 EKF State Update Unit, 41 Mission Flight Setting Screen, 42 Map Display Area, 43 Instruction GUI Display Area
Claims
1. a state specifying unit that selects a self-location estimating unit to be used for estimating a position of the moving object from among a plurality of self-location estimating units having different estimation accuracies based on an index, and determines a state of the moving object; a behavior control unit that controls the behavior of the moving object according to a behavior plan created by a specified behavior task based on a state of the moving object; Equipped with The self-location estimation unit includes a vision-based self-location estimation unit that uses information acquired from an image sensor, and a Lidar-based self-location estimation unit that uses information acquired from a Lidar (Light Detection And Ranging) sensor. Behavior control device.
2. a state specifying unit that selects a self-location estimating unit to be used for estimating a position of the moving object from among a plurality of self-location estimating units having different estimation accuracies based on an index, and determines a state of the moving object; a behavior control unit that controls the behavior of the moving object according to a behavior plan created by a specified behavior task based on a state of the moving object; an index designation unit that designates the self-location estimation unit selected by the state identification unit for each of the indexes; Equipped with the index designation unit designates the self-location estimation unit for each of the predetermined indices in accordance with a priority that is to be prioritized when the self-location estimation unit estimates a self-location of the moving object for each of the predetermined indices; When the priority is a moving speed, the index designation unit designates the self-location estimation unit that can obtain the fastest moving speed within an effective range of speeds at which the self-location estimation unit can function effectively. Behavior control device.
3. The index designation unit designates the self-location estimation unit for each of the predetermined indices in accordance with a priority that is to be prioritized when the self-location estimation unit estimates the self-location of the moving body for each of the predetermined indices. The behavior control device according to claim 2 .
4. When the priority is estimation accuracy, the index designation unit designates the self-location estimation unit that can obtain the highest estimation accuracy within an effective range of speed at which the self-location estimation unit can function effectively. The behavior control device according to claim 3 .
5. The indicator is information obtained from the image sensor or the Lidar sensor. The behavior control device according to any one of claims 1 to 4.
6. The state specifying unit is a nearest neighbor index selection unit that selects the index that is the nearest neighbor in accordance with the behavior of the moving object from among the indices when the moving object behaves; an observation information selection unit that selects, from among a plurality of pieces of observation information output as a result of each of the plurality of self-location estimation units estimating the position of the moving object, the observation information output from the self-location estimation unit that is designated based on the index selected by the nearest index selection unit; a state update unit that obtains a latest state of the moving object based on the observation information selected by the observation information selection unit; have The behavior control device according to any one of claims 1 to 5.
7. a state specifying unit that selects a self-location estimating unit to be used for estimating a position of the moving object from among a plurality of self-location estimating units having different estimation accuracies based on an index, and determines a state of the moving object; a behavior control unit that controls the behavior of the moving object according to a behavior plan created by a specified behavior task based on a state of the moving object; an index designation unit that designates the self-location estimation unit selected by the state identification unit for each of the indexes; a display unit that visualizes the self-location estimation unit designated for each of the indicators by the indicator designation unit, and visualizes and displays the self-location estimation unit used for each route following the indicators; Equipped with In the index designation unit, a plurality of the indexes are designated by a user who designates the behavioral task, and the self-location estimation unit is designated for each of the indexes. Behavior control device.
8. The display unit displays an effective range of the speed of the moving body in which a specific self-location estimation unit can function effectively based on the indicator designated by the self-location estimation unit, and visually displays that anything outside the effective range cannot be designated. The behavior control device according to claim 7 .
9. The display unit visualizes and displays the estimation accuracy of the self-location estimation unit used on the route along each route. The behavior control device according to claim 7 .
10. The display unit, in response to the speed of the moving body being specified, visually displays that the self-position estimation unit, which cannot function effectively at that speed, cannot be specified. The behavior control device according to claim 7 .
11. an obstacle map creation unit that creates an obstacle map showing an area where obstacles exist near the route based on distance measurements performed while the moving object moves according to the action plan; Further equipped with the display unit displays the obstacle map by superimposing it on a map showing an area in which the moving object is to be moved; In the index designation unit, the position of the index on the map is finely adjusted and redesignated by the user. The behavior control device according to claim 7 .
12. The display unit displays an error range for a range in which the obstacle exists, superimposed on the map, according to a distance measurement error of distance measurement when creating the obstacle map. The behavior control device according to claim 11.
13. The display unit displays an error ellipse, which indicates an error range based on the estimation accuracy of the self-position estimation unit, on the map by superimposing the error ellipse on the map. The behavior control device according to claim 12.
14. A behavior control device that controls the behavior of a moving object, selecting a self-location estimation unit to be used for estimating the position of the moving body from among a plurality of self-location estimation units having different estimation accuracies based on an index, and obtaining a state of the moving body; Controlling the behavior of the moving object according to a behavior plan created by a specified behavior task based on a state of the moving object; Including, The self-location estimation unit includes a vision-based self-location estimation unit that uses information acquired from an image sensor, and a Lidar-based self-location estimation unit that uses information acquired from a Lidar (Light Detection And Ranging) sensor. Behavioral control methods.
15. A behavior control device that controls the behavior of a moving object, selecting a self-location estimation unit to be used for estimating the position of the moving body from among a plurality of self-location estimation units having different estimation accuracies based on an index, and obtaining a state of the moving body; Controlling the behavior of the moving object according to a behavior plan created by a specified behavior task based on a state of the moving object; Specifying the self-location estimation unit to be selected for each of the indices; Including, specifying the self-location estimation unit for each of the predetermined indices according to a priority to be given when the self-location estimation unit estimates the self-location of the moving object for each of the predetermined indices; When the priority is estimation accuracy, the self-location estimation unit that can obtain the highest estimation accuracy within an effective range of altitude and speed in which the self-location estimation unit can function effectively is specified. Behavioral control methods.
16. A behavior control device that controls the behavior of a moving object, selecting a self-location estimation unit to be used for estimating the position of the moving body from among a plurality of self-location estimation units having different estimation accuracies based on an index, and obtaining a state of the moving body; Controlling the behavior of the moving object according to a behavior plan created by a specified behavior task based on a state of the moving object; Specifying the self-location estimation unit to be selected for each of the indices; Including, When the self-location estimation unit and the speed of the moving body in a predetermined index are specified, the lowest altitude at which the specified self-location estimation unit functions effectively is specified for each of the predetermined indexes. Behavioral control methods.
17. A behavior control device that controls the behavior of a moving object, selecting a self-location estimation unit to be used for estimating the position of the moving body from among a plurality of self-location estimation units having different estimation accuracies based on an index, and obtaining a state of the moving body; Controlling the behavior of the moving object according to a behavior plan created by a specified behavior task based on a state of the moving object; Specifying the self-location estimation unit to be selected for each of the indicators; visualizing the self-location estimation unit designated for each of the indicators, and visualizing and displaying the self-location estimation unit used for each route following the indicators; Including, A user instructs the creation of the action plan to specify a plurality of the indices and to specify the self-location estimation unit for each of the indices. Behavioral control methods.
Citation Information
Patent Citations
Position estimation device
JP2017215199A
Vehicle control system, vehicle control method and program
WO2018230461A1
Rubber composition for solid golf ball
JP1987089750A
