Moving object, control method and program

Abstract

To further safely stop.SOLUTION: A plane information calculating unit calculates plane information on a plane based on the plane sensed in an external environment, and a movement control unit controls a movement to the plane based on the calculated plane information. The movement control unit further controls a movement to a plane corresponding to a superimposed image selected by a user on a picked-up image among the superimposed images superimposed at a position on the picked-up image of the external environment, the position corresponding to the plane from which the plane information is calculated. The technology according to the present disclosure is applicable to a moving object like a drone.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present disclosure relates to a moving object, a control method, and a program, and more particularly to a moving object, a control method, and a program that enable a moving object to be stopped more safely. [Background technology]

[0002] Conventionally, there are mobile objects equipped with sensors for observing the external environment in order to move autonomously without colliding with obstacles in the external environment. Mobile objects include autonomously moving drones, vehicles, ships, autonomous mobile robots such as vacuum cleaners, and devices that move attached to the mobile object. For example, cameras, sonar, radar, LiDER (Light Detection and Ranging, Laser Imaging Detection and Ranging), etc. are mainly used as sensors.

[0003] In this context, Patent Document 1 discloses a technology in which a landing site search device, which searches for a landing site for an aircraft, identifies the landing site by evaluating the surface conditions of candidate landing sites based on distance information of the surface obtained from stereo camera images. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2001-328600 A Summary of the Invention [Problem to be solved by the invention]

[0005] An autonomous vehicle needs to be able to stop safely to prevent breakdowns in the vehicle itself. However, there is a risk that the vehicle may stop in an inappropriate place, such as on an inclined surface.

[0006] The present disclosure has been made in light of these circumstances, and makes it possible to stop more safely. [Means for solving the problem]

[0007] The moving body of the present disclosure includes a plane information calculation unit that calculates plane information of a plane based on a plane sensed in an external environment, and a movement control unit that controls movement to the plane based on the calculated plane information, and the movement control unit is a moving body that controls movement to the plane corresponding to a superimposed image selected by a user on a captured image of the external environment, among superimposed images that are superimposed on a position corresponding to the plane for which the plane information was calculated on the captured image.

[0008] The control method disclosed herein is a control method for a moving body, which calculates plane information of a plane based on a plane sensed in an external environment, controls movement to the plane based on the calculated plane information, and controls movement to the plane corresponding to a superimposed image selected by a user on a captured image in which the external environment is captured, among superimposed images superimposed at a position corresponding to the plane for which the plane information was calculated.

[0009] The program disclosed herein is a program for causing a processor to execute a process of calculating plane information of a plane based on a plane sensed in an external environment, controlling movement to the plane based on the calculated plane information, and controlling movement to the plane corresponding to a superimposed image selected by a user on a captured image of the external environment, among the superimposed images superimposed at a position corresponding to the plane for which the plane information was calculated, on the captured image.

[0010] In the present disclosure, plane information of a plane sensed in the external environment is calculated based on the plane, movement to the plane is controlled based on the calculated plane information, and movement to the plane corresponding to a superimposed image selected by a user on a captured image of the external environment among the superimposed images superimposed at a position corresponding to the plane for which the plane information was calculated on the captured image is controlled. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating a moving body to which the technology according to the present disclosure is applied. [Diagram 2] FIG. 2 is a diagram showing the appearance of a moving object. [Diagram 3] FIG. 2 is a block diagram showing a configuration example of a moving body. [Figure 4] 4 is a block diagram showing an example of a functional configuration of a control unit; [Diagram 5] 13 is a flowchart illustrating the flow of a movement control process. [Figure 6] 11A and 11B are diagrams illustrating an example of a method for detecting a plane. [Figure 7] 11A and 11B are diagrams illustrating an example of a detection result of a plane. [Figure 8] FIG. 13 is a diagram illustrating calculation of a safety degree of a plane. [Figure 9] 13 is a block diagram showing another example of the functional configuration of the control unit. FIG. [Figure 10] 13 is a flowchart illustrating the flow of a movement control process. [Figure 11] FIG. 1 is a diagram illustrating detection of a dynamic object. [Figure 12] FIG. 1 is a diagram illustrating semantic segmentation. [Figure 13] FIG. 13 is a diagram illustrating calculation of a safety degree of a plane. [Figure 14] FIG. 13 is a diagram illustrating an example of a stopping stability rate. [Figure 15] FIG. 13 is a diagram illustrating calculation of a safety degree of a plane. [Figure 16]FIG. 2 illustrates an example of the configuration of a controller. [Figure 17] FIG. 13 is a diagram showing an example of a superimposed image. [Figure 18] FIG. 13 is a diagram showing an example of a superimposed image. [Figure 19] FIG. 13 is a diagram showing an example of a superimposed image. [Figure 20] 13 is a flowchart illustrating the flow of a movement control process using a superimposed image. [Figure 21] FIG. 13 is a diagram illustrating setting of a stop target point. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Modes for carrying out the present disclosure (hereinafter, referred to as embodiments) will be described below. The description will be made in the following order.

[0013] 1. Overview of the technology disclosed herein 2. Configuration of moving objects 3. First embodiment (calculation of safety of a plane based on a plane map) 4. Second embodiment (calculation of safety of a plane based on a plane map and external environment information) 5. Displaying superimposed images and specifying stopping locations

[0014] <1. Overview of the Technology Relating to the Present Disclosure> 1, the moving body 10 to which the technology according to the present disclosure is applied is configured to calculate the safety level of a plane that exists in the external environment in the moving space, and to control the movement to the plane based on the calculated safety level. The safety level is an index of the safety after the moving body 10 stops on the plane, and the ease with which the moving body 10 stops on the plane.

[0015] Specifically, the moving body 10 detects planes P1, P2, and P3 in the external environment using sensor data acquired by a sensor (not shown). The moving body 10 calculates the safety level of each of the planes P1, P2, and P3 based on plane information about the detected planes P1, P2, and P3.

[0016] Then, the moving body 10 moves to and stops at the plane that has the highest degree of safety among the calculated planes P1, P2, and P3.

[0017] The mobile body includes autonomously moving drones, vehicles, ships, autonomously moving robots such as vacuum cleaners, as well as devices that move attached to the mobile body. In the following, an example in which the technology according to the present disclosure is applied mainly to a drone flying in the air will be described, but the technology according to the present disclosure can be applied to autonomously moving robots such as autonomous vehicles that move on land, autonomously navigating ships that move on or underwater, and autonomously moving vacuum cleaners that move indoors, in addition to drones.

[0018] <2. Configuration of moving object> FIG. 2 is a diagram showing the appearance of a moving object to which the technology according to the present disclosure (the present technology) is applied.

[0019] As described above, the moving body 20 shown in Fig. 2 is configured as a drone. The moving body 20 moves by flying, but when the moving body 20 is configured as an autonomous vehicle, it moves on land, and when the moving body 20 is configured as an autonomous ship, it moves on or underwater. Also, when the moving body 20 is configured as an autonomous vacuum cleaner, it moves indoors.

[0020] The moving body 20 is equipped with a sensor 21 for observing the external environment so as to move autonomously without colliding with obstacles in the external environment.

[0021] The sensor 21 may be any sensor capable of acquiring a three-dimensional shape of the external environment, and may be configured to include, for example, a camera, a stereo camera, a depth sensor such as a ToF (Time of Flight) sensor, as well as a sonar, a radar, a LiDAR, etc. The sensor 21 may also be configured to include a spectroscopic sensor or a polarized sensor capable of acquiring the material and the degree of unevenness of a flat surface present in the external environment. The sensor data collected by the sensor 21 is used, for example, for controlling the movement of the mobile object 20.

[0022] The moving body 20 may be configured to move autonomously, or may be configured to move according to signals from a controller (not shown) for operating the moving body 20, which may be constituted by a transmitter, a PC (Personal Computer), or the like.

[0023] For example, an autonomous drone needs to land safely to prevent breakdown of the drone. However, there is a risk that the drone may land in an inappropriate location, such as on an inclined ground. Even when a pilot manually flies the drone by operating the controller, the pilot needs to recognize a suitable location for landing.

[0024] Therefore, the moving body 20 of the present technology is configured to detect a place suitable for stopping (landing) using the sensor 21 mounted on the moving body 20, and move to a safer place.

[0025] (Building blocks of moving objects) FIG. 3 is a block diagram showing an example of the configuration of the moving object 20. As shown in FIG.

[0026] The moving body 20 includes a control unit 51, a communication unit 52, a storage unit 53, and a moving mechanism .

[0027] The control unit 51 is composed of a processor such as a CPU (Central Processing Unit), a memory, and the like, and controls the communication unit 52, the storage unit 53, the moving mechanism 54, and the sensor 21 by executing a predetermined program. For example, the control unit 51 controls the moving mechanism 54 based on sensor data collected by the sensor 21.

[0028] The communication unit 52 is composed of a network interface and the like, and performs wireless or wired communication with a controller for operating the mobile object 20 or any other device. For example, the communication unit 52 may perform direct communication with the device to be the communication partner, or may perform network communication via a base station or repeater such as Wi-Fi (registered trademark), 4G, or 5G. The communication unit 52 also receives GPS information transmitted from a GPS satellite.

[0029] The storage unit 53 is configured with a non-volatile memory such as a flash memory, and stores various information under the control of the control unit 51. For example, the storage unit 53 stores (stores) a plane map in which plane information about a plane detected in the external environment is mapped onto a three-dimensional space.

[0030] The moving mechanism 54 is a mechanism for moving the moving body 20, and includes a flight mechanism, a running mechanism, a propulsion mechanism, and the like. In this example, the moving body 20 is configured as a drone, and the moving mechanism 54 is configured as a flight mechanism, such as a motor and a propeller. When the moving body 20 is configured as an autonomous vehicle, the moving mechanism 54 is configured as a running mechanism, such as a wheel, and when the moving body 20 is configured as an autonomous ship, the moving mechanism 54 is configured as a propulsion mechanism, such as a screw propeller. The moving mechanism 54 is driven under the control of the control unit 51 to move the moving body 20.

[0031] <3. First embodiment> (Functional block of the control unit) FIG. 4 is a block diagram showing an example of a functional configuration of the control unit 51.

[0032] The functional blocks of the control unit 51 shown in FIG. 4 are realized by a processor constituting the control unit 51 executing a predetermined program.

[0033] The control unit 51 is made up of a sensor data acquisition unit 71 , a plane detection unit 72 , a self-position estimation unit 73 , a map construction unit 74 , a safety level calculation unit 75 , a movement control unit 76 , and an image generation unit 77 .

[0034] The sensor data acquisition unit 71 acquires sensor data from the sensor 21 and supplies the data to the plane detection unit 72 and the self-position estimation unit 73 .

[0035] The plane detection unit 72 detects a plane existing in the external environment (movement space) based on the sensor data from the sensor data acquisition unit 71, and extracts plane information about the detected plane. The extracted plane information is supplied to the map construction unit 74 and the safety level calculation unit 75.

[0036] The self-position estimation unit 73 estimates the position of itself (the mobile body 20) based on the GPS information received by the communication unit 52, and supplies position information indicating the position to the map construction unit 74. In addition, the self-position estimation unit 73 may estimate its own position based on the sensor data from the sensor data acquisition unit 71 by SLAM (Simultaneous Localization and Mapping).

[0037] Map construction unit 74 constructs a planar map based on the position information from self-position estimation unit 73 and the plane information from plane detection unit 72. The constructed planar map is supplied to safety level calculation unit 75.

[0038] Note that a pre-constructed plan view may be stored in storage unit 53, and the plan view may be read from storage unit 53 and updated every time new sensor data is acquired by sensor data acquisition unit 71. In addition to reading the plan view from storage unit 53, the pre-constructed plan view may be read from an external device or a server on a network via communication unit 52.

[0039] The safety degree calculation unit 75 calculates the safety degree for each plane corresponding to the plane information, based on the plane map from the map construction unit 74 and the plane information from the plane detection unit 72. The calculated safety degree for each plane is supplied to the movement control unit 76 and the image generation unit 77.

[0040] The movement control unit 76 controls the movement of the moving object 20 on the plane based on the safety level from the safety level calculation unit 75 .

[0041] The image generation unit 77 generates a superimposed image to be superimposed on the captured image of the external environment at a position corresponding to each plane for which the safety level has been calculated, based on the safety level from the safety level calculation unit 75. The generated superimposed image is transmitted via the communication unit 52 to a controller or the like on which the captured image of the external environment is displayed.

[0042] (Flow of movement control process) Next, a flow of a movement control process for the moving object 20 will be described with reference to the flowchart of Fig. 5. The process of Fig. 5 is executed, for example, when the moving object 20 moves along a predetermined route and before stopping (arriving) at a destination location.

[0043] In step S11 , the sensor data acquisition unit 71 acquires sensor data from the sensor 21 .

[0044] In step S12 , the plane detection unit 72 detects a plane that exists in the external environment based on the sensor data acquired by the sensor data acquisition unit 71 .

[0045] FIG. 6 is a diagram illustrating an example of a method for detecting a plane.

[0046] 6A, the sensor data acquisition unit 71 acquires point cloud data 100 as depth data from a stereo camera or a ToF sensor constituting the sensor 21. The sensor 21 is configured as a stereo camera or a ToF (Time of Flight) sensor capable of acquiring a three-dimensional shape.

[0047] Next, the plane detection unit 72 groups the acquired point cloud data 100. For example, the point cloud data 100 is grouped based on the position information and distance information of each point constituting the point cloud data 100. In B of Fig. 6, the point cloud data 100 is grouped into three point cloud data groups G1, G2, and G3.

[0048] Next, the plane detection unit 72 calculates the flatness of each point cloud data group, and determines the point cloud data group whose calculated flatness exceeds a predetermined level as a plane candidate. The flatness is a value that represents the smoothness (uniformity) of a plane. In C of FIG. 6, two point cloud data groups G1 and G3 are determined as plane candidates.

[0049] Then, the plane detection unit 72 calculates the size of the point cloud data group determined as a plane candidate, and detects the point cloud data group whose calculated size exceeds a predetermined size as a plane. In D of Fig. 6, the point cloud data group G1 is detected as a plane.

[0050] In this manner, a plane existing in the external environment is detected, but the method of detecting the plane is not limited to the example in Fig. 6, and may be realized by a predetermined plane detection algorithm using the sensor data acquired by the sensor data acquisition unit 71. Furthermore, a plane existing in the external environment may be detected by deep learning.

[0051] FIG. 7 is a diagram illustrating an example of a plane detection result.

[0052] For example, as shown in A of Fig. 7, the detected plane is represented by a normal vector 121, a two-dimensional planar region 122, and their coordinates in the world coordinate system. The shape of the two-dimensional planar region 122 corresponds to the shape of the point cloud data group described above.

[0053] 7B, the detected plane may be represented using an approximate two-dimensional planar region 123 instead of the two-dimensional planar region 122. The approximate two-dimensional planar region 123 is represented by a rectangular plane that approximates the two-dimensional planar region 122.

[0054] Furthermore, as shown in FIG. 7C, the detected plane may be expressed by a set 131 of points constituting the point cloud data group and their coordinates in the world coordinate system.

[0055] In this way, the detected plane is represented as a two-dimensional plane in the world coordinate system.

[0056] Then, the plane detection unit 72 extracts the position, size, and inclination of the plane, which are determined from the above-mentioned coordinates, as plane information for the detected plane. The inclination of the plane is the inclination with respect to the gravitational acceleration direction, and can be calculated based on sensor data from an acceleration sensor provided in the moving body 20. In addition, when the sensor 21 is configured to include a spectroscopic sensor or a polarized sensor, the plane information may include the material of the plane and the degree of unevenness of the plane.

[0057] Such plane information serves as an index for calculating the safety level of each plane.

[0058] Returning to the flowchart of FIG. 5, in step S13, the map construction unit 74 constructs a plane map based on the position information representing its own position estimated by the self-position estimation unit 73 and the plane information extracted by the plane detection unit 72.

[0059] In the planar map, the planar information is mapped (associated) with the coordinates explained with reference to FIG. 7 in the three-dimensional space in the world coordinate system, using the own position information as a reference.

[0060] In step S14, safety level calculation unit 75 calculates the safety level of the plane detected by plane detection unit 72, based on the plane map constructed by map construction unit 74 and the plane information mapped onto the plane map.

[0061] For example, the safety level calculation unit 75 calculates the safety level of a plane based on the degree to which the plane information mapped on the plane map satisfies preset conditions. The preset conditions include, for example, (1) the inclination of the plane with respect to the direction of gravitational acceleration is 5° or less, (2) the size of the plane is 1 m 2 (3) The surface condition is other than water or gravel.

[0062] For example, it is assumed that the plane detection unit 72 detects five planes A to E.

[0063] As shown in Figure 8, Plane A has a slope of 1.2° and a size of 6.0m. 2 Plane B is concrete. Plane B has a slope of 5.0° and a size of 4.5m. 2 The surface condition of the plane is gravel. Plane C has a slope of 12.0° and a size of 2.0m. 2 The plane is in a grassy state. Plane D has a slope of 3.5° and a size of 0.9m. 2 The surface condition of the plane is concrete. Plane E has a slope of 2.3° and a size of 3.7m. 2 The surface condition is gravel.

[0064] Here, plane A satisfies all three of the above conditions (1) to (3), plane B satisfies two, plane C satisfies two, plane D satisfies two, and plane E satisfies two, so in this case, plane A, which satisfies all of conditions (1) to (3), is calculated to have the highest safety level.

[0065] Of course, depending on the conditions that are set, the safety level of a plane other than plane A can be calculated as the highest value.

[0066] In step S15, the movement control unit 76 controls the movement of the moving body 20 to the plane based on the safety degree calculated by the safety degree calculation unit 75. Specifically, the movement control unit 76 controls the movement mechanism 54 to move and stop the moving body 20 on the plane with the highest safety degree among the planes for which the safety degrees have been calculated, as a target stopping position.

[0067] In step S16, it is determined whether or not the moving body 20 has completed moving to the plane having the highest degree of safety. For example, it is determined whether or not the moving body 20 configured as a drone has landed on the plane having the highest degree of safety.

[0068] The process of step S15, i.e., the control of the movement based on the safety level, is repeated until it is determined that the movement of the moving body 20 is completed. Then, for example, when the moving body 20 configured as a drone lands on a plane with the highest safety level and it is determined that the movement of the moving body 20 is completed, the process proceeds to step S17.

[0069] In step S17, the movement control unit 76 feeds back the result of the movement to the plane as plane information. Specifically, the movement control unit 76 notifies the map construction unit 74 that the moving body 20 has safely stopped on the plane that is the target of the stopping position. The map construction unit 74 adds, as plane information, history information indicating that the moving body 20 has safely stopped on the plane that is the target of the stopping position in the constructed planar map.

[0070] According to the above processing, the safety level of a plane existing in the external environment is calculated, and movement onto the plane is controlled based on the calculated safety level, so that the moving body 20 can stop more safely without stopping at a place that is inappropriate for stopping.

[0071] In the above embodiment, the safety level of a plane is calculated based only on the plane information of the plane. However, whether a plane is safe or not depends not only on the condition of the plane itself but also on the condition of the surroundings of the plane.

[0072] Therefore, hereinafter, an embodiment will be described in which the safety level of a plane is calculated based on plane information about the plane and external environment information about the external environment in which the plane exists.

[0073] <4. Second embodiment> (Functional block of the control unit) FIG. 9 is a block diagram showing an example of a functional configuration of the control unit 51 according to the present embodiment.

[0074] The control unit 51 in FIG. 9 includes an external environment recognition unit 151 in addition to the same configuration as the control unit 51 in FIG.

[0075] The external environment recognition unit 151 acquires external environment information by recognizing the state of the external environment (movement space) based on the sensor data from the sensor data acquisition unit 71. The external environment information includes, for example, information indicating the presence or absence of an obstacle in the external environment and attributes of flat surfaces existing in the external environment (whether it is a road surface, a park plaza, an indoor floor surface, etc.). The acquired external environment information is supplied to the safety level calculation unit 75.

[0076] (Flow of movement control process) Next, the flow of the movement control process for the moving object 20 according to this embodiment will be described with reference to the flowchart of FIG.

[0077] The processes in steps S31 to S33 and S36 to S38 in the flowchart of FIG. 10 are similar to the processes in steps S11 to S17 in the flowchart of FIG. 5, respectively, and therefore will not be described.

[0078] That is, in step S34, the external environment recognition unit 151 recognizes the state of the external environment based on the sensor data from the sensor data acquisition unit 71. Specifically, the external environment recognition unit 151 detects an obstacle (specifically, a dynamic object) in the external environment and determines the attributes of a plane existing in the external environment.

[0079] For example, suppose that a captured image 210 as shown in the upper part of Fig. 11 is captured by the sensor data acquisition unit 71 configured as a camera. The captured image 210 includes three people H1, H2, and H3.

[0080] As shown in the lower part of Fig. 11, the external environment recognition unit 151 performs person detection on the captured image 210. In the captured image 210 in the lower part of Fig. 11, a frame F1 indicating that a person H1 has been detected, a frame F2 indicating that a person H2 has been detected, and a frame F3 indicating that a person H3 has been detected are superimposed and displayed.

[0081] In the example of FIG. 11, a person is detected as a dynamic object in the external environment, but animals such as dogs and cats, and other moving objects (for example, drones) other than a person may also be detected.

[0082] Also, for example, it is assumed that a captured image 220 as shown in the upper part of Fig. 12 is captured by the sensor data acquisition unit 71 configured as a camera. The captured image 220 shows a road on which a car is traveling.

[0083] The external environment recognition unit 151 uses semantic segmentation based on machine learning such as deep learning to determine the attributes of the subject in the captured image 220 on a pixel-by-pixel basis, and labels each pixel with the attribute. This results in a processed image 230 as shown in the lower part of Fig. 12. In the processed image 230, the attributes of the subject are determined to be automobiles, roads, sidewalks, houses, walls, trees, sky, etc.

[0084] In this manner, the external environment recognition unit 151 acquires external environment information by recognizing the state of the external environment.

[0085] Then, in step S35, safety level calculation unit 75 calculates the safety level of the plane detected by plane detection unit 72 based on the plane map, the plane information, and the external environment information acquired by external environment recognition unit 151.

[0086] For example, the safety level calculation unit 75 calculates the safety level of a plane based on the degree to which the plane information and the external environment information satisfy preset conditions. The preset conditions include, for example, (1) the inclination of the plane with respect to the direction of gravitational acceleration is 5° or less, (2) the size of the plane is 1 m 2 (3) the surface condition must be other than water or gravel, and (4) there must be no nearby objects within a radius of 2 m.

[0087] For example, it is assumed that the plane detection unit 72 detects five planes A to E.

[0088] As shown in Figure 13, Plane A has a slope of 1.2° and a size of 6.0m. 2 The plane is concrete and there are no nearby objects. Plane B has an inclination of 5.0° and a size of 4.5m. 2 The surface is gravel, and there are two trees nearby. Plane C has an inclination of 12.0° and a size of 2.0 m. 2 The plane is covered with grass and there are no nearby objects. Plane D has an inclination of 3.5° and a size of 0.9m. 2 The plane is concrete, and there are two cars nearby. Plane E has an inclination of 2.3° and a size of 3.7m. 2 The surface condition is gravel, and there are three bicycles as nearby objects.

[0089] Here, all four of the planes A satisfy the above-mentioned conditions (1) to (4), two of the planes B, three of the planes C, two of the planes D, and two of the planes E. Therefore, in this case, the safety level of the plane A, which satisfies all of the conditions (1) to (4), is calculated to be the highest.

[0090] The safety level calculation unit 75 may also calculate the safety level of a plane based on the success rate of a process required for the moving body 20 to stop on a plane. For example, the product of the stopping stability rate calculated from plane information (flatness) and the non-collision probability calculated from external environment information is calculated as the safety level of the plane. For example, even for planes of the same inclination and size, the non-collision probability is higher on a sidewalk than on a roadway, and higher on a park lawn than on a sidewalk. Also, for example, indoors, the non-collision probability is higher on a table top where people do not step than on a floor where people pass by.

[0091] The stopping stability rate may also be calculated based on an experiment.

[0092] Fig. 14 shows an example of a stopping stability ratio calculated based on an experiment. In the example of Fig. 14, when the inclination of the plane is from 0° to a certain angle, the stopping stability ratio only drops slightly from 100%. However, when the inclination of the plane exceeds a certain angle, the stopping stability ratio drops sharply. When the inclination of the plane is 30° or more, the stopping stability ratio becomes 0%, and the moving body 20 cannot stop (land) safely.

[0093] Furthermore, the product of the above-mentioned stopping stability rate and the in-area stopping probability calculated from the control error of the aircraft as external environment information and the size of the plane may be calculated as a stopping success rate indicating the safety of the plane.

[0094] For example, it is assumed that the plane detection unit 72 detects five planes A to E.

[0095] As shown in FIG. 15, the plane A has a stopping stability rate of 99% and a stopping probability within the region of 99%, so that the stopping success rate is 99%. The plane B has a stopping stability rate of 98% and a stopping probability within the region of 80%, so that the stopping success rate is 78.4%. The plane C has a stopping stability rate of 90% and a stopping probability within the region of 20%, so that the stopping success rate is 18%. The plane D has a stopping stability rate of 99% and a stopping probability within the region of 15%, so that the stopping success rate is 14.85%. The plane E has a stopping stability rate of 99% and a stopping probability within the region of 60%, so that the stopping success rate is 59.4%. In FIG. 15, the inclination and magnitude of each of the planes A to E are the same as those in FIG. 8 and FIG. 13.

[0096] Thus, in the example of FIG. 15, the stopping success rate of plane A is the highest, and the safety degree of plane A is calculated to be the highest value.

[0097] According to the above processing, movement onto the plane is controlled based on the safety of the plane calculated based on the plane information and the external environment information, so that the moving body 20 can stop more safely without stopping at a place that is inappropriate for stopping and without colliding with an obstacle after stopping.

[0098] <5. Displaying superimposed images and specifying stopping locations> FIG. 16 is a diagram showing an example of the configuration of a controller for operating the moving body 20. As shown in FIG.

[0099] 16 is configured such that a smartphone 310 is attached to a dedicated transmitter. As described above, the moving object 20 may be configured to move according to a signal from the controller 300, or may be configured to move autonomously.

[0100] 16, a captured image of the external environment captured by the sensor 21 configured as a camera during the movement of the moving object 20 is displayed on a screen 320 of the smartphone 310. The captured image may be a moving image or a still image.

[0101] In the example of FIG. 16, an image captured by a moving object 20 configured as a drone while flying over a living room is displayed on a screen 320.

[0102] (Display of superimposed images) At positions on the captured image displayed on the screen 320 corresponding to the planes for which the safety levels have been calculated, superimposed images generated by the image generating unit 77 based on the calculated safety levels are displayed.

[0103] For example, superimposed images 351, 352, and 353 imitating planes are displayed at positions on photographed image 331 shown in Fig. 17 corresponding to a plane for which a safety level greater than a predetermined value has been calculated. Superimposed image 351 is superimposed at a position on photographed image 331 corresponding to the top surface of a table, and superimposed image 352 is superimposed at a position on photographed image 331 corresponding to the seat surface of a sofa. Superimposed image 353 is superimposed at a position on photographed image 331 corresponding to the floor surface of the living room.

[0104] The superimposed images 351, 352, and 353 may be displayed in a color corresponding to the safety level of the corresponding plane, such as green if the safety level is relatively high, yellow if the safety level is medium, and red if the safety level is lower than medium.

[0105] For example, since people will not step on the top of a table, superimposed image 351 is displayed in green, since there is a possibility that people may sit on the seat of a sofa, superimposed image 352 is displayed in yellow, and since there is a high possibility that people will step on the floor of a living room, superimposed image 352 is displayed in red.

[0106] In the example of FIG. 16, a superimposed image resembling a plane is displayed on the photographed image 331, but the display form of the superimposed image is not limited to this.

[0107] For example, as shown in FIG. 18, arrow-shaped superimposed images 361, 362, and 363 may be displayed on a captured image 331 at positions corresponding to planes for which a safety level greater than a predetermined value has been calculated.

[0108] Furthermore, as shown in FIG. 19, superimposed images 371, 372, and 373 resembling flags may be displayed on a captured image 331 at positions corresponding to planes for which a safety level greater than a predetermined value has been calculated.

[0109] In addition, the above-mentioned superimposed image may be displayed in a color corresponding to the safety level of the corresponding plane, or may be displayed at a size corresponding to the safety level or may be displayed blinking at a speed corresponding to the safety level.

[0110] (Designation of stop location) If the moving body 20 is configured to move autonomously, a plane corresponding to a superimposed image selected by a user on a captured image displayed on a screen 320 having a touch panel function may be specified as a stopping place for the moving body 20.

[0111] For example, when a user touches a superimposed image 351 in the captured image 331 of FIG. 17, the moving body 20 configured as a drone is controlled to land on the top surface of a table corresponding to the superimposed image 351.

[0112] Here, the flow of the movement control process of the moving object 20 using the superimposed image will be described with reference to the flowchart of FIG.

[0113] In step S51, image generating unit 77 generates a superimposed image based on the planar map generated by map constructing unit 74 and the degree of safety calculated by safety degree calculating unit 75.

[0114] In step S52, the communication unit 52 transmits the superimposed image generated by the image generation unit 77 to the controller 300 (smartphone 310).

[0115] As a result, a superimposed image is displayed on the captured image displayed on the screen 320 of the smartphone 310 at a position corresponding to a plane for which a safety level greater than a predetermined value has been calculated.

[0116] In this state, when the user selects any of the superimposed images in the captured image displayed on screen 320 of controller 300 (smartphone 310), controller 300 transmits a signal indicating that the superimposed image has been selected to moving object 20.

[0117] In step S53, the movement control unit 76 determines, based on a signal from the controller 300, whether or not a superimposed image has been selected by the user in the photographed image displayed on the controller 300.

[0118] The process of step S53 is repeated until it is determined that a superimposed image has been selected, and when it is determined that a superimposed image has been selected, the process proceeds to step S54.

[0119] In step S54, the movement control unit 76 controls the movement of the moving object 20 to the plane corresponding to the selected superimposed image.

[0120] At this time, the movement control unit 76 sets a stop target point for the player's aircraft within the planar area to which the player will move.

[0121] For example, when the planar area to which the user moves is represented by point cloud data, the point at the center of gravity of the point cloud data is set as the stop target point.

[0122] Also, a point located farthest from the edge of the planar area to which the object is to be moved may be set as the target stopping point.

[0123] For example, as shown in A of Fig. 21, the movement control unit 76 generates candidate points CP that are arranged at regular intervals across the entire planar region 411. Then, as shown in B of Fig. 21, the movement control unit 76 calculates the distance from each of the generated candidate points CP to the edge of the planar region 411 in directions at regular angular intervals (eight directions in the example of Fig. 21). The candidate point CP with the largest minimum value among the distances calculated in this way is set as the stop target point.

[0124] Furthermore, if the planar area to which the object is to be moved is shaped like a circle or a rectangle, the center of gravity of the shape may be set as the stop target point.

[0125] According to the above process, movement to a plane specified by the user is controlled among planes with a relatively high degree of safety, so that the moving body 20 can be stopped more safely at a location desired by the user.

[0126] The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed from a network or a program recording medium.

[0127] The embodiments of the technology according to the present disclosure are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the technology according to the present disclosure.

[0128] Furthermore, the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0129] Furthermore, the technology according to the present disclosure can have the following configuration. (1) a safety degree calculation unit that calculates a safety degree of a plane based on plane information about a plane existing in an external environment; a movement control unit that controls movement onto the plane based on the calculated safety level; A moving body comprising: (2) The safety level calculation unit calculates the safety level based on a plane map in which the plane information is mapped to coordinates of the plane in a three-dimensional space. A moving object as described in (1). (3) The plane information includes the position, size, and inclination of the plane. A moving object as described in (2). (4) The inclination is relative to the direction of gravitational acceleration. A moving object as described in (3). (5) The plane information further includes a material of the plane. A mobile object according to (3) or (4). (6) The plane information further includes a degree of unevenness of the plane. A moving body according to any one of (3) to (5). (7) The vehicle further includes a map constructing unit that constructs the plane map based on its own position information and the plane information about the plane detected in the external environment using sensor data. A moving body according to any one of (2) to (6). (8) The safety level calculation unit calculates the safety level based on the plan map and external environment information regarding the external environment. A moving body according to any one of (2) to (7). (9) The external environment information includes information indicating the presence or absence of an obstacle in the external environment. A moving object as described in (8). (10) The obstacle is a dynamic object. A moving object as described in (9). (11) The external environment information includes attributes of the plane that exists in the external environment. A moving body according to any one of (8) to (10). (12) The attributes are determined by semantic segmentation. A moving object as described in (11). (13) The external environment recognition unit acquires the external environment information by recognizing a state of the external environment based on sensor data. A moving body according to any one of (8) to (12). (14) The movement control unit controls the movement to the plane having the highest degree of safety. A moving body according to any one of (1) to (13). (15) The moving body according to (14), wherein the movement control unit feeds back the result of the movement to the plane as the plane information. (16) The apparatus further includes an image generating unit that generates a superimposed image to be superimposed on a photographed image of the external environment at a position corresponding to the plane for which the safety level is calculated. A moving body according to any one of (1) to (15). (17) The image generating unit generates the superimposed image that is displayed in a color corresponding to the safety level. A moving object as described in (16). (18) The movement control unit controls the movement to the plane corresponding to the superimposed image selected by a user on the captured image. A mobile object according to (16) or (17). (19) The moving body, Calculating a safety level of a plane based on plane information about a plane present in an external environment; and controlling movement to the plane based on the calculated safety level. Control methods. (20) The processor: Calculating a safety level of a plane based on plane information about a plane present in an external environment; and controlling movement to the plane based on the calculated safety level. A program for executing a process. [Explanation of symbols]

[0130] 10 Mobile object, 20 Mobile object, 21 Sensor, 51 Control unit, 52 Communication unit, 53 Memory unit, 54 Mobile mechanism, 71 Sensor data acquisition unit, 72 Plane detection unit, 73 Self-position estimation unit, 74 Map construction unit, 75 Safety degree calculation unit, 76 Mobile control unit, 77 Image generation unit, 151 External environment recognition unit

Claims

1. a plane information calculation unit that calculates plane information of a plane based on a plane sensed in an external environment; a movement control unit that controls movement onto the plane based on the calculated plane information; Equipped with The movement control unit controls movement to the plane corresponding to the superimposed image selected by a user on the captured image among superimposed images superimposed at a position on the captured image of the external environment corresponding to the plane for which the plane information is calculated. Mobile body.

2. The plane information includes the position, size, and inclination of the plane. The moving body according to claim 1 .

3. The inclination is relative to the direction of gravitational acceleration. The moving body according to claim 2.

4. The plane information further includes a material of the plane. The moving body according to claim 2.

5. The plane information further includes a degree of unevenness of the plane. The moving body according to claim 2.

6. The present invention further includes a map construction unit that constructs a plane map in which the plane information is mapped to coordinates of the plane in a three-dimensional space based on the position information of the vehicle itself and the plane information. The moving body according to claim 1 .

7. The superimposed image may be generated by an image generating unit. The moving body according to claim 1 .

8. The image generating unit generates the superimposed image that is displayed in a color according to the plane information. The moving body according to claim 7.

9. A method for controlling a moving object, comprising: Calculating plane information of a plane based on a plane sensed in an external environment; Controlling movement onto the plane based on the calculated plane information; Among superimposed images superimposed at positions on a photographed image of the external environment corresponding to the plane for which the plane information is calculated, a movement to the plane corresponding to the superimposed image selected by a user on the photographed image is controlled. Control methods.

10. The processor: Calculating plane information of a plane based on a plane sensed in an external environment; Controlling movement onto the plane based on the calculated plane information; Among superimposed images superimposed at positions on a photographed image of the external environment corresponding to the plane for which the plane information is calculated, a movement to the plane corresponding to the superimposed image selected by a user on the photographed image is controlled. A program for executing a process.

Images