Information processing method, information processing device, and program

Abstract

To enable high-speed autonomous flight of a flying object.SOLUTION: A three-dimensional real-time observation result is generated based on self-position estimation information and three-dimensional distance measurement information. A preliminary map corresponding to the three-dimensional real-time observation result is acquired. The three-dimensional real-time observation result is aligned with the preliminary map. After the alignment, the three-dimensional real-time observation result is expanded based on the preliminary map. A flight route is set based on the expanded three-dimensional real-time observation result. For a flying object such as a drone, a flight route that is long to some extent can be accurately calculated all at a time by a global action plan, and high-speed autonomous flight of the flying object becomes possible.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] This technology is 、 The present invention relates to an information processing method, an information processing device, and a program. [Background technology]

[0002] When a drone, which is an aerial vehicle, flies autonomously, it plots a flight path to its destination using a global action plan and then flies along that flight path repeatedly. Because route calculation takes time, in order to fly at high speed, it is necessary to calculate a fairly long route at once, and it is also necessary to plot routes in unobserved areas. For example, if a route is drawn assuming that there is nothing in the unobserved area, there is the inconvenience of a collision if an obstacle suddenly appears in an area that cannot be observed until the very last moment.

[0003] For example, Patent Document 1 describes a technology that creates an integrated map by overlaying a pre-stored environmental information map with information on observed obstacles, and controls the movement of a robot along a predetermined route while avoiding obstacles on the integrated map. Furthermore, for example, Patent Document 2 describes a technology that estimates the vehicle's self-position by matching registered images included in map data with observed images taken from the vehicle. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2007-249632 [Patent Document 2] Japanese Patent Application Publication No. 2019-045892 Summary of the Invention [Problem to be solved by the invention]

[0005] The purpose of this technology is to enable high-speed autonomous flight of an aircraft. [Means for solving the problem]

[0006] The concept of this technology is: a generating unit that generates a 3D real-time observation result based on the self-location estimation information and the 3D ranging information; an acquisition unit that acquires a preliminary map corresponding to the 3D real-time observation result; a registration unit that registers the three-dimensional real-time observation result with the preliminary map; and an expansion unit that expands the 3D real-time observation results based on the preliminary map after the alignment. It is located in the information processing device.

[0007] In this technology, a generating unit generates a 3D real-time observation result based on self-location estimation information and 3D ranging information. For example, the 3D real-time observation result may be a 3D proprietary grid map. An acquiring unit acquires a preliminary map corresponding to the 3D real-time observation result.

[0008] The alignment unit aligns the 3D real-time observation results with the prior map. Then, the expansion unit expands the 3D real-time observation results based on the prior map. For example, the system may further include an environmental structure recognition unit that performs plane detection on the 3D real-time observation results, and the expansion unit may use the results of the plane detection to expand the plane based on information from the prior map. In this case, for example, the environmental structure recognition unit may further perform semantic segmentation on the 3D real-time observation results, and the expansion unit may use the results of the semantic segmentation to expand the plane if the semantics are continuous.

[0009] In this way, this technology aligns the 3D real-time observation results with a pre-defined map, and then expands the 3D real-time observation results based on the pre-defined map. Therefore, by using the expanded 3D real-time observation results, it is possible to grasp the state of unobserved areas in advance, and for example, in the case of an aerial vehicle such as a drone, it is possible to accurately calculate a fairly long flight path at once using a global action plan, enabling high-speed autonomous flight of the aerial vehicle.

[0010] Another concept of the present technology is a generating unit that generates a 3D real-time observation result based on the self-location estimation information and the 3D ranging information; an acquisition unit that acquires a preliminary map corresponding to the 3D real-time observation result; a registration unit that registers the three-dimensional real-time observation result with the preliminary map; an expansion unit that expands the 3D real-time observation result based on the preliminary map after the alignment; The aircraft is equipped with a flight planning unit that sets a flight path based on the expanded 3D real-time observation results. It's on the aircraft.

[0011] In this technology, a generating unit generates 3D real-time observation results based on self-location estimation information and 3D ranging information. An acquiring unit acquires a preliminary map corresponding to the 3D real-time observation results. For example, the acquiring unit may acquire the preliminary map from another aircraft via communication. In this case, for example, the preliminary map may be a map based on the 3D real-time observation results generated by the other aircraft.

[0012] For example, in this case, the preliminary map may be a map obtained by performing a process of cutting the 3D real-time observation results at a certain height and converting them into a bird's-eye view. Also, for example, in this case, the preliminary map may be a map obtained by performing a process of reducing the resolution of the 3D real-time observation results to an extent that communication is possible.

[0013] The alignment unit aligns the 3D real-time observation results with the pre-established map. The expansion unit expands the 3D real-time observation results based on the pre-established map. The action planning unit then sets a flight path based on the expanded 3D real-time observation results.

[0014] For example, the system may further include an environmental structure recognition unit that performs plane detection on the 3D real-time observation results, and the extension unit may use the plane detection results to extend the plane based on information from the prior map. In this case, for example, the environmental structure recognition unit may further perform semantic segmentation on the 3D real-time observation results, and the extension unit may use the semantic segmentation results to extend the plane when the semantics are continuous.

[0015] In this way, this technology aligns the 3D real-time observation results with a pre-defined map, then expands the 3D real-time observation results based on the pre-defined map, and sets a flight path based on this expanded 3D real-time observation result. As a result, for example, in the case of an aerial vehicle such as a drone, it is possible to accurately calculate a fairly long flight path at once using a global action plan, enabling high-speed autonomous flight of the aerial vehicle. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 is a diagram illustrating an outline of the autonomous flight operation of a drone as an air vehicle. [Figure 2] FIG. 1 is a diagram illustrating an outline of alignment and expansion. [Figure 3] FIG. 1 is a block diagram illustrating an example configuration of a drone. [Figure 4] 10 is a flowchart showing an example of a processing procedure for redrawing a flight path. [Figure 5] FIG. 10 is a diagram illustrating a state in which a drone acquires a preliminary map from another drone through communication. DETAILED DESCRIPTION OF THE INVENTION

[0017] The following describes modes for carrying out the invention (hereinafter referred to as "embodiments") in the following order: 1. Embodiment 2. Variations

[0018] <1. Embodiment> 1 shows a schematic diagram of the autonomous flight operation of a drone 10. In an observation area 20, the drone 10 generates 3D real-time observation results, such as a 3D proprietary grid map, based on self-location estimation information and 3D ranging information. Furthermore, in an unobserved area 30, the drone 10 expands the 3D real-time observation results based on a preliminary map. The unobserved area 30 includes, for example, areas that cannot be observed due to obstacles, areas outside the measurement range of a sensor, and the like.

[0019] Here, the preliminary map is a simple map that describes rough information about the environment in which the drone 10 will fly. For example, this preliminary map is a two-dimensional or three-dimensional map that shows the positions and sizes of walls, buildings, etc. More specifically, this includes two-dimensional or three-dimensional maps, topographical maps, building floor plans, etc. stored on a server on the cloud.

[0020] This preliminary map may be stored in storage by the drone 10. In order for the drone 10 to fly at high speed, it is necessary to store a preliminary map covering a relatively wide range. If the preliminary map is simple, the data volume is small, and the drone 10 can store a preliminary map covering a relatively wide range. This preliminary map only needs to show the rough positions and sizes of obstacles.

[0021] In addition, this preliminary map is always stored on a cloud server, and the drone 10 can download and use the preliminary map of the required range from the cloud server each time. If the preliminary map is simple, the data volume is small and it can be downloaded in a short time.

[0022] When expanding the 3D real-time observation results based on the preliminary map, the drone 10 aligns the 3D real-time observation results with the preliminary map. In this case, the 3D real-time observation results are first aligned with the dimensions of the preliminary map. For example, if the preliminary map is two-dimensional, a map of a certain range from the drone's height in the 3D real-time observation results is convolved into two dimensions. Next, well-known alignment methods such as ICP (Iterative Closest Points) and NDT (Normal Distributions Transform) are used to align the results with the map.

[0023] After alignment, the drone 10 expands the 3D real-time observation results based on the pre-map. The method of this expansion will be described below. In this case, a plane is detected from the 3D real-time observation results, and if a space corresponding to that plane is found in the pre-map, the plane is expanded. In this case, semantic segmentation is further performed on the 3D real-time observation results, and the results are used to expand the plane if the semantics are continuous. By further utilizing the results of semantic segmentation in this way, erroneous expansion can be suppressed.

[0024] In this case, if a space corresponding to a plane detected from the 3D real-time observation results is found in the prior map, and the semantics (walls, roads, ground, buildings, etc.) are continuous at the connection point between the 3D real-time observation results and the prior map related to that plane, the plane detected from the 3D real-time observation results is extended based on the prior map.

[0025] Figure 2 shows a schematic overview of the alignment and expansion. Figure 2(a) shows a 3D real-time observation result observed by a drone 10. In the illustrated example, a bottom and a wall are present in the 3D real-time observation result.

[0026] Figure 2(b) shows the state in which the 3D real-time observation results obtained by the drone 10 have been aligned to match the preliminary map (2D in the illustrated example). As mentioned above, this alignment is performed using well-known alignment techniques such as ICP and NDT. This alignment corrects the positional deviations of walls, roads, etc. in the 3D real-time observation results so that they match the preliminary map.

[0027] 2(c) shows a state in which the 3D real-time observation results observed by the drone 10 are expanded based on a preliminary map (2D in the illustrated example). In this case, the wall portion of the 3D real-time observation results is detected as a plane, and since a space corresponding to this plane exists in the preliminary map, the wall portion of the 3D real-time observation results is extended toward the preliminary map, thereby expanding the 3D real-time observation results.

[0028] In this case, semantic segmentation determines the semantics of the bottom part of the 3D real-time observation results, and it is assumed that this is the same as the semantics of the spatial part of the subsequent prior map, confirming the continuity of the semantics.

[0029] Returning to Figure 1, the drone 10 performs a global action plan based on the expanded 3D real-time observation results and sets a flight path to the destination. The drone 10 then flies along this flight path 40, creating control information required for that flight as a local action plan. This control information includes information such as the speed and acceleration of the drone 10, as well as corrective path information based on obstacle detection.

[0030] "Drone configuration example" 3 shows an example of the configuration of the drone 10. The drone 10 includes a drone-mounted PC 100, a drone control unit 200, a sensor unit 300, and an external storage 400.

[0031] The sensor unit 200 includes a stereo camera, a LiDAR (Light Detection and Ranging), etc. The external storage 400 stores a preliminary map. This preliminary map is a simple two-dimensional or three-dimensional map, a topographical map, a building floor plan, etc. that corresponds to a fairly wide area over which the drone 10 will fly. In this case, the preliminary map may be stored in the external storage 400 from the beginning, or a preliminary map of a required area may be obtained from a server on the cloud and stored in the external storage 400.

[0032] The drone-mounted PC 100 has a self-position estimation unit 101, a 3D ranging unit 102, a real-time observation result management unit 103, an environmental structure recognition unit 104, a pre-map acquisition unit 105, a positioning unit 106, an expansion unit 107, a global behavior planning unit 108, and a local behavior planning unit 109.

[0033] The self-position estimation unit 101 estimates the self-position based on the sensor output of the sensor unit 300. In this case, for example, a relative position from the activation position is estimated. The three-dimensional ranging unit 103 acquires depth information of the surrounding environment based on the sensor output of the sensor unit 300.

[0034] The real-time observation result management unit 103 creates a three-dimensional real-time observation result (for example, a three-dimensional exclusive grid map) based on the self-position estimated by the self-position estimation unit 101 and the depth information of the surrounding environment obtained by the three-dimensional ranging unit 102. In this case, the three-dimensional real-time observation result is generated by adding the depth information of the surrounding environment along with the self-position.

[0035] The environmental structure recognition unit 104 recognizes the environmental structure based on the 3D real-time observation results generated by the real-time observation result management unit 103. Specifically, it performs plane detection and semantic segmentation on the 3D real-time observation results.

[0036] The advance map acquisition unit 105 acquires from the external storage 400 an advance map corresponding to the 3D real-time observation results generated by the real-time observation result management unit 103. In this case, the range of the advance map needs to be a fairly wide range that encompasses the range of the 3D real-time observation results, since the 3D real-time observation results are expanded based on this advance map.

[0037] The alignment unit 106 refers to the results of plane detection and semantic segmentation obtained by the environmental structure recognition unit 104, and uses well-known alignment techniques such as ICP and NDT to correct the position of the 3D real-time observation results and align them with the pre-map (see Figure 2(b)).

[0038] After alignment, the expansion unit 107 expands the 3D real-time observation results based on the prior map, based on the plane detection and semantic segmentation results obtained by the environmental structure recognition unit 104 (see FIG. 2(c)). In this case, if a space corresponding to a plane detected from the 3D real-time observation results is found in the prior map, the plane is expanded. In this case, if the semantics are continuous at the connection point between the 3D real-time observation results and the prior map related to the plane, the plane is expanded.

[0039] The global action planning unit 108 performs global action planning and sets a flight path to the destination based on the expanded 3D real-time observation results obtained by the expansion unit 107. The local action planning unit 109 creates control information required to fly along the flight path set by the global action plan.

[0040] The drone control unit 200 receives control information obtained by the local behavior planning unit 109 of the drone-mounted PC 100, and controls the motors to drive the propellers so that the drone 10 flies along the set flight path.

[0041] The flowchart in Figure 4 shows an example of the processing procedure for redrawing a flight path. In step ST1, the drone-mounted PC 100 starts processing when a flight path redrawing management unit (not shown in Figure 3) instructs the drone to redraw a flight path. The flight path redrawing management unit instructs the drone to redraw a flight path if the flight path is unreasonable, for example, if there is an unexpected large obstacle on the already set flight path. The flight path redrawing management unit also instructs the drone to redraw a flight path at regular intervals or after flying a certain distance.

[0042] Next, in step ST2, the drone-mounted PC 100 causes the real-time observation result management unit 103 to generate new 3D real-time observation results and update the 3D real-time observation results. Next, in step ST3, the drone-mounted PC 100 causes the advance map acquisition unit 105 to acquire from the external storage 400 a 2D or 3D advance map corresponding to the updated real-time observation results.

[0043] Next, in step ST4, the drone-mounted PC 100 recognizes the environmental structure from the 3D real-time observation results in the environmental structure recognition unit 104. Specifically, plane detection and semantic segmentation are performed on the 3D real-time observation results.

[0044] Next, in step ST5, the drone-mounted PC 100 uses the alignment unit 106 to refer to the results of plane detection and semantic segmentation, and uses well-known alignment techniques such as ICP and NDT to correct the position of the 3D real-time observation results and align them with the preliminary map.

[0045] Next, in step ST6, the drone-mounted PC 100, in the expansion unit 107, expands the 3D real-time observation results based on the prior map, based on the results of plane detection and semantic segmentation. In this case, if a space corresponding to a plane detected from the 3D real-time observation results is found in the prior map, the plane is expanded. In this case, if the semantics are continuous at the connection point between the 3D real-time observation results and the prior map related to the plane, the plane is expanded.

[0046] Next, in step ST7, the drone-mounted PC 100 performs global action planning based on the expanded 3D real-time observation results in the global action planning unit 108, and sets a flight path to the destination. After that, the drone-mounted PC 100 ends the series of processes in step ST8.

[0047] As described above, in the drone 10 shown in Fig. 1, the 3D real-time observation results are aligned with a pre-arranged map, then the 3D real-time observation results are expanded based on the pre-arranged map, and a global action plan is performed based on the expanded 3D real-time observation results to set a flight path. Therefore, the expanded 3D real-time observation results make it possible to grasp the state of unobserved areas in advance, and for example, in an aircraft such as a drone, a fairly long flight path can be calculated accurately at once using a global action plan, enabling high-speed autonomous flight of the drone 10.

[0048] In the above description, an example has been shown in which the drone 10 acquires a pre-planned map from the external storage 400. As another example, the drone 10 may acquire a pre-planned map from another drone 10A through communication. Figure 5 shows a schematic diagram of this situation.

[0049] Although detailed description will be omitted, the drone 10A is configured similarly to the drone 10. The drone 10A transmits to the drone 10 a preliminary map obtained by converting the 3D real-time observation results into a simple map format. For example, this preliminary map is a map obtained by performing a process of cutting the 3D real-time observation results at a certain height and converting them into a bird's-eye view. Also, for example, this preliminary map is a map obtained by performing a process of reducing the resolution of the 3D real-time observation results to an extent that communication is possible.

[0050] In the example of Fig. 5, there is one other drone 10A, but the number of other drones 10A that transmit the preliminary map to the drone 10 is not limited to one and may be two or more. The greater the number of other drones 10A, the wider the range of the preliminary map transmitted to the drone 10.

[0051] In this way, by sending and sharing a preliminary map from the other drone 10A to the drone 10, it becomes possible to effectively utilize the 3D real-time observation results obtained by the other drone 10A. In this case, it becomes possible for the drone 10 to avoid dead ends or the like that have been identified by the other drone 10A without observing them.

[0052] <2. Modifications> In the above-described embodiment, an example has been shown in which the flying object is a drone. Although detailed description will be omitted, the present technology can be similarly applied to other flying objects.

[0053] Furthermore, while the preferred embodiments of the present disclosure have been described in detail with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person skilled in the art of the present disclosure can conceive of various modified or altered examples within the scope of the technical ideas described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0054] Furthermore, the effects described herein are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that will be apparent to those skilled in the art from the description of this specification, in addition to or in place of the above-described effects.

[0055] The present technology can also be configured as follows. (1) a generating unit that generates a 3D real-time observation result based on self-location estimation information and 3D ranging information; an acquisition unit that acquires a preliminary map corresponding to the 3D real-time observation result; a registration unit that registers the three-dimensional real-time observation result with the preliminary map; and an expansion unit that expands the 3D real-time observation results based on the preliminary map after the alignment. Information processing device. (2) further comprising an environmental structure recognition unit that performs plane detection on the 3D real-time observation results; The expansion unit uses the result of the plane detection to expand the plane based on the information of the preliminary map. The information processing device according to (1) above. (3) The environmental structure recognition unit further performs semantic segmentation on the 3D real-time observation results, The extension unit uses the results of the semantic segmentation to extend the plane if the semantics are continuous. The information processing device according to (2) above. (4) The above 3D real-time observation results are a 3D proprietary grid map. The information processing device according to any one of (1) to (3). (5) a procedure for generating a 3D real-time observation result based on the self-location estimation information and the 3D ranging information; A procedure for obtaining a preliminary map corresponding to the above 3D real-time observation results; a step of aligning the three-dimensional real-time observation results with the preliminary map; After the registration, the method includes a procedure for expanding the 3D real-time observation results based on the preliminary map. Information processing methods. (6) The computer generating means for generating a three-dimensional real-time observation result based on the self-location estimation information and the three-dimensional ranging information; an acquisition means for acquiring a preliminary map corresponding to the 3D real-time observation result; a positioning means for positioning the three-dimensional real-time observation result with the preliminary map; After the above alignment, the system functions as an extension means for extending the above 3D real-time observation results based on the above prior map. program. (7) a generating unit that generates a 3D real-time observation result based on the self-location estimation information and the 3D ranging information; an acquisition unit that acquires a preliminary map corresponding to the 3D real-time observation result; a registration unit that registers the three-dimensional real-time observation result with the preliminary map; an expansion unit that expands the 3D real-time observation result based on the preliminary map after the alignment; The aircraft is equipped with a flight planning unit that sets a flight path based on the expanded 3D real-time observation results. Flying vehicle. (8) The acquisition unit acquires the advance map from another aircraft via communication. The aircraft described in (7) above. (9) The preliminary map is a map based on the three-dimensional real-time observation results generated by the other aircraft. The aircraft described in (8) above. (10) The above-mentioned preliminary map is a map obtained by cutting the above-mentioned 3D real-time observation results at a certain height and converting them into a bird's-eye view. The aircraft described in (9) above. (11) The above-mentioned preliminary map is a map obtained by processing the above-mentioned 3D real-time observation results to reduce the resolution to an extent that the above-mentioned communication is possible. The aircraft described in (9) above. (12) An environmental structure recognition unit that detects planes from the three-dimensional real-time observation results is further provided. The expansion unit uses the result of the plane detection to expand the plane based on the information of the preliminary map. An aircraft described in any one of (7) to (11). (13) The environmental structure recognition unit further performs semantic segmentation on the 3D real-time observation results; The extension unit uses the results of the semantic segmentation to extend the plane if the semantics are continuous. 13. The aircraft described in 12. [Explanation of symbols]

[0056] 10,10A... Drone 20. Observation area 30. Unobserved Area 100···Drone-mounted PC 101... Self-position estimation section 102...3D distance measurement section 103 Real-time Observation Results Management Department 104...Environmental structure recognition unit 105 Pre-map acquisition section 106 Alignment part 107 Extension 108 Global Action Planning Department 109 Local Action Planning Department 200 Drone control unit 300 Sensor section 400...External Storage

Claims

1. A step of acquiring three-dimensional distance measurement information based on sensor data from a sensor mounted on a moving object; generating a three-dimensional real-time observation result of the surrounding environment of the mobile object based on the self-position estimation information of the mobile object and the three-dimensional ranging information; a step of obtaining a preliminary map corresponding to the three-dimensional real-time observation result; a procedure for dynamically setting a movement route of the moving object from a current position of the moving object as a starting point through an unobserved area of the three-dimensional ranging information based on the three-dimensional real-time observation result and the advance map; Information processing methods.

2. and registering the three-dimensional real-time observation results with the preliminary map. In the step of setting the moving path, a moving path of the moving object is dynamically set based on the aligned three-dimensional real-time observation result and the advance map. The information processing method according to claim 1 .

3. In the step of setting the movement route, the three-dimensional real-time observation result is expanded based on the advance map, and the movement route of the moving body is dynamically set based on the expanded three-dimensional real-time observation result. The information processing method according to claim 1 .

4. further comprising a step of detecting a plane from the three-dimensional real-time observation result; In the procedure for setting the travel route, the plane detected by the plane detection in the three-dimensional real-time observation result is expanded based on the information of the preliminary map. The information processing method according to claim 3 .

5. In the procedure for setting the travel route, the result of semantic segmentation performed on the 3D real-time observation result is used, and if a space corresponding to the plane detected by the plane detection exists in the prior map and the semantics are continuous, the plane detected by the plane detection in the 3D real-time observation result is expanded. The information processing method according to claim 4.

6. The above-mentioned preliminary map includes a preliminary map based on 3D real-time observation results generated by another mobile object. The information processing method according to claim 1 .

7. The moving body or the other moving body includes an air vehicle. The information processing method according to claim 6.

8. The above-mentioned preliminary map is a map obtained by cutting the 3D real-time observation results generated by another mobile body at a certain height and converting it into a bird's-eye view. The information processing method according to claim 6.

9. The above-mentioned preliminary map is a map obtained by processing the 3D real-time observation results generated by another mobile body to reduce the resolution to a level that allows communication. The information processing method according to claim 6.

10. The above 3D real-time observation results include a 3D proprietary grid map. The information processing method according to claim 1 .

11. An acquisition unit that acquires three-dimensional ranging information based on sensor data from a sensor mounted on a moving object; a generating unit that generates a three-dimensional real-time observation result of the surrounding environment of the moving object based on the self-position estimation information of the moving object and the three-dimensional ranging information; an acquisition unit that acquires a preliminary map corresponding to the three-dimensional real-time observation result; a setting unit that dynamically sets a movement route of the moving object from a current position of the moving object as a starting point through an unobserved area of the three-dimensional ranging information based on the three-dimensional real-time observation result and the advance map; Information processing device.

12. Computer, an acquisition means for acquiring three-dimensional distance measurement information based on sensor data from a sensor mounted on the moving object; generating means for generating a three-dimensional real-time observation result of the surrounding environment of the mobile object based on the self-position estimation information of the mobile object and the three-dimensional ranging information; an acquisition means for acquiring a preliminary map corresponding to the three-dimensional real-time observation result; The system functions as a setting means for dynamically setting a moving path of the moving object from the current position of the moving object as a starting point through an unobserved area of the three-dimensional ranging information based on the three-dimensional real-time observation result and the advance map. program.

Images