Information processing device, information processing method, and moving body control system

Abstract

The present disclosure relates to an information processing device, an information processing method, and a moving body control system that make it possible to control the movement of a moving body more efficiently. A position information estimation unit estimates moving body position information by using point cloud data acquired by two or more three-dimensional point cloud acquisition devices provided in an environment including the moving body so that at least a part of the scanning ranges overlap, and a movement control unit controls the movement of the moving body on the basis of the estimated position information. The present disclosure can be applied to a mobile body control system that controls the movement of a mobile body in an environment in which the mobile body cannot receive a GNSS signal or in environmental conditions in which a stereo camera cannot be used.

Description

Information processing device, information processing method, and mobile object control system

[0001] The present disclosure relates to an information processing device, an information processing method, and a mobile object control system, and more particularly to an information processing device, an information processing method, and a mobile object control system that enable more efficient control of the movement of a mobile object.

[0002] In drone flight control, drones receive GNSS signals to recognize their own position, allowing them to move and hover stably anywhere in the air. However, flight control using GNSS signals is not possible in environments where GNSS signals from the sky are blocked, such as when conducting drone inspections indoors or under structures such as bridges. In such environments, drones may be equipped with multiple stereo cameras, and aircraft control may be performed using distance measurement and self-position estimation using the stereo cameras. However, accurate self-position estimation is not possible when the surroundings are dark or the subject does not have distinctive features.

[0003] In response to this, Patent Documents 1 and 2 disclose a technology in which a device having a laser scanner function and a surveying function acquires the position of a drone based on reflected light from a reflecting prism equipped on the drone.

[0004] JP 2019-117127 A JP 2019-39867 A

[0005] Equipping a mobile object such as a drone with components such as a GNSS receiver, stereo camera, and reflecting prism for position acquisition increases the weight of the mobile object and ultimately reduces the mobile object's travel time.

[0006] The present disclosure has been made in consideration of such circumstances, and aims to enable more efficient control of the movement of a moving body.

[0007] The information processing device of the present disclosure is an information processing device that includes a position information estimation unit that estimates position information of a moving body using point cloud data acquired by two or more three-dimensional point cloud acquisition devices that exist in an environment including a moving body, with at least a portion of their scanning ranges overlapping, and a movement control unit that controls the movement of the moving body based on the estimated position information.

[0008] The information processing method disclosed herein includes estimating position information of a moving body using point cloud data acquired by two or more three-dimensional point cloud acquisition devices present in an environment including the moving body, with at least a portion of the scanning ranges overlapping, and controlling the movement of the moving body based on the estimated position information.

[0009] The mobile body control system of the present disclosure is a mobile body control system that includes two or more three-dimensional point cloud acquisition devices that exist in an environment including a mobile body so that at least a portion of their scanning ranges overlap, a position information estimation unit that estimates position information of the mobile body using point cloud data acquired by the two or more three-dimensional point cloud acquisition devices, and a movement control unit that controls the movement of the mobile body based on the estimated position information.

[0010] In the present disclosure, position information of a moving body is estimated using point cloud data acquired by two or more three-dimensional point cloud acquisition devices present in an environment including a moving body, with at least a portion of the scanning ranges overlapping, and the movement of the moving body is controlled based on the estimated position information.

[0011] 1 is a diagram illustrating an example configuration of a mobile object control system to which the technology according to the present disclosure is applied. FIG. 1 is a diagram illustrating an application example of a mobile object control system. FIG. 1 is a diagram illustrating a scanning range of a laser scanner. FIG. 2 is a diagram illustrating the number of laser scanners. FIG. 2 is a diagram illustrating an example arrangement of laser scanners. FIG. 3 is a diagram illustrating another example arrangement of laser scanners. FIG. 4 is a diagram illustrating yet another example arrangement of laser scanners. FIG. 5 is a diagram illustrating yet another example arrangement of laser scanners. FIG. 6 is a diagram illustrating yet another example arrangement of laser scanners. FIG. 7 is a diagram illustrating an example of a movable laser scanner. FIG. 8 is a diagram illustrating use of a portable laser scanner. FIG. 9 is a diagram illustrating control of a laser scanner. FIG. 10 is a diagram illustrating control of a laser scanner. FIG. 11 is a diagram illustrating setting of coordinate information of a laser scanner. FIG. 12 is a diagram illustrating setting of coordinate information of a laser scanner. FIG. 13 is a block diagram illustrating an example functional configuration of an information processing unit. FIG. 14 is a flowchart illustrating the flow of flight preparation processing for a drone. FIG. 15 is a flowchart illustrating the flow of flight control processing for a drone. FIG. 16 is a block diagram illustrating an example configuration of computer hardware.

[0012] Modes for carrying out the present disclosure (hereinafter referred to as embodiments) will be described below in the following order.

[0013] 1. Prior art and its problems 2. Configuration of a mobile object control system to which the technology disclosed herein is applied and application examples thereof 3. Specific examples of laser scanners 4. Setting coordinate information for laser scanners 5. Estimating drone position information 6. Configuration of information processing unit and control of drones 7. Example configurations of computer hardware

[0014] <1. Prior art and its problems> In drone flight control, typical methods for making a drone fly stably or hover stably at a certain position include self-position estimation using GNSS (Global Navigation Satellite System) signals and self-position estimation using a stereo camera.

[0015] Self-location estimation using GNSS signals is a method of obtaining a three-dimensional position by receiving GNSS signals using a GNSS antenna equipped on a drone. However, if there are obstructions above or around the drone and the GNSS signals cannot be received, self-location estimation is not possible. If self-location estimation is not possible, even without operation by remote controller, the drone may be blown away by the wind it encounters or may be unable to continue hovering in a certain position. In other words, the drone cannot be moved to the intended location or kept hovering in the air.

[0016] Self-localization using a stereo camera can estimate the drone's position and movement amount from images acquired by the stereo camera without requiring a GNSS signal, enabling it to move to and stay in the air at the intended location. However, feature points within the image are required to estimate movement amount from images acquired by a stereo camera. For example, self-localization using a stereo camera cannot be used when the image is uniform without feature points or when appropriate images cannot be obtained due to environmental conditions such as a dark place.

[0017] Whether it's a GNSS antenna or a stereo camera, it needs to be installed on the drone. The lighter the drone's weight, the longer its flight time will be, but if the drone is equipped with a self-localization system, the weight of the drone will increase and flight time will decrease.

[0018] Drones are also equipped with sensors that obtain information about the drone's surroundings to prevent it from coming into contact with surrounding structures during flight. The stereo camera mentioned above is used to estimate the drone's own position and movement amount, as well as to obtain information about surrounding obstacles. Distance-measuring sensors such as infrared sensors and ultrasonic sensors are also used for this purpose. Because these sensors can only obtain information within a relatively limited range, multiple types and multiple sensors must be installed to obtain information about the drone in all directions. In other words, increasing the number of sensors for safety reasons increases the drone's weight and shortens its flight time.

[0019] The technology disclosed herein is for stably operating a mobile body such as a drone even in an environment where the mobile body cannot receive GNSS signals or where a stereo camera cannot be used, by providing a means for acquiring position information of the mobile body necessary for controlling the movement of the mobile body outside the mobile body. The technology disclosed herein makes it possible to reduce the types and number of sensors mounted on the mobile body, thereby reducing the weight of the mobile body and extending its travel time.

[0020] 2. Configuration of a mobile object control system to which the technology according to the present disclosure is applied and application examples thereof (Configuration of a mobile object control system) FIG. 1 is a diagram illustrating an example of the hardware configuration of a mobile object control system to which the technology according to the present disclosure is applied.

[0021] The mobile object control system 1 shown in FIG. 1 is configured to include a drone 10, a remote controller 20, multiple laser scanners 30, and a point cloud data processing server 40.

[0022] The drone 10 and the remote controller 20 are connected to each other wirelessly. The drone 10 and the laser scanner 30, and the remote controller 20 and the laser scanner 30 are also connected to each other wirelessly. However, the drone 10 and the remote controller 20 may be connected to each other via a wire, or the remote controller 20 and some of the laser scanners 30 may be connected to each other via a wire. Multiple laser scanners 30 may be connected to each other wirelessly or via a wire, or may be connected via a network NW such as the Internet. In addition to the point cloud data processing server 40, the drone 10 may also be connected to the network NW.

[0023] (Configuration of Drone) The drone 10 is configured as one of the moving bodies in the technology according to the present disclosure.

[0024] The drone 10 includes a communication unit 11 , a sensor 12 , a camera 13 , a power unit 14 , and an information processing unit 15 .

[0025] The communication unit 11 is a communication module that performs wireless communication with external devices such as the remote controller 20 and the laser scanner 30. The communication unit 11 can also perform wired communication with external devices. The communication unit 11 supplies information received by wireless communication or wired communication to the information processing unit 15, and transmits information supplied from the information processing unit 15 by wireless communication or wired communication.

[0026] The sensor 12 is a sensor for obtaining information about the surroundings of the drone 10. The sensor 12 may be a stereo camera or a distance measurement sensor such as an infrared sensor or an ultrasonic sensor. The sensor 12 may also be a GNSS receiver capable of receiving GNSS signals via a GNSS antenna (not shown). The drone 10 according to the technology disclosed herein may be equipped with a minimum number of sensors as the sensor 12. The sensor data obtained by the sensor 12 is supplied to the information processing unit 15.

[0027] The camera 13 is an imaging device for aerial photography, surveying, and inspection using the drone 10.

[0028] The power unit 14 is composed of a motor and a propeller (rotor) that serve as a drive source for the drone 10. The power unit 14 is driven based on a control signal from the information processing unit 15, causing the drone 10 to fly (move) and stay in the air.

[0029] The information processing unit 15 is configured with one or more boards on which various processors are mounted, etc. The information processing unit 15 controls each part of the drone 10 and performs various calculations (information processing).

[0030] (Configuration of Remote Controller) The remote controller 20 is a user terminal operated by a user (the pilot of the drone 10). The remote controller 20 may be configured as a monitor-integrated proportional controller (transmitter), or as a transmitter connectable to a mobile terminal such as a smartphone or tablet terminal.

[0031] The remote controller 20 includes a communication unit 21, a display output unit 22, an operation input unit 23, a laser output unit 24, a light receiving unit 25, a sensor 26, a camera 27, and an information processing unit 28.

[0032] The communication unit 21 is a communication module that performs wireless communication with external devices such as the drone 10 and the laser scanner 30. The communication unit 21 can also perform wired communication with external devices. The communication unit 21 supplies information received by wireless communication or wired communication to the information processing unit 28, and transmits information supplied from the information processing unit 28 by wireless communication or wired communication.

[0033] The display output unit 22 is a monitor that presents (displays) various information to the user based on the presentation information from the information processing unit 28. The display output unit 22 may be configured integrally with the remote controller 20, or may be a display screen of a mobile terminal such as a smartphone or a tablet terminal.

[0034] The operation input unit 23 is a button or stick that accepts user operations, a touch panel that is integrated with the display output unit 22, or the like. The operation input unit 23 supplies input information corresponding to user operations to the information processing unit 28.

[0035] The laser output unit 24 is a light-emitting unit that scans an object by emitting laser light (pulsed laser light or frequency-continuously modulated light for laser scanning). In contrast, the light-receiving unit 25 receives laser light reflected from the object to acquire three-dimensional point cloud data (hereinafter simply referred to as point cloud data). In other words, the laser output unit 24 and the light-receiving unit 25 constitute a three-dimensional point cloud acquisition device that realizes a laser scanning function. Alternatively, the laser output unit 24 and the light-receiving unit 25 may be configured as a time-of-flight (ToF) camera in which the laser output unit 24 does not perform scanning, and the light-receiving unit 25 has multiple pixels. The point cloud data acquired by the three-dimensional point cloud acquisition device (light-receiving unit 25) is supplied to the information processing unit 28. The laser output unit 24 and the light-receiving unit 25 do not necessarily have to be provided in the remote controller 20.

[0036] The sensor 26 is configured with a GNSS receiver or the like that can receive GNSS signals via a GNSS antenna (not shown).

[0037] The camera 27 is an imaging device capable of capturing images of the surroundings of the remote controller 20 (user). The camera 27 may be configured integrally with the remote controller 20, or may be a built-in camera of a mobile terminal such as a smartphone or a tablet terminal.

[0038] The information processing unit 28 is configured with one or more boards on which various processors are mounted, etc. The information processing unit 28 controls each part of the remote controller 20 and performs various calculations (information processing).

[0039] (Configuration of Laser Scanner) The laser scanner 30 is configured as a relatively small laser scanner having a laser scanning function. The laser scanner 30 is fixedly disposed in an environment including the drone 10. The laser scanner 30 may also be mounted on a laser scanner-equipped device such as a dock that serves as a takeoff and landing point for the drone 10, an automobile (vehicle), or an autonomously mobile robot such as a transport robot.

[0040] The laser scanner 30 includes a communication unit 31, a laser output unit 32, a light receiving unit 33, a sensor 34, a camera 35, and an information processing unit 36.

[0041] The communication unit 31 is a communication module that performs wireless communication with external devices such as the drone 10 and the remote controller 20. The communication unit 31 can also perform wireless or wired communication with other laser scanners 30. The communication unit 31 supplies information received by wireless or wired communication to the information processing unit 36, and transmits information supplied from the information processing unit 36 ​​by wireless or wired communication.

[0042] The laser output unit 32 is an emitter that emits laser light to scan an object. Meanwhile, the light-receiving unit 33 receives the laser light reflected from the object to acquire point cloud data. In other words, the laser output unit 32 and the light-receiving unit 33 constitute a 3D point cloud acquisition device that realizes a laser scanning function. Alternatively, the laser output unit 32 and the light-receiving unit 33 may be configured as a Time of Flight (ToF) camera, in which the laser output unit 32 does not perform scanning, and the light-receiving unit 33 has multiple pixels. The point cloud data acquired by the 3D point cloud acquisition device (light-receiving unit 33) is supplied to the information processing unit 36.

[0043] The sensor 34 is configured with a GNSS receiver or the like that can receive GNSS signals via a GNSS antenna (not shown).

[0044] The camera 35 is an imaging device capable of capturing images of the environment including the drone 10.

[0045] The information processing unit 36 ​​is configured with one or more boards on which various processors are mounted, etc. The information processing unit 36 ​​controls each part of the laser scanner 30 and performs various calculations (information processing).

[0046] (Point Cloud Data Processing Server) The point cloud data processing server 40 processes point cloud data acquired by the laser scanning function of the remote controller 20 or the laser scanner 30 and received directly or via the network NW. The processing results of the point cloud data are transmitted to the drone 10 or the remote controller 20 via the network NW.

[0047] (Application Example of Mobile Object Control System) FIG. 2 is a diagram illustrating an application example of the mobile object control system 1 described with reference to FIG.

[0048] Figure 2 shows an example of a bridge inspection site where it is difficult for the drone 10 to receive GNSS signals from the sky and the environment is dimly lit below the bridge BR. Specifically, the left side of Figure 2 shows a side view of the bridge inspection site, and the right side shows a top view of the site below the superstructure SS that constitutes the bridge BR.

[0049] In the example of Figure 2, a user operates a remote controller 20 to inspect the pier P1 as an inspection target IT using a drone 10 in the space below the superstructure SS and between the piers P1 and P2 that make up the bridge BR. Also, in the example of Figure 2, two laser scanners 30A and 30B are placed so that the inspection target IT (pier P1) is included in their scanning range. The laser scanners 30A and 30B are present in an environment that includes the drone 10 so that at least a portion of their scanning ranges overlap each other.

[0050] At a bridge inspection site such as that shown in FIG. 2, the condition of high-positioned structures such as bridge piers P1 and P2, which cannot be directly observed by humans, is confirmed by a main camera (camera 13) mounted on the drone 10. In this case, the main camera may be a relatively large imaging device that can obtain bright images even in dimly lit environments. Furthermore, a light may be attached to the main camera to illuminate the imaging direction to compensate for the amount of light. Furthermore, a configuration may be adopted in which the main camera (main sensor) can be changed from among multiple types of cameras (sensors) depending on the purpose of the imaging.

[0051] On the other hand, stereo cameras mounted on drones and used for flight control of typical drones have relatively small sensors and lenses to minimize their impact on the drone's weight, which can make them difficult to operate in dimly lit environments.

[0052] In contrast, the technology disclosed herein achieves flight control of the drone 10 by positioning two or more laser scanners 30 and acquiring point cloud data of the surrounding environment, including the drone 10 and the structure to be inspected, without using information such as GNSS signals or stereo cameras on the drone 10.

[0053] 3. Specific Examples of Laser Scanners Specific examples of the scanning range (scanning area), number, and arrangement of the laser scanner 30 will be described.

[0054] (Scanning Range of Laser Scanner) FIG. 3 is a diagram illustrating the scanning range of the laser scanner 30. As shown in FIG.

[0055] In the example of Figure 3, only one side of the bridge pier P1 is set as the inspection target IT, and a spherical fan-shaped scanning range RA is shown centered on the laser scanner 30A. Depending on the type of laser scanner 30A and the control of the scanning range, the spread angle of the laser light (the central angle of the fan shape) may be narrowed or widened so that, for example, a 360° range centered on the laser scanner 30A can be scanned.

[0056] 3, the area that is in the shadow of the drone 10 as seen from the laser scanner 30A is not scanned. However, the area that is not in the shadow of the drone 10 as seen three-dimensionally can be scanned.

[0057] (Number of Laser Scanners) The number of laser scanners 30 to be arranged can be changed depending on the inspection target and inspection range.

[0058] For example, as shown in the left diagram of Figure 4 (similar to Figure 3), if only one side of the pier P1 is the inspection target IT, one laser scanner 30A can be positioned so that the scanning range RA of the laser scanner 30A includes the inspection target IT.

[0059] On the other hand, as shown in the right diagram of Figure 4, when the entire surface of the pier P1 is the inspection target IT, the two laser scanners 30A and 30B are positioned so that the scanning ranges RA and RB of the laser scanners 30A and 30B include the inspection target IT. By positioning the laser scanners 30A and 30B in this way, it is desirable to eliminate blind spots when flying the drone 10.

[0060] 1 , the drone 10, the remote controller 20, and the laser scanner 30 can transmit and receive data to and from each other by being connected to each other wirelessly or by wire, or by being connected via a network NW. Processing of the point cloud data used for flight control of the drone 10 and visualization of the information can be performed by the drone 10, the remote controller 20, and the laser scanner 30, as well as by a point cloud data processing server 40 via the network NW.

[0061] (Laser Scanner Arrangement) FIG. 5 is a diagram showing an example of the arrangement of the laser scanner 30 according to the inspection target.

[0062] In the example of Figure 5, the entire surface of the pier P1 is the inspection target IT for the main camera of the drone 10.

[0063] In this case, as shown in the right diagram of Figure 5, by placing the laser scanners 30A and 30B at positions that form diagonal corners around the inspection target IT (pier P1) in top view, it becomes possible to acquire point cloud data of the inspection target IT without any blind spots. Note that the placement of the laser scanner 30 may be changed depending on the actual condition and position of the inspection target.

[0064] 5, the scanning ranges RA and RB of the laser scanners 30A and 30B can be set and the orientations and heights of the laser scanners 30A and 30B can be adjusted so that the inspection target IT and the flight range including the takeoff and landing points of the drone can be covered in a side view. The installation height of the laser scanner 30 can be determined based on the flatness of the ground, the dryness of the ground such as the presence or absence of puddles, and the like, and the laser scanner 30 can be placed on the ground like the laser scanner 30A, or can be placed away from the ground using a tripod or the like like the laser scanner 30B.

[0065] FIG. 6 is a diagram showing another example of the arrangement of the laser scanner 30 according to the inspection target.

[0066] In the example of Figure 6, the entire underside of the superstructure SS sandwiched between piers P1 and P2 is the inspection target IT for the main camera of drone 10.

[0067] In this case, as shown in the left diagram of Figure 6, by positioning laser scanner 30A so that the main scanning range RA is the upward direction where the inspection target IT is located when viewed from the side, and by positioning laser scanner 30B so that the main scanning range RB is the flight range of drone 10, it is possible to obtain point cloud data without any blind spots.

[0068] 6, it is desirable to place the laser scanners 30A and 30B along the longitudinal center line of the superstructure SS (inspection target IT) so that, in a top view, they can cover the inspection target IT and the flight range including the takeoff and landing points of the drone 10. However, the placement locations may be changed depending on the flatness of the ground and the dryness conditions such as the presence or absence of puddles, and the number of placed laser scanners 30 may be increased if the distance between pier P1 and pier P2 is long.

[0069] FIG. 7 is a diagram showing yet another example of the arrangement of the laser scanner 30. In FIG.

[0070] In the example of Figure 7, as in the example of Figure 6, the entire underside of the superstructure SS sandwiched between the piers P1 and P2 is the inspection target IT for the main camera of the drone 10.

[0071] In the example of FIG. 7 , laser scanners 30A and 30B are disposed at the takeoff and landing point DK of the drone 10. Specifically, the laser scanner 30A is disposed so that its main scanning range RA is the upward (vertical) direction of the takeoff and landing point DK, and the laser scanner 30B is disposed so that its main scanning range RB is the lateral (horizontal) direction of the takeoff and landing point DK. The takeoff and landing point DK may be configured as an airborne marker, dock, station, or the like used as a takeoff and landing surface for the drone 10. When the takeoff and landing point DK is configured as a dock, the laser scanners 30A and 30B may be mounted on the dock, or only the laser scanner 30A may be mounted on the dock and the laser scanner 30B may be provided separately. When the takeoff and landing point DK is configured as an airborne marker or station, the laser scanners 30A and 30B are installed at predetermined positions around the airborne marker or station.

[0072] In this way, by positioning the laser scanner 30 (three-dimensional point cloud acquisition device) so that the scan range is above (or above and sideways from) the takeoff and landing point DK, the flight range of the drone 10 can be easily included in the scan range without the user having to consider the placement or installation height of the three-dimensional point cloud acquisition device. Also, for example, when the laser scanner 30 is retrofitted to the takeoff and landing point DK serving as a dock, the location on the dock where the laser scanner 30 should be mounted may be displayed on the display output unit 22 of the remote controller 20, for example.

[0073] 8, 9, and 10 are diagrams showing further examples of the arrangement of the laser scanner 30. FIG.

[0074] In the example of Figure 8, the entire surface of the pier P1 is the inspection target IT for the main camera of the drone 10.

[0075] 8, vehicles CR1 and CR2 are equipped with laser scanners 30A and 30B, respectively. The laser scanner 30A is mounted so that its main scanning range RA is in the traveling direction of the vehicle CR1, and the laser scanner 30B is mounted so that its main scanning range RB is in the traveling direction of the vehicle CR2.

[0076] In this case, as shown in Fig. 8, by placing the vehicles CR1 and CR2 at positions diagonally opposite the inspection target IT (pier P1) in top view, it becomes possible to acquire point cloud data of the inspection target IT without any blind spots. Note that the placement of the vehicles CR1 and CR2 may be changed depending on the actual condition and position of the inspection target.

[0077] In the example of Figure 9, each of the opposing sides of pier P1 and pier P2 is designated as an inspection target IT by the main camera of drone 10.

[0078] 9, four laser scanners 30A, 30B, 30C, and 30D are mounted on a vehicle CR1 as a laser scanner-mounted device. Of these, laser scanner 30A is mounted so that its main scanning range RA is in the traveling direction of vehicle CR1, and laser scanner 30B is mounted so that its main scanning range RB is to the right of the traveling direction of vehicle CR1.

[0079] In this case, as shown in Figure 9, the vehicle CR is positioned so that, when viewed from above, the scanning range RA of the laser scanner 30A includes the inspection target IT of the pier P1, and the scanning range RB of the laser scanner 30B includes the inspection target IT of the pier P2.

[0080] In the example of Figure 10, the entire underside of the superstructure SS sandwiched between piers P1 and P2 is the inspection target IT for the main camera of drone 10.

[0081] 10, two laser scanners 30A and 30B are arranged on a vehicle CR1 as a laser scanner-equipped device. Specifically, the laser scanner 30A is arranged so that its main scanning range RA is in the upward direction (vertical direction) of the vehicle CR1, and the laser scanner 30B is arranged so that its main scanning range RB is in the traveling direction (horizontal direction) of the vehicle CR1.

[0082] In this case, by placing the vehicle CR1 below the superstructure SS so that the main scanning range RA is the upward direction where the inspection target IT is located, and the main scanning range RB is the flight range of the drone 10, it becomes possible to acquire point cloud data without any blind spots.

[0083] (Movable Laser Scanner) At least one of the three-dimensional point cloud acquisition devices that realizes the laser scanning function may be movable in the environment including the drone 10, in addition to being fixedly disposed.

[0084] For example, as shown in the left diagram of FIG. 11 , a remote controller 20, which is a user terminal, may be equipped with a 3D point cloud acquisition device (laser output unit and light receiving unit) that realizes a laser scanning function, replacing the fixedly installed laser scanner 30. In this case, the user can move around to acquire point cloud data of the surrounding environment, such as surrounding structures, in advance, and create a 3D map of the surrounding environment from the acquired point cloud data. When flying the drone 10, the position of the drone 10 can be acquired by pointing the scan range RC of the 3D point cloud acquisition device mounted on the remote controller 20 toward the drone 10, thereby determining where the drone 10 is currently flying on the pre-created 3D map. In addition, unlike a fixedly installed laser scanner 30, the scan range RC can be dynamically changed by moving the user operating the remote controller 20 along with the movement of the drone 10.

[0085] 11, the drone 10 itself may be equipped with a 3D point cloud acquisition device (laser output unit and light receiving unit) that realizes a laser scanning function, in addition to the remote controller 20. In this case, the weight of the drone 10 increases, but it becomes possible to complement blind spots that are shaded by the drone 10 in the scanning range of the fixedly arranged laser scanner 30 or the scanning range RC of the 3D point cloud acquisition device mounted on the remote controller 20, with the scanning range RD of the drone 10.

[0086] Here, a 3D point cloud acquisition device that can be carried and moved by a user, such as the 3D point cloud acquisition device mounted on the remote controller 20, is referred to as a portable laser scanner. The portable laser scanner is not limited to a 3D point cloud acquisition device mounted on the remote controller 20, but may be, for example, a handheld scanner that can be held in one hand. The mobile object control system 1 may be provided with a plurality of portable laser scanners, such as the remote controller 20 or the handheld scanner, rather than being limited to one.

[0087] FIG. 12 is a diagram illustrating the use of the portable laser scanner.

[0088] 12, due to the placement of the fixedly installed laser scanner 30A and the remote controller 20 (portable laser scanner) operated by the user, blind spots are created in parts of the surroundings of the drone 10 within their respective scan ranges RA and RC. If there is a blind spot in the direction of travel of the drone 10, which moves in accordance with input from the remote controller 20 or movement commands based on a preset flight path, it becomes impossible to know the situation in the direction of travel, such as the presence or absence of obstacles OB such as trees, which raises safety concerns.

[0089] In such a case, as shown in the right diagram of Fig. 12, the user holding the portable laser scanner (remote controller 20) can move to a position that eliminates the blind spot and change the scanning range RC of the portable laser scanner, thereby ensuring the safety of the movement of the drone 10. In this case, the mobile object control system 1 may estimate where the portable laser scanner should move and in what direction the scanning range RC should be to eliminate the blind spot, and may present destination information indicating the destination of the portable laser scanner and the direction of the scanning range RC as presentation information to be presented on the display output unit 22 of the remote controller 20.

[0090] (Laser Scanner Control) Generally, laser scanners emit laser light radially from a light source, which allows for dense point cloud data to be obtained at close range, but sparse point cloud data at long range. Obtaining dense point cloud data at long range requires increasing the scanning density of the laser light, which increases the scanning time, resulting in a lower frame rate and poor real-time performance. Furthermore, increasing the amount of point cloud data obtained increases the data processing load, resulting in increased processing time and power consumption.

[0091] The technology disclosed herein proposes controlling the laser scanner 30 to obtain environmental (spatial) information necessary for flight control of the drone 10 without increasing the amount of point cloud data obtained.

[0092] For example, as shown in the left diagram of Figure 13, when the laser scanner 30A is positioned so that the scan range RA includes the inspection target IT (one side of the pier P1), the laser scanner 30A is controlled so that the area IM_L away from the inspection target IT is considered to be of low importance and the point cloud data is sparse. Alternatively, for the area IM_L of low importance, laser light scanning may not be performed and point cloud data may not be acquired. On the other hand, in order to obtain precise position information of the drone 10, the area IM_H around the drone 10 is considered to be of high importance and the laser scanner 30A is controlled so that the point cloud data is dense.

[0093] As described above, the point cloud data is dense at positions closer to the laser scanner 30 and sparse at positions farther away. Therefore, structures close to the laser scanner 30 and structures farther away may be recognized to make the density of the overall point cloud data uniform. For example, as shown in the right diagram of FIG. 13 , the laser scanner 30A may be controlled so that the density of the point cloud data of a portion st1 of the pier P1 facing the scan range RA and a portion st2 of the drone 10 facing the scan range RA is uniform.

[0094] In the mobile object control system 1, point cloud data is acquired by a plurality of laser scanners 30 (three-dimensional point cloud acquisition devices).

[0095] For example, as shown in the left diagram of Figure 14, if point cloud data is acquired using two fixedly installed laser scanners 30A and 30A' for the same structure (bridge pier P1) as the inspection target IT, duplicate point cloud data will be acquired in the overlapping portion OL of the scan ranges RA and RA'. In such a case, as shown in the right diagram of Figure 14, the scan range RA' of the laser scanner 30A' can be changed to prevent the laser scanner 30A' from acquiring point cloud data in the overlapping portion OL. This reduces the amount of point cloud data acquired and also increases the density of point cloud data for other important areas.

[0096] 4. Setting Coordinate Information of Laser Scanners In order to integrate point cloud data of structures, drones 10, etc. obtained by multiple laser scanners 30 and to associate the data with real-world coordinates, it is necessary to set coordinate information for each laser scanner 30.

[0097] The point cloud data acquired by a single laser scanner 30 represents an independent space without being associated with real-world coordinates. If the inspection target area is not complex and is relatively small, and flight control of the drone 10 is possible using point cloud data acquired by a single laser scanner 30, including the flight range of the drone 10, there is no need to associate the point cloud data with real-world coordinates.

[0098] However, in actual inspection work, even within an area that can be covered by one laser scanner 30, there is a demand for storing the position information of the drone 10 in association with real-world coordinates, and for later referencing the point cloud data associated with real-world coordinates when checking the data or re-flying the drone 10. Furthermore, when integrating point cloud data obtained by multiple laser scanners 30 to control the flight of the drone 10, it is necessary to manually integrate the point cloud data or integrate it by associating it with real-world coordinates.

[0099] As explained with reference to Figure 1, in the mobile object control system 1, the remote controller 20 and drone 10 equipped with a laser scanner 30 and a 3D point cloud acquisition device can be equipped with a GNSS antenna and various sensors as needed.

[0100] Generally, at bridge inspection sites such as those described above, it is often difficult to receive GNSS signals below the bridge BR (superstructure SS).

[0101] Consider the case in which a laser scanner 30A equipped with a GNSS antenna is used in FIG. 15 . In this case, as shown in the left diagram, real-world coordinate information of the laser scanner 30A is first acquired at a location away from the superstructure SS where GNSS signals can be received. Then, as shown in the right diagram, the laser scanner 30A is moved below the superstructure SS, and the coordinate information of the laser scanner 30A below the superstructure SS can be set based on the amount of movement. At this time, the amount of movement of the laser scanner 30A may be manually input to the mobile object control system 1. Furthermore, if the laser scanner 30A is equipped with sensors such as an IMU (Inertial Measurement Unit), a magnetic sensor, a barometer, or a stereo camera, the amount of movement may be calculated from sensor data acquired by these sensors and input to the mobile object control system 1.

[0102] The coordinate information of the laser scanner 30 may be set based on GNSS signals or manually. For example, a known point may be selected as the location where the laser scanner 30 is to be placed. The known point may be a specific point for which coordinate information has already been obtained by, for example, surveying, or may be a point for which coordinate information has been obtained by, for example, conducting a new survey at an inspection site. By placing the laser scanner 30 at the known point, manually setting the coordinate information of the known point, and manually setting the placement direction, the obtained point cloud data can be associated with real-world coordinates. Furthermore, the manually set information may be corrected based on sensor data acquired by a sensor mounted on the laser scanner 30.

[0103] Furthermore, the coordinate information of the laser scanner 30 may be set by matching the point cloud data obtained by the laser scanner 30 with point cloud data of the surrounding environment that has already been obtained by some method. Specifically, when point cloud data of the inspection site associated with real-world coordinates has already been obtained, such as point cloud data previously obtained at the inspection site or a dynamic map (high-precision three-dimensional map information) for autonomous driving, the point cloud data obtained at the site can be matched with the past point cloud data to be associated with real-world coordinates.

[0104] For example, as shown in the left diagram of Fig. 16, assume that point cloud data pd1 and pd2 for bridge piers P1 and P2, respectively, have been obtained as point cloud data near a past site, and point cloud data pd11 and coordinate information c30 of the laser scanner 30 have been obtained at the site. In this case, as shown in the right diagram of Fig. 16, the point cloud data pd11 obtained at the site can be rotated and moved based on the coordinate information c30 of the laser scanner 30, and matched with the past point cloud data pd1.

[0105] Matching with past point cloud data may be performed manually or automatically using image recognition or machine learning. Furthermore, the display output unit 22 and the operation input unit 23 of the remote controller 20 may be used as a means for performing matching. In this case, the point cloud data as shown in FIG. 16 may be visualized as presentation information and displayed on the display output unit 22.

[0106] Furthermore, when matching point cloud data obtained by multiple laser scanners 30, instead of referring to point cloud data near the previous site, point cloud data obtained by one laser scanner 30 at the site may be matched with point cloud data obtained by another laser scanner 30. Note that if there is no common data between point cloud data obtained by different laser scanners 30, each point cloud data may be associated with an independent coordinate space.

[0107] 5. Estimation of Drone Position Information An example of a method for estimating the position information of the drone 10 using point cloud data acquired by the laser scanner 30 will be described.

[0108] For example, as shown in the left diagram of Fig. 17, the position information of the drone 10 may be estimated by matching point cloud data acquired by a laser scanner 30A or the like with a point cloud pattern 10pp of the drone 10 that has been acquired in advance. Matching with the point cloud pattern 10pp may be performed while the drone 10 is in a landing state (before takeoff) using point cloud recognition or machine learning.

[0109] 17 , the position information of the drone 10 may be estimated using sensor data acquired by various sensors, such as the sensor 34 mounted on the laser scanner 30A and the sensor 26 mounted on the remote controller 20. For example, a microphone, one of the various sensors, may detect propeller sounds during takeoff and landing or flight of the drone 10, or specific sounds output from the drone 10 itself, and the position information of the drone 10 may be estimated by linking the sensor data (sound) from multiple positions. The position information of the drone 10 may also be estimated using image recognition or machine learning from image data acquired by a camera, one of the various sensors. Furthermore, the position information of the drone 10 may be estimated by combining point cloud data acquired by a 3D point cloud acquisition device, such as the laser scanner 30A, with sensor data acquired by various sensors.

[0110] Furthermore, the display output unit 22 of the remote controller 20 may be configured to display point cloud data acquired at the site, and the user may specify the position of the drone 10 in the point cloud data displayed on the display output unit 22, thereby acquiring position information of the drone 10. In this case, the point cloud pattern specified by the user in the point cloud data can be recognized and tracked.

[0111] In addition, if the drone 10 itself is equipped with a three-dimensional point cloud acquisition device, the position information of the drone 10 may be estimated by matching the point cloud data of the surrounding environment with the point cloud data acquired by the three-dimensional point cloud acquisition device installed on the drone 10.

[0112] When displaying the point cloud data acquired by the laser scanner 30 on the display output unit 22 of the remote controller 20, RGB information (color information) may be added. If the laser scanner 30 has the function of directly acquiring color information by learning the reflection intensity pattern of laser light and RGB data, that color information may be displayed. Furthermore, if the point cloud data acquired by the laser scanner 30 does not include RGB data, color information of structures and the like may be estimated and learned from data acquired by the main camera mounted on the drone 10 or various sensors mounted on the remote controller 20 and the laser scanner 30, and reflected in the displayed point cloud data. Furthermore, the color information may be supplemented by referencing map photo data obtained from a map photo service via the Internet. If previously acquired point cloud data includes RGB data, the color information may be supplemented by referencing that RGB data.

[0113] 6. Configuration of Information Processing Unit and Control of Drone> The configuration of the information processing unit that realizes the above-described embodiment and the control of the drone 10 by the information processing unit will be described.

[0114] (Configuration of Information Processing Unit) FIG. 18 is a block diagram showing an example of the functional configuration of the information processing unit.

[0115] 18 may be realized by the information processing unit 15 of the drone 10, or may be realized by the information processing unit 28 of the remote controller 20. In addition, each function of the information processing unit 100 may be realized by the information processing unit 36 ​​of any of the laser scanners 30, or some of the functions of the information processing unit 100 may be realized by the point cloud data processing server 40.

[0116] The information processing unit 100 is configured to include a coordinate information setting unit 110 , a position information estimation unit 120 , a movement control unit 130 , and a presentation information generation unit 140 .

[0117] The coordinate information setting unit 110 sets the coordinate information for each of the plurality of laser scanners 30 .

[0118] The position information estimation unit 120 estimates the position information of the drone 10 using point cloud data acquired by two or more 3D point cloud acquisition devices that exist in an environment including the drone 10 and whose scanning ranges overlap at least partially. The 3D point cloud acquisition devices here may be included in the remote controller 20 or the drone 10 itself, as well as in the multiple laser scanners 30 that make up the mobile object control system 1.

[0119] The point cloud data acquired by the three-dimensional point cloud acquisition device includes point cloud data of at least a portion of the object imaged by the imaging device (camera 13 of drone 10) mounted on the mobile body. Depending on the situation, the point cloud data acquired by the three-dimensional point cloud acquisition device may include point cloud data of the entire object imaged by camera 13. The point cloud data acquired by the three-dimensional point cloud acquisition device also includes point cloud data of obstacles present in the surrounding environment of the mobile body (drone 10).

[0120] Furthermore, it is desirable that the point cloud data acquired by the 3D point cloud acquisition device include all point cloud data within the flight range of the drone 10. In this case, the flight range includes the takeoff point and landing point of the drone 10.

[0121] The movement control unit 130 controls the flight (movement) of the drone 10 based on the position information estimated by the position information estimation unit 120. Specifically, the movement control unit 130 generates control information for controlling the flight (movement) of the drone 10 based on the estimated position information and the surrounding environment information of the drone 10 included in the point cloud data, and supplies the control information to the power unit 14 of the drone 10.

[0122] The presentation information generation unit 140 generates presentation information to be presented on the display output unit 22 of the remote controller 20 and supplies the generated presentation information to the display output unit 22 as appropriate. For example, the presentation information generation unit 140 generates presentation information including location information of a movable three-dimensional point cloud acquisition device such as a portable laser scanner. The presentation information may include destination information indicating the destination of the movable three-dimensional point cloud acquisition device.

[0123] (Control of the drone) As an example of control of the drone 10 by the information processing unit 100, flight preparation processing of the drone 10 and flight control processing of the drone 10 will be described.

[0124] First, the flow of the drone 10 flight preparation process performed by the information processing unit 100 will be described with reference to the flowchart in Fig. 19. The process in Fig. 19 may be started, for example, when the user operates the remote controller 20 to start the drone 10 flying while the drone 10 is in a landing state (before takeoff).

[0125] In step S11 , the information processing unit 100 confirms the presence of each laser scanner 30 in the mobile object control system 1 , and then starts acquiring point cloud data acquired by each laser scanner 30 .

[0126] In step S12, the coordinate information setting unit 110 determines whether the coordinate information of each laser scanner 30 has already been set.

[0127] If it is determined that the coordinate information of each laser scanner 30 has not been set, the process proceeds to step S13 , where the coordinate information setting unit 110 sets the coordinate information of each laser scanner 30 .

[0128] For example, the coordinate information setting unit 110 sets the coordinate information based on GNSS information acquired by the laser scanner 30. Alternatively, the coordinate information setting unit 110 sets the coordinate information based on user input. Furthermore, the coordinate information setting unit 110 may set the coordinate information by matching using point cloud data acquired by a laser scanner 30 for which coordinate information has been set in advance. Alternatively, the coordinate information setting unit 110 may set the coordinate information by matching using point cloud data acquired by multiple laser scanners 30.

[0129] This allows the point cloud data acquired by each laser scanner 30 to be associated with real-world coordinates.

[0130] If it is determined that the coordinate information of each laser scanner 30 has already been set, step S13 is skipped and the process proceeds to step S14.

[0131] In step S14, the position information estimation unit 120 determines whether the position of the drone 10 has already been estimated.

[0132] If it is determined that the position of the drone 10 has not been estimated, the process proceeds to step S15, where the position information estimation unit 120 estimates the position (position information) of the drone 10 in the point cloud data acquired by each laser scanner 30.

[0133] For example, the position information estimation unit 120 estimates the position information by matching the point cloud data acquired by each laser scanner 30 with a point cloud pattern of the drone 10 that has been acquired in advance. The position information estimation unit 120 also estimates the position information using the point cloud data acquired by each laser scanner 30 and sensor data acquired by a predetermined sensor. Furthermore, the position information estimation unit 120 may estimate the position information based on a position specified by a user in the point cloud data acquired by each laser scanner 30. The position information estimation unit 120 may also estimate the position information by matching the point cloud data acquired by a laser scanner 30 installed in the environment with point cloud data acquired by a three-dimensional point cloud acquisition device mounted on the drone 10.

[0134] Furthermore, if three-dimensional map information has been created from point cloud data acquired in advance, the location information estimation unit 120 may estimate the location information based on the three-dimensional map information. For example, the location information estimation unit 120 can estimate the location information by matching the point cloud data of the surrounding environment of the drone 10 with the three-dimensional map information.

[0135] Then, in step S16, the position information estimation unit 120 determines whether or not the user has requested correction of the position of the drone 10. If it is determined that correction of the position of the drone 10 has been requested, step S15 is repeated.

[0136] After the position of the drone 10 in the point cloud data has been estimated in this manner, or if it is determined in step S14 that the position of the drone 10 has already been estimated, the process proceeds to step S17.

[0137] In step S17, the position information estimation unit 120 starts tracking the drone 10 based on the estimated position information of the drone 10.

[0138] Once the flight preparation process for the drone 10 is completed in this manner, the flight control process for the drone 10 can begin.

[0139] Next, the flow of the flight control process of the drone 10 by the information processing unit 100 will be described with reference to the flowcharts of Figures 20 and 21. The process of Figures 20 and 21 starts after the flight preparation process of the drone 10 described with reference to the flowchart of Figure 19 has been completed, and in a state where the position information of the drone 10 and the surrounding environment information have been acquired as point cloud data.

[0140] In step S21, the movement control unit 130 sets a separation distance, which is a distance from structures and people that must be maintained for safe flight of the drone 10. The separation distance may be set based on instructions from the remote controller 20 according to the environment in which the drone 10 is flying, for example.

[0141] In step S22, the movement control unit 130 determines whether or not there is an obstacle within the distance around the drone 10 based on the position information of the drone 10 and the surrounding environment information acquired as point cloud data.

[0142] If it is determined that there are no obstacles within the clearance distance around the drone 10, the process proceeds to step S23, where the movement control unit 130 determines whether there is a blind spot in the direction of travel of the drone 10 based on the position information of the drone 10 and the surrounding environment information acquired as point cloud data. The direction of travel of the drone 10 may be calculated based on an input from the remote controller 20 or a preset flight path.

[0143] If it is determined that there is no blind spot in the direction of travel of the drone 10, the process proceeds to step S24, where the movement control unit 130 moves the drone 10 in accordance with a movement command input from the remote controller 20 or based on a preset flight path. Thereafter, the process returns to step S22, and the subsequent processes are repeated.

[0144] On the other hand, if it is determined in step S22 that there is an obstacle within the clearance distance around the drone 10, the process proceeds to step S25.

[0145] In step S25, the movement control unit 130 determines whether the drone 10 is moving toward an obstacle based on input from the remote controller 20 or a preset flight path. If it is determined that the drone 10 is not moving toward an obstacle, the possibility of a collision with the obstacle is low, and the process proceeds to step S23, where it is determined whether there is a blind spot in the direction of travel of the drone 10. If it is determined that the drone 10 is moving toward an obstacle, the process proceeds to step S26.

[0146] In step S26, the movement control unit 130 determines whether or not the drone 10 will land on the ground based on input from the remote controller 20 or a preset flight path. If it is determined that the drone 10 will land on the ground, the possibility of a collision with an obstacle is low, and the process proceeds to step S23, where it is determined whether or not there is a blind spot in the direction of travel of the drone 10. If it is determined that the drone 10 will not land on the ground, the process proceeds to step S27.

[0147] In step S27, the movement control unit 130 determines whether to relax the separation distance setting based on an instruction from the remote controller 20. If it is determined that the separation distance setting should be relaxed, it is determined that a collision with an obstacle can be avoided by the user's operation, and the process proceeds to step S23, where it is determined whether there is a blind spot in the direction of travel of the drone 10. If it is determined that the separation distance setting should not be relaxed, it is determined that there is a possibility of a collision with an obstacle, and the process proceeds to step S28.

[0148] In step S28, the movement control unit 130 invalidates the movement command for the drone 10 based on the input from the remote controller 20 or the preset flight path. At this time, the movement control unit 130 stops (hovers) the drone 10 at that position and waits for the next movement command.

[0149] Now, if it is determined in step S23 that there is a blind spot in the direction of travel of the drone 10, the process proceeds to step S29 in FIG.

[0150] In step S29, the information processing unit 100 determines whether or not a portable laser scanner is present in the mobile object control system 1. If it is determined that a portable laser scanner is not present in the mobile object control system 1, the process proceeds to step S30.

[0151] In step S30, the presentation information generation unit 140 generates presentation information indicating that there is a blind spot in the direction of travel of the drone 10. The presentation information is presented on the display output unit 22 of the remote controller 20. After that, the process returns to step S24 in Fig. 20, and the drone 10 moves based on the movement command. In this case, the user moves the drone 10, taking into account that there is a blind spot in the direction of travel of the drone 10.

[0152] On the other hand, if it is determined in step S29 that a portable laser scanner is present in the mobile object control system 1, the process proceeds to step S31.

[0153] In step S31, the presentation information generator 140 generates presentation information (destination information) indicating the destination of the portable laser scanner (movable 3D point cloud acquisition device). The destination information is also presented on the display output unit 22 of the remote controller 20.

[0154] Thereafter, in step S32, the information processing unit 100 determines whether to wait for the portable laser scanner to move. Specifically, the information processing unit 100 determines whether the user has selected to move the portable laser scanner in accordance with the destination information presented on the display output unit 22 of the remote controller 20.

[0155] If it is determined that the user has not waited for the portable laser scanner to move, i.e., that the user has selected not to move the portable laser scanner, the process proceeds to step S30, where presentation information is generated to indicate that there is a blind spot in the direction of travel of the drone 10. The user will move the drone 10, taking into account that there is a blind spot in the direction of travel of the drone 10.

[0156] If it is determined that the movement of the portable laser scanner is being waited for, i.e., that the user has selected to move the portable laser scanner, the process returns to step S28 in Fig. 28 and the movement command for the drone 10 is invalidated. In this case, too, the movement control unit 130 stops (hovers) the drone 10 at that position and puts it into a standby state for the next movement command.

[0157] The above processing enables the drone 10 to fly safely.

[0158] According to the technology disclosed herein, by providing a means for acquiring the position information of a mobile body, such as a drone, necessary for controlling the movement of the mobile body, outside the mobile body, it becomes possible to more efficiently control the movement of the mobile body even in an environment where the mobile body cannot receive GNSS signals or where a stereo camera cannot be used.

[0159] In other words, by minimizing the number of parts installed on the drone, such as the GNSS receiver, stereo camera, and reflecting prism, it is possible to reduce the drone's weight and extend its flight time.

[0160] In addition, by eliminating blind spots in the surrounding environment, it will be possible to expand the drone's flight range and improve the safety and operational flexibility of drone-based inspections.

[0161] Furthermore, by reducing the data processing load of point cloud data, which requires time and power consumption, it is possible to shorten inspection times and extend system operating times by reducing power consumption.

[0162] It should be noted that the mobile body in the technology disclosed herein is not limited to a drone, but may also be any of a variety of robots that can move by acquiring information from the outside, such as a transport robot, a rescue robot, or a cleaning robot.

[0163] 7. Example of Computer Hardware Configuration The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed from a program recording medium into a computer incorporated in dedicated hardware, a general-purpose personal computer, or the like.

[0164] 22 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program. The information processing unit 100 that may be included in the drone 10, the remote controller 20, or the laser scanner 30, and the point cloud data processing server 40 may be configured, for example, by a computer 300 having a configuration similar to that shown in FIG.

[0165] A CPU (Central Processing Unit) 301 , a ROM (Read Only Memory) 302 , and a RAM (Random Access Memory) 303 are interconnected by a bus 304 .

[0166] An input / output interface 305 is also connected to the bus 304. An input unit 306 including a keyboard, a mouse, etc., and an output unit 307 including a display, a speaker, etc. are connected to the input / output interface 305. Also connected to the input / output interface 305 are a storage unit 308 including a hard disk, a nonvolatile memory, etc., a communication unit 309 including a network interface, etc., and a drive 310 that drives removable media 311.

[0167] In the computer 300 configured as described above, the CPU 301 performs the above-described series of processes by, for example, loading a program stored in the storage unit 308 into the RAM 303 via the input / output interface 305 and the bus 304 and executing the program.

[0168] The program executed by the CPU 301 is provided, for example, by being recorded on a removable medium 311 or via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting, and is installed in the storage unit 308 .

[0169] The program executed by computer 300 may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0170] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0171] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0172] The embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present disclosure.

[0173] For example, the embodiment of the present disclosure can be configured as a cloud computing system in which a single function is shared and processed collaboratively by multiple devices via a network.

[0174] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0175] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0176] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0177] Furthermore, the technology disclosed herein may have the following configurations. (1) An information processing device including: a position information estimation unit that estimates position information of a moving object using point cloud data acquired by two or more 3D point cloud acquisition devices that are present in an environment including the moving object such that at least a portion of the scanning ranges overlap; and a movement control unit that controls movement of the moving object based on the estimated position information. (2) The information processing device described in (1), in which the point cloud data includes point cloud data of at least a portion of an image capture target of an image capture device mounted on the moving object. (3) The information processing device described in (2), in which the point cloud data includes point cloud data of the entire image capture target. (4) The information processing device described in any of (1) to (3), in which the point cloud data includes point cloud data of obstacles present in the surrounding environment of the moving object. (5) The information processing device described in any of (1) to (4), in which the moving object is a drone. (6) The information processing device described in (5), in which the point cloud data includes point cloud data of all points within the flight range of the drone. (7) The information processing device according to (6), wherein the flight range includes a takeoff point and a landing point of the drone. (8) The information processing device according to any one of (1) to (7), wherein at least one of the three-dimensional point cloud acquisition devices is fixedly disposed in the environment. (9) The information processing device according to any one of (1) to (8), wherein at least one of the three-dimensional point cloud acquisition devices is movable in the environment. (10) The information processing device according to (9), wherein the movable three-dimensional point cloud acquisition device is mounted on a user terminal. (11) The information processing device according to (9), wherein the movable three-dimensional point cloud acquisition device is mounted on the moving body. (12) The information processing device according to (9), further comprising a presentation information generation unit that generates presentation information including location information of the movable three-dimensional point cloud acquisition device. (13) The information processing device according to (12), wherein the presentation information includes destination information of the movable three-dimensional point cloud acquisition device. (14) The information processing device according to (13), wherein the presentation information generation unit generates the presentation information including the destination information when there is a blind spot in the traveling direction of the moving object.(15) The information processing device according to any one of (1) to (14), wherein the location information estimation unit estimates the location information based on three-dimensional map information created from the point cloud data acquired in advance. (16) The information processing device according to (15), wherein the location information estimation unit estimates the location information by matching the point cloud data of the surrounding environment of the moving body with the three-dimensional map information. (17) The information processing device according to any one of (1) to (14), wherein the location information estimation unit estimates the location information by matching the point cloud data acquired by two or more of the three-dimensional point cloud acquisition devices with a point cloud pattern of the moving body acquired in advance. (18) The information processing device according to any one of (1) to (14), wherein the location information estimation unit estimates the location information using the point cloud data acquired by two or more of the three-dimensional point cloud acquisition devices and sensor data acquired by a predetermined sensor. (19) An information processing method comprising: estimating position information of a moving object using point cloud data acquired by two or more three-dimensional point cloud acquisition devices present in an environment including a moving object with at least a portion of its scanning range overlapping, and controlling movement of the moving object based on the estimated position information. (20) A mobile object control system comprising: two or more three-dimensional point cloud acquisition devices present in an environment including a moving object with at least a portion of its scanning range overlapping, a position information estimation unit that estimates position information of the moving object using point cloud data acquired by the two or more three-dimensional point cloud acquisition devices, and a movement control unit that controls movement of the moving object based on the estimated position information.

[0178] REFERENCE SIGNS LIST 1 Mobile object control system, 10 Drone, 14 Power unit, 15 Information processing unit, 20 Remote controller, 22 Display output unit, 23 Operation input unit, 24 Laser output unit, 25 Light receiving unit, 28 Information processing unit, 30 Laser scanner, 32 Laser output unit, 33 Light receiving unit, 36 Information processing unit, 40 Point cloud data processing server, 100 Information processing unit, 110 Coordinate position setting unit, 120 Position information estimation unit, 130 Movement control unit, 140 Presentation information generation unit

Claims

1. An information processing device comprising: a position information estimation unit that estimates the position information of a moving object using point cloud data acquired by two or more 3D point cloud acquisition devices that exist in an environment including the moving object and whose scanning ranges overlap at least partially; and a movement control unit that controls the movement of the moving object based on the estimated position information.

2. The information processing device according to claim 1, wherein the point cloud data includes point cloud data of at least a portion of an image capture target of an imaging device mounted on the moving body.

3. The information processing device according to claim 2, wherein the point cloud data includes point cloud data of the entire image capturing target.

4. The information processing device according to claim 1, wherein the point cloud data includes point cloud data of obstacles present in the surrounding environment of the moving object.

5. The information processing device according to claim 1, wherein the moving object is a drone.

6. The information processing device according to claim 5, wherein the point cloud data includes all point cloud data within the flight range of the drone.

7. The information processing device according to claim 6, wherein the flight range includes a takeoff point and a landing point of the drone.

8. The information processing device according to claim 1, wherein at least one of the three-dimensional point cloud acquisition devices is fixedly disposed in the environment.

9. The information processing device according to claim 1, wherein at least one of the three-dimensional point cloud acquisition devices is movable in the environment.

10. The information processing device according to claim 9, wherein the movable 3D point cloud acquisition device is mounted on a user terminal.

11. The information processing device according to claim 9, wherein the movable 3D point cloud acquisition device is mounted on the mobile body.

12. The information processing device according to claim 9, further comprising a presentation information generating unit that generates presentation information including arrangement information of the movable three-dimensional point cloud acquisition device.

13. The information processing device according to claim 12, wherein the presented information includes destination information of the movable 3D point cloud acquisition device.

14. The information processing device according to claim 13, wherein the presentation information generation unit generates the presentation information including the destination information when there is a blind spot in the traveling direction of the moving object.

15. The information processing device according to claim 1, wherein the location information estimation unit estimates the location information based on three-dimensional map information created from the point cloud data acquired in advance.

16. The information processing device according to claim 15, wherein the location information estimation unit estimates the location information by matching the point cloud data of the surrounding environment of the mobile object with the three-dimensional map information.

17. The information processing device according to claim 1, wherein the location information estimation unit estimates the location information by matching the point cloud data acquired by two or more of the three-dimensional point cloud acquisition devices with a point cloud pattern of the moving body that has been acquired in advance.

18. The information processing device according to claim 1, wherein the location information estimation unit estimates the location information using the point cloud data acquired by two or more of the three-dimensional point cloud acquisition devices and sensor data acquired by a predetermined sensor.

19. An information processing method comprising: estimating position information of a moving object using point cloud data acquired by two or more 3D point cloud acquisition devices present in an environment including the moving object, with at least a portion of the scanning ranges overlapping; and controlling the movement of the moving object based on the estimated position information.

20. A mobile object control system comprising: two or more 3D point cloud acquisition devices present in an environment including a mobile object with at least a portion of their scanning ranges overlapping; a position information estimation unit that estimates position information of the mobile object using point cloud data acquired by the two or more 3D point cloud acquisition devices; and a movement control unit that controls the movement of the mobile object based on the estimated position information.

Images