Mobile body, program, and position estimation method

Abstract

To suppress a decrease in precision of a position and posture of a mobile body during autonomous movement even when a map is created with distortion.SOLUTION: A mobile body enabling teaching travel by an operator and autonomous movement of a moving route comprises: an information creation unit which creates calculation information which can calculate the displacement between a nodal point through which the mobile body passed during the teaching travel and a location through which the mobile body passed during the autonomous movement in association with each nodal point of the moving route and a plurality of external sensors; a displacement calculation unit which calculates a displacement about each of the plurality of external sensors on the basis of the calculation information and sensor values of the external sensors during autonomous movement; a position estimation unit which estimates the calculated displacement as a position and posture of each nodal point of the moving route with reference to the position and posture of the mobile body; an integration processing unit which integrates the position and posture of each nodal point of the moving route estimated about each of the plurality of external sensors; and an autonomous movement processing unit which autonomously moves the moving route on the basis of the position and posture of each nodal point of the moving route integrated by the integration processing unit.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a moving object, a program, and a position estimation method. [Background technology]

[0002] In the past, when inspecting outdoor structures (pipes, meters, valves, etc.) in chemical plants and the like for leaks or abnormalities, inspectors would patrol the area where the structures were installed and visually inspect them, but in recent years, technology has been developed to automate the above inspection work using autonomous mobile objects. Mobile objects can move autonomously to their destinations using GNSS (Global Navigation Satellite System) positioning technology in outdoor environments and SLAM (Simultaneous Localization and Mapping) technology in relatively narrow indoor environments.

[0003] There is known a technology that enables a moving body to travel seamlessly both indoors and outdoors (for example, see Patent Document 1). In Patent Document 1, SLAM technology is used to create an environmental map in advance using topographical data obtained as a result of distance measurement by a distance sensor, and during autonomous movement, the environmental map is matched with distance data at each point in time to estimate the self-position. Summary of the Invention [Problem to be solved by the invention]

[0004] However, with conventional technology, for example, when an environmental map is created in an environment where GNSS positioning accuracy is continuously deteriorating, the map may be distorted, resulting in a decrease in the accuracy of the position and attitude of the moving body during autonomous movement.

[0005] In view of the above problems, the present invention provides a technique for suppressing a decrease in the accuracy of the position and orientation of a moving body during autonomous movement, even if a distorted map is created. [Means for solving the problem]

[0006] In view of the above problems, the present invention provides an instruction driving method that stores a position and posture when a travel path is moved by manual operation by an operator, and a moving body capable of autonomously moving along the travel path, the information creation unit creating calculation information that corresponds to each node of the travel path and a plurality of external sensors, and that is capable of calculating a deviation between a node passed during instruction driving and a point passed during autonomous movement, a deviation calculation unit that calculates the deviation for each of the plurality of external sensors based on the calculation information and the sensor value of the external sensor during autonomous movement, a position estimation unit that estimates the calculated deviation as the position and posture of each node of the travel path based on the position and posture of the moving body, an integration processing unit that integrates the positions and postures of each node of the travel path estimated for each of the plurality of external sensors, and an autonomous movement processing unit that autonomously moves the travel path based on the positions and postures of each node of the travel path integrated by the integration processing unit. Effect of the Invention

[0007] The present invention can provide a technique for suppressing a decrease in the accuracy of the position and orientation of a moving body during autonomous movement, even if a distorted map is created. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a conceptual diagram of teaching travel and autonomous movement in a comparative technique. [Diagram 2] FIG. 2(a) is a diagram showing a schematic diagram of waypoints stored during teaching travel, and FIG. 2(b) is a diagram explaining a method of estimating position and attitude during autonomous movement. [Diagram 3] FIG. 1 illustrates an example of an overall configuration of a communication system. [Figure 4] FIG. 2 is an example of an external perspective view of a moving body. [Figure 5A] FIG. 2 is a block diagram showing an example of a hardware configuration of a moving body. [Figure 5B]FIG. 2 is a block diagram showing a hardware configuration of an example of a display device. [Figure 6] FIG. 1 illustrates an example of a functional configuration of a communication system. [Figure 7] FIG. 13 is a conceptual diagram illustrating an example of a destination candidate management table. [Figure 8] FIG. 13 is a conceptual diagram illustrating an example of a route information management table. [Figure 9] FIG. 4 is a conceptual diagram illustrating an example of a map information management table. [Figure 10] FIG. 11 is a sequence diagram illustrating an example of a process for registering a destination candidate. [Figure 11] FIG. 13 is a diagram illustrating an example of an operation screen. [Figure 12] FIG. 11 is a sequence diagram showing an example of a manual operation process of a moving object. [Figure 13] 13 is a flowchart showing an example of a process for registering destination candidates according to a moving state of a mobile object; [Figure 14] FIG. 11 is a sequence diagram showing an example of a process for setting a destination of a moving body; [Figure 15] FIG. 13 illustrates an example of a selection screen. [Figure 16] FIG. 13 illustrates an example of a selection screen. [Figure 17] 13 is a flowchart illustrating an example of an autonomous movement process of a moving object. [Figure 18] FIG. 1 is a conceptual diagram of a topological map. [Figure 19] FIG. 1 is a block diagram of an example of movement control according to an embodiment of the present invention using an external sensor and a local map. [Figure 20] FIG. 11 is a flowchart illustrating an example of a process in a route teaching phase. [Figure 21] FIG. 11 is a flowchart illustrating an example of a process in an autonomous movement phase. [Figure 22] FIG. 1 is a schematic diagram showing the matching calculation between a 2D occupancy grid map and a 2D-LiDAR scan point cloud. [Diagram 23] FIG. 13 is a flowchart illustrating an example of a flow of evaluation of b0 based on the amount of obstacles. [Figure 24] FIG. 2 is an example of a hardware block diagram of the display device when the display device is an HMD. [Diagram 25] FIG. 11 is a diagram showing an example of image data displayed by a display device that is an HMD. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A moving object and a position estimation method performed by the moving object will be described below as an example of an embodiment of the present invention with reference to the drawings.

[0010] [First embodiment] <Additional information on positioning technology> GNSS positioning technology is used in open outdoor environments, while autonomous navigation technology (SLAM technology) using maps created using distance sensors (LiDAR: Light Detection And Ranging) and cameras has been developed for relatively small indoor environments. In addition, technology is also known that uses multiple sensors to create maps and estimate self-location in order to estimate position and attitude with high accuracy even in vast areas where the GNSS positioning accuracy is not necessarily high.

[0011] In order to estimate the position and attitude with high accuracy, there are technologies that aim to create an accurate environmental map and estimate the self-location using multiple sensors such as GNSS, LiDAR, and cameras. Most of these technologies use a method of "reconstructing the entire environment into a single two-dimensional or three-dimensional map and estimating the location of the moving object 10 within that map."

[0012] However, even with multiple sensors, it is still difficult to accurately restore the environment without distortion, and manual map correction is often required. Furthermore, on an inaccurate map with distortion, there is a possibility that the self-location estimation results from each sensor may differ, leading to unstable driving.

[0013] As a concrete example, we will explain position and attitude estimation using GNSS and LiDAR. If the GNSS positioning accuracy is good, accurate maps can be created by correcting the distortions in maps created using LiDAR to match the GNSS coordinate system. However, if the GNSS positioning accuracy continues to deteriorate, the distortions in maps created using LiDAR cannot be properly corrected, and the maps in that area become inaccurate.

[0014] Furthermore, if accurate GNSS positioning happens to be possible in a location where a map was created using LiDAR during autonomous movement, a discrepancy will occur between the self-location estimation result obtained by matching the LiDAR scan data with the map and the GNSS positioning result. In this case, the mobile unit will autonomously move using one of these self-location estimation results or an intermediate result as the final estimation result, which may lead to inaccurate driving. Alternatively, in order to prevent inaccurate driving, the operator must manually correct the distorted parts of the map to match the GNSS coordinates.

[0015] Therefore, in this embodiment, even in a vast area where the GNSS positioning accuracy is not necessarily high, the vehicle can autonomously move with high accuracy without manually correcting the map. This also enables an operator without specialized knowledge to autonomously move the vehicle along a route that the vehicle has traveled manually.

[0016] FIG. 1 is a conceptual diagram of teaching travel and autonomous movement in the comparative technology. The comparative technology is a technology that clarifies the processing of the present embodiment by comparing it with the present embodiment. The comparative technology is not necessarily a conventional technology. Teaching refers to an operator manually moving a moving object 10 and having the moving object 10 memorize a moving route 9.

[0017] As shown in FIG. 1(a), first, an operator manually drives a moving body 10 as a teaching run. The moving body 10 memorizes a travel route 9 while creating an environmental map. Next, as shown in FIG. 1(b), the moving body 10 autonomously moves along the memorized travel route 9. In this way, in the comparative technology, although the moving body moves along the travel route 9, a local map of waypoints described below is not memorized during teaching run.

[0018] 2(a) is a schematic diagram showing a waypoint 8 stored during instruction travel in this embodiment. The moving body 10 of this embodiment generates a topological map during instruction travel. The topological map is a map representation in a graph format consisting of waypoints 8 (nodes, described later) that exist discretely on the travel route, and edges that represent the relative positional relationships between the waypoints 8. Each waypoint 8 is associated with a plurality of local maps 91, 92 that are generated independently by a plurality of sensors and have the waypoint 8 as the origin.

[0019] Fig. 2(b) is a diagram explaining a method of estimating position and attitude during autonomous movement. During autonomous movement, the moving body 10 calculates the deviation of the position and attitude of the waypoint 8, which is estimated from the local map associated with the waypoint 8 and the position and attitude based on the sensor value (sensor measurement value) detected by the sensor during autonomous movement. Since the moving body 10 uses its own position as the origin during autonomous movement, this deviation is the position and attitude of the waypoint 8 as seen from the moving body 10 during autonomous movement. Note that "seen" means that the position and attitude of the moving body 10 are used as the reference.

[0020] The moving body 10 weights the position and attitude based on the GNSS and the position and attitude based on the SLAM according to the positioning accuracy of the GNSS and the LiDAR, and integrates the deviation amount (i.e., the position of the waypoint 8 as seen from the moving body 10).

[0021] In this way, the mobile body 10 of this embodiment has an independent local map for each sensor in the form of a topological map, calculates the reliability of position estimation from the sensor values ​​during teaching driving and autonomous movement for the self-position and self-attitude estimated independently for each, and integrates them into the final position and attitude by weighting according to the reliability. Therefore, there is no need to manually correct the map so that the position estimation results by each sensor are consistent, and high-precision autonomous movement is possible even if the positioning accuracy of any sensor is reduced in the route teaching phase.

[0022] <Terminology> The teaching run refers to a run in which the operator manually operates the mobile body 10 with a controller, moves the mobile body 10 along a desired path along which the mobile body 10 should move autonomously, and stores the position and posture of the mobile body 10 when the mobile body 10 moves along the path. The autonomous run refers to the mobile body 10 automatically moving on or along the path stored in the teaching run.

[0023] The external sensor is a sensor that measures the environment (i.e., the outside world) in which the moving body 10 is placed. Examples of the external sensor include GNSS, LiDAR, and a camera that can acquire distance information. The internal sensor is a sensor that measures the state (i.e., the inside) of the moving body 10. Examples of the internal sensor include a speed sensor, an acceleration sensor, an angular velocity sensor, and a direction sensor.

[0024] A node is a point that constitutes a travel route. In this embodiment, the node is described using the term node or waypoint.

[0025] The position is information that determines the location of the moving body 10 based on the origin, and the attitude is information that determines the direction of the moving body 10 based on a specific direction. The attitude may be in a three-dimensional space or a two-dimensional space.

[0026] The calculated information is information that allows for calculating the deviation between the nodes passed during the teaching run and the points passed during the autonomous run. The calculated information may differ depending on the external sensor. In GNSS, the calculated information is the position and attitude in the GNSS coordinate system, and in LiDAR, it is a two-dimensional occupancy grid map. In this embodiment, the calculated information is described in terms of a local map.

[0027] The local map refers to a map of at least a part of the travel route, but does not cover the entire travel route. The local map may be a map of an area in which the external sensor can measure the environment, with the waypoint 8 as the origin.

[0028] Integration means combining two or more things into one.

[0029] Using the position and attitude of the moving body 10 as a reference means a position and attitude relative to the position and attitude of the moving body 10. In this embodiment, the expression "as seen from" the moving body 10 is sometimes used.

[0030] <System configuration> Fig. 3 is a diagram showing an example of the overall configuration of a communication system 1. The communication system 1 shown in Fig. 3 is a system in which an operator can remotely control a moving object 10 that moves within a predetermined base.

[0031] The communication system 1 includes a mobile object 10 and a display device 60 located at a predetermined base. The mobile object 10 and the display device 60 constituting the communication system 1 can communicate with each other via a communication network 100. The communication network 100 is constructed by the Internet, a mobile communication network, a LAN (Local Area Network), and the like. Note that the communication network 100 may include not only wired communication but also wireless communication networks such as 3G (3rd Generation), 4G (4th Generation), 5G (5th Generation), Wi-Fi (Wireless Fidelity) (registered trademark), WiMAX (Worldwide Interoperability for Microwave Access), or LTE (Long Term Evolution).

[0032] The mobile body 10 is a robot that is installed at a target base and can move autonomously within the target base. This autonomous movement includes an operation of autonomously moving within the target base by imitation learning (machine learning) the route traveled in the past within the target base and using the result of the imitation learning. In addition, the autonomous movement may be an operation of autonomously moving within the target base according to a preset moving route or an operation of autonomously moving within the target base using a technology such as line tracing. Furthermore, the mobile body 10 can also be moved by manual operation from an operator at a remote location. That is, the mobile body 10 can move within the target base while switching between autonomous movement and manual operation by the operator. In addition, the mobile body 10 can perform a predetermined task such as inspection, maintenance, transportation, or light work while moving within the target base. Here, the mobile body 10 means a robot in a broad sense, and may be a robot that can perform both autonomous movement and movement by remote operation by an operator. In addition, the mobile body 10 also includes a car that can run by switching between automatic driving and manual operation by operation from a remote location. Furthermore, the mobile object 10 also includes an air vehicle such as a drone, a multicopter, or an unmanned air vehicle.

[0033] The target base where the mobile object 10 is installed may be, for example, an outdoor base such as a business office, a factory, a construction site, a substation, a farm, a field, a plantation, a cultivated land, or a disaster site, or an indoor base such as an office, a school, a factory, a warehouse, a commercial facility, a hospital, or a nursing home. In other words, the target base may be any base where there is a need to have the mobile object 10 take on a task that has traditionally been performed manually.

[0034] The display device 60 is a computer such as a notebook PC (Personal Computer) that is located at a management base different from the target base and is used by an operator who performs a predetermined operation on the mobile object 10. The operator, at the management base such as an office, uses an operation screen displayed on the display device 60 to perform a movement operation on the mobile object 10 or an operation for executing a predetermined task. For example, the operator remotely controls the mobile object 10 while viewing an image of the target base displayed on the display device 60.

[0035] 3 shows an example in which one mobile object 10 and one display device 60 are connected via the communication network 100, the display device 60 may be configured to be connectable to a plurality of mobile objects 10 located at one target base, or may be configured to be connectable to mobile objects 10 located at different target bases. Also, while FIG. 3 shows an example in which the display device 60 is located at a management base in a remote location different from the target base where the mobile object 10 is installed, the display device 60 may be configured to be located within the target base where the mobile object 10 is installed. Furthermore, the display device 60 is not limited to a notebook PC, and may be, for example, a desktop PC, a tablet terminal, a smartphone, or a wearable terminal.

[0036] <Outline of the moving object> 4(a) is a perspective view of the exterior of the moving body 10 according to this embodiment. The moving body 10 includes track-type traveling bodies 10a and 10b and a main body 50.

[0037] The track-type running bodies 10a and 10b are units that serve as the means of movement of the moving body 10. The track-type running bodies 10a and 10b are track (crawler) type running bodies using metal or rubber belts. The track-type running bodies have a larger ground contact area than running bodies that run on tires such as automobiles, and can run stably, for example, even in environments with poor footing. In addition, while running bodies that run on tires require a turning space when turning, a running device equipped with track-type running bodies can perform so-called pivot turns, so that it can turn smoothly even in a limited space.

[0038] The main body 50 is a support that supports the track-type vehicles 10a, 10b in a runnable state, and is also a control device that controls the driving of the moving body 10. The main body 50 also has a battery mounted thereon that supplies power to drive the track-type vehicles 10a, 10b.

[0039] 4(b) is a front view (view taken along arrow P) of the moving body 10 according to this embodiment. The main body 50 of the moving body 10 includes an emergency stop switch 143, a status display lamp 144, and a cover portion 146. The emergency stop switch 143 is an operating means that is pressed by a person in the vicinity of the moving body 10 when stopping the moving body 10 during movement.

[0040] The status indicator lamp 144 is a notification means for notifying the status of the moving object 10. For example, when the status of the moving object 10 changes, such as when the remaining battery level is low, the status indicator lamp 144 lights up to notify people around the moving object 10 of the change in status. The status indicator lamp 144 also lights up when there is a risk of an abnormality occurring, such as when the presence of an obstacle that hinders the movement of the moving object 10 is detected. Note that while FIG. 4(b) shows an example in which the moving object 10 is provided with two status indicator lamps 144, the number of status indicator lamps 144 may be one or three or more. The notification means may also be configured to notify the status of the moving object 10 not only by the status indicator lamp 144 but also by a warning sound emitted from a speaker.

[0041] The lid portion 146 is provided on the upper surface of the main body 50, and seals the inside of the main body 50. The lid portion 146 also has a ventilation portion 35a having a vent hole for ventilating the inside of the main body 50.

[0042] The two track-type vehicles 10a, 10b are installed with the main body 50 in between, so that the mobile unit 10 can move. The number of track-type vehicles is not limited to two, and may be three or more. For example, the mobile unit 10 may be installed in a movable state, such as by arranging three track-type vehicles in three rows in parallel. Also, for example, the mobile unit 10 may have four track-type vehicles arranged in the front, rear, left and right directions like tires of an automobile.

[0043] FIG. 4(c) is a side view (view from the Q arrow) of the moving body 10 according to this embodiment. The tracked moving body 10a has a triangular shape. For example, when the front-rear size of the moving body is restricted, the triangular tracked moving body 10a can increase the ground contact area within the limited front-rear size, thereby improving stability during movement. On the other hand, a so-called tank-type crawler, in which the upper side (driving wheel side) is longer than the lower side (rolling wheel side), has a smaller overall ground contact area and becomes unstable when the front-rear size is restricted. In this way, the tracked moving body 10a is effective in increasing the mobility of a relatively small moving body 10.

[0044] 4, the moving body 10 includes a distance measuring sensor 112 for detecting the horizontal direction on the front side of the moving direction of the main body 50. The distance measuring sensor 112 for detecting the horizontal direction is a LiDAR, and in this embodiment, is a MEMS (Micro Electro Mechanical Systems) type LiDAR for 2D scanning. The LiDAR may be for 3D scanning.

[0045] Distance measurement sensor 112 irradiates an object with laser light, and measures the distance to the object and the direction in which the object is located based on the measurement result of the time it takes for the light to hit the object and bounce back.

[0046] 4, the moving body 10 is provided with a distance measuring sensor 113 for detecting oblique directions on the front side of the main body 50 in the traveling direction. The distance measuring sensor 113 for detecting oblique directions is a LiDAR, and in this embodiment, it is a MEMS type LiDAR for 2D scanning. The distance measuring sensor 113 is attached at an angle below the horizontal so that it faces diagonally downward at a predetermined inclination angle with respect to the horizontal traveling road surface.

[0047] The distance measurement sensor 113 irradiates an object (road obstacle) with laser light, and measures the distance to the object (road obstacle) and the direction in which the object (road obstacle) is located based on the measurement of the time it takes for the light to hit the object (road obstacle) and bounce back.

[0048] Incidentally, the optimum installation position of the distance measuring sensor 113 depends on the width and length of the tracks (crawlers) of the track-type traveling bodies 10a, 10b, and on the size, width, depth, and height of the object to be detected.

[0049] Although the moving body 10 in FIG. 4 is a caterpillar-propelled type, the shape and movement method of the moving body 10 are not limited to the embodiment in FIG. 2, and may be a bipedal type, a multipedal type, a drone type, a flying type, or an underwater type.

[0050] <Hardware configuration example> Next, an example of the hardware configuration of the moving body 10 according to the present embodiment will be described with reference to Figures 5A and 5B. Figure 5A is a block diagram showing an example of the hardware configuration of the moving body 10.

[0051] As shown in FIG. 5A, the moving body 10 in this embodiment has a CPU (Central Processing Unit) 101, a memory 102, an auxiliary storage device 103, a camera 111, a distance measurement sensor 112 for detecting horizontal directions, a distance measurement sensor 113 for detecting diagonal directions, a satellite positioning system 114, an IMU (Inertial Measurement Unit) 115, a battery 121, a start SW 142, an emergency stop SW 143, motor drivers 122a, 122b, running motors 132a, 132b, brake drivers 123a, 123b, brake motors 133a, 133b, and a power supply SW 141.

[0052] The CPU 101 controls the entire moving body 10. The memory 102 is a temporary storage area for the CPU 101 to execute a moving program and the like.

[0053] The camera 111 includes a spherical camera, a stereo camera, an infrared camera, etc. The distance measuring sensor 112 for detecting a horizontal direction and the distance measuring sensor 113 for detecting an oblique direction are, for example, LiDARs.

[0054] The satellite positioning system 114 receives radio waves from satellites and, based on the reception results, measures the position on the earth of the mobile unit 10. The satellite positioning system 114 is also called a GNSS.

[0055] The IMU 115 is an inertial positioning sensor that uses an acceleration sensor and an angular velocity sensor to estimate the position and attitude (translational motion and rotational motion in three orthogonal axial directions) of the mobile object 10. The IMU 115 is an example of an internal sensor.

[0056] The battery 121 is a power source for moving the moving object 10. The start SW 142 is a switch for starting the moving object 10. The emergency stop SW 143 is a switch for arbitrarily stopping the movement of the moving object 10.

[0057] The motor drivers 122a and 122b are drivers for the traveling motors 132a and 132b provided on the two track-type traveling bodies 10a and 10b, respectively. The brake drivers 123a and 123b are drivers for the braking motors 133a and 133b provided on the two track-type traveling bodies 10a and 10b, respectively. The power supply SW 141 is a switch that turns the power supply of the moving body 10 on and off.

[0058] The mobile program executed by the mobile body 10 of this embodiment is provided as a file in an installable or executable format recorded on a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, or a DVD (Digital Versatile Disc).

[0059] The mobile program executed by the mobile object 10 of the present embodiment may be configured to be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. The mobile program executed by the mobile object 10 of the present embodiment may be provided or distributed via a network such as the Internet. The mobile program executed by the mobile object 10 of the present embodiment may be provided by being pre-installed in a ROM or the like.

[0060] <<Display device>> Fig. 5B is a diagram showing an example of a hardware configuration of the display device 60 according to the present embodiment. As shown in Fig. 5B, the display device 60 is constructed by a computer 500, and includes a CPU 501, a ROM 502, a RAM 503, a HD (Hard Disk) 504, a HDD (Hard Disk Drive) controller 505, a display 506, an external device connection I / F (Interface) 508, a network I / F 509, a bus line 510, a keyboard 511, a pointing device 512, a DVD-RW (Digital Versatile Disk Rewritable) drive 514, and a media I / F 516.

[0061] Among these, the CPU 501 controls the operation of the entire computer 500. The ROM 502 stores programs such as IPL used to drive the CPU 501. The RAM 503 is used as a work area for the CPU 501. The HD 504 stores various data such as programs. The HDD controller 505 controls reading or writing of various data from the HD 504 under the control of the CPU 501. The display 506 displays various information such as a cursor, a menu, a window, a character, or an image. The external device connection I / F 508 is an interface for connecting various external devices. In this case, the external device is, for example, a USB (Universal Serial Bus) memory or a printer. The network I / F 509 is an interface for data communication using a network. The bus line 510 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 501 shown in FIG. 5B.

[0062] Moreover, the keyboard 511 is a type of input means having a plurality of keys used for inputting characters, numbers, various instructions, etc. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The DVD-RW drive 514 controls reading and writing of various data from a DVD-RW 513 as an example of a removable recording medium. Note that the DVD-RW drive 514 is not limited to a DVD-RW, and may be a DVD-R or the like. The media I / F 516 controls reading and writing (storing) of data from a recording medium 515 such as a flash memory.

[0063] <Functional configuration> Next, a functional configuration of the communication system 1 according to the embodiment will be described with reference to Fig. 6 to Fig. 9. Fig. 6 is a diagram showing an example of a functional configuration of the communication system 1. Fig. 6 shows devices or terminals shown in Fig. 1 that are related to processing or operations described below.

[0064] <<Functional configuration of mobile unit (control device)>> The moving body 10 includes a transmitting / receiving unit 31, a determining unit 32, a photographing control unit 33, a state detecting unit 34, a position information acquiring unit 35, a destination candidate acquiring unit 36, a route information generating unit 37, a route information managing unit 38, a moving destination setting unit 39, a moving control unit 40, an image recognition processing unit 41, a mode setting unit 42, an autonomous moving processing unit 43, a manual operation processing unit 44, a task executing unit 45, an information creating unit 46, an image processing unit 47, a learning unit 48, and a storing / reading unit 49. Each of these units is a function or a functioning means realized by any of the components shown in FIG. 5A operating according to an instruction from the CPU 101 according to the moving program deployed on the memory 102. The moving body 10 also includes a memory unit 3000 constructed by the memory 102 shown in FIG. 5A.

[0065] The transmitting / receiving unit 31 transmits and receives various data or information to and from other devices or terminals via the communication network 100 .

[0066] The determination unit 32 performs various determinations and is realized by the processing of the CPU 101. The shooting control unit 33 is mainly realized by the processing of the CPU 101 for the camera 111 and controls the shooting processing for the camera 111.

[0067] The state detection unit 34 is mainly realized by the processing of the CPU 101 on the camera 111 and the distance measuring sensors 112 and 113, and detects the state of the moving body 10 or the surroundings of the moving body 10 using various sensors. The state detection unit 34 measures the distance to an object (obstacle) present around the moving body 10, for example, and outputs it as distance data. The state detection unit 34 may also be configured to acquire the position by using the distance data and matching it with an environmental map by applying Simultaneous Localization and Mapping (SLAM). Here, SLAM is a technology that can simultaneously perform self-location estimation and environmental map creation. Furthermore, the state detection unit 34 detects the direction in which the moving body 10 is facing, for example, using the IMU 115 or the like.

[0068] The location information acquisition unit 35 is realized by processing of the CPU 101 for the satellite positioning system 114, and acquires location information indicating the current location of the mobile object 10 using the satellite positioning system 114. The location information acquisition unit 35 acquires, for example, coordinate information indicating the latitude and longitude of the current location of the mobile object 10 using the satellite positioning system 114.

[0069] The destination candidate acquisition unit 36 ​​is mainly realized by the processing of the CPU 101, and acquires destination candidate images that indicate candidates for destinations to which the moving body 10 can move. Specifically, the destination candidate acquisition unit 36 ​​acquires, as the destination candidate images, images captured by the imaging control unit 33 that indicate a part of an area within the base where the moving body 10 is located.

[0070] The route information generating unit 37 is mainly realized by the processing of the CPU 101, and generates route information indicating a moving route of the mobile body 10. The route information generating unit 37 generates route information indicating a route from the current position to the final destination based on the positions of the destination candidates selected by the operator of the mobile body 10. The route information is generated, for example, by connecting each waypoint 8 from the current position to the final destination with a straight line, or by using information on obstacles obtained from the captured images or the state detection unit 34 to minimize the moving time while avoiding obstacles.

[0071] The route information management unit 38 is mainly realized by the processing of the CPU 101 , and manages the route information generated by the route information generation unit 37 by using the route information management DB 3002 .

[0072] The destination setting unit 39 is mainly realized by the processing of the CPU 101, and sets the destination of the moving body 10. The destination setting unit 39 sets the destination to which the moving body 10 is currently heading as the destination, from among destination candidates selected by the operator of the moving body 10, based on, for example, the current position of the moving body 10 acquired by the position information acquisition unit 35 and the route information generated by the route information generation unit 37.

[0073] The movement control unit 40 is mainly realized by the processing of the CPU 101 for the motor drivers 122a, 122b, and controls the movement of the moving body 10 by driving the running motors 132a, 132b. The movement control unit 40 moves the moving body 10 in response to a drive command from the autonomous movement processing unit 43 or the manual operation processing unit 44, for example.

[0074] The image recognition processing unit 41 is mainly realized by the processing of the CPU 101, and performs image recognition processing on the captured image acquired by the photography control unit 33. For example, the image recognition processing unit 41 detects whether or not a specific subject is captured in the acquired captured image by image recognition. The specific subject is, for example, an obstacle on the movement path of the moving object 10 or in the vicinity of the movement path, an intersection such as a crossroads or an L-shaped intersection, or a sign or signal installed at a base.

[0075] The mode setting unit 42 is mainly realized by the processing of the CPU 101, and sets an operation mode indicating the operation of moving the moving body 10. The mode setting unit 42 sets either an autonomous movement mode in which the moving body 10 moves autonomously, or a manual operation mode in which the moving body 10 is moved by manual operation by an operator.

[0076] The autonomous movement processing unit 43 is mainly realized by the processing of the CPU 101, and controls the autonomous movement processing of the moving body 10. The autonomous movement processing unit 43 outputs a drive command for the moving body 10 to the movement control unit 40 so as to pass through a movement route indicated in the route information generated by the route information generating unit 37, for example.

[0077] The autonomous movement processing unit 43 includes a calculation unit 43a, a deviation calculation unit 43b, a position estimation unit 43c, and an integration processing unit 43d. The calculation unit 43a calculates the position and orientation of the moving object 10 based on the measurement values ​​of a plurality of external sensors and internal sensors during autonomous movement, using an existing method such as a Kalman filter.

[0078] The deviation calculation unit 43b calculates the deviation between the local map and the sensor value of each of the multiple external sensors. The position estimation unit 43c estimates the calculated deviation as the position and orientation of the waypoint 8 as seen from the moving body 10. The integration processing unit 43d integrates the position and orientation of the waypoint 8 estimated for each of the multiple external sensors.

[0079] The manual operation processing unit 44 is mainly realized by the processing of the CPU 101, and controls the manual operation processing of the moving body 10. The manual operation processing unit 44 outputs a drive command for the moving body 10 to the movement control unit 40 in response to a manual operation command transmitted from the display device 60, for example.

[0080] The task execution unit 45 is mainly realized by the processing of the CPU 101, and executes a predetermined task by the moving body 10 in response to a request from an operator. Here, the predetermined task executed by the task execution unit 45 includes, for example, a photographing process for inspecting equipment in a base, or light work using a movable arm.

[0081] The information creation unit 46 is mainly realized by the processing of the CPU 101, and manages map information indicating an environmental map of a target base where the mobile object 10 is installed, using the map information management DB 3003. The information creation unit 46 creates a plurality of local maps based on a plurality of external sensors, for example, and manages them in association with the external sensors.

[0082] The image processing unit 47 is mainly realized by the processing of the CPU 101, and generates a display image to be displayed on the display device 60. The image processing unit 47 generates a display image to be displayed on the display device 60, for example, by performing processing on the captured image acquired by the shooting control unit 33.

[0083] The learning unit 48 is mainly realized by the processing of the CPU 101, and learns a movement route for autonomous movement of the moving body 10. The learning unit 48 performs imitation learning (machine learning) of a movement route for autonomous movement, for example, based on a captured image acquired during a movement operation in a manual operation mode by the manual operation processing unit 44 and detection data by the state detection unit 34. The autonomous movement processing unit 43 performs autonomous movement of the moving body 10, for example, based on learning data that is a result of imitation learning by the learning unit 48.

[0084] The storage / reading unit 49 is realized mainly by the processing of the CPU 101 , and stores various data (or information) in the storage unit 3000 and reads various data (or information) from the storage unit 3000 .

[0085] <<Destination candidate management table>> Fig. 7 is a conceptual diagram showing an example of a destination candidate management table. The destination candidate management table is a table for managing destination candidates indicating potential destinations of the mobile object 10. A destination candidate management DB 3001 configured by the destination candidate management table as shown in Fig. 7 is constructed in the storage unit 3000.

[0086] The destination candidate management table manages destination candidate data in which a candidate ID for identifying a destination candidate, location information indicating the location of the destination candidate, and photographed image data of a part of an area in the destination candidate base are associated with each base ID for identifying a base where the mobile body 10 is installed. Among these, the location information is represented by coordinate information including latitude and longitude indicating the location of the destination candidate in the base. Note that, when the mobile body 10 is an aircraft such as a drone, the location information includes information such as the speed, attitude, or altitude of the aircraft in addition to coordinate information indicating the latitude and longitude. Also, the photographed image data indicates an image data file photographed by the mobile body 10 at the location of the destination candidate in the base. Here, the destination candidate is an example of a candidate for the moving destination of the mobile body 10. The candidate for the moving destination of the mobile body 10 includes not only the candidate for the destination of the mobile body 10 but also, for example, a candidate for a location to be excluded from the moving route of the mobile body 10.

[0087] <<Route information management table>> Fig. 8 is a conceptual diagram showing an example of a route information management table. The route information management table is a table for managing route information indicating a moving route of the moving object 10. In the storage unit 3000, a route information management DB 3002 configured by the route information management table as shown in Fig. 8 is constructed.

[0088] The route information management table manages, for each base ID that identifies the base where the mobile body 10 is installed, a route ID that identifies the movement route of the mobile body 10 and route information that indicates the movement route of the mobile body 10 in association with each other. Among these, the route information indicates the future movement route of the mobile body 10 in the order of the future destinations. The route information is generated by the route information generation unit 37 when the mobile body 10 starts moving.

[0089] In this embodiment, a travel route connecting multiple destinations and a travel route stored during instruction driving are described, but part or the entirety of the former travel route is the latter travel route, and no distinction will be made strictly between the two.

[0090] <<Map information management table>> Fig. 9 is a conceptual diagram showing an example of a map information management table. The map information management table is a table for managing local maps generated for each node during teaching travel. A map information management DB 3003 configured by the map information management table shown in Fig. 9 is constructed in the storage unit 3000.

[0091] The map information management table in FIG. 9 stores a first local map generated by GNSS with the position and attitude of each node as the origin, and a second local map generated by SLAM with the position and attitude of each node as the origin. The first local map and the second local map are generated in a data format suitable for each map. In this embodiment, the "local map" is "calculation information capable of calculating the deviation between the node passed during teaching travel and the point passed during autonomous travel", and the format of the local map differs depending on the external sensor. For example, the first local map represents the position and attitude of the GNSS coordinate system and is not a general map. Details of the local map will be described in FIG. 18.

[0092] The map information management table may further include a local map captured by a camera. The camera may be a stereo camera or a monocular camera capable of acquiring distance information. The local map captured by the camera includes the same information as the local map captured by the SLAM.

[0093] <<Learning Data Management DB>> The learning data management DB 3004 constructed in the storage unit 3000 will be described. The learning data management DB 3004 manages learning data that is a result of learning of the autonomous movement route of the moving body 10 by the learning unit 48. The learning data management DB 3004 accumulates, for example, photographed image data and various detection data acquired from the moving body 10, and stores learning data that is a result of imitation learning (machine learning) for each base or each moving body 10. The moving body 10 moves autonomously based on the learning data stored in the learning data management DB 3004. Note that the photographed image data photographed by the special imaging device 12a or the general imaging device 12b may include PTZ (Pan-Tilt-Zoom) parameters for specifying the imaging direction of the special imaging device 12a or the general imaging device 12b. The photographed image data including the PTZ parameters is stored (saved) in the storage unit 3000 (learning data management DB 3004) of the moving body 10. Further, the PTZ parameters may be linked to the position information of the destination candidate and stored in the storage unit 3000, and the coordinate data (x, y, θ) indicating the attitude of the moving body 10 when the captured image data of the destination candidate is acquired may be stored in the storage unit 3000 at the same time. This makes it possible to correct the attitude of the moving body 10 using the PTZ parameters and the coordinate data (x, y, θ) when a deviation occurs in the actual stopping position of the moving body 10 relative to the destination. Note that some of the data, such as the data of the autonomous movement route (GNSS positioning history) of the moving body 10 and the captured image data of the destination candidate used for display on the display device 60, may be stored in a cloud computing service such as AWS (Amazon Web Services) (trademark).

[0094] <<Functional configuration of the display device>> Next, the functional configuration of the display device 60 will be described with reference to Fig. 6. The display device 60 has a transmitting / receiving unit 51, a receiving unit 52, a display control unit 53, a determining unit 54, a manual operation command generating unit 55, an autonomous movement request information generating unit 56, an image processing unit 57, and a storing / reading unit 59. Each of these units is a function or a means for performing a function that is realized by operating any of the components shown in Fig. 5B by an instruction from the CPU 501 according to the display device program expanded on the RAM 503. The display device 60 also has a storage unit 5000 constructed by the ROM 502, HD 504, or recording medium 521 shown in Fig. 5B.

[0095] The transmission / reception unit 51 is realized mainly by the processing of the CPU 501 for the network I / F 509, and transmits and receives various data or information to and from other devices or terminals.

[0096] The reception unit 52 is mainly realized by the processing of the CPU 501 on the keyboard 511 or the pointing device 512, and receives various selections or inputs from the operator. The display control unit 53 is mainly realized by the processing of the CPU 501, and displays various screens on a display unit such as a display 506. The determination unit 54 is mainly realized by the processing of the CPU 501, and performs various determinations.

[0097] The manual operation command generation unit 55 is mainly realized by the processing of the CPU 501, and generates a manual operation command for moving the moving body 10 by manual operation in response to an input operation by the operator. The autonomous movement request information generation unit 56 is mainly realized by the processing of the CPU 501, and generates autonomous movement request information for autonomously moving the moving body 10. The autonomous movement request information generation unit 56 transmits autonomous movement request information to be transmitted to the moving body 10 based on, for example, information on a destination candidate selected by the operator.

[0098] The image processing unit 57 is mainly realized by processing of the CPU 501, and generates a display image to be displayed on a display unit such as the display 506. The image processing unit 57 generates a display image to be displayed on the display device 60, for example, by processing a photographed image acquired by the moving object 10. Note that the communication system 1 may be provided with the functions of either the image processing unit 47 of the moving object 10 or the image processing unit 57 of the display device 60.

[0099] The storage / reading unit 59 is realized mainly by the processing of the CPU 501 , and stores various data (or information) in the storage unit 5000 and reads various data (or information) from the storage unit 5000 .

[0100] <Process or Action> <<Destination candidate registration process>> First, a process of registering a destination candidate for the moving object 10 will be described with reference to Fig. 10 to Fig. 13. Fig. 10 is a sequence diagram showing an example of a process of registering a destination candidate.

[0101] The display device 60 starts operating the moving object 10 in response to a predetermined input operation by the operator (step S11). Then, the transmitting / receiving unit 51 transmits an operation start request to the moving object 10 (step S12). As a result, the transmitting / receiving unit 31 of the moving object 10 receives the operation start request transmitted from the display device 60.

[0102] Next, the photographing control unit 33 starts the photographing process using the special photographing device 12a and the general photographing device 12b (step S13). The photographing control unit 33 acquires photographed image data photographed by the special photographing device 12a and the general photographing device 12b. Here, the image acquired by the photographing process in step S13 is a moving image. Then, the transmitting / receiving unit 31 transmits the photographed image data acquired in step S13 to the display device 60 (step S14). As a result, the transmitting / receiving unit 51 of the display device 60 receives the photographed image data transmitted from the moving object 10.

[0103] Next, the display control unit 53 of the display device 60 causes the operation screen 200 including the captured image data received in step S14 to be displayed on a display unit such as the display 506 (step S15). 11 is a diagram showing an example of an operation screen 200. The operation screen 200 shown in FIG.

[0104] The operation screen 200 includes a base display area 210 in which a photographed image (planar image) photographed by the general photographing device 12b, which is the photographed image data received in step S14, is displayed, and a base display area 230 in which a photographed image (special image) photographed by the special photographing device 12a, which is the photographed image data received in step S14, is displayed. Of these, the special image displayed in the base display area 230 is an image photographed in all directions within the base, such as a celestial sphere image, a wide-angle image, or a hemispherical image photographed by the special photographing device 12a, as described above. The special image may also be an image obtained by the special photographing device 12a that generates an image in all directions by synthesizing images photographed while rotating the general photographing device. In addition, in the base display area 230, a moving object display image 235 indicating the presence of the moving object 10 on the special image is displayed superimposed on the special image. In addition, in the operation screen 200, coordinate information represented by latitude (Lat) and longitude (Lon) is displayed as position information indicating the current position of the moving object 10. When the moving body 10 is an air vehicle such as a drone, the operation screen 200 may be configured to display information such as the speed, attitude, or altitude of the air vehicle in addition to the coordinate information of latitude and longitude. Also, the operation screen 200 may display live streaming images in which images captured by the special imaging device 12a and the general imaging device 12b are distributed in real time via a computer network such as the Internet.

[0105] The operation screen 200 also includes operation icons 250 for remotely controlling the moving body 10. The operation icons 250 include direction buttons to be pressed when requesting movement of the moving body 10 in a horizontal direction (forward, backward, right rotation, left rotation). The operator can remotely control the moving body 10 by selecting the operation icons 250 while viewing a planar image displayed in the base display area 210 and a special image such as a celestial sphere image displayed in the base display area 230. Note that, although an example of remotely controlling the movement of the moving body 10 by accepting a selection for the operation icons 250 displayed on the operation screen 200 will be described, the movement operation of the moving body 10 may be configured to be performed by a dedicated controller such as a keyboard or a game pad equipped with a joystick.

[0106] Furthermore, the operation screen 200 includes a destination candidate registration button 270 that is pressed when registering a destination candidate using the planar image displayed in the base display area 210 or the special image displayed in the base display area 230. In the following explanation, an example will be explained in which the operator presses the destination candidate registration button 270 to register a destination candidate using the special image displayed in the base display area 230. Note that the same process is also performed when the operator presses the destination candidate registration button 270 to register a destination using the planar image displayed in the base display area 210. Furthermore, the operation screen 200 includes a destination setting button 290 that is pressed when setting a destination for the moving object 10.

[0107] Next, the display device 60 uses the operation screen 200 displayed in step S15 to manually operate the moving object 10 (step S16). The process of step S16 will be described in detail later.

[0108] Next, when the operator presses the destination candidate registration button 270 on the operation screen 200, the reception unit 52 receives the destination candidate registration for the area appearing in the special image displayed in the base display area 230 (step S17). Then, the transmission / reception unit 51 transmits a destination candidate registration request to the mobile object 10 (step S18). This destination registration request is, for example, a request to capture an image of the area appearing in the special image displayed in the base display area 230. As a result, the mobile object 10 receives the destination candidate registration request transmitted from the display device 60.

[0109] Next, the position information acquisition unit 35 of the moving body 10 acquires position information indicating the current position of the moving body 10 using the satellite positioning system 114 or the like (step S19). Specifically, the position information acquisition unit 35 acquires coordinate information including latitude and longitude indicating the current position of the moving body 10. If the moving body 10 is an air vehicle such as a drone, the position information acquisition unit 35 acquires information such as the speed, attitude, or altitude of the air vehicle as position information in addition to the coordinate information including latitude and longitude. In addition, the destination candidate acquisition unit 36 ​​acquires a photographed image taken by the special photography device 12a at the current position of the moving body 10 as a destination candidate image indicating a destination candidate (step S20). Here, the photographed image acquired as the destination candidate image by the destination candidate acquisition unit 36 ​​is a still image.

[0110] Then, the storage / reading unit 49 stores the destination candidate data including the location information acquired in step S19 and the photographed image acquired in step S20 in the destination candidate management DB 3001 (see FIG. 7) (step S21). The storage / reading unit 49 registers the acquired location information and photographed image as destination candidate data, each associated with a candidate ID that identifies the destination candidate.

[0111] In this way, the communication system 1 displays the images captured by the moving body 10 on the display device 60 of the operator remotely operating the moving body 10, thereby enabling the operator to remotely operate the moving body 10 while checking the situation around the moving body 10 in real time. Furthermore, the communication system 1 can register in advance captured images of areas within bases that are potential destinations for the moving body 10 when it is moved autonomously, in response to an input operation by the operator when manually operating the moving body 10.

[0112] <<Manual operation process>> Next, the manual operation process for the moving object 10 in step S16 will be described in detail with reference to Figures 12 and 13. Figure 12 is a sequence diagram showing an example of the manual operation process for the moving object.

[0113] First, the reception unit 52 of the display device 60 receives the selection of the operation icon 250 of the operation screen 200 displayed in step S15 by the input operation of the operator (step S31). The manual operation command generation unit 55 generates a manual operation command according to the content of the operation icon 250 selected in step S31 (step S32). In the example of FIG. 11, since the operator has selected the "forward (↑)" icon, in this case, the manual operation command generation unit 55 generates a command for moving the moving body 10 forward. Then, the transmission / reception unit 51 transmits the manual operation command generated in step S32 to the moving body 10 (step S33). As a result, the moving body 10 receives the manual operation command transmitted from the display device 60. Note that, in the input operation of the operation icon 250 by the operator, if the operator selects the "reverse (↓)" icon while the moving body 10 is moving forward, the captured image may be switched to a screen behind the moving body 10, and the moving body 10 may be made to reverse (go backward) from that point on. Furthermore, the transmission of the manual operation command from the display device 60 to the mobile object 10 may be configured to be performed via a managed cloud platform such as AWS IoT Core (trademark).

[0114] Next, the mode setting unit 42 sets the operation mode of the moving body 10 to the manual operation mode (step S34). Then, the manual operation processing unit 44 outputs a drive command based on the manual operation command received in step S33 to the movement control unit 40. The movement control unit 40 executes the movement process of the moving body 10 in response to the drive command by the manual operation processing unit 44 (step S35). In addition, the learning unit 48 performs imitation learning (machine learning) of the movement path in response to the manual operation by the manual operation processing unit 44 (step S36). The learning unit 48 performs imitation learning of the movement path based on, for example, the captured image acquired during the movement operation in the manual operation mode by the manual operation processing unit 44 and the detection data by the state detection unit 34. Note that the learning unit 48 may be configured to perform imitation learning of the movement path using only the captured image acquired during the manual operation, or may be configured to perform imitation learning of the movement path using both the captured image and the detection data by the state detection unit 34. Furthermore, the captured images used for imitation learning by the learning unit 48 may be captured images acquired during autonomous movement by the autonomous movement processing unit 43 in the autonomous movement mode.

[0115] Next, the mode setting unit 42 of the moving body 10 sets the operation mode of the moving body 10 to the manual operation mode (step S34). Then, the manual operation processing unit 44 outputs a drive command based on the manual operation command received in step S33 to the movement control unit 40. The movement control unit 40 executes the movement process of the moving body 10 in response to the drive command by the manual operation processing unit 44 (step S35). In addition, the learning unit 48 performs imitation learning (machine learning) of a movement path in response to the manual operation by the manual operation processing unit 44 (step S36). The learning unit 48 performs imitation learning of the movement path based on, for example, a captured image acquired during a movement operation in the manual operation mode by the manual operation processing unit 44 and detection data by the state detection unit 34. The movement path to be learned includes the position and posture of the moving body 10.

[0116] Then, the moving object 10 executes a process of registering the destination candidates during the moving operation (step S37). Fig. 13 is a flowchart showing an example of a process of registering the destination candidates according to the moving state of the moving object.

[0117] First, the determination unit 32 determines whether or not a predetermined task has been executed by the task execution unit 45 (step S51). Specifically, the task execution unit 45 executes a predetermined task by the moving body 10 in response to, for example, a task execution request from an operator or a preset schedule. Then, the determination unit 32 determines whether or not the predetermined task can be executed by the task execution unit 45. Here, the predetermined task includes, for example, a photographing process for inspecting equipment in a base, light work using a movable arm, and the like.

[0118] For example, when the mobile object 10 moves within a base and enters an area to be inspected, it checks the inspection targets such as meters, valves, etc. At this time, the mobile object 10 stops moving and performs a photographing operation to obtain an inspection image, and this operation can be used as a trigger to register a destination candidate.

[0119] If the determination unit 32 determines that the task executing unit 45 has executed the predetermined task (YES in step S51), the process proceeds to step S56. On the other hand, if the determination unit 32 determines that the task executing unit 45 has not executed the predetermined task (NO in step S51), the process proceeds to step S52.

[0120] Next, the determination unit 32 determines whether the moving operation of the moving body 10 has stopped (step S52). Specifically, when the determination unit 32 detects that the drive control of the moving mechanism 15 by the movement control unit 40 has stopped, the determination unit 32 determines that the moving operation of the moving body 10 has stopped. When the determination unit 32 determines that the moving operation of the moving body 10 has stopped (YES in step S52), the process proceeds to step S56. On the other hand, when the determination unit 32 determines that the moving operation of the moving body 10 has not stopped (moving operation is in progress) (NO in step S52), the process proceeds to step S53.

[0121] Next, the determination unit 32 determines whether an intersection has been detected around the moving object 10 (step S53). Specifically, the image recognition processing unit 41 performs image recognition processing on the captured image acquired by the special imaging device 12a or the general imaging device 12b. Then, the determination unit 32 determines whether an intersection has been detected in the captured image by the processing by the image recognition processing unit 41. For example, the moving object 10 can acquire images seen from different directions on the moving path at once regardless of the moving direction by acquiring special images captured in all directions at a characteristic point such as an intersection. Here, an intersection is an example of a specific subject detected by the image recognition processing unit 41. The specific subject may be an intersection, an obstacle on or around the moving path of the moving object 10, or a sign or signal installed at a base. In addition, the specific subject differs depending on the type of moving object 10, such as when the moving object 10 moves on a road surface or when the moving object 10 such as a drone flies, and information on the specific subject to be detected is set in advance in the moving object 10.

[0122] If the judgment unit 32 judges that an intersection has been detected around the moving object 10 (YES in step S53), the process proceeds to step S56. On the other hand, if the judgment unit 32 judges that an intersection has not been detected around the moving object 10 (NO in step S52), the process proceeds to step S54.

[0123] Next, the determination unit 32 determines whether the current position of the mobile object 10 is close to a destination candidate registered in the destination candidate management DB 3001 (step S54). Specifically, the determination unit 32 refers to the position information of the destination candidate stored in the destination candidate management DB 3001, and determines whether there is a destination series close to the current position of the mobile object 10 acquired by the position information acquisition unit 35. Here, the determination of whether the destination candidate is close may be made based on, for example, the difference between the position of the destination candidate set in advance and the current position of the mobile object 10. Note that the determination of whether the destination candidate is close may be made using image recognition between a captured image corresponding to the destination candidate (destination candidate image) and an image captured at the current position of the mobile object 10.

[0124] If the determination unit 32 determines that the current location of the moving object 10 is close to a destination candidate registered in the destination candidate management DB 3001 (YES in step S54), the process proceeds to step S56. On the other hand, if the determination unit 32 determines that the current location of the moving object 10 is far from the destination candidate registered in the destination candidate management DB 3001 (NO in step S54), the process proceeds to step S55.

[0125] Next, the determination unit 32 determines whether or not the direction indicated in the manual operation command transmitted from the display device 60 has changed (step S55). Specifically, the determination unit 32 determines whether or not the moving direction indicated in the manual operation command has changed based on the manual operation command received in step S33. In addition, the determination unit 32 may determine that the moving direction has changed, for example, when the manual operation command indicates a characteristic operation. The characteristic operation includes, for example, a certain amount of steering operation or an accelerator or brake operation performed by moving through an intersection, a curve, or the like.

[0126] If the determination unit 32 determines that the operation direction indicated in the manual operation command transmitted from the display device 60 has been changed (YES in step S55), the process proceeds to step S56. On the other hand, if the determination unit 32 determines that the operation direction indicated in the manual operation command transmitted from the display device 60 has not been changed (NO in step S55), the process ends.

[0127] Then, in step S56, the location information acquisition unit 35 acquires location information indicating the current location of the moving body 10 using the satellite positioning system 114 or the like. Also, the destination candidate acquisition unit 36 ​​acquires images captured by the special photography device 12a at the current location of the moving body 10 as destination candidate images indicating destination candidates of the moving body 10 (step S57).

[0128] Then, the storage / reading unit 49 stores the destination candidate data including the location information acquired in step S19 and the photographed image acquired in step S20 in the destination candidate management DB 3001 (see FIG. 7) (step S58). The storage / reading unit 49 registers the acquired location information and photographed image as destination candidate data, each associated with a candidate ID that identifies the destination candidate.

[0129] In this way, the moving body 10 can automatically capture images of the surroundings of the moving body 10 using the imaging device 12 mounted on the moving body 10 based on a predetermined judgment criterion according to the moving state of the moving body 10, and can register captured images showing potential destinations of the moving body 10 in advance. Note that the judgment criterion by the judgment unit 32 shown in steps S51 to S55 is not limited to this, and is set appropriately according to the content of the manual operation of the moving body 10 and the environment around the moving body 10. The judgment criterion by the judgment unit 32 includes, for example, a change in the state of the moving body 10 that the operator wants to recognize when selecting a destination of the moving body 10, or a change in the situation of a base or an environment within the base.

[0130] <<Destination setting process>> Next, a process of setting a destination of the moving object 10 using the destination candidates registered by the above process will be described with reference to Fig. 14 to Fig. 17. Fig. 14 is a sequence diagram showing an example of a process of setting a destination of the moving object.

[0131] First, the accepting unit 52 of the display device 60 accepts the selection of the destination setting button 290 by an input operation on the operation screen 200 by the operator (step S71).

[0132] Next, the transmitting / receiving unit 51 transmits a destination candidate acquisition request to the mobile body 10, which indicates a request to acquire data indicating destination candidates (step S72). This destination candidate acquisition request includes a base ID that identifies the base where the mobile body 10 is installed. As a result, the transmitting / receiving unit 31 of the mobile body 10 transmits the destination candidate acquisition request transmitted from the display device 60.

[0133] Next, the storage / reading unit 49 searches the destination candidate management DB 3001 (see FIG. 7) using the base ID indicated in the destination acquisition request received in step S72 as a search key, and reads out destination candidate data associated with the same base ID as the received base ID (step S73).

[0134] Then, the transmitting / receiving unit 31 transmits the destination candidate data read in step S73 to the requesting display device 60 (step S74). This destination candidate data includes a set of the multiple candidate IDs, the captured images, and the location information read in step S73. Thus, the transmitting / receiving unit 51 of the display device 60 receives and acquires the destination candidate data transmitted from the moving object 10.

[0135] Next, the display control unit 53 of the display device 60 causes a display unit such as the display 506 to display the selection screen 400 including the destination candidate data received in step S74 (step S75). Specifically, the image processing unit 57 generates the selection screen 400 based on the received destination candidate data. Then, the display control unit 53 causes the selection screen 400 generated by the image processing unit 57 to be displayed. Note that the generation process of the selection screen 400 may be performed by the image processing unit 47 of the moving object 10. In this case, the image processing unit 47 generates the selection screen 400 based on the destination candidate data read out in step S73. Then, in step S74, the transmission / reception unit 31 transmits screen data of the selection screen 400 including the destination candidate data generated by the image processing unit 47 to the display device 60.

[0136] Fig. 15 and Fig. 16 are diagrams showing an example of a selection screen. A selection screen 400 shown in Fig. 15 and Fig. 16 is a display screen for allowing the operator to select a destination to which the moving object 10 will move.

[0137] The selection screen 400 includes an image display area 410 in which a plurality of captured images (destination candidate images) 415 (415a to 417f) included in the destination candidate data received in step S74 are displayed in a selectable manner. The selection screen 400 also includes an OK button 430 to be pressed when the selection is completed, and a cancel button 435 to be pressed to abort the selection process. The operator uses input means such as a pointing device 512 to select a captured image 415 displayed in the image display area 410.

[0138] The reception unit 52 of the display device 60 receives the selection of a destination candidate image for setting an area indicated by a photographed image 415 displayed in the image display area 410 as a destination by the operator selecting the photographed image 415 (step S76). The selection screen 400 shown in FIG. 16 shows a state in which, for example, photographed images 415a, 415b, and 415c are selected by the operator. When the photographed image 415 displayed in the image display area 410 is selected, the selected photographed image is displayed in the selection result display area 420 on the selection screen 400 as shown in FIG. 16. The selection result display area 420 shows the selected photographed images in the order selected by the operator (No. 1 to No. 3). The operator selects, for example, a photographed image showing an area indicating a position where the moving object 10 is to execute a predetermined task, or a photographed image showing an area indicating a branch point on a route such as an intersection in a base when the moving object 10 is to be moved to a position where the moving object 10 is to execute a predetermined task, as the destination of the moving object 10.

[0139] In this way, the display device 60 uses a captured image 415 (candidate destination image) that shows candidate destinations for the moving body 10, captured in advance by the moving body 10, to select the area shown in the captured image 415 as the destination for the moving body 10, thereby improving operability in selecting the destination for the moving body 10 compared to conventional methods in which the operator is required to specify a character string such as location information or a keyword.

[0140] Next, when the selection of the destination candidate image is accepted in step S76 and the operator presses the OK button 430, the autonomous movement request information generating unit 56 of the display device 60 generates autonomous movement request information (step S77). This autonomous movement request information includes the candidate ID associated with the photographed image 415 whose selection was accepted in step S76, and information indicating the order in which the selection was accepted in step S76.

[0141] Next, the transmitting / receiving unit 51 transmits the autonomous movement request information generated in step S77 to the moving body 10 (step S78). As a result, the transmitting / receiving unit 31 of the moving body 10 receives the autonomous movement request information transmitted from the display device 60.

[0142] Then, the moving object 10 starts the autonomous movement process based on the autonomous movement request information received in step S78 (step S79).

[0143] <<Autonomous movement processing>> Next, the autonomous movement processing of the moving body 10 in step S79 will be described in detail with reference to Fig. 17. Fig. 17 is a flowchart showing an example of the autonomous movement processing of the moving body.

[0144] First, when the autonomous movement request information is received in step S78, the mode setting unit 42 of the moving object 10 sets the operation mode of the moving object 10 to the autonomous movement mode (step S91).

[0145] Next, the route information generating unit 37 generates route information indicating an autonomous movement route of the moving body 10 based on the autonomous movement request information received in step S78 (step S92). Specifically, the route information generating unit 37 generates a movement route for autonomously moving the moving body 10 to an area indicated by the photographed image 415 in accordance with the order in which the operator selected the photographed image 415, based on the candidate ID indicated in the received autonomous movement request information and the order in which the photographed image 415 specified by the candidate ID was selected. Note that the route information generating unit 37 may be configured to generate route information based on the information on the photographed position of the photographed image 415 stored in the destination candidate management DB 3001, not limited to the order in which the photographed image 415 was selected, so that, for example, the distance or movement time of the movement route based on the selected photographed image 415 is shortened.

[0146] Then, the route information management unit 38 stores the route information generated in step S92 in the route information management DB 3002 (see FIG. 8) in association with the base ID received in step S72 (step S93).

[0147] Next, the location information acquisition unit 35 acquires location information indicating the current location of the moving object 10 (step S94).

[0148] The destination setting unit 39 sets the destination of the moving body 10 based on the current position of the moving body 10 acquired by the position information acquisition unit 35 and the route information generated in step S92 (step S95). Specifically, the destination setting unit 39 sets, for example, the position indicated by the next destination among multiple destinations specified by the candidate IDs indicated in the generated route information as the destination. For example, when autonomous movement is about to start, the destination setting unit 39 sets, as the destination, the position indicated by the destination specified by the first candidate ID indicated in the route information. Then, the destination setting unit 39 generates a movement route from the acquired current position of the moving body 10 to the set destination. The method of generating the movement route by the destination setting unit 39 is a method of connecting the current position and the destination with a straight line, or a method of minimizing the movement time while avoiding obstacles by using the information of the obstacles obtained by the photographed image or the state detection unit 34.

[0149] In addition, the generation of the travel route by the destination setting unit 39 is not limited to a configuration using the location information of the destination candidates (destinations) registered in the destination candidate management DB 3001, but may also be a configuration in which the location shown in the captured image 415 set as the destination is identified by image recognition processing by the image processing unit 47 using the captured image captured by the photographing device 12.

[0150] Then, the movement control unit 40 moves the moving body 10 to the movement destination set to pass through the movement route generated in step S92. In this case, the movement control unit 40 moves the moving body 10 autonomously in response to a drive command from the autonomous movement processing unit 43 (step S96). The autonomous movement processing by the autonomous movement processing unit 43 will be described in detail later.

[0151] Then, when the moving object 10 reaches the final destination (YES in step S97), the movement control unit 40 ends the process. On the other hand, the movement control unit 40 continues the autonomous movement process by repeating the process from step S94 until the moving object 10 reaches the final destination (NO in step S97).

[0152] In this way, the moving body 10 can perform autonomous movement with a position shown in the captured image 415 selected by the operator as the destination of the moving body 10. Furthermore, when operating in the autonomous movement mode, the moving body 10 can perform autonomous movement using the generated route information and learning data learned in the manual operation mode.

[0153] <Integration of deviations from travel path using local maps> Next, a process of calculating deviations from the travel route by comparing multiple local maps associated with waypoint 8 with the sensor values ​​of the external sensors during teaching travel and integrating the deviations for the multiple external sensors will be described. First, a topological map will be described.

[0154] FIG. 18 is a conceptual diagram of a topological map. In this embodiment, as described above, the moving body 10 has N external sensors (GNSS, distance measurement sensor (LiDAR), camera, etc.) and internal sensors (IMU, encoder, etc.), and can calculate the relative movement amount (odometry) of the moving body 10. The moving body 10 is an autonomous mobile system of a teaching iterative type that uses a map in the form of a topological map. A topological map is a map in the form of a graph consisting of a node set V and an edge set E. An element e of the edge set E is ij is two nodes v i and v j is defined between v i and v j The relative positional relationship (an example of relative positional information) is as follows.

[0155] The information creation unit 46 creates a node v that is an element of the node set V. i , node v iIn the present embodiment, the local map (local map) generated using the sensor values ​​of the external and internal sensors is stored. In addition, in this embodiment, "information that enables calculation of the deviation (position and attitude) between the points passed during the teaching run and the points passed during the autonomous movement" is expressed as "map". For example, when a two-dimensional lattice map (an H×W image in which 1 is stored in the lattice where an obstacle exists and 0 is stored in the lattice where no obstacle exists) created by the LiDAR during the teaching run and a LiDAR scan point cloud during the autonomous movement are given, the position and attitude of the origin (= the point passed during the teaching run = node) of the two-dimensional lattice map as seen from the moving body 10 during the autonomous movement can be calculated by optimization calculation or the like. Therefore, when an external sensor called LiDAR is used, the two-dimensional lattice map can be called a "map". On the other hand, when the position (latitude, longitude) and azimuth angle observed by GNSS during taught driving and the position (latitude, longitude) and azimuth angle observed by GNSS during autonomous movement are given, it is possible to calculate the position and attitude of the points (nodes) passed during taught driving as seen from the mobile body 10, so in this embodiment, the "position (latitude, longitude) and azimuth angle observed by GNSS during taught driving" is also referred to as a "map."

[0156] The autonomous movement of the moving body 10 consists of two phases: a "route teaching phase" and an "autonomous movement phase." The "route teaching phase" corresponds to teaching driving, in which an operator manually operates the moving body 10 with a controller and makes the moving body 10 move along the desired moving path along which the moving body 10 should move autonomously. The "autonomous movement phase" corresponds to autonomous movement, in which the moving body 10 moves autonomously so as to trace (move on or along) the moving path moved in the "route teaching phase."

[0157] Fig. 19 is a block diagram showing the movement control of this embodiment using an external sensor and a local map. Fig. 19 shows the movement control in the autonomous movement phase. (1) First, the storage device stores a first local map and a second local map that are generated based on the sensor values ​​of the first external sensor and the second external sensor, respectively, in the route teaching phase. (2) In the autonomous movement phase, the first and second external sensors detect sensor values. The internal sensor also detects sensor values, and the position and orientation are calculated by integrating the sensor values ​​of the first or second external sensor. (3) The moving body 10 compares the first local map with the sensor value of the first external sensor to calculate the deviation from the moving path using the first external sensor (deviation from the node). The moving body 10 also compares the second local map with the sensor value of the second external sensor to calculate the deviation from the moving path using the second external sensor (deviation from the node). As will be described later, the deviation from the node represents the position and attitude of the waypoint 8 as seen from the moving body 10. The moving body 10 calculates the reliability of the first external sensor based on the first local map and the first external sensor, and calculates the reliability of the second external sensor based on the second local map and the second external sensor. (4) The moving body 10 weights the position and orientation detected by the moving body 10 by the reliability and integrates the position and orientation using the first external sensor and the position and orientation using the second external sensor. Since the integrated position and orientation are deviations from the moving path, integration of the position and orientation is equivalent to integration of deviations from the moving path. (5) The moving body 10 uses the integrated position and attitude obtained from multiple external sensors for control.

[0158] <Route teaching phase processing> The route teaching phase will be described in detail with reference to Figures 20 and 21. Figure 20 is a flow chart for explaining the process of the route teaching phase. This process is executed in step S16 of Figure 10.

[0159] First, the information creation unit 46 initializes the topological map G by setting both V and E to an empty set (S101). The information creation unit 46 creates a local map generated using the sensor value of the k-th external sensor or the external sensor as m k At the start of the route teaching phase, the information creation unit 46 adds the first node v0 to V. v0 contains the m k (=:m i k) are all preserved.

[0160] A person manually operates the moving body 10 (S102). The movement control unit 40 executes a movement process for the moving body 10 in response to a drive command from the manual operation processing unit 44 (S103).

[0161] The determination unit 32 periodically determines whether the moving object 10 has moved a certain distance or rotated a certain angle or more (S104). This certain distance is appropriately set depending on the processing load, memory usage, accuracy of the position and route, etc., and may be, for example, one meter to several meters.

[0162] If the answer is Yes in step S104, the information creation unit 46 adds a new node v1 to V and a new edge e 0,1 Add (S105). 0,1 The relative movement amount and the relative rotation amount from the time v0 was added, which were acquired by the internal sensor, are stored in v1. The position of this node vi is called waypoint 8 in Figs. 1 and 2. The information creation unit 46 increments i, which is the node number, by 1 (S106). For example, until the route teaching phase is ended by the operator (No in S107), the information generating unit 46 similarly generates the node v every time the moving body 10 moves a certain distance or turns a certain angle. i and Edge e i-1,i Add more.

[0163] The node set V and edge set E are G={V,E} V = {v0,…,v n} E = {e 0,1 ,…,e n-1,n} v i ={m i 0 ,…,m i N} e i-1,i ={Δx i-1,i , Δy i-1,i , Δθ i-1,i} It can be expressed as follows.

[0164] <Autonomous movement phase processing> 21 is a flow chart for explaining the process of the autonomous movement phase, which is executed in step S79 of FIG.

[0165] Before the autonomous movement starts, the moving body 10 is placed at the movement start point. The sensor value of the k-th external sensor in the autonomous movement phase is denoted by s k The specific calculation methods in steps S201 to S203 and S206 will be described later.

[0166] · S201 The deviation calculation unit 43b calculates the sensor value s of the kth external sensor during the autonomous movement of the moving body 10. k and a certain node v during the teaching run of the mobile unit 10 i The k-th local map m i k From the viewpoint of the mobile unit 10, the node v i Calculate the position and orientation deviation of node v from the viewpoint of the moving object 10. i The position and orientation of the node v i Since the position and orientation of the moving body 10 are the same, the node v i The position and orientation of node v i The position estimation unit 43c estimates the calculated deviation as the deviation between the position and posture of the node v i Position and orientation of x^ i k ={x^ i k ,y^ i k ,θ^ i k}. Note that "^" is a hat, but due to font restrictions it is shown to the upper right of the x, not directly above it. The arithmetic unit that performs this operation is F k "Calculation device F k :(s k ,m i k ) →x^ ik On the other hand, the local map m i k and the sensor value s from the kth external sensor k Since the accuracy of is unknown, we use the following formula to determine the degree of confidence: i k will be integrated.

[0167] · S202 The integrated processing unit 43d is a calculation device F k Estimated confidence in b i k The calculation device that performs this calculation is called B k "Calculation device B k :(s k ,m i k ) → b i k " is expressed as

[0168] · S203 The integration processing unit 43d integrates the node v estimated by each external sensor. i Position and orientation of x i k The weights are assigned based on the reliability of the data and integrated. The calculation unit that performs this calculation is called "Calculation unit I: (x^ i 0 ,…,x^ i N ,b i 0 ,…,b i N ) →x^ i Node v i Position and orientation of x i k is the local map m i k Since the deviation of the position and posture of the moving object 10 from the origin (node) of i Position and orientation of x i k The integration of includes the integration of deviations.

[0169] · S204 The autonomous movement processing unit 43 is a node v i Position and orientation of x iand the relative position between nodes (connections) stored on each edge {Δx j-1,i ,Δy j-1,j ,Δθ j-1,j}, the taught moving path in the moving object coordinate system (i.e., each node v j The calculation device that performs this calculation is called "Calculation device T:x^ i ,e 0,1 ,…,e n-1,n ) → (x^0,…,x^ n The calculated path is expressed as " i The position and orientation of node v i This makes the moving path pass through the nodes v and v . i Each time it passes through node v i The deviation at node v is suppressed. i The subsequent movement path is recalculated, so that the moving body 10 can move along a movement path with a small deviation from the taught movement path.

[0170] · S205 The calculation unit 43a calculates the integrated node v i The position and orientation of the internal sensor are integrated with the position and orientation of the external sensor. k The autonomous movement processing unit 43 calculates the position and orientation by applying the position and orientation based on the current position and orientation of the moving body 10 to a Kalman filter or the like. Based on the current position and orientation of the moving body 10, the autonomous movement processing unit 43 calculates the position and orientation of the moving body 10 based on the taught moving path (x^0, ..., x^ n ) is moved.

[0171] · S206 The autonomous movement processing unit 43 determines the movement path (x^0, ..., x^ n ) at node v i Position and orientation of x^ i Depending on whether the position and orientation of the moving object 10 approaches within a threshold value, the node number i of the position and orientation estimation target is judged to be advanced to the next. i ,(x^0,…,x^ n)) is represented by "0 (do not advance i) or 1 (advance i)".

[0172] ·S207 At the start of autonomous movement, the movement control unit 40 sets i = 0 and increments i by 1.

[0173] ·S208 The movement control unit 40 repeats the process of FIG. 21 until i becomes greater than the number of nodes n. The number of nodes n is the number of nodes stored until the final destination.

[0174] By steps S201 to S204 in FIG. 21, the instructed movement path in the mobile coordinate system is calculated. The mobile body 10 moves along the instructed movement path (S205). Then, while repeatedly calculating in steps S201 to S205, the mobile body 10 sets i←i + 1 according to the result of step S206. By repeating this, the mobile body 10 can move along the instructed movement path.

[0175] <<Example of Processing When Using GNSS and 2D-LiDAR>> Detailed examples of steps S201 to S203 and S206 in the autonomous movement phase will be described.

[0176] m i 0 is the position and orientation estimated from the GNSS positioning history and odometry by the internal sensor, and the positioning state (good / bad). Also, m i 1 is the 2D occupancy grid map in the mobile coordinate system generated from the history of the scan point cloud of the 2D-LiDAR and odometry by the internal sensor. On the other hand, s 0 is the position and orientation and positioning state estimated from the GNSS positioning history and odometry by the internal sensor, and s 1 is the scan point cloud of the 2D-LiDAR.

[0177] S201 F0 is m i 0 and s 0By comparing the position and attitude of node v from the mobile unit 10 based on the GNSS positioning results, i Position and orientation of x^ i 0 Calculate m i 0 is node v i GNSS coordinate system position and attitude (x i,GNSS , y i,GNSS ,θ i,GNSS ), s 0 are the position and attitude (x GNSS , y GNSS ,θ GNSS ) from the viewpoint of the mobile unit 10. i The process of calculating the position and orientation of is calculated using equations (1) and (2). The calculation using equations (1) and (2) corresponds to F0.

[0178]

number

[0179]

number

[0180] Next, F1 is m i 1 The two-dimensional occupancy map of s 1 By matching the scan point cloud of the 2D-LiDAR of the i Position and orientation of x^ i 1 (An example of the second position and orientation) m i 1 is node vi A two-dimensional occupancy grid map with s as the origin 1 is the LiDAR scan point cloud {(x j ,y j ) j=1, ..., N} (N is the number of scan points). At this time, the autonomous movement processing unit 43 finds a way of moving the two-dimensional occupancy grid map in the xy plane such that the sum of all the grid values ​​(the grid where an obstacle is present has a 1) corresponding to the positions of each point of the scan point cloud is the largest (i.e., the scan point cloud matches the map). As a result, the m i 1 (i.e., node v in the 2D occupancy grid map) i This series of processes corresponds to F1.

[0181] Fig. 22 is a diagram showing a schematic diagram of a matching calculation between a two-dimensional occupancy grid map and a 2D-LiDAR scan point cloud. Fig. 22(a) shows a two-dimensional occupancy grid map 93 and a 2D-LiDAR scan point cloud 94 before matching. In the two-dimensional occupancy grid map 93, grids in which obstacles exist are indicated by diagonal lines. In addition, the 2D-LiDAR scan point cloud 94 shows the direction and distance of the obstacles.

[0182] FIG. 22(b) shows a state in which the two-dimensional occupancy grid map 93 has been moved so that the grid in which the obstacle exists matches the 2D-LiDAR scan point cloud 94 most closely. The amount of movement in the x and y directions and the amount of rotation of the two-dimensional occupancy grid map 93 when the two-dimensional occupancy grid map 93 is moved are deviations. In addition, in the state of FIG. 22(b), the moving body 10 is at the origin 70 (=node v i ), so the amount of movement and rotation in the x and y directions of the two-dimensional occupancy grid map 93 corresponds to the node v i Position and orientation of x^ i 1 It is.

[0183] S202 b0 is m i 0 and s 0b1 is set to 1 only when both GNSS positioning states are "good", otherwise it is set to 0. i 1 The number of obstacles in the 2D occupancy grid map of s is sufficiently large, and 1 If the number of scan points in the scan point cloud 94 owned by m is sufficiently large, it is set to 1, otherwise it is set to 0. Also, b0 and b1 are not limited to 1 or 0, and may take values ​​according to the accuracy of the local map and the sensor value. For example, i 0 and s 0 The GNSS positioning state of each is evaluated in N steps and the sum is designated as b0, and the amount of obstacles in the two-dimensional occupancy grid map and the number of scan points in the scan point cloud 94 are evaluated in N steps and the sum is designated as b1.

[0184] S203 b i 0 or b i 1 If either is nonzero, I(x^ i 0 ,x^ i 1 ,b i 0 ,b i 1 ) = (b i 0 x^ i 0 + b i 1 x^ i 1 ) / (b i 0 + b i 1 ) Let us assume that.

[0185] b i 0 = b i 1 If = 0, the previous node v i The estimated position and orientation are calculated based on the estimated position and orientation of the robot and the odometry using an internal sensor (dead reckoning).

[0186] S206 In step S203, for the integrated x^ i ={x^ i , y^ i , θ^ i}, if the difference from √(x^ i 2 + y^ i 2 ) is greater than a certain threshold, it is set to 0; if it is smaller, it is set to 1. x^ i , y^ i are the current positions of the moving body 10 calculated by the calculation unit 43a.

[0187] <<Processing Example When Using GNSS and 3D-LiDAR>> Subsequently, for the flowchart in FIG. 21, a case of processing the scan point cloud by "3D-LiDAR" instead of the scan point cloud by "2D-LiDAR" will be described. Different from "2D-LiDAR", detailed examples of steps S201 to S203 and S206 in the autonomous movement phase will be described.

[0188] m i 0 is the position, orientation, and positioning state (good / bad) estimated from the GNSS positioning history and the odometry by the internal sensor. Also, m i 1 is a three-dimensional point cloud map in the moving body coordinate system generated from the scan point cloud history of the 3D-LiDAR and the odometry by the internal sensor. On the other hand, s 0 is the position, orientation, and positioning state estimated from the GNSS positioning history and the odometry by the internal sensor, and s 1 is the scan point cloud of the 3D-LiDAR.

[0189] S201 F0 calculates the position and orientation x^ i 0 of the node v seen from the moving body 10 by comparing the positions and orientations based on the GNSS positioning results of m 0 and s i . m i 0 is the node v i 0 is the node vi GNSS coordinate system position and attitude (x i,GNSS , y i,GNSS ,θ i,GNSS ), s 0 are the position and attitude (x GNSS , y GNSS ,θ GNSS ) from the viewpoint of the mobile unit 10. i The process of calculating the position and orientation of is performed using the above formulas (1) and (2). The calculation using formulas (1) and (2) corresponds to F0.

[0190] As is clear from equations (1) and (2), the node v seen from the mobile unit 10 i The position and attitude of node v in the GNSS coordinate system are i Position and orientation of (x i,GNSS , y i,GNSS ,θ i,GNSS ) is a deviation of the position and attitude of the moving object 10 in the GNSS coordinate system with respect to the reference point (an example of a first position and attitude).

[0191] Next, F1 is m i 1 3D point cloud map and s 1 By matching the scan point cloud of the 3D-LiDAR of the i Position and orientation of x^ i 1 (An example of the second position and orientation) m i 1 is node v i A 3D point cloud map with s as the origin. 1 is the 3D-LiDAR scan point cloud {(x j ,y j ) j=1, ..., N} (N is the number of scan points). At this time, the autonomous movement processing unit 43 finds a movement method that minimizes the error with the scan point cloud when the 3D point cloud map is moved by NDT matching (see Non-Patent Document 1). As a result, m i 1 (i.e., node v in the 2D occupancy grid map) iThis series of processes corresponds to F1.

[0192] S202 b0 is m i 0 and s 0 b1 is set to 1 only when both GNSS positioning states are "good", otherwise it is set to 0. i 1 When the amount of obstacles within the field of view of the 3D-LiDAR on the 3D point cloud map held by the 3D-LiDAR is sufficiently large, the value is set to 1, and when not, the value is set to 0. However, when calculating the amount of obstacles, the autonomous movement processing unit 43 performs the following process.

[0193] Fig. 23 is a flow chart for explaining the flow of evaluation of b0 based on the amount of obstacles. As shown in Fig. 23, when autonomous movement starts, the autonomous movement processing unit 43 performs preprocessing of removing ground point clouds (step S2) when reading the point cloud map (step S1) and converting it into a two-dimensional occupancy grid map (step S3). The ground removal is performed, for example, as follows. The inclination of the moving body 10 is detected using an internal sensor, and a ground plane is defined. The distance from this ground plane to each point in the point cloud map is calculated, and points within a threshold value are determined to be ground, and the points are removed.

[0194] The autonomous movement processing unit 43 calculates the relative position and orientation in the two-dimensional occupancy grid map as seen from the moving body 10 during autonomous movement (step S4). The autonomous movement processing unit 43 calculates the amount of obstacles in the 3D-LiDAR field of view based on the number of occupancy grids (step S5). The autonomous movement processing unit 43 normalizes the amount of obstacles by the perimeter of the two-dimensional occupancy grid map (step S6). The autonomous movement processing unit 43 evaluates b0 by the normalized amount of obstacles (step S7). b0 is set to 1 when the normalized amount of obstacles is greater than a threshold, and 0 otherwise. The amount of obstacles is normalized to eliminate dependency on the range (size) of the point cloud map. Also, b0 and b1 are not limited to 1 or 0, and may take values ​​according to the accuracy of the local map and the sensor value. For example, m i 0 and s 0The GNSS positioning state of each of the sensors is evaluated in N steps, and the sum is designated as b0. The amount of obstacles on the 3D point cloud map and the number of scan points of the scan point cloud 94 are evaluated in N steps, and the sum is designated as b1.

[0195] S203 b i 0 or b i 1 If either is nonzero, I(x^ i 0 ,x^ i 1 ,b i 0 ,b i 1 ) = (b i 0 x^ i 0 + b i 1 x^ i 1 ) / (b i 0 + b i 1 ) Let us assume that.

[0196] b i 0 = b i 1 If = 0, the previous node v i The estimated position and orientation are calculated based on the estimated position and orientation of the robot and the odometry using an internal sensor (dead reckoning).

[0197] S206 x^ integrated in step S203 i ={x^ i ,y^ i ,θ^ i}, √(x^ i 2 +y^ i 2 If the difference between x^ and the threshold is greater than a threshold, it is set to 0. If it is smaller, it is set to 1. i , y^ i is the current position of the moving object 10 calculated by the calculation unit 43a.

[0198] <Major Effects> The mobile body 10 of this embodiment has an independent local map for each sensor in the form of a topological map, calculates the reliability of position estimation from the sensor values ​​during teaching driving and autonomous movement for the self-position and attitude estimated independently for each, and integrates them into the final position and attitude by weighting according to the reliability. Therefore, there is no need to manually correct the map so that the position estimation results by each sensor are consistent, and high-precision autonomous movement is possible even if the positioning accuracy of any sensor is reduced during the route teaching phase.

[0199] In addition, there are two advantages to calculating reliability when using 3D-LiDAR: The reliability of map matching (= the amount of obstacles around the moving object 10) is calculated using the map point cloud instead of the real-time point cloud, so preprocessing for reliability calculation can be performed in advance. In other words, the computational load in real time is small. When calculating the reliability by looking at the real-time point cloud, the reliability may increase due to obstacles (vehicles, people, etc.) passing temporarily around the moving object 10, and an erroneous position estimation result may be adopted. However, when using the map point cloud, there is no such concern.

[0200] [Second embodiment] In this embodiment, an embodiment in which the display device 60 is an HMD (head mounted display) will be described. The HMD is a type of display device that is shaped like goggles and is worn on the head so that two displays are placed over both eyes. The HMD realizes stereoscopic vision by displaying parallax images to the left and right eyes. Stereoscopic vision is not necessarily possible. In addition to a type that simply displays images stereoscopically, there are types of HMDs called VR goggles (Virtual Reality), AR goggles (Augmented Reality), and MR goggles (Mixed Reality). The VR goggles display an artificial virtual world. The AR goggles read the real world three-dimensionally and display virtual information superimposed on the real world. The MR goggles are an extension of the AR goggles, and display something that is not actually there three-dimensionally superimposed on the real world. In this embodiment, since the operator views image data captured by the moving body 10 with the HMD, it is considered that a simple HMD or AR goggles will be used.

[0201] Fig. 24 is an example of a hardware block diagram of the display device 60 when the display device 60 is an HMD. As shown in Fig. 24, the display device 60, which is an HMD, has a configuration as a computer. The display device 60 includes a CPU 201, a ROM 202, a RAM 203, an external device connection I / F 205, a display 207, an operation unit 208, a media I / F 209, a bus line 211, a speaker 212, an electronic compass 218, a gyro sensor 219, and an acceleration sensor 220.

[0202] Of these, the CPU 201 controls the overall operation of the HMD. The ROM 202 stores programs such as the IPL used to drive the CPU 201. The RAM 203 is used as a work area for the CPU 201.

[0203] The external device connection I / F 205 is an interface for connecting various external devices. In this case, the external devices are the communication management server 3, the earphones with microphone 6, etc.

[0204] The display 207 is a type of display unit, such as a liquid crystal display or an organic EL (Electro Luminescence) display, that displays various images.

[0205] The operation unit 208 is an input means for selecting and executing various instructions such as various operation buttons, a power switch, physical buttons, and an eye-gaze operation circuit that detects and operates the operator's line of sight, selecting the object to be processed, moving the cursor, etc.

[0206] The media I / F 209 controls reading and writing (storing) of data from and to a recording medium 209m such as a flash memory. The recording medium 209m includes DVDs and Blu-ray Discs (registered trademarks).

[0207] The speaker 212 is a circuit that converts an electric signal into a physical vibration to generate sound such as music or voice. The electronic compass 218 calculates the direction of the HMD from the earth's magnetism and outputs the direction information. The gyro sensor 219 is a sensor that detects changes in angle (roll angle, pitch angle, yaw angle) accompanying the movement of the HMD.

[0208] The acceleration sensor 220 is a sensor that detects acceleration in three axial directions. The bus line 211 is an address bus, a data bus, or the like for electrically connecting the components such as the CPU 201.

[0209] The display device 60 of this embodiment can detect the attitude of the display device 60 by a signal from the gyro sensor 219, and can therefore identify the direction in which the operator is looking. The camera 111 can capture a celestial sphere image, a wide-angle image, a hemispherical image, or the like, and therefore the display device 60 can display the direction in which the operator is looking on the display 207 of the HMD. This makes it easier to view various directions than operating the area displayed in the base display area 210 with a pointing device 512 such as a mouse.

[0210] Furthermore, the position of the moving object 10 when the operator is looking in any direction on the HMD is identified by the state detection unit 34. Therefore, the display device 60 can display notes regarding buildings and the like that are in the direction in which the operator is looking from the position of the moving object 10 by superimposing them on the image data transmitted from the moving object 10. In this case, a system administrator or the like sets notes regarding buildings using an environmental map created by the moving object 10.

[0211] FIG. 25 shows an example of image data displayed by the display device 60, which is an HMD. The image data in FIG. 25 shows the current position 601 of the moving body 10 and the name 602 of the building shown in the image data, which is "experimental building". The moving body 10 receives the direction in which the operator is looking from the display device 60, and identifies the building in the direction in which the operator is looking from the position of the moving body 10 on the environmental map. The photographing control unit 33 superimposes the current position and the name of the building on the photographed image data, and the transmitting and receiving unit 31 transmits the image data to the display device 60. In addition to the name 602 of the building, details of accidents that have occurred in the past, events that should be observed with caution, etc. may be displayed as notes. Therefore, the operator can easily understand which direction he or she is looking at and what he or she is looking at.

[0212] <Major Effects> In addition to the effects of the first or second embodiment, the moving body 10 of this embodiment makes it easier for the operator to view image data captured by the moving body 10 by configuring the display device 60 as an HMD.

[0213] <Other application examples> The above describes the best mode for carrying out the present invention using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0214] For example, in this embodiment, two positions and orientations obtained mainly using two external sensors are integrated, but three or more positions and orientations obtained using three or more external sensors may be integrated.

[0215] Furthermore, although the technology of the present embodiment is effective in vast locations where the GNSS positioning accuracy is not necessarily high, it can also be suitably applied to small indoor environments and locations where the GNSS positioning accuracy is high.

[0216] As an example of an external sensor, location information by UWB (Ultra-Wideband) may be used. In the UWB method, a communication tag is attached to the moving object 10, and sensors are installed at intervals of 30 to 40 meters in the moving environment. The location of the communication tag can be identified by having two or more sensors receive radio waves emitted by the communication tag.

[0217] In addition, the configuration examples in Fig. 6 and the like are divided according to main functions to facilitate understanding of the processing by the moving body 10 and the display device 60. The method of dividing the processing units and their names do not limit the present invention. The processing by the moving body 10 and the display device 60 can be divided into more processing units depending on the processing content. Also, it can be divided so that one processing unit includes more processes.

[0218] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and a device such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), or a conventional circuit module designed to execute each function described above.

[0219] Embodiments of the present invention provide significant improvements in computer power and functionality. These improvements allow operators to utilize computers that provide more efficient and robust interaction with tables, the methods of storing and presenting information in information processing devices. Furthermore, embodiments of the present invention provide a better user experience through the use of more efficient, powerful and robust user interfaces. Such user interfaces provide better interaction between humans and machines.

[0220] <Additional Notes> [Appendix 1] A moving body capable of teaching travel in which a position and a posture when the moving path is moved by manual operation of an operator are stored, and autonomous travel in which the moving body moves at the stored position with the stored posture, an information creation unit that creates calculation information that can calculate a deviation between a node passed during teaching travel and a point passed during autonomous travel, by associating each node of the travel path and a plurality of external sensors; a deviation calculation unit that calculates the deviation for each of the plurality of external sensors based on the calculation information and a sensor value of the external sensor during autonomous movement; a position estimation unit that estimates the calculated deviation as a position and an orientation of each node of the moving path based on the position and orientation of the moving object; an integration processing unit that integrates the positions and orientations of each node of the movement path estimated for each of the plurality of external sensors; an autonomous movement processing unit that autonomously moves along the movement path based on the positions and orientations of each node of the movement path integrated by the integration processing unit; A mobile object having the above configuration. [Appendix 2] having one or more internal sensors for detecting the amount and direction of movement; During teaching travel in which the moving body is manually moved, The moving body described in Appendix 1, wherein the information creation unit stores the calculation information obtained by each of the multiple external sensors and the internal sensor for each node of the movement path in a memory in association with the external sensor. [Appendix 3] the integration processing unit weights the positions and orientations of the nodes based on the positions and orientations of the moving body estimated for each of the plurality of external sensors, based on the reliability of the calculation information and the reliability of the sensor values ​​of the external sensors during autonomous movement; 3. The moving body according to claim 1 or 2, wherein the position and orientation of the node estimated for each of the multiple external sensors are integrated. [Appendix 4] When the integration processing unit integrates the position and orientation of any one of the nodes estimated for each of the plurality of external sensors during autonomous movement, A moving body according to any one of appendices 1 to 3, comprising an autonomous movement processing unit that calculates the position and attitude of the node after any one of the nodes based on the integrated positions and attitudes of the nodes and the relative position information between the nodes included in the calculation information, and calculates the movement path with the position and attitude of the moving body at the any one of the nodes as the origin. [Appendix 5] 5. The moving body according to any one of claims 1 to 4, wherein the plurality of external sensors are two or more of a GNSS, a 2D-LiDAR, and a camera capable of obtaining distance information. [Appendix 6] When the first external sensor is a GNSS, the calculated information is a position and attitude in a GNSS coordinate system. When the second external sensor is a 2D-LiDAR, the calculated information is a two-dimensional occupancy grid map. the position estimation unit estimates, for the first external sensor, a deviation of a position and attitude in the GNSS coordinate system, which is a sensor value during autonomous movement, from the position and attitude in the GNSS coordinate system included in the calculation information, as a first position and attitude of the node based on the position and attitude of the moving body; For the second external sensor, the two-dimensional occupancy grid map is compared with a scan point cloud that is a sensor value during autonomous movement, and a deviation of a position and an orientation of the moving body with respect to an origin of the two-dimensional occupancy grid map is estimated as a second position and orientation of the node based on the position and orientation of the moving body; The moving body according to claim 5, wherein the integration processing unit integrates the first position and attitude and the second position and attitude. [Appendix 7] 5. The moving body according to claim 1, wherein the plurality of external sensors are two or more of a GNSS, a 3D-LiDAR, and a camera capable of obtaining distance information. [Appendix 8] When the first external sensor is a GNSS, the calculated information is a position and attitude in a GNSS coordinate system. When the second external sensor is a 3D-LiDAR, the calculated information is a three-dimensional point cloud map. the position estimation unit estimates, for the first external sensor, a deviation of a position and attitude in the GNSS coordinate system, which is a sensor value during autonomous movement, from the position and attitude in the GNSS coordinate system included in the calculation information, as a first position and attitude of the node based on the position and attitude of the moving body; For the second external sensor, the 3D point cloud map is compared with a scan point cloud which is a sensor value during autonomous movement, and a deviation of a position and an attitude of the moving body with respect to an origin of the 3D point cloud map is estimated as a second position and an attitude of the node based on the position and attitude of the moving body; 8. The moving body according to claim 7, wherein the integration processing unit integrates the first position and orientation and the second position and orientation. [Explanation of symbols]

[0221] 1. Communication Systems 10 Mobile 60 Display device [Prior art documents] [Patent documents]

[0222] [Patent Document 1] JP 2018-206004 A [Non-patent literature]

[0223] [Non-Patent Document 1] P. Biber & W. Strasser (2003), https: / / ieeexplore.ieee.org / document / 1249285)

Claims

1. A teaching travel that stores the position and posture when the moving path is moved by manual operation of an operator, and a moving body that can move autonomously along the moving path, an information creation unit that creates calculation information that can calculate a deviation between a node passed during teaching travel and a point passed during autonomous travel, by associating each node of the travel path and a plurality of external sensors; a deviation calculation unit that calculates the deviation for each of the plurality of external sensors based on the calculation information and a sensor value of the external sensor during autonomous movement; a position estimation unit that estimates the calculated deviation as a position and an orientation of each node of the moving path based on the position and orientation of the moving object; an integration processing unit that integrates the positions and orientations of each node of the movement path estimated for each of the plurality of external sensors; an autonomous movement processing unit that autonomously moves along the movement path based on the positions and orientations of each node of the movement path integrated by the integration processing unit; A mobile object having the above configuration.

2. having one or more internal sensors for detecting the amount and direction of movement; During teaching travel in which the moving body is manually moved, The moving body according to claim 1 , wherein the information creation unit stores the calculation information obtained by each of the plurality of external sensors and the internal sensor for each of the nodes of the movement path in a memory in association with the external sensor.

3. the integration processing unit weights the positions and orientations of the nodes based on the positions and orientations of the moving body estimated for each of the plurality of external sensors, based on the reliability of the calculation information and the reliability of the sensor values ​​of the external sensors during autonomous movement; The moving body according to claim 1 or 2, wherein the positions and orientations of the nodes estimated for each of the plurality of external sensors are integrated.

4. When the integration processing unit integrates the position and orientation of any one of the nodes estimated for each of the plurality of external sensors during autonomous movement, The moving body according to claim 1, further comprising an autonomous movement processing unit that calculates the position and orientation of the node after any one of the nodes based on the integrated positions and orientations of the nodes and the relative position information between the nodes included in the calculation information, and calculates the movement path with the position and orientation of the moving body at the any one of the nodes as the origin.

5. The moving body according to claim 1 , wherein the plurality of external sensors are two or more of a GNSS, a 2D-LiDAR, and a camera capable of obtaining distance information.

6. When the first external sensor is a GNSS, the calculated information is a position and attitude in a GNSS coordinate system. When the second external sensor is a 2D-LiDAR, the calculated information is a two-dimensional occupancy grid map. the position estimation unit estimates, for the first external sensor, a deviation of a position and attitude in the GNSS coordinate system, which is a sensor value during autonomous movement, from the position and attitude in the GNSS coordinate system included in the calculation information, as a first position and attitude of the node based on the position and attitude of the moving body; For the second external sensor, the two-dimensional occupancy grid map is compared with a scan point cloud which is a sensor value during autonomous movement, and a deviation of a position and an orientation of the moving body with respect to an origin of the two-dimensional occupancy grid map is estimated as a second position and an orientation of the node based on the position and orientation of the moving body; The moving body according to claim 5 , wherein the integration processing unit integrates the first position and orientation with the second position and orientation.

7. The moving body according to claim 1 , wherein the plurality of external sensors are two or more of a GNSS, a 3D-LiDAR, and a camera capable of obtaining distance information.

8. When the first external sensor is a GNSS, the calculated information is a position and attitude in a GNSS coordinate system. When the second external sensor is a 3D-LiDAR, the calculated information is a three-dimensional point cloud map. the position estimation unit estimates, for the first external sensor, a deviation of a position and attitude in the GNSS coordinate system, which is a sensor value during autonomous movement, from the position and attitude in the GNSS coordinate system included in the calculation information, as a first position and attitude of the node based on the position and attitude of the moving body; For the second external sensor, the three-dimensional point cloud map is compared with a scan point cloud which is a sensor value during autonomous movement, and a deviation of a position and an attitude of the moving body with respect to an origin of the three-dimensional point cloud map is estimated as a second position and an attitude of the node based on the position and attitude of the moving body; The moving body according to claim 7 , wherein the integration processing unit integrates the first position and orientation with the second position and orientation.

9. A method for estimating a position of a moving body that can autonomously move along a travel path and a teaching travel that stores a position and a posture when the moving body moves along the travel path by manual operation of an operator, comprising: A process of creating calculation information that can calculate a deviation between a node passed during teaching travel and a point passed during autonomous travel by corresponding each node of the travel path and a plurality of external sensors; A process of calculating the deviation for each of the plurality of external sensors based on the calculation information and a sensor value of the external sensor during autonomous movement; a process of estimating the calculated deviation as a position and an orientation of each node of the moving path based on the position and orientation of the moving object; A process of integrating the positions and orientations of each node of the movement path estimated for each of the plurality of external sensors; A process of autonomously moving along the movement path based on the positions and orientations of each node of the integrated movement path; A location estimation method that performs the above.

10. A teaching travel in which a position and a posture when a moving path is moved by a manual operation of an operator are stored, and a moving body capable of autonomously moving along the moving path is provided. an information creation unit that creates calculation information that can calculate a deviation between a node passed during teaching travel and a point passed during autonomous travel, by associating each node of the travel path and a plurality of external sensors; a deviation calculation unit that calculates the deviation for each of the plurality of external sensors based on the calculation information and a sensor value of the external sensor during autonomous movement; a position estimation unit that estimates the calculated deviation as a position and an orientation of each node of the moving path based on the position and orientation of the moving object; an integration processing unit that integrates the positions and orientations of each node of the movement path estimated for each of the plurality of external sensors; an autonomous movement processing unit that autonomously moves the movement path based on the positions and orientations of the nodes of the movement path integrated by the integration processing unit; A program to function as a

Images