Moving body, position estimation method, computer program product, and storage medium

Abstract

The invention relates to a moving body, a computer program product and a position estimation method, and aims to reduce precision reduction of autonomous movement of the moving body. The mobile body has: an information generation unit for associating each node on a movement path with a plurality of external sensors, and generating calculation information for calculating a deviation between a node through which travel is guided and a location through which the mobile body passes during autonomous movement; the deviation calculation part is used for calculating deviation according to the calculation information and the sensor value of the external sensor during autonomous movement; a position estimation unit for estimating the obtained deviation as the position and orientation of each node on the movement path based on the position and orientation of the moving body; an integration processing unit for integrating the estimated position and orientation of each node; and an autonomous movement processing unit for autonomously moving through the movement path on the basis of the integrated position and orientation of each node.

Description

Technical Field

[0001] The present invention relates to a mobile object, a position estimation method, a computer program product, a storage medium, and a computer device. Background Art

[0002] In the past, when inspecting outdoor structures (piping, instruments, valves, etc.) such as chemical plants for leaks or abnormalities, inspectors patrolled the locations where the structures were installed and conducted visual inspections. In recent years, a method has been developed to address the above inspection work by allowing autonomous mobile bodies to perform unmanned operations. The mobile body utilization method uses GNSS (Global Navigation Satellite System) positioning technology in outdoor environments and SLAM (Simultaneous Localization and Mapping) positioning technology in relatively narrow indoor environments to autonomously move to the destination.

[0003] Currently, there are technologies that allow mobile objects to move seamlessly indoors and outdoors. For example, Patent Document 1 (Japanese Patent Application Publication No. 2018-206004) discloses a technical solution that uses SLAM technology to pre-produce environmental maps using terrain data obtained from distance measurement results of distance sensors. During autonomous movement, the environmental map and distance data at each moment are matched to perform self-position estimation.

[0004] However, in the prior art, when an environmental map is produced in an environment where the GNSS positioning accuracy continues to decrease, for example, the produced map may be deformed, resulting in a decrease in the accuracy of the position and posture of the moving object during autonomous movement. Summary of the invention

[0005] In view of the above problems, the present invention provides a technical solution that can reduce the decrease in accuracy of the position and posture of a moving object during autonomous movement when a created map is deformed.

[0006] In order to solve the above-mentioned problems, the present invention provides a mobile body that can perform guided driving and autonomous movement, wherein the guided driving enables the mobile body to memorize the position and posture when it moves through a moving path according to the manual operation of an operator, and the autonomous driving is the mobile body autonomously moving through the moving path, characterized in that it has: an information generating unit for corresponding each node on the moving path to a plurality of external sensors, and generating calculation information for calculating the deviation between the node passed during the guided driving and the place passed during the autonomous movement; a deviation calculating unit for calculating the deviation for each of the external sensors based on the calculation information and the sensor value of the external sensor during the autonomous movement; a position estimating unit for estimating the deviation obtained for each of the external sensors as the position and posture of each node on the moving path based on the position and posture of the mobile body; an integration processing unit for integrating the position and posture of each node on the moving path estimated by each of the external sensors; and an autonomous movement processing unit for autonomously moving through the moving path based on the position and posture of each node on the moving path integrated by the integration processing unit.

[0007] The present invention has the effect of reducing the decrease in accuracy of the position and posture of a moving object during autonomous movement even when a created map is deformed. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 It is a schematic diagram comparing guided driving and autonomous movement in technology.

[0009] Figure 2 , (a) is a schematic diagram of route points memorized during guided driving, and (b) is a schematic diagram of a method for estimating position and posture during autonomous movement.

[0010] Figure 3 This is a schematic diagram of the overall structure of an example of a communication system.

[0011] Figure 4 This is an example of the appearance of a moving object.

[0012] Figure 5A This is an example of a hardware structure block diagram of a mobile object.

[0013] Figure 5B This is an example hardware structure block diagram of a display device.

[0014] Figure 6 This is an example functional block diagram of a communication system.

[0015] Figure 7 This is a conceptual diagram of an example of a destination candidate management table.

[0016] Figure 8This is a conceptual diagram of an example of a route information management table.

[0017] Fig. 9 This is a conceptual diagram of an example of a map information management table.

[0018] Fig.10 This is a sequence diagram of an example of destination candidate registration processing.

[0019] Fig.11 This is a diagram of an example of an operation screen.

[0020] Fig.12 This is a timing diagram of an example of manual operation processing of a moving object.

[0021] Fig.13 This is a flowchart of an example of destination candidate registration processing according to the moving state of the moving object.

[0022] Fig.14 This is a timing diagram of an example of destination setting logic for a moving object.

[0023] Fig.15 This is a diagram of an example selection screen.

[0024] Fig.16 This is a diagram of an example selection screen.

[0025] Fig.17 This is a flowchart of an example of autonomous movement processing of a mobile object.

[0026] Fig.18 It is a conceptual diagram of the topology map.

[0027] Fig.19 This is an example of a movement control block diagram of the present embodiment using an external sensor and a local map.

[0028] Fig. 20 This is a processing flow chart of the path guidance stage.

[0029] Fig.21 This is a processing flow chart of the autonomous mobility stage.

[0030] Fig. 22 It is a schematic diagram of the matching calculation between the two-dimensional occupancy grid map and the scanning point cloud of 2D-LiDAR.

[0031] Fig.23 This is a flowchart of an example of b0 evaluation based on the obstacle amount.

[0032] Fig.24 This is an example of a hardware block diagram of a display device when the display device is an HMD.

[0033] Fig.25 This is a schematic diagram of an example of image data displayed by the HMD display device.

[0034] Explanation of symbols

[0035] 1 Communication system

[0036] 10 Mobile

[0037] 60 Display device DETAILED DESCRIPTION

[0038] Hereinafter, a mode for implementing the present invention will be described using a moving object and a method for estimating a position of the moving object as an example with reference to the drawings.

[0039] [First embodiment]

[0040] <Supplementary explanation of positioning technology>

[0041] The maps used by SLAM (self-driving automatic mapping) technology are created using GNSS positioning technology in open outdoor environments and LiDAR (Light Detection And Ranging) and cameras in relatively narrow indoor environments. In order to estimate the position and posture with high accuracy even in large places where GNSS positioning accuracy is not high, multiple sensors can be used for map creation and self-position estimation.

[0042] In order to estimate the position and posture with high accuracy, there are currently technical solutions that use multiple sensors such as GNSS, LiDAR, and cameras to obtain accurate environmental map production and self-position estimation. Most of these technical solutions adopt the method of "restoring the entire environment into a two-dimensional or three-dimensional map and estimating the position of the mobile body 10 in the map."

[0043] However, even with multiple sensors, it is still difficult to avoid deformation and accurately restore the environment. In many cases, maps need to be manually modified. On incorrect maps with deformation, the self-position estimation results of each sensor may be biased, resulting in unstable driving.

[0044] Estimation of the position and posture of GNSS and LiDAR. If the positioning accuracy of GNSS is good, the deformation of the map created by LiDAR can be compensated to conform to the GNSS coordinate system to create an accurate map. However, as the positioning accuracy of GNSS continues to decline, the deformation of the map created by LiDAR cannot be compensated well, and this part of the map will become inaccurate.

[0045] Furthermore, if a location where LiDAR is used to create a map during autonomous movement accidentally obtains a GNSS position with good accuracy, the self-position estimation result obtained by matching the LiDAR scan data with the map will deviate from the GNSS positioning result. As a result, the mobile body uses one of these self-position estimation results or an intermediate result as the final estimation result for autonomous movement, which may lead to incorrect driving. Or the operator needs to manually correct the deformed part in the map to match the GNSS coordinates to avoid incorrect driving.

[0046] In this regard, even in a wide area where the GNSS positioning accuracy is not high, the present embodiment can perform autonomous movement with high accuracy without correcting the map by manual operation. Thus, an operator without professional knowledge can manually operate the mobile object to autonomously move along the movement path it has traveled.

[0047] Figure 1 This is a conceptual diagram of guided driving and autonomous movement in comparison technology. Comparison technology is a technology that clarifies the processing of this embodiment by comparing it with this embodiment. Comparison technology is not a prior art. Guidance means that the operator manually moves the mobile body 10 so that the mobile body 10 remembers the movement path 9.

[0048] like Figure 1 As shown in (a), first, as a guide for driving, the operator manually drives the moving body 10. The moving body 10 memorizes the moving path 9 while creating an environmental map. Figure 1 As shown in (b), the moving body 10 moves autonomously on the memorized moving path 9. Thus, in the comparative technique, although the moving body 10 moves through the moving path 9, the partial map of the route points described below is not memorized when guiding driving.

[0049] Figure 2 (a) shows the route point 8 memorized during the driving guidance in this embodiment. The mobile body 10 of this embodiment generates a topological map during the driving guidance. The topological map is a map representation in the form of a graph consisting of route points 8 (nodes described later) that exist discretely on the moving path and edges that represent the relative positional relationship between the route points 8. Each route point 8 corresponds to a plurality of local maps 91, 92 that are independently generated by a plurality of sensors and have the route point 8 as the origin.

[0050] Figure 2(b) is a schematic diagram of a method for estimating the position and posture during autonomous movement. During autonomous movement, the moving body 10 obtains the deviation from the position and posture of the moving body 10 estimated from the position and posture obtained based on the local map corresponding to the route point 8 and the sensor value (sensor measurement value) detected by the sensor during autonomous movement, and uses the deviation as the position and posture of the route point 8. The moving body 10 uses its own position as the origin during autonomous movement, so the deviation is the position and posture of the route point 8 seen from the moving body 10 during autonomous movement. The "seen" here is based on the position and posture of the moving body 10.

[0051] The moving object 10 weights the position and posture based on GNSS and the position and posture based on SLAM according to the positioning accuracy of GNSS and LiDAR, and integrates the deviation amount (ie, the position of the route point 8 viewed from the moving object 10 ).

[0052] In this way, the mobile body 10 of this embodiment has each sensor's independent local map in the form of a topological map. For each local map, the position estimation reliability of the self-position and self-posture estimated independently is calculated based on the sensor values ​​during guided driving and autonomous movement, and then weighted according to the reliability to integrate the final position and posture. Therefore, this embodiment does not need to correct the map through manual work to integrate the position estimation results of each sensor, and even if the positioning accuracy of a certain sensor decreases during the route guidance stage, high-precision autonomous movement can be achieved.

[0053] <Terms>

[0054] Herein, guided driving means that the operator manually operates the driving of the moving body 10 through a controller, so that the moving body 10 moves on the moving path to be moved autonomously, and memorizes the position and posture of the moving body 10 when moving on the moving path. Autonomous movement means that the moving body 10 automatically moves on or along the moving path memorized in guided driving.

[0055] The external sensor refers to a sensor that measures the environment (i.e., the external environment) of the mobile body 10. Examples of external sensors include GNSS antennas, LiDAR, and cameras that can obtain distance information. The so-called internal sensors refer to sensors that measure the state (i.e., the internal environment) of the mobile body 10. Examples of internal sensors include velocity sensors, acceleration sensors, angular velocity sensors, and azimuth sensors.

[0056] A node is a location that constitutes a movement path. In this embodiment, nodes or route points are used for explanation.

[0057] The position is information for specifying the location of the moving body 10 based on the origin, and the posture is information for specifying the orientation of the moving body 10 based on a predetermined direction. The posture may be a posture in a three-dimensional space or a posture in a two-dimensional space.

[0058] The calculation information refers to information that can be used to calculate the deviation between the node passed during guided driving and the location passed during autonomous movement. The calculation information may vary depending on the external sensor. In GNSS, the calculation information is the position and posture of the GNSS coordinate system, while in LiDAR, the calculation information is a two-dimensional occupancy grid map. In this embodiment, the term "local map" is used to explain the calculation information.

[0059] The local map refers to at least a portion of the map within the moving path that does not cover the entire moving path. The local map may be a map with the route point 8 as the origin and within the range where the external sensor can perform environmental measurement.

[0060] Integration means combining two or more things into one.

[0061] The position and posture based on the position and posture of the moving object 10 refer to the relative position and posture with respect to the position and posture of the moving object 10. In the present embodiment, expressions such as "seen from" the moving object 10 are sometimes used.

[0062] <System Configuration>

[0063] Figure 3 This is a schematic diagram of the overall structure of an example of a communication system. Figure 3 The communication system 1 shown is a system in which an operator can remotely control a moving object 10 that moves within a predetermined location.

[0064] The communication system 1 includes a mobile body 10 and a display device 60 located at a predetermined location. The mobile body 10 and the display device 60 constituting the communication system 1 can communicate via a communication network 100. The communication network 100 includes the Internet, a mobile communication network, a LAN (Local Area Network), etc. In addition to wired communication, the communication network 100 may also include 3G (3rd Generation), 4G (4th Generation), 5G (5 th Wireless communication networks such as Wi-Fi (Wireless Generation), Wi-Fi (Wireless Fidelity) (registered trademark), WiMAX (Worldwide Interoperability for Microwave Access) or LTE (Long Term Evolution).

[0065] The mobile body 10 is a robot that is set at the target location and can move autonomously in the target location. Autonomous movement includes imitating learning (mechanical learning) the path that has been moved in the target location in the past, and using the result of imitating learning to autonomously move in the target location. Autonomous movement can also be autonomous movement in the target location according to a preset movement path, or autonomous movement in the target location using a line tracking or other technology. Furthermore, the mobile body 10 can also move by manual operation of a remote operator. That is, the mobile body 10 can move in the target location while switching between autonomous movement and manual operation by the operator. The mobile body 10 can, for example, move in the target base while performing specified tasks such as inspection, maintenance, transportation or light manual labor. Here, the mobile body 10 is a robot in a broad sense, as long as it is a robot that can move autonomously and move by remote operation of the operator. The mobile body 10 also includes a car that travels by switching between automatic operation and manual operation through remote control operation. Furthermore, the mobile body 10 also includes aircraft such as drones, multi-axis helicopters or unmanned aerial vehicles.

[0066] The target location where the mobile body 10 is installed is, for example, an outdoor location such as a business office, a factory, a construction site, a substation, a farm, a field, a garden, a cultivated land, or a disaster site, or an indoor location such as an office, a school, a factory, a warehouse, a commercial facility, a hospital, or a nursing facility. In other words, the target location is a place where there is a need to replace the tasks that were previously performed manually with the mobile body 10.

[0067] The display device 60 is a computer such as a notebook PC (Personal Computer) used by an operator who is located at a management location different from the target location and performs a predetermined operation on the mobile object 10. The operator uses the operation screen displayed on the display device 60 at the management location such as an office to perform a movement operation on the mobile object 10 or an operation for executing a predetermined task. For example, the operator performs a remote operation of the mobile object 10 while viewing an image of the target location displayed on the display device 60.

[0068] Figure 3 In the example in which one moving object 10 and one display device 60 are connected via the communication network 100 , the display device 60 may be connected to a plurality of moving objects 10 installed at one target location, or may be connected to moving objects 10 located at different target locations. Figure 3 The display device 60 is shown as being located at a management location that is far away from the target location where the moving body 10 is located, but the display device 60 may be located within the target location where the moving body 10 is located. The display device 60 is not limited to a notebook PC, and may be, for example, a desktop PC, a tablet terminal, a smartphone, or a wearable terminal.

[0069] <Overview of mobile body>

[0070] Figure 4 (a) is a perspective view of the appearance of a mobile body 10 according to the present embodiment. The mobile body 10 includes crawler-type traveling bodies 10a and 10b and a main body 50.

[0071] The crawler traveling bodies 10a and 10b are mobile equipment units serving as the moving body 10. The crawler traveling bodies 10a and 10b are crawler traveling bodies using metal belts or rubber belts. Compared with traveling bodies that travel on tires such as automobiles, crawler traveling bodies have a larger ground contact area and can travel stably even in a harsh ground environment. Traveling bodies that travel on tires require a turning space when performing a rotational movement. In contrast, a traveling device having a crawler traveling body can perform a so-called over-the-ground turn, and therefore can smoothly perform a rotational movement even in a limited space.

[0072] The main body 50 is not only a support body for supporting the crawler-type traveling bodies 10a and 10b in a walkable state, but also a control device for controlling the driving of the traveling body 10. The main body 50 is also equipped with a battery for supplying power to drive the crawler-type traveling bodies 10a and 10b.

[0073] Figure 4 (b) is a front view (view in the direction of arrow P) of the mobile body 10 according to the present embodiment. The main body 50 of the mobile body 10 includes an emergency stop switch 143, a status indicator light 144, and a cover 146. The emergency stop switch 143 is an operating device that can be pressed by people around the mobile body 10 to stop the moving mobile body 10.

[0074] The status indicator light 144 is a notification device for notifying the status of the mobile body 10. The status indicator light 144 lights up when the status of the mobile body 10 changes, such as when the remaining battery level decreases, and notifies people around the mobile body 10 of the change in the status. The status indicator light 144 may also light up when an abnormality may occur, for example, when the presence of an obstacle that hinders the movement of the mobile body 10 is detected. Figure 4 (b) shows that the moving object 10 includes two status indicators 144, but the number of status indicators 144 may be one or three or more. The notification device may be configured to notify the status of the moving object 10 by, for example, sounding a warning sound from a speaker in addition to the status indicators 144.

[0075] The cover 146 is provided on the upper surface of the main body 50 to seal the inside of the main body 50. The cover 146 has a ventilation portion 35a having a ventilation hole for ventilation of the inside of the main body 50.

[0076] The two crawler-type traveling bodies 10a and 10b are arranged on both sides of the main body 50 in a state where the moving body 10 can move. The number of crawler-type traveling bodies is not limited to two, and can be three or more. For example, the moving body 10 can be arranged in a state where three crawler-type traveling bodies are arranged in parallel in three rows, etc., so that the moving body 10 can move. For another example, the moving body 10 can also be four crawler-type traveling bodies arranged front and back, left and right like the tires of a motor vehicle.

[0077] Figure 4 (c) is a side view (view in the direction of arrow Q) of the mobile body 10 involved in this embodiment. The crawler-type traveling body 10a has a triangular shape. For example, when the front and rear dimensions of the traveling body are restricted, the triangular crawler-type traveling body 10a can increase the ground contact area within the limited front and rear dimensions, thereby improving stability during movement. On the other hand, when the front and rear dimensions are restricted, the upper part (driving wheel side) is longer than the lower part (rotating wheel side), that is, the so-called tank-type crawler track, the overall ground contact area becomes smaller and becomes unstable. In this way, the crawler-type traveling body 10a is more effective in improving the mobility of a relatively small mobile body 10.

[0078] like Figure 4 As shown, the mobile body 10 is provided with a distance measuring sensor 112 for horizontal direction detection on the front surface of the main body 50 in the forward direction. The distance measuring sensor 112 for horizontal direction detection is a LiDAR, and in this embodiment, it is a MEMS (Micro Electro Mechanical Systems) LiDAR for 2D scanning. LiDAR can also be used for 3D scanning.

[0079] The distance measuring sensor 112 irradiates a laser beam to an object, and measures the distance to the object and the direction in which the object exists based on the measurement result of the time taken for the laser beam to return after irradiating the object.

[0080] In addition, Figure 4 As shown, the moving body 10 has a distance measuring sensor 113 for oblique detection on the front surface of the main body 50 in the forward direction. The distance measuring sensor 113 for oblique detection is a LiDAR, and in this embodiment, it is a MEMS-type LiDAR for 2D scanning. The distance measuring sensor 113 is installed at a certain angle below the horizontal, tilted downward relative to the horizontal moving road surface, and has a predetermined tilt angle.

[0081] The distance measuring sensor 113 irradiates a laser beam to an object (road obstacle), and based on the measurement result of the time it takes for the laser beam to return after irradiating the object, measures the distance to the object and the direction in which the object exists.

[0082] The installation position of the distance measuring sensor 113 has an optimal value according to the track width and length of the crawler traveling bodies 10a and 10b and the size, width, depth and height of the object to be detected.

[0083] Figure 4 The mobile body 10 is a crawler driving type. In addition, the shape and movement of the mobile body 10 are not limited to a bipedal walking type, a multi-legged walking type, an unmanned aerial vehicle type, a flying type, an underwater moving type, etc. Figure 2 way.

[0084] <Hardware Configuration Example>

[0085] Next, according to Figure 5A and Figure 5B An example of the hardware configuration of the moving object 10 according to the present embodiment will be described. Figure 5A This is a block diagram showing an example of the hardware configuration of the mobile object 10 .

[0086] like Figure 5A As shown, the mobile body 10 of the present embodiment has a CPU (Central Processing Unit) 101, a memory 102, an auxiliary storage device 103, a camera 111, a distance sensor 112 for horizontal direction detection, a distance sensor 113 for oblique direction detection, a satellite positioning system 114, an IMU (Inertial Measurement Unit) 115, a battery 121, a start SW 142, an emergency stop SW 143, motor drivers 122a, 122b, driving motors 132a, 132b, brake drivers 123a, 123b, brake motors 133a, 133b, and a power supply SW 141.

[0087] The CPU 101 controls the entire moving object 10. The memory 102 is a temporary storage area used by the CPU 101 to execute a moving program and the like.

[0088] The camera 111 includes a general photographing device that can obtain a planar image, such as a digital SLR camera and a small digital camera, and also includes a special photographing device that can obtain a special image, such as a spherical camera, a stereo camera, and an infrared camera. The mobile body 10 may have a plurality of cameras 111, or may have both a general photographing device that can obtain a planar image and a special photographing device that can obtain a special image. The distance measuring sensor 112 for horizontal direction detection and the distance measuring sensor 113 for oblique direction detection are, for example, LiDAR.

[0089] The satellite positioning system 114 includes an antenna for receiving radio waves from a satellite, and measures the position of the mobile object 10 on the earth based on the reception result. The satellite positioning system 114 is also called GNSS.

[0090] The IMU 115 is an inertial positioning sensor that estimates the position and posture (translation and rotation in three orthogonal axis directions) of the moving object 10 using an acceleration sensor and an angular velocity sensor. The IMU 115 is an example of an internal limit sensor.

[0091] The battery 121 is a power source for the movement of the moving body 10. The start SW 142 is a switch for starting the moving body 10. The emergency stop switch 143 is a switch for stopping the movement of the moving body 10 arbitrarily.

[0092] The motor drivers 122a and 122b are drivers for the travel motors 132a and 132b respectively provided on the two crawler travel bodies 10a and 10b. The brake drivers 123a and 123b are drivers for the brake motors 133a and 133b respectively provided on the two crawler travel bodies 10a and 10b. The power supply SW141 is a switch for turning on or off the power supply of the mobile body 10.

[0093] The mobile program executed in the mobile object 10 of this embodiment is provided as an installable or executable file stored in a computer-readable storage medium such as a CD-ROM, a floppy disk (FD), a CD-R, or a DVD (Digital Versatile Disc).

[0094] The mobile program executed in the mobile body 10 of this embodiment may be stored in a computer connected to a network such as the Internet and provided by downloading via the network. The mobile program executed in the mobile body 10 of this embodiment may also be provided or published via a network such as the Internet. In addition, the mobile program executed in the mobile body 10 of this embodiment may be provided by being pre-assembled in a ROM or the like.

[0095] Display device

[0096] Figure 5B FIG. 6 is a block diagram of an example of the structure of the display device 60 according to the present embodiment. Figure 5B As shown, the display device 60 is composed of a computer 500, and has a CPU 501, a ROM 502, a RAM 503, an HD 504, an HDD controller 505, a display 506, an external device connection I / F (interface) 508, a network I / F 509, a bus 510, a keyboard 511, an indicating device 512, a DVD-RW drive 514, and a medium I / F 516.

[0097] Among them, CPU501 controls the overall operation of computer 500. ROM502 stores programs such as IPL for driving CPU501. RAM503 is used as a work area for CPU501. HD504 stores various data such as programs. HDD controller 505 controls the reading or writing of various data in HD504 according to the control of CPU501. Display 506 displays various information such as cursor, menu, window, text or image. Peripheral connection I / F508 is an interface for connecting various external devices. The peripherals here are, for example, USB memory or printer. Network I / F509 is an interface for data communication using the network. Bus 510 is used Figure 5B The various components such as the CPU 501 shown are electrically connected to an address bus, a data bus, etc.

[0098] The keyboard 511 is an input device having a plurality of keys for inputting text, numerical values, or various instructions. The pointing device 512 is an input device for selecting and executing various instructions, selecting a processing object, moving a cursor, etc. The DVD-RW drive 514 controls reading or writing various data in a DVD-RW 513 as an example of a removable recording medium. The DVD-RW drive 514 is not limited to a DVD-RW, but may also be a DVD-R, etc. The medium I / F 516 controls reading or writing (storing) data in a recording medium 515 such as a flash memory.

[0099] <Functional Configuration>

[0100] Next use Figures 6 to 9 The functional configuration of the communication system 1 according to the embodiment will be described. Figure 6 This is a functional block diagram of an example of the communication system 1. Figure 6 express Figure 1 The device or terminal shown is related to the processing or action described later.

[0101] Functional structure of mobile device (control device)

[0102] The mobile body 10 includes a transceiver 31, a determination unit 32, a photography control unit 33, a state detection unit 34, a position information acquisition unit 35, a destination candidate acquisition unit 36, a route information generation unit 37, a route information management unit 38, a mobile destination setting unit 39, a mobile control unit 40, an image recognition processing unit 41, a mode setting unit 42, an autonomous mobile processing unit 43, a manual operation processing unit 44, a task execution unit 45, an information generation unit 46, an image processing unit 47, a learning unit 48, and a storage reading unit 49. These units are Figure 5A The mobile body 10 has a function or a device that performs a function when one of the components shown in the figure executes a command of the CPU 101 according to a mobile program developed on the memory 102. Figure 5AThe memory 102 shown in the figure constitutes the storage unit 3000.

[0103] The transceiver unit 31 is used to transmit and receive various data or information with other devices or terminals via the communication network 100 .

[0104] The determination unit 32 is implemented by the CPU 101 executing processing, and is used to perform various determinations. The photography control unit 33 is mainly implemented by the CPU 101 executing processing on the camera 111, and is used to control the photography processing on the camera 111.

[0105] The state detection unit 34 is mainly implemented by the CPU 101 executing processing on the camera 111 and the distance measuring sensors 112113, and is used to detect the state of the mobile body 10 or the surroundings of the mobile body 10 using various sensors. The state detection unit 34, for example, measures the distance to the objects (obstacles) around the mobile body 10 and outputs it as distance data. The state detection unit 34 can also use, for example, the distance data to obtain the position by comparing it with the environmental map using SLAM (Synchronous Localization and Mapping). Here, SLAM is a technology that can simultaneously perform self-position estimation and environmental map creation. Furthermore, the state detection unit 34 also uses, for example, IMU115 to detect the orientation of the mobile body 10.

[0106] The position information acquisition unit 35 is implemented by the CPU 101 executing processing on the satellite positioning system 114, and is used to acquire position information indicating the current position of the moving object 10 using the satellite positioning system 114. The position information acquisition unit 35 acquires coordinate information indicating the latitude and longitude of the current position of the moving object 10 using the satellite positioning system 114, for example.

[0107] The destination candidate acquisition unit 36 ​​is mainly implemented by the processing of the CPU 101, and is used to acquire a destination candidate image, which indicates a destination candidate of the mobile body 10. Specifically, the destination candidate acquisition unit 36 ​​acquires a photographic image taken by the photography control unit 33 and indicating a partial area in the location where the mobile body 10 is located, as the destination candidate image.

[0108] The path information generating unit 37 is mainly implemented by the processing of the CPU 101, and is used to generate path information indicating the moving path of the mobile body 10. The path information generating unit 37 generates path information indicating the path from the current position to the final destination based on the destination candidate position selected by the operator of the mobile body 10. The path information generating method adopts, for example, a method of connecting the route points 8 from the current position to the final destination with a straight line, or a method of minimizing the moving time while avoiding obstacles by obtaining information about obstacles from the photographic image or the state detecting unit 34.

[0109] The route information management unit 38 is mainly implemented by the processing of the CPU 101 , and manages the route information generated by the route information generation unit 37 using the route information management DB 3002 .

[0110] The destination setting unit 39 is mainly implemented by the processing of the CPU 101, and is used to set the destination of the mobile body 10. The destination setting unit 39 sets the destination to be traveled to from among the destination candidates selected by the operator of the mobile body 10 as the destination, based on the current position of the mobile body 10 acquired by the position information acquisition unit 35 and the route information generated by the route information generation unit 37, for example.

[0111] The movement control unit 40 is mainly implemented by the CPU 101 processing the motor drivers 122a and 122b, and is used to control the movement of the moving body 10 by driving the travel motors 132a and 132b. The movement control unit 40 moves the moving body 10 according to a drive instruction from the autonomous movement processing unit 43 or the manual operation processing unit 44, for example.

[0112] The image recognition processing unit 41 is mainly implemented by the processing of the CPU 101, and is used to perform image recognition processing on the photographic image obtained by the photography control unit 33. The image recognition processing unit 41 detects whether a specific subject is included in the obtained photographic image through image recognition. The specific subject is, for example, an obstacle on the moving path of the mobile body 10 or around the moving path, an intersection such as a crossroad or an L-shaped road, or a sign or signal set at a location.

[0113] The mode setting unit 42 is mainly implemented through the processing of CPU 101, and is used to set an action mode representing an action for moving the mobile body 10. The mode setting unit 42 sets one of an autonomous movement mode for moving the mobile body 10 autonomously and a manual operation mode for moving the mobile body 10 through manual operation of an operator.

[0114] The autonomous movement processing unit 43 is mainly implemented by the processing of the CPU 101, and is used to control the autonomous movement processing of the moving body 10. For example, the autonomous movement processing unit 43 outputs a drive instruction of the moving body 10 to the movement control unit 40 so that the moving body 10 passes through the movement path represented by the path information generated by the path information generation unit 37.

[0115] The autonomous movement processing unit 43 includes a calculation unit 43a, a deviation calculation unit 43b, a position estimation unit 43c, and an integration processing unit 43d. During autonomous movement, the calculation unit 43a calculates the position and posture of the mobile body 10 based on the measurement values ​​of multiple external sensors and internal sensors using existing methods such as Kalman filters.

[0116] The deviation calculation unit 43b is used to calculate the deviation between the local map and the sensor value of the external sensor for each of the plurality of external sensors. The position estimation unit 43c is used to estimate the obtained deviation as the position and posture of the route point 8 as seen from the moving body 10. The integration processing unit 43d is used to integrate the position and posture of the route point 8 estimated by each of the plurality of external sensors.

[0117] The manual operation processing unit 44 is mainly realized by the processing of the CPU 101 and controls the manual operation processing of the moving body 10. The manual operation processing unit 44 outputs a drive command of the moving body 10 to the moving control unit 40 based on a manual operation command sent from the display device 60, for example.

[0118] The task execution unit 45 is mainly realized by the processing of the CPU 101, and is used to execute a predetermined task of the mobile body 10 according to a request from the operator. Here, the predetermined task executed by the task execution unit 45 includes, for example, photographic processing for inspecting equipment in a location, or light manual labor using a movable arm.

[0119] The information generation unit 46 is mainly realized by processing of the CPU 101, and manages map information of an environment map representing a target location set by the mobile body 10 using the map information management DB 3003. The information generation unit 46 generates a plurality of local maps based on a plurality of external sensors, for example, and manages them in correspondence with the external sensors.

[0120] The image processing unit 47 is mainly realized by processing of the CPU 101, and generates a display image to be displayed on the display device 60. The image processing unit 47 generates a display image to be displayed on the display device 60 by processing a photographic image acquired by the photographic control unit 33, for example.

[0121] The learning unit 48 is mainly implemented by the processing of the CPU 101, and is used to learn the movement path for the autonomous movement of the mobile body 10. The learning unit 48 performs simulation learning (mechanical learning) on ​​the movement path of the autonomous movement based on, for example, the photographic image obtained by the manual operation processing unit 44 when the movement action is performed in the manual operation mode and the detection data by the state detection unit 34. The autonomous movement processing unit 43 performs the autonomous movement of the mobile body 10 based on, for example, the learning data of the simulation learning result of the learning unit 48.

[0122] The storage and reading unit 49 is mainly realized by the processing of the CPU 101 , and is used to store various data (or information) in the storage unit 3000 or read various data (or information) from the storage unit 3000 .

[0123] Destination candidate management table

[0124] Figure 72 is a conceptual diagram of an example of a destination candidate management table. The destination candidate management table is used to manage destination candidates indicating destination candidates of the mobile body 10. The storage unit 3000 is constructed as follows: Figure 7 The candidate destination management DB 3001 is composed of the candidate destination management table shown.

[0125] The destination candidate management table manages a candidate ID for identifying each location ID of the location where the mobile body 10 is located, location information indicating the location of the destination candidate, and destination candidate data associated with photographic image data of a portion of the area within the location that is the destination candidate. The location information is represented by coordinate information including the latitude and longitude indicating the location of the destination candidate within the location. When the mobile body 10 is an aircraft such as a drone, the location information includes information such as the speed, posture or altitude of the aircraft in addition to the coordinate information indicating the latitude and longitude. The photographic image data represents an image data document captured by the mobile body 10 at the location of the destination candidate within the location. Here, the destination candidate is an example of a mobile destination candidate of the mobile body 10. The mobile destination candidates of the mobile body 10 include not only the destination candidates of the mobile body 10,

[0126] Path information management table

[0127] Figure 8 FIG. 2 is a schematic diagram of an example of a path information management table. The path information management table manages path information representing the moving path of the moving body 10. The storage unit 3000 is constructed as follows: Figure 8 The route information management table shown is configured as the route information management DB 3002.

[0128] The path information management table manages a path ID for identifying a moving path of the moving body 10 in association with path information indicating the moving path of the moving body 10 for each location ID for identifying a location where the moving body 10 is installed. The path information indicates the moving path that the moving body 10 will pass through in the order of future moving destinations. The path information is generated by the path information generation unit 37 when the moving body 10 starts moving.

[0129] Although the present embodiment describes a movement route connecting a plurality of destinations and a movement route memorized by guided driving, a part or all of the former movement route is the latter movement route, and therefore no strict distinction is made in the description.

[0130] Map information management table

[0131] Fig. 9 2 is a schematic diagram of an example of a map information management table. The map information management table manages the local map of each node generated during the driving guidance. Fig. 9The map information management table shown in FIG. 3 is configured into the map information management DB 3003.

[0132] Fig. 9 The map information management table stores a first local map generated by GNSS with the position and posture of each node as the origin, and a second local map generated by SLAM with the position and posture of each node as the origin. The first local map and the second local map are generated in a data format suitable for their respective maps. The "local map" in this embodiment refers to "the calculation information that can be used to calculate the deviation between the nodes passed during guided driving and the places passed during autonomous movement", and the form of the local map varies with different external sensors. For example, the first local map represents the position and posture of the GNSS coordinate system, rather than a general map. For specific local maps, see below. Fig.18 illustrate.

[0133] The map information management table may also have a local map taken by a camera. The camera may be a stereo camera or a single-lens camera capable of obtaining distance information. The local map taken by the camera has the same information as the local map produced by SLAM.

[0134] Learning Data Management DB

[0135] The following describes the learning data management DB 3004 constructed in the storage unit 3000. The learning data management DB 3004 manages learning data that is the result of the learning unit 48 learning the autonomous movement path of the mobile body 10. The learning data management DB 3004 accumulates, for example, photographic image data and various detection data obtained from the mobile body 10, and at the same time stores the imitation learning (mechanical learning) results, that is, the learning data, for each location or each mobile body 10. The mobile body 10 moves autonomously based on the learning data stored in the learning data management DB 3004. The captured image data captured by the camera 111, such as a special photographic device or a general photographic device, may also include PTZ (Pan-Tilt-Zoom) parameters for determining the shooting direction of the special photographic device or the general photographic device. The photographic image data containing the PTZ parameters are stored in the storage unit 3000 (learning data management DB 3004) of the mobile body 10. The PTZ parameters and the position information of the candidate destination may be associated and stored in the storage unit 3000, and the coordinate data (x, y, θ) indicating the posture of the mobile body 10 when the photographic image data of the candidate destination is obtained may be stored in the storage unit 3000. In this way, when a deviation occurs from the actual stop position of the mobile body 10 at the destination, the PTZ parameters and the coordinate data (x, y, θ) may be used to compensate for the posture of the mobile body 10. In addition, part of the information such as the autonomous movement path (GNSS positioning history) of the mobile body 10 and the photographic image information of the candidate destination to be displayed on the display device 60 may be stored in a cloud computing service such as AWS (Amazon Web Services) (trademark).

[0136] Functional composition of the display device

[0137] Next use Figure 6 The functional structure of the display device 60 is described. The display device 60 includes a transceiver 51, a receiver 52, a display controller 53, a determiner 54, a manual operation command generator 55, an autonomous movement request information generator 56, an image processor 57, and a storage reader 59. These components are Figure 5B The display device 60 has a function or a device that realizes a function by operating one of the components shown in the figure according to a display device program developed on the RAM 503 in accordance with a command from the CPU 501. Figure 5B The storage unit 5000 is composed of the ROM 502 , HD 504 , or recording medium 521 shown.

[0138] The transceiver unit 51 is mainly implemented by the CPU 501 processing the network I / F 509 , and is used to send and receive various data or information with other devices or terminals.

[0139] The receiving unit 52 is mainly realized by the CPU 501 processing the keyboard 511 or the pointing device 512, and is used to accept various selections or inputs from the operator. The display control unit 53 is mainly realized by the CPU 501 processing, and is used to control the display unit such as the display 506 to display various screens. The judgment unit 54 is realized by the CPU 501 processing, and is used to make various judgments.

[0140] The manual operation instruction generating unit 55 is mainly realized by the processing of the CPU 501, and is used to generate a manual operation instruction for moving the mobile body 10 by manual operation according to the input operation of the operator. The autonomous movement request information generating unit 56 is mainly realized by the processing of the CPU 501, and generates autonomous movement request information for requesting the mobile body 10 to move autonomously. The autonomous movement request information generating unit 56 is used to send the autonomous movement request information to the mobile body 10 according to the information of the destination candidate selected by the operator, for example.

[0141] The image processing unit 57 is mainly realized by the processing of the CPU 501, and is used to generate a display image to be displayed on a display unit such as the display 506. The image processing unit 57 generates a display image to be displayed on the display device 60 by processing, for example, a photographic image obtained by the mobile body 10. The communication system 1 only needs to have the functions of the image processing unit 47 of the mobile body 10 and the image processing unit 57 of the display device 60.

[0142] The storage and reading unit 59 is mainly realized by the processing of the CPU 501 , and stores various data (or information) in the storage unit 5000 or reads various data (or information) from the storage unit 5000 .

[0143] <Processing or Operation>

[0144] Destination candidate registration processing

[0145] First use Figures 10 to 13 The process of registering the destination candidate of the mobile body 10 will be described. Fig.10 This is a diagram showing an example of the destination candidate registration process.

[0146] The display device 60 starts the operation of the mobile body 10 in response to the operator's predetermined input operation or the like (step S11). Then, the transceiver 51 transmits an operation start request to the mobile body 10 (step S12). The transceiver 31 of the mobile body 10 receives the operation start request transmitted by the display device 60.

[0147] Next, the photography control unit 33 starts photography processing using the camera 111 such as the special photography device and the general photography device (step S13). The photography control unit 33 obtains the photography image data taken by the special photography device and the general photography device. Here, the image obtained by the photography processing of step S13 is a moving picture. Then, the transceiver 31 sends the photography image data obtained in step S13 to the display device 60 (step S14). The transceiver 51 of the display device 60 receives the photography image data sent by the mobile body 10.

[0148] Next, the display control unit 53 of the display device 60 causes the display unit such as the display 506 to display the operation screen 200 included in the captured image data received in step S14 (step S15 ). Fig.11 This is a diagram of an example of an operation screen. Fig.11 The operation screen 200 shown is a display screen for an operator to remotely operate the moving object 10 .

[0149] The operation screen 200 includes a location display area 210 and a location display area 230. The location display area 210 displays a photographic image (plane image) taken by a general photographing device in the photographic image data received in step S14, and the location display area 230 displays a photographic image (special image) taken by a special photographing device in the photographic image data received in step S14. As described above, the special image is an image taken in all directions within the location, such as a full spherical image, a wide-angle image, or a hemispherical image taken by the special photographing device. The special image may also be an image obtained by a special photographing device that generates an all-directional image by synthesizing an image taken by a general photographing device while rotating. A moving body representation image 235 indicating the presence of the moving body 10 is displayed on the location display area 230 by being superimposed on the special image. Coordinate information represented by latitude (Lat) and longitude (Lon) is displayed on the operation screen 200, indicating the position information of the current position of the moving body 10. When the mobile object 10 is an aircraft such as a drone, in addition to the latitude and longitude coordinate information, information such as the speed, posture, or altitude of the aircraft can also be displayed on the operation screen 200. Furthermore, images captured by special camera devices and general camera devices can be transmitted in real time via a computer network such as the Internet and displayed as live streaming video images on the operation screen 200.

[0150] The operation screen 200 includes operation icons 250 for remotely controlling the moving object 10. The operation icons 250 include direction indication buttons that are pressed when requesting the moving body 10 to move in the horizontal direction (forward, backward, right rotation, left rotation). The operator can select the operation icon 250 while viewing the special images such as the plane image displayed on the location display area 210 and the full celestial sphere image displayed on the location display area 230, and perform remote control of the moving body 10. Here, an example of remotely controlling the movement of the moving body 10 by accepting the selection of the operation icon 250 displayed on the operation screen 200 is described, but the movement of the moving body 10 can also be operated by a keyboard, a dedicated controller such as a game pad with a game controller, or the like.

[0151] Furthermore, the operation screen 200 includes a destination candidate registration button 270, which can be pressed when registering a destination candidate using the plane image displayed in the location display area 210 or the special image displayed in the location display area 230. The following example describes a case where the operator presses the destination candidate registration button 270 to register a destination candidate using the special image displayed in the location display area 230. The same process is also performed when the operator presses the destination candidate registration button 270 to register a destination using the plane image displayed in the location display area 210. The operation screen 200 also includes a destination setting button 290 that is pressed when setting the destination of the mobile body 10.

[0152] Next, the display device 60 uses the operation screen 200 displayed in step S15 to perform a manual operation of the moving object 10 (step S16). The details of the process in step S16 will be described later.

[0153] Next, the operator presses the destination candidate registration button 270 on the operation screen 200, and the receiving unit 52 receives the destination candidate registration for the area in the special image displayed on the location display area 230 (step S17). Then, the transceiver unit 51 transmits a destination candidate registration request to the mobile body 10 (step S18). The destination registration request is, for example, a request to take a picture of the area in the special image displayed on the location display area 230. The mobile body 10 receives the destination candidate registration request transmitted by the display device 60.

[0154] Next, the position information acquisition unit 35 of the mobile body 10 uses the satellite positioning system 114 or the like to acquire position information indicating the current position of the mobile body 10 (step S19). Specifically, the position information acquisition unit 35 acquires coordinate information including the latitude and longitude indicating the current position of the mobile body 10. When the mobile body 10 is an aircraft such as a drone, in addition to acquiring the coordinate information including the latitude and longitude, the position information acquisition unit 35 also acquires information such as the speed, posture or altitude of the aircraft as position information. The destination candidate acquisition unit 36 ​​acquires a photographic image taken by a special photographic device at the current position of the mobile body 10 as a destination candidate image indicating a destination candidate (step S20). Here, the photographic image acquired by the destination candidate acquisition unit 36 ​​as the destination candidate image is a still image.

[0155] Then, the storage and reading unit 49 stores the candidate destination data including the position information acquired in step S19 and the photographic image acquired in step S20 in the candidate destination management DB 3001 (see Figure 7 ) (step S21). The storage reading unit 49 registers the acquired position information and photographic image as destination candidate data, and the destination candidate data is associated with a candidate ID for identifying the destination candidate.

[0156] In this way, the communication system 1 displays the photographic image captured by the mobile body 10 on the display device 60 of the operator who remotely controls the mobile body 10, so that the operator can remotely control the mobile body 10 while checking the surrounding conditions of the mobile body 10 in real time. The communication system 1 can register in advance the photographic image of the area within the location that becomes the destination candidate when the mobile body 10 moves autonomously in response to the input operation of the operator when the mobile body 10 is manually operated.

[0157] Manual operation processing

[0158] Next, use Fig.12 and Fig.13 The manual operation process on the moving body 10 in step S16 is described in detail. Fig.12 This is a timing diagram of an example of manual operation processing of a moving object.

[0159] First, the receiving unit 52 of the display device 60 receives the selection of the operation icon 250 of the operation screen 200 displayed in step S15 through the input operation of the operator (step S31). The manual operation instruction generating unit 55 generates a manual operation instruction according to the content of the operation icon 250 selected in step S31 (step S32). Fig.11In the example, since the operator is in a state where the "Forward (↑)" icon is selected, the manual operation instruction generating unit 55 generates an instruction for moving the moving body 10 forward. Then, the transceiver unit 51 sends the manual operation instruction generated in step S32 to the moving body 10 (step S33). As a result, the moving body 10 receives the manual operation instruction sent from the display device 60. When the operator selects the "Backward (↓)" icon during the input operation of the operation icon 250 while the moving body 10 is moving forward, the photographic image can also be switched to the rear screen of the moving body 10, and the moving body 10 can be moved backward from that moment on. The sending of the manual operation instruction from the display device 60 to the moving body 10 can also be via a management cloud platform such as AWS IoT Core (registered trademark).

[0160] Next, the mode setting unit 42 sets the action mode of the mobile body 10 to the manual operation mode (step S34). Then, the manual operation processing unit 44 outputs a drive instruction based on the manual operation instruction received in step S33 to the mobile control unit 40. The mobile control unit 40 executes the movement processing of the mobile body 10 according to the drive instruction of the manual operation processing unit 44 (step S35). The learning unit 48 performs imitation learning (mechanical learning) of the movement path corresponding to the manual operation of the manual operation processing unit 44 (step S36). The learning unit 48 performs imitation learning of the movement path, for example, based on the photographic image obtained when the manual operation processing unit 44 performs the movement action in the manual operation mode and the detection data obtained by the state detection unit 34. The learning unit 48 can perform imitation learning of the movement path using only the photographic image obtained during the manual operation, or can perform imitation learning of the movement path using both the photographic image and the detection data of the state detection unit 34. The photographic image used in the imitation learning of the learning unit 48 can also be a photographic image obtained when the autonomous movement processing unit 43 moves autonomously in the autonomous movement mode.

[0161] The learning of the movement path includes the position and posture of the moving object 10 .

[0162] Then, the mobile body 10 executes a registration process of the candidate destination during the moving operation (step S37). Fig.13 This is a flowchart showing an example of a destination candidate registration process according to the moving state of a moving object.

[0163] First, the judgment unit 32 judges whether the task execution unit 45 has executed the prescribed task (step S51). Specifically, the task execution unit 45 executes the prescribed task of the mobile body 10, for example, according to a task execution request from an operator or a preset schedule. Then, the judgment unit 32 judges whether the task execution unit 45 can execute the prescribed task. Here, the prescribed task includes, for example, photographic processing for checking equipment in a location, or light physical labor using a movable arm.

[0164] For example, after the mobile body 10 moves into the area of ​​the inspection object in the site, it performs the operation of confirming the inspection object such as an instrument or a valve. At this time, the mobile body 10 can stop moving to perform a photography operation for obtaining an inspection image, and the registration of the destination candidate is performed using this action as a trigger.

[0165] If the determination unit 32 determines that the task execution unit 45 has executed the specified task (Yes in step S51), the process proceeds to step S56. If the determination unit 32 determines that the task execution unit 45 has not executed the specified task (No in step S51), the process proceeds to step S52.

[0166] Next, the determination unit 32 determines whether the moving action of the moving body 10 has stopped (step S52). Specifically, when the determination unit 32 detects that the moving control unit 40 stops the driving control of the traveling motors 132a and 132b, it determines that the moving action of the moving body 10 has stopped. When the determination unit 32 determines that the moving action of the moving body 10 has stopped ("Yes" in step S52), the process transfers to step S56. When the determination unit 32 determines that the moving action of the moving body 10 has not stopped (is moving) (No in step S52), the process transfers to step S53.

[0167] Next, the judgment unit 32 judges whether an intersection is detected around the mobile body 10 (step S53). Specifically, the image recognition processing unit 41 performs image recognition processing on the photographic image obtained by the camera 111 such as a special photographic device or a general photographic device. Then, the judgment unit 32 judges whether an intersection is detected in the photographic image through the processing of the image recognition processing unit 41. For example, at a characteristic location such as an intersection, by obtaining a special image captured in all directions, the mobile body 10 can obtain images seen from different directions on the moving path at one time regardless of the moving direction. Here, the intersection is an example of a specific subject detected by the image recognition processing unit 41. In addition to being an intersection, the specific subject can also be an obstacle on the moving path or around the moving path of the mobile body 10, or a sign or signal set in a location. Different types of mobile bodies 10 have different specific subjects when the mobile body 10 moves on the road or when the mobile body 10 such as a drone is flying, and the information of the specific subject that needs to be detected is preset in the mobile body 10.

[0168] When the determination unit 32 determines that an intersection is detected around the moving body 10 (Yes in step S53), the process moves to step S56. When the determination unit 32 determines that no intersection is detected around the moving body 10 (No in step S52), the process moves to step S54.

[0169] Next, the determination unit 32 determines whether the current position of the mobile body 10 is close to the destination candidate registered in the destination candidate management DB 3001 (step S54). Specifically, the determination unit 32 refers to the location information of the destination candidate stored in the destination candidate management DB 3001, and determines whether there is a destination candidate close to the current position of the mobile body 10 acquired by the location information acquisition unit 35. Here, the determination of whether it is close to the destination candidate is made, for example, based on the difference between the preset location of the destination candidate and the current location of the mobile body 10. The determination of whether it is close to the destination candidate can also be made by performing image recognition on a photographic image corresponding to the destination candidate (destination candidate image) and a photographic image of the current location of the mobile body 10.

[0170] When the determination unit 32 determines that the current position of the mobile body 10 is close to the destination candidate registered in the destination candidate management DB 3001 (Yes in step S54), the process moves to step S56. When the determination unit 32 determines that the current position of the mobile body 10 is far from the destination candidate registered in the destination candidate management DB 3001 (No in step S54), the process moves to step S55.

[0171] Next, the determination unit 32 determines whether the direction indicated by the manual operation instruction sent by the display device 60 has changed (step S55). Specifically, the determination unit 32 determines whether the moving direction indicated in the manual operation instruction has changed based on the manual operation instruction received in step S33. The determination unit 32 may also determine that the moving direction has changed when, for example, the manual operation instruction indicates a characteristic operation. The characteristic operation includes, for example, a certain amount of steering operation or accelerator or brake operation performed along with movement at an intersection or a turn.

[0172] When the determination unit 32 determines that the operation direction indicated by the manual operation instruction sent from the display device 60 has changed (Yes in step S55), the process transfers to step S56. When the determination unit 32 determines that the operation direction indicated by the manual operation instruction sent from the display device 60 has not changed (No in step S55), the process ends.

[0173] Then, in step S56, the position information acquisition unit 35 acquires position information indicating the current position of the mobile body 10 using the satellite positioning system 114 or the like. The destination candidate acquisition unit 36 ​​acquires a photographic image taken by a special photographing device at the current position of the mobile body 10 as a destination candidate image indicating a destination candidate of the mobile body 10 (step S57).

[0174] Then, the storage and reading unit 49 stores the candidate destination data including the position information acquired in step S56 and the photographic image acquired in step S57 in the candidate destination management DB 3001 (see Figure 7) (step S58). The storage reading unit 49 registers the acquired position information and photographic image as destination candidate data respectively associated with the candidate ID for identifying the destination candidate.

[0175] In this way, the mobile body 10 automatically photographs the surroundings of the mobile body 10 using the photographing device 12 mounted on the mobile body 10 according to the prescribed judgment criterion of the moving state of the mobile body 10, and the photographic images representing the candidate moving destinations of the mobile body 10 can be registered in advance. The judgment criterion of the judgment unit 32 shown in steps S51 to S55 is not limited to this, and can be appropriately set according to the manual operation content of the mobile body 10 and the surrounding environment of the mobile body 10. The judgment criterion of the judgment unit 32 includes, for example, the state change of the mobile body 10 that the operator is expected to recognize when the operator selects the destination of the mobile body 10, or the condition of the place or the environmental change in the place.

[0176] Destination setting process

[0177] Then use Figures 14 to 17 , a process of setting the destination of the mobile body 10 using the destination candidates registered by the above-mentioned process will be described. Fig.14 This is a sequence diagram of an example of the moving body destination setting process.

[0178] First, the reception unit 52 of the display device 60 receives the selection of the destination setting button 290 in response to the operator's input operation on the operation screen 200 (step S71 ).

[0179] Next, the transceiver 51 transmits a destination candidate acquisition request to the mobile body 10 (step S72), requesting acquisition of data indicating destination candidates. The destination candidate acquisition request includes a location ID for identifying the location of the mobile body 10. The transceiver 31 of the mobile body 10 transmits the destination candidate acquisition request transmitted by the display device 60.

[0180] Next, the storage reading unit 49 searches the destination candidate management DB 3001 (see FIG. 10 ) using the location ID indicated by the destination acquisition request received in step S72 as a search key. Figure 7 ), thereby reading the destination candidate data associated with the same location ID as the received location ID (step S73).

[0181] Then, the transceiver 31 transmits the destination candidate data read in step S73 to the display device 60 that issued the request (step S74). The destination candidate data includes a combination of a plurality of candidate IDs, photographic images, and position information read in step S73. Accordingly, the transceiver 51 of the display device 60 receives the destination candidate data transmitted from the mobile body 10, thereby acquiring the destination candidate data.

[0182] Next, the display control unit 53 of the display device 60 displays the selection screen 400 included in the destination candidate data received in step S74 on a display unit such as the display 506 (step S75). Specifically, the image processing unit 57 generates the selection screen 400 based on the received destination candidate data. Then, the display control unit 53 displays the selection screen 400 generated by the image processing unit 57. The generation process of the selection screen 400 may also be performed by the image processing unit 47 of the mobile body 10. In this case, the image processing unit 47 generates the selection screen 400 based on the destination candidate data read in step S73. Then, in step S74, the transceiver 31 sends the screen data of the selection screen 400 including the destination candidate data generated by the image processing unit 47 to the display device 60.

[0183] Fig.15 and Fig.16 This is a diagram of an example selection screen. Fig.15 and Fig.16 The selection screen 400 shown is a display screen that allows the operator to select a destination for the moving object 10 .

[0184] The selection screen 400 includes an image display area 410 on which a plurality of photographic images (destination candidate images) 415 (415a to 417f) included in the destination candidate data received in step S74 are selectively displayed. The selection screen 400 also includes an OK button 430 to be pressed when the selection is completed, and a cancel button 435 to be pressed when the selection process is terminated. The operator selects the photographic image 415 displayed on the image display area 410 using an input device such as a pointing device 512.

[0185] The accepting unit 52 of the display device 60 accepts selection of a destination candidate image for setting the area indicated by the photographic image as the destination, when the operator selects the photographic image 415 displayed in the image display area 410 (step S76 ). Fig.16 The selection screen 400 shown in FIG. 4 shows a state in which the operator has selected, for example, photographic images 415a, 415b, and 415c. When the photographic image 415 displayed in the image display area 410 is selected, as shown in FIG. Fig.16 As shown, the selected photographic images are displayed in the selection result display area 420 of the selection screen 400. The selection result display area 420 displays the selected photographic images in the order (No. 1 to No. 3) selected by the operator. The operator selects, as the moving destination of the moving body 10, for example, a photographic image of an area showing a location where the moving body 10 is to perform a predetermined task, or a photographic image of an area showing a branch point on a path such as an intersection within a location when the moving body 10 is to move to the location where the predetermined task is to be performed.

[0186] In this way, the display device 60 uses the photographic image 415 (destination candidate image) representing the candidate moving destination of the mobile body 10 taken in advance by the mobile body 10, and selects the area represented by the photographic image 415 as the moving destination of the mobile body 10. Compared with the previous method of allowing the operator to specify a character string such as location information or keywords, the operability of selecting the moving destination of the mobile body 10 can be improved.

[0187] Next, after the operator presses the OK button 430 upon receiving the destination candidate image selection in step S76, the autonomous movement request information generating unit 56 of the display device 60 generates autonomous movement request information (step S77). The autonomous movement request information includes the candidate ID associated with the photographic image 415 selected in step S76 and information indicating the order in which the selection was received in step S76.

[0188] Next, the transceiver 51 transmits the autonomous movement request information generated in step S77 to the moving object 10 (step S78 ). In this way, the transceiver 31 included in the moving object 10 receives the autonomous movement request information transmitted from the display device 60 .

[0189] Then, the moving object 10 starts the autonomous movement process according to the autonomous movement request information received in step S78 (step S79).

[0190] Autonomous mobile processing

[0191] Next reference Fig.17 , the autonomous movement processing of the moving body 10 in step S79 is described in detail. Fig.17 This is a flowchart of an example of autonomous movement processing of a moving object.

[0192] First, upon receiving the autonomous movement request information in step S78 , the mode setting unit 42 of the moving object 10 sets the operation mode of the moving object 10 to the autonomous movement mode (step S91 ).

[0193] Next, the path information generating unit 37 generates path information indicating the autonomous movement path of the mobile body 10 based on the autonomous movement request information received in step S78 (step S92). Specifically, the path information generating unit 37 generates a movement path for the mobile body 10 to autonomously move to the area indicated by the photographic image 415 in the order in which the operator selects the photographic image 415, based on the candidate ID indicated by the received autonomous movement request information and the selection order of the photographic image 415 determined by the candidate ID. The path information generating unit 37 is not limited to the selection order of the photographic image 415, and may generate path information based on the information of the photographic position of the photographic image 415 stored in the destination candidate management DB 3001, for example, in such a manner that the distance or movement time of the movement path based on the selected photographic image 415 is shortened.

[0194] Then, the route information management unit 38 associates the route information generated in step S92 with the location ID received in step S72 and stores it in the route information management DB 3002 (see Figure 8 ) (step S93).

[0195] Next, the position information acquisition unit 35 acquires position information indicating the current position of the moving object 10 (step S94 ).

[0196] The mobile destination setting unit 39 sets the mobile destination of the mobile body 10 based on the current position of the mobile body 10 obtained by the position information acquisition unit 35 and the path information generated in step S92 (step S95). Specifically, the mobile destination setting unit 39 sets the position indicating the destination to be traveled to next among multiple destinations, for example, determined by the candidate ID indicated by the generated path information, as the mobile destination. For example, when autonomous movement starts from now, the mobile destination setting unit 39 sets the position indicated by the destination determined by the initial candidate ID indicated in the path information as the mobile destination. Then, the mobile destination setting unit 39 generates a moving path from the current position of the mobile body 10 obtained to the set mobile destination. The mobile path generation method of the mobile destination setting unit 39 adopts a method of connecting the current position and the mobile destination with a straight line, or a method of minimizing the moving time while avoiding obstacles using information on obstacles obtained from a photographic image or the state detection unit 34.

[0197] The mobile destination setting unit 39 is not limited to using the location information of the destination candidate (mobile destination) registered in the destination candidate management DB 3001 to generate the mobile route. It can also be performed by the image processing unit 47 on the photographic image taken by the photographic device 12 to find the location captured in the set photographic image 415 as the mobile destination.

[0198] Then, the movement control unit 40 causes the moving body 10 to move to the set moving destination via the moving path generated in step S92. In this case, the movement control unit 40 responds to the driving instruction of the autonomous movement processing unit 43 and causes the moving body 10 to move autonomously (step S96). The autonomous movement processing of the autonomous movement processing unit 43 will be described in detail below.

[0199] Then, if the moving body 10 reaches the final destination (Yes in step S97), the movement control unit 40 ends the process. Otherwise, the movement control unit 40 repeats the process from step S94 and continues the autonomous movement process until the moving body 10 reaches the final destination (No in step S97).

[0200] In this way, the mobile body 10 can autonomously move with the position captured in the photographic image 415 selected by the operator as the destination of the mobile body 10. Furthermore, when the mobile body 10 operates in the autonomous movement mode, it can autonomously move using the generated path information and the learning data learned in the manual operation mode.

[0201] <Deviation from the movement path based on the local map>

[0202] Next, the process of comparing the sensor values ​​of multiple local maps and external sensors corresponding to the route point 8 during the guided driving, calculating the deviation from the moving path, and integrating the deviations of multiple external sensors is described. First, the topological map is described.

[0203] Fig.18 It is a conceptual diagram of a topological map. As described above, in this embodiment, the mobile body 10 has N external sensors (GNSS, ranging sensors (LiDAR), cameras, etc.) and internal sensors (IMU or encoders, etc.), which can calculate the relative movement amount (odometer) of the mobile body 10. The mobile body 10 is an autonomous mobile system of a guided iterative type using a topological map-style map. The so-called topological map refers to a map in the form of a graph consisting of a node set V and an edge set E. The element eij of the edge set E is defined between two nodes vi and vj, and has a relative position relationship between vi and vj (an example of relative position information).

[0204] The information generation unit 46 stores a local map (local map) generated in the node vi using the sensor values ​​of the external sensor and the internal sensor in the node vi, which is an element of the node set V. In the present embodiment, "information capable of calculating the deviation (position and posture) between the place passed during guided driving and the place passed during autonomous movement" is represented as a "map". For example, when a two-dimensional grid map (an H×W image storing 1 in the grid where obstacles exist and 0 in the grid where obstacles do not exist) produced by LiDAR during guided driving and a scanning point cloud of LiDAR during autonomous movement are given, the position and posture of the origin (=the place passed during guided driving=node) of the two-dimensional grid map as seen from the mobile body 10 in autonomous movement can be obtained by optimization calculation, etc. Therefore, when using an external sensor such as LiDAR, the two-dimensional grid map can be called a "map". On the other hand, when the position (latitude, longitude), azimuth observed by GNSS during guided driving and the position (latitude, longitude), azimuth observed by GNSS during autonomous movement are given, since the position and posture of the places (nodes) passed during the guided driving as seen from the mobile body 10 can be calculated, in this embodiment, the "position (latitude, longitude), azimuth observed by GNSS during guided driving" is also expressed as a "map".

[0205] The autonomous movement of the mobile body 10 consists of two stages: a "route guidance stage" and an "autonomous movement stage". The "route guidance stage" corresponds to guided driving, and the operator manually operates the mobile body 10 through a controller to make the mobile body 10 travel on the moving path to be autonomously moved. The "autonomous movement stage" corresponds to autonomous movement, and the mobile body 10 moves autonomously in a manner that tracks the moving path moved in the "route guidance stage" (moves on or along the moving path).

[0206] Fig.19 This is a block diagram of movement control according to the present embodiment using external sensors and a local map. Fig.19 The mobile control in the autonomous mobile stage is shown. (1) First, the first local map and the second local map generated by the sensor values ​​of the first external sensor and the second external sensor in the path guidance stage are stored in the storage device. (2) In the autonomous mobile stage, the first external sensor and the second external sensor detect the sensor values ​​respectively. The infield sensor also detects the sensor value, and the position and posture are calculated by accumulating the sensor value of the first external sensor or the second external sensor. (3) The mobile body 10 compares the sensor value of the first local map and the first external sensor to calculate the deviation from the mobile path using the first external sensor (deviation from the node). In addition, the mobile body 10 compares the sensor value of the second local map and the second external sensor to calculate the deviation from the mobile path using the second external sensor (deviation from the node). As described later, the deviation from the node represents the position and posture of the route point 8 as seen from the mobile body 10. The mobile body 10 calculates the reliability of the first external sensor based on the first local map and the first external sensor, and calculates the reliability of the second external sensor based on the second local map and the second external sensor. (4) The mobile body 10 weights the position and posture detected by the mobile body 10 with reliability, and integrates the position and posture using the first external sensor and the position and posture using the second external sensor. Since the integrated position and posture deviate from the moving path, the integration of the position and posture is equivalent to the integration of the deviation from the moving path. (5) The mobile body 10 is controlled using the position and posture integrated from the positions and postures of multiple external sensors.

[0207] <Processing of the Path Guidance Phase>

[0208] refer to Fig. 20 , Fig.21 Describe the route indication phase in detail. Fig. 20 is a flowchart of the path guidance phase process. Fig.10 The process is performed in step S16.

[0209] First, the information generation unit 46 sets V and E to empty sets and initializes the topological map G (S101). The information generation unit 46 sets the sensor value of the kth external sensor or the local map generated using the external sensor to m. k At the beginning of the path guidance phase, the information generation unit 46 adds the initial node v0 to V. v0 stores all the m nodes at that time. k (=:m i k ).

[0210] The transceiver 31 receives a manual operation command for manually operating the moving body 10 (S102). The moving control unit 40 executes a moving process of the moving body 10 in response to a driving command from the manual operation processing unit 44 (S103).

[0211] The judgment unit 32 periodically judges whether the movement of the moving body 10 reaches a threshold, for example, whether the moving body 10 moves more than a certain distance or rotates more than a certain angle (S104). Here, the threshold can be reasonably set according to the processing load, memory usage, position and path accuracy, etc., and when the threshold is a certain distance, for example, it is 1 meter to several meters.

[0212] If the answer is yes in step S104, the information generation unit 46 adds the new node v1 to V and adds the new edge e 0，1 Add E (S105). 0，1 The relative movement and rotation obtained by the internal sensor when adding from v0 are stored in . Figure 2 The position of the node vi is referred to as route point 8. The information generation unit 46 increases the node number i by 1 (S106). For example, during the period from the operation of the operator to the end of the route guidance phase (No in S107), the information generation unit 46 adds a node v whenever the moving body 10 moves more than a certain distance or rotates more than a certain angle. i and edge i-1，i .

[0213] The node set V and edge set E can be expressed as G = {V, E} V = {v0, ..., v n}E={e 0，1 ,…,e n-1，n} i ={m i 0 ,…,m i N}e i-1，i ={Δx i-1，i , Δy i-1，i , Δθ i-1，i}.

[0214] <Processing during autonomous mobility>

[0215] Fig.21 is a flowchart of the autonomous mobility phase. Fig.14 The process is performed in step S79.

[0216] The moving body 10 is placed at the moving start point before the autonomous moving starts. The sensor value of the kth external sensor in the autonomous moving stage is set to s k The specific calculation method of steps S201 to S203 and S206 will be described later.

[0217] S201: The deviation calculation unit 43b calculates the sensor value s of the kth external sensor when the moving body 10 moves autonomously. k and a certain node v of the mobile body 10 during the guided driving i The kth local map w saved in i k , calculate the node v seen from mobile body 10 i The deviation of the position and posture of the node v as seen from the moving body 10. i The position and posture of the moving body 10 are based on the node v as the origin. i Therefore, the node v based on the position and posture of the moving body 10 is i The position and posture of the node v i The position estimation unit 43c estimates the obtained deviation as a node v based on the position and posture of the moving body 10. i Position and posture x^ i k ={x^ i k , y^ i k , θ^ i k Here, "^" is a hat, but due to font limitations, it is not displayed directly above x, y, and θ but in the upper right corner. Set the computing device that performs this calculation to F, and use "computing device F k :(s k , w i k )→x^ i k " indicates. On the other hand, due to the local map m i k and the sensor value s of the kth external sensor k The accuracy of x^ is unknown, so as follows, the reliability of i k to integrate.

[0218] S202: The integration processing unit 43d calculates the relative value of the computing device F k The estimated reliability of b i k The computing device for this operation is B k , using "computing device B k :(s k , m i k ) → b i k "express.

[0219] S203: The integration processing unit 43d integrates the node v estimated by each external sensor according to the reliability. i The position and posture of x i k Perform weighting and integration. Assume that the computing device for performing the operation is I, and use "computing device I: (x^ i 0 , …, x^ i N , b i 0 , …, b i N )→x^ i " indicates that node v i The position and posture of x i k is the position and posture of the mobile object 10 and the local map m i k The deviation of the origin (node), so the node v i The position and posture of x i k The integration of includes the integration of deviations.

[0220] S204: The autonomous movement processing unit 43 generates a signal according to the node v i The position and posture of x i The relative position {Δx j-1，i , Δy j-1，j , Δθ j-1，j}, calculate the guided moving path in the moving body coordinate system (i.e., each node v j The position and posture of the calculation). Let the calculation device that performs the calculation be T, and use "calculation device T: x^ i , e 0，1 ,…,e n-1，n )→(x^0,…,x^ n )". The obtained moving path becomes the one passing through the node v i The position and posture of the origin node v iThus, whenever the moving body 10 passes through the node v i When , we can recalculate the node v i Later node v i Therefore, the moving body 10 can move on a moving path with a small deviation from the guided moving path.

[0221] S205: The calculation unit 43a converts the integrated node v i The position and posture of the internal sensor are taken as the origin, and the position and posture of the sensor value of the internal sensor and the sensor value s based on multiple external sensors are accumulated. k The position and posture of the moving body 10 are applied to a Kalman filter, etc. to obtain the position and posture. The autonomous movement processing unit 43 causes the moving body 10 to move along the guided movement path (x^0, ..., x^ n ) moves in the same way as above.

[0222] S206: The autonomous movement processing unit 43 calculates the position and posture of the moving body 10 relative to the guided movement path (x^0, ..., x^ n ) in the node v i Position and posture x^ i Whether it is close to the threshold, it is judged whether the node number i of the position and posture estimation object has advanced to the next one. Let the judgment device for making this judgment be J, and use "judgment device J: (x^ i , (x^0, ..., x^ n ))→0 (do not advance i to the next one) or 1 (i advances to the next one)".

[0223] S207: The movement control unit 40 sets i=0 when the autonomous movement starts, and increments i by 1.

[0224] S208: The movement control unit 40 repeats Fig.21 The process continues until i is greater than the number of nodes n. The number of nodes n is the number of nodes saved until the final destination.

[0225] pass Fig.21 The guided moving path in the moving body coordinate system is obtained by steps S201 to S204. The moving body 10 moves along the guided moving path (S205). Then, the moving body 10 repeatedly performs the calculations of steps S201 to S205, and sets i←i+1 according to the result of step S206. By repeating this process, the moving body 10 can move along the indicated moving path.

[0226] Processing example when using GNSS and 2D-LiDAR

[0227] A specific example of steps S201 to S203 and S206 in the autonomous movement phase is described below.

[0228] Assume m i 0 The position and posture in the GNSS coordinate system and the positioning status (good / bad) estimated based on the GNSS positioning history and the mileage measurement of the internal sensor, m i 1 It is a two-dimensional occupancy grid map of the mobile body coordinate system generated from the scan point cloud history of 2D-LiDAR and the mileage measurement of the internal sensor. On the other hand, let s 0 It is the position, posture and positioning status in the GNSS coordinate system estimated based on the GNSS positioning history and the mileage measurement of the internal sensor. 1 It is the scanning point cloud of 2D-LiDAR.

[0229] S201: F0 by comparing m i 0 and 0 With the position and posture of the GNSS positioning result, calculate the node v seen from the mobile body 10 i Position and posture x^ i 0 . Let m i 0 For node v i The position and attitude of the GNSS coordinate system (x i,GNSS ,y i,GNSS ,θ i,GNSS ), s 0 is the position and posture (x GNSS ,y GNSS ,θ GNSS ). At this time, the node v seen from the moving object 10 i The position and posture of can be obtained by equation (1) and equation (2). The calculation using equation (1) and equation (2) is equivalent to F0.

[0230]

[0231] From equations (1) and (2), we can see that the node v seen by the mobile object 10 is i The position and posture are relative to the node v in the GNSS coordinate system i The position and posture (x i,GNSS ,y i,GNSS ,θ i,GNSS ) is a deviation of the position and posture of the mobile body 10 in the GNSS coordinate system (an example of a first position and posture).

[0232] Next, F1 passes m i 1 With a two-dimensional occupancy grid map and s 1 The matching calculation of the 2D-LiDAR scanning point cloud is performed to calculate the node v seen from the mobile body 10 i Position and posture x^ i 1 (An example of the second position and posture). Let m i 1 For node v i The two-dimensional occupancy grid map with s as the origin, 1 is the LiDAR scanning point cloud of the mobile body 10 during autonomous movement {(x j ,y j )j=1,……,N} (N is the number of scanning points). At this time, when the autonomous movement processing unit 43 moves the two-dimensional occupied grid map in the xy plane, it finds a movement method that maximizes the value of the sum of all the grid values ​​corresponding to the positions of each point in the scanning point cloud (the grid with obstacles has 1) (i.e., the scanning point cloud matches the map). In this way, the m seen from the moving body 10 can be obtained. i 1 The origin (i.e., the node v on the two-dimensional occupancy grid map) i This series of processing is equivalent to F1.

[0233] Fig. 22 It is a schematic diagram of the matching calculation between the two-dimensional occupancy grid map and the scanning point cloud of 2D-LiDAR. Fig. 22 (a) shows a two-dimensional occupancy grid map 93 before matching and a 2D-LiDAR scan point cloud 94. Grids where obstacles exist in the two-dimensional occupancy grid map 93 are indicated by oblique lines. The 2D-LiDAR scan point cloud 94 shows the direction and distance of the obstacle.

[0234] Fig. 22 (b) is a state where the two-dimensional occupied grid map 93 is moved to make the grid with obstacles and the 2D-LiDAR scanning point cloud 94 optimally match. When the two-dimensional occupied grid map 93 is moved, the movement amount and rotation amount of the two-dimensional occupied grid map 93 in the x and y directions deviate. Fig. 22 In the state (b), the mobile body 10 should be located at the origin 70 (= node v i ), so the movement and rotation amounts in the x and y directions of the two-dimensional occupied grid map 93 are the node v seen from the moving body 10. i Position and posture x^ i 1 .

[0235] S202: b0 only in m i0 and 0 If all GNSS positioning states are "good", it is 1; otherwise, it is 0. i 1 The number of obstacles in the two-dimensional occupied grid map is large enough, and s 1 If the number of scan points in the scan point cloud 94 is sufficient, it is 1, otherwise it is 0. b0 and b1 are not limited to 1 or 0, but can also be values ​​corresponding to the accuracy of the local map and sensor value. i 0 and 0 The total number of GNSS positioning states is s 0 , the number of obstacles on the two-dimensional occupied grid map and the number of scan points of the scan point cloud 94 are evaluated in N stages, and the total is b1.

[0236] S203: If bi0 or bi1 is not 0, then I(x^ i 0 ,x^ i 1 ,b i 0 ,b i 1 )=(b i 0 x^ i 0 +b i 1 x^ i 1 ) / (b i 0 +b i 1 ).

[0237] In b i 0 =b i 1 =0, the estimated result of the position and posture of the previous node vi and the position and posture estimated by the odometer measurement of the internal sensor are used as the estimation result (dead reckoning).

[0238] S206: For the x^ integrated in step S203 i ={x^ i ,y^ i ,θ^ i}, if with √(x^ i 2 +y^ i 2 ) is greater than the threshold value, and is 1 if it is less than the threshold value. i ,y^i It is the current position of the moving object 10 obtained by the calculation unit 43a.

[0239] Processing examples when using GNSS and 3D-LiDAR

[0240] Next use Fig.21 The flowchart of the present invention illustrates the case of processing a scanned point cloud of "3D-LiDAR" instead of "2D-LiDAR". The following is a detailed example of steps S201 to S203 and S206 in the autonomous movement phase, which are different from "2D-LiDAR".

[0241] Assume m i 0 The position and posture estimated from the GNSS positioning history and the mileage measurement of the internal sensor, as well as the positioning status (good / bad), are assumed to be i 1 It is a three-dimensional point cloud map of the mobile body coordinate system generated from the scanning point cloud history of 3D-LiDAR and the mileage measurement of the internal sensor. 0 It is the position, posture and positioning status estimated based on the GNSS positioning history and the mileage measurement of the internal sensor. 1 It is the scanning point cloud of 3D-LiDAR.

[0242] S201: F0 by comparing m i 0 and 0 With the position and posture of the GNSS positioning result, calculate the node v seen from the mobile body 10 i Position and posture x^ i 0 . m i 0 For node v i The position and attitude of the GNSS coordinate system (x i,GNSS ,y i,GNSS ,θ i,GNSS ), s 0 is the position and posture (x) of the GNSS coordinate system of the mobile object 10 during autonomous movement. GNSS ,y GNSS ,θ GNSS ). At this time, the node v seen from the moving object 10 i The calculation process of the position and posture of is obtained by the above-mentioned formula (1) and formula (2). The calculation based on formula (1) and formula (2) is equivalent to F0.

[0243] From equations (1) and (2), we can see that the node v seen by the moving object 10 is i The position and posture of the mobile body 10 in the GNSS coordinate system are the position and posture of the mobile body 10 in the GNSS coordinate system relative to the node v in the GNSS coordinate system.i The position and posture (x i,GNSS ,y i,GNSS ,θ i,GNSS ) (an example of a first position and posture).

[0244] Next, F1 passes through m i 1 The 3D point cloud map and s 1 The matching calculation of the 3D-LiDAR scanning point cloud is performed to calculate the node v seen from the mobile body 10. i Position and posture x^ i 1 (An example of the second position and posture). Let m i 1 is the node v i The 3D point cloud map with s as the origin. 1 is the 3D-LiDAR scanning point cloud of the mobile body 10 during autonomous movement {(x j ,y j )j=1,……,N}(N is the number of scan points). At this time, the autonomous mobile processing unit 43 finds a moving method that minimizes the error with the scanned point cloud when moving the three-dimensional point cloud map through NDT matching (see non-patent document 1: P.Biber&W.Strasser(2003), https: / / ieeexplore.ieee.org / document / 1249285) etc. In this way, the origin of mi 1 seen from the mobile body 10 (i.e., the position and posture of the node vi on the two-dimensional occupied grid map) can be obtained. This series of processing is equivalent to F1.

[0245] S202: b0 only in m i 0 and 0 If all GNSS positioning states are "good", it is 1; otherwise, it is 0. i 1 If the number of obstacles in the field of view of the 3D-LiDAR on the three-dimensional point cloud map is sufficient, the value is 1, and if not, the value is 0. However, when calculating the number of obstacles, the autonomous movement processing unit 43 performs the following processing.

[0246] Fig.23 This is a flow chart of the evaluation of b0 based on the number of obstacles. Fig.23As shown, the autonomous movement processing unit 43 performs pre-processing at the beginning of autonomous movement, reads in the point cloud map (step S1), removes the ground point cloud when the point cloud map is read in (step S2), and then transforms it into a two-dimensional occupied grid map (step S3). The ground removal is implemented as follows. The inclination of the moving body 10 is detected by the inner boundary sensor, and the ground plane is defined. The distance from the ground plane to each point on the point cloud map is calculated, and the point within the threshold is judged as the ground and removed.

[0247] During autonomous movement, the autonomous movement processing unit 43 calculates the relative position and posture on the above-mentioned two-dimensional occupied grid map as seen from the mobile body 10 (step S4). The autonomous movement processing unit 43 calculates the number of obstacles in the 3D-LiDAR field of view based on the number of occupied grids (step S5). The autonomous movement processing unit 43 standardizes the number of obstacles using the outer perimeter of the above-mentioned two-dimensional occupied grid map (step S6). The autonomous movement processing unit 43 evaluates b0 using the standardized number of obstacles (step S7). b0 is 1 when the standardized number of obstacles > the threshold, and 0 otherwise. The purpose of standardizing the number of obstacles is to eliminate the dependence on the range (size) of the point cloud map. b0 and b1 are not limited to 1 or 0, and can also take values ​​corresponding to the accuracy of the local map and sensor values. For example, m is evaluated at N levels respectively. i 0 and 0 The GNSS positioning status possessed is summed up as b0, and the number of obstacles on the three-dimensional point cloud map and the number of scanning points of the scanning point cloud 94 are evaluated using N levels, summed up as b1.

[0248] S203: If b i 0 or b i 1 is not 0, then let I(x^ i 0 ,x^ i 1 ,b i 0 ,b i 1 )=(b i 0 x^ i 0 +b i 1 x^ i 1 ) / (b i 0 +b i 1 ).

[0249] When b i 0 =bi 1 = 0, the node b will be i 0 The estimated results of the position and posture of the vehicle and the estimated position and posture by the mileage measurement of the internal sensor are used as the estimation result (radar calculation).

[0250] S206: For the x^ integrated in step S203 i ={x^ i ,y^ i ,θ^ i}, if the difference between the ratio and √(x^i2+y^i2) is greater than the threshold, it is 0, if it is less, it is 1. i ,y^ i It is the current position of the moving object 10 obtained by the calculation unit 43a.

[0251] <Main Effect>

[0252] The mobile body 10 of this embodiment has independent local maps of sensors in the form of topological maps. For the independently estimated own positions and postures, the reliability of the position estimation is calculated based on the sensor values ​​during guided driving and autonomous movement, and weighted according to the reliability to integrate the final position and posture. Therefore, there is no need to modify the map by manual operation to match the position estimation results of each sensor, and even if the positioning accuracy of a certain sensor is reduced during the path guidance stage, high-precision autonomous movement can be performed.

[0253] Reliability calculation using 3D-LiDAR has the following two advantages.

[0254] Since the reliability of map matching (=the number of obstacles around the mobile body 10) is calculated using the map point cloud instead of the real-time point cloud, the pre-processing of the reliability calculation can be performed in advance. In other words, the real-time calculation load is small.

[0255] When observing the real-time point cloud for reliability calculation, the reliability may increase due to the temporary passage of obstacles (vehicles, people, etc.) around the moving body 10, and an erroneous position estimation result may be used. However, there is no need to worry about this problem when using the map point cloud.

[0256] [Second embodiment]

[0257] In this embodiment, an embodiment in which the display device 60 is an HMD (head mounted display) is described. HMD is a type of display having two displays with a goggle-like shape, which are respectively mounted on the head, and most of the display covers both eyes. HMD realizes stereoscopic display by displaying parallax images in front of the left and right eyes. Stereoscopic display is not necessarily possible. In addition to the type of HMD that simply displays images in stereo, there are also types called VR goggles (Virtual Reality), AR goggles (Augmented Reality), and MR goggles (Mixed Reality). VR goggles can display an artificial virtual world. AR goggles can read the real world in stereo and display imaginary information overlapping with the real world. MR goggles are an extension of AR goggles, and can three-dimensionally superimpose things that do not actually exist in the real world. In this embodiment, in order to allow the operator to view the image data captured by the mobile body 10 through the HMD, it is possible to consider using a simple HMD or AR goggles.

[0258] Fig.24 60 is an example of a hardware block diagram of the display device 60 when the display device 60 is an HMD. Fig.24 As shown, the display device 60 as the HMD has a computer structure and includes a CPU 201 , a ROM 202 , a RAM 203 , a peripheral connection I / F 205 , a display 207 , an operation unit 208 , a medium I / F 209 , a bus 211 , a speaker 212 , an electronic compass 218 , a gyro sensor 219 , and an acceleration sensor 220 .

[0259] The CPU 201 controls the overall operation of the HMD. The ROM 202 stores programs such as IPL for driving the CPU 201. The RAM 203 is used as a work area for the CPU 201.

[0260] The peripheral connection I / F 205 is an interface for connecting various external devices, such as the communication management server 3 and the earphone 6 with a microphone.

[0261] The display 207 is a display unit such as a liquid crystal or an organic EL (Electro Luminescence) that displays various images.

[0262] The operation unit 208 is an input device for selecting and executing various instructions such as various operation buttons, a power switch, a physical button, a line of sight operation circuit that detects the operator's line of sight and performs operations, selecting a processing object, moving a cursor, etc.

[0263] The medium I / F 209 controls reading or writing (saving) of data in a recording medium 209m such as a flash memory. The recording medium 209m includes a DVD, a Blu-ray Disc (registered trademark), and the like.

[0264] The speaker 212 is a circuit that converts an electrical signal into physical vibration to generate sounds such as music or voice. The electronic compass 218 calculates the orientation of the HMD based on the magnetic force of the earth and outputs orientation information. The gyro sensor 219 is a sensor that detects angular changes (roll angle, pitch angle, yaw angle) accompanying the movement of the HMD.

[0265] The acceleration sensor 220 is a sensor that detects acceleration in three-axis directions. The bus 211 is an address bus, a data bus, etc. for electrically connecting various components such as the CPU 201.

[0266] The display device 60 of this embodiment can detect the posture of the display device 60 based on the signal of the gyro sensor 219, so it can determine the direction the operator is looking at. The camera 111 can capture a full spherical image, a wide-angle image, a hemispherical image, etc., so the display device 60 can display the direction the operator is looking at on the display 207 of the HMD. Therefore, it is easier to see various directions than operating the area displayed on the location display area 210 using a pointing device 512 such as a mouse.

[0267] In addition, the position of the mobile body 10 when the operator looks at an arbitrary direction on the HMD is determined by the state detection unit 34. Therefore, the display device 60 can display annotations of buildings and the like located in the direction that the operator looks at from the position of the mobile body 10 by superimposing them on the image data transmitted from the mobile body 10. In this case, the system administrator or the like uses the environment map created by the mobile body 10 to set annotations related to the buildings.

[0268] Fig.25 An example of image data displayed by the display device 60 as the HMD is displayed. Fig.25 The current position 601 of the mobile body 10 and the building 602 named "laboratory building" photographed in the image data are displayed in the image data. The mobile body 10 receives the direction the operator is looking at from the display device 60, and uses the environment map to determine the building located in the direction the operator is looking at from the position of the mobile body 10. The photography control unit 33 overlays the current position and the building name on the photographed image data, and the transceiver unit 31 sends the image data to the display device 60. In addition to the building name 602, the contents of past accidents and phenomena that need to be observed can also be displayed as annotations. Therefore, it is easy for the operator to understand which direction he is looking at and what he is looking at.

[0269] <Main Effect>

[0270] In addition to the effects of the first embodiment or the second embodiment, the mobile body 10 of the present embodiment makes it easier for the operator to view image data captured by the mobile body 10 by configuring the display device 60 with an HMD.

[0271] <Other Application Examples>

[0272] The best mode for carrying out the present invention has been described above using the embodiments, but the present invention is not limited to these embodiments, and various modifications and substitutions can be made without departing from the spirit of the present invention.

[0273] For example, the present embodiment mainly integrates two positions and postures obtained by two external sensors, but three or more positions and postures obtained by more than one external sensor may be integrated.

[0274] The technology of this embodiment is not only effective in wide locations where the GNSS positioning accuracy is not necessarily high, but can also be well applied to narrow indoor environments or locations where the GNSS positioning accuracy is high.

[0275] As an example of an external sensor, location information based on UWB (Ultra-Wideband) can also be used. In the UWB method, a communication tag is installed on the mobile body 10, and sensors are set at intervals of 30m to 40m in the mobile environment. The location of the communication tag can be determined by receiving the radio waves emitted by the communication tag through two or more sensors.

[0276] Figure 6 The above are examples of division according to main functions in order to facilitate understanding of the processing of the mobile body 10 and the display device 60. The present invention does not limit the division method and name of the processing unit. The processing of the mobile body 10 and the display device 60 can also be divided into more processing units according to the processing content. It can also be divided into one processing unit to include more processing.

[0277] Each function of the above-described embodiments can be implemented by one or more processing circuits. The "processing circuit" herein includes a processor that is programmed by software to perform each function, such as a processor installed by a circuit, and an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), an existing circuit module, and other devices designed to perform the above functions.

[0278] Embodiments of the present invention bring great improvements to the capabilities and functionality of computers. These improvements enable operators to utilize computers that provide more efficient and robust interactions with the way information is stored and presented in information processing devices. In addition, embodiments of the present invention provide a better user experience by using a more efficient, powerful and robust user interface. Such a user interface can provide better human-computer interaction.

[0279] <Remarks>

[0280] [1] A mobile body capable of guided driving and autonomous movement, wherein the guided driving means that the mobile body memorizes the position and posture when it moves through a moving path according to manual operation of an operator, and the autonomous driving means that the mobile body moves through the memorized position in the memorized posture, characterized in that:

[0281] An information generating unit, for matching each node on the moving path with a plurality of external sensors, and generating calculation information for calculating the deviation between the node passed during the guided driving and the location passed during the autonomous driving;

[0282] a deviation calculation unit, configured to calculate the deviation for each of the external sensors according to the calculation information and the sensor value of the external sensor during the autonomous movement;

[0283] A position estimating unit for estimating the deviation obtained for each of the external sensors as a position and posture of each node on the moving path based on the position and posture of the moving body;

[0284] an integration processing unit, used for integrating the position and posture of each node on the moving path estimated by each of the external sensors; and

[0285] The autonomous movement processing unit is used to autonomously move through the movement path according to the position and posture of each node on the movement path integrated by the integration processing unit.

[0286] [2] The moving object according to [1], characterized in that:

[0287] Have one or more internal sensors to detect the amount and direction of movement,

[0288] The information generation unit further generates the calculation information using the sensor value of the internal sensor.

[0289] When the moving body is guided to move manually, the information generating unit stores the calculated information obtained by the plurality of external sensors and the internal sensors respectively in correspondence with the external sensors for each node on the moving path in the storage unit.

[0290] [3] The moving object according to [1] or [2], characterized in that:

[0291] The integration processing unit weights the position and posture of the node based on the position and posture of the moving body estimated for each external sensor according to the reliability of the calculated information and the reliability of the sensor value of the external sensor during autonomous movement, and integrates the position and posture of the node estimated for each external sensor.

[0292] [4] The moving object according to any one of [1] to [3], characterized in that:

[0293] It has an autonomous movement processing unit, which is used to calculate the position and posture of the node after any node based on the integrated position and posture of the node and the relative position information between the nodes included in the calculation information when the integration processing unit integrates the position and posture of the node estimated by each external sensor on any node during autonomous movement, and calculate the movement path with the position and posture of the moving body at any node as the origin.

[0294] [5] According to any one of [1] to [4], the plurality of external sensors are two or more of GNSS, 2D-LiDAR, and a camera capable of obtaining distance information.

[0295] [6] The moving object according to [5], characterized in that:

[0296] When the first external sensor is GNSS, the calculation information is the position and posture of the GNSS coordinate system; when the second external sensor is 2D-LiDAR, the calculation information is a two-dimensional occupancy grid map.

[0297] The position estimation unit estimates, for the first external sensor, the sensor value during autonomous movement relative to the position and posture of the GNSS coordinate system included in the calculation information, i.e., the deviation of the position and posture of the GNSS coordinate system, as the first position and posture of the node based on the position and posture of the moving body. For the second external sensor, the position estimation unit compares the two-dimensional occupancy grid map with the sensor value during autonomous movement, i.e., the scan point cloud, and estimates the deviation of the position and posture of the moving body relative to the origin of the two-dimensional occupancy grid map as the second position and posture of the node based on the position and posture of the moving body.

[0298] The integration processing unit integrates the first position and posture and the second position and posture.

[0299] [7] The mobile object according to any one of [1] to [4], characterized in that the plurality of external sensors are two or more of GNSS, 3D-LiDAR, and a camera capable of obtaining distance information.

[0300] [8] The moving object according to [7], characterized in that:

[0301] When the first external sensor is GNSS, the calculation information is the position and posture of the GNSS coordinate system; when the second external sensor is 3D-LiDAR, the calculation information is a three-dimensional point cloud map.

[0302] The position estimation unit estimates, for the first external sensor, the sensor value during autonomous movement relative to the position and posture of the GNSS coordinate system included in the calculation information, i.e., the deviation of the position and posture of the GNSS coordinate system, as the first position and posture of the node based on the position and posture of the moving body. For the second external sensor, the position estimation unit compares the three-dimensional point cloud map with the sensor value during autonomous movement, i.e., the scanned point cloud, and estimates the deviation of the position and posture of the moving body relative to the origin of the three-dimensional point cloud map as the second position and posture of the node based on the position and posture of the moving body.

[0303] The integration processing unit integrates the first position and posture and the second position and posture.

Claims

1. A mobile body capable of guided driving and autonomous movement, wherein the guided driving enables the mobile body to memorize the position and posture when it moves through a moving path according to manual operation of an operator, and the autonomous driving is the mobile body autonomously moving through the moving path, characterized in that: have: An information generating unit, for matching each node on the moving path with a plurality of external sensors, and generating calculation information for calculating the deviation between the node passed during the guided driving and the location passed during the autonomous driving; a deviation calculation unit, configured to calculate the deviation for each of the external sensors according to the calculation information and the sensor value of the external sensor during the autonomous movement; A position estimating unit for estimating the deviation obtained for each of the external sensors as a position and posture of each node on the moving path based on the position and posture of the moving body; An integration processing unit, used for integrating the position and posture of each node on the moving path estimated by each of the external sensors; as well as, The autonomous movement processing unit is used to autonomously move through the movement path according to the position and posture of each node on the movement path integrated by the integration processing unit.

2. The mobile body according to claim 1, characterized in that: Have one or more internal sensors to detect the amount and direction of movement, The information generation unit further generates the calculation information using the sensor value of the internal sensor. When the moving body is guided to move manually, the information generating unit stores the calculated information obtained by the plurality of external sensors and the internal sensors respectively in correspondence with the external sensors for each node on the moving path in the storage unit.

3. The mobile body according to claim 1 or 2, characterized in that: The integration processing unit weights the position and posture of the node based on the position and posture of the moving body estimated for each external sensor according to the reliability of the calculated information and the reliability of the sensor value of the external sensor during autonomous movement, and integrates the position and posture of the node estimated for each external sensor.

4. The moving object according to claim 1, characterized in that It has an autonomous movement processing unit, which is used to calculate the position and posture of the node after any node based on the integrated position and posture of the node and the relative position information between the nodes included in the calculation information when the integration processing unit integrates the position and posture of the node estimated by each external sensor on any node during autonomous movement, and calculate the movement path with the position and posture of the moving body at any node as the origin.

5. The moving object according to claim 1, characterized in that: The plurality of external sensors are two or more of GNSS, 2D-LiDAR, and a camera capable of obtaining distance information.

6. The moving object according to claim 5, characterized in that: When the first external sensor is GNSS, the calculation information is the position and posture of the GNSS coordinate system; when the second external sensor is 2D-LiDAR, the calculation information is a two-dimensional occupancy grid map. The position estimation unit estimates, for the first external sensor, the sensor value during autonomous movement relative to the position and posture of the GNSS coordinate system included in the calculation information, i.e., the deviation of the position and posture of the GNSS coordinate system, as the first position and posture of the node based on the position and posture of the moving body. For the second external sensor, the position estimation unit compares the two-dimensional occupancy grid map with the sensor value during autonomous movement, i.e., the scan point cloud, and estimates the deviation of the position and posture of the moving body relative to the origin of the two-dimensional occupancy grid map as the second position and posture of the node based on the position and posture of the moving body. The integration processing unit integrates the first position and posture and the second position and posture.

7. The moving object according to claim 1, characterized in that The plurality of external sensors are two or more of GNSS, 3D-LiDAR, and a camera capable of obtaining distance information.

8. The moving object according to claim 7, characterized in that: When the first external sensor is GNSS, the calculation information is the position and posture of the GNSS coordinate system; when the second external sensor is 3D-LiDAR, the calculation information is a three-dimensional point cloud map. The position estimation unit estimates, for the first external sensor, the sensor value during autonomous movement relative to the position and posture of the GNSS coordinate system included in the calculation information, i.e., the deviation of the position and posture of the GNSS coordinate system, as the first position and posture of the node based on the position and posture of the moving body. For the second external sensor, the position estimation unit compares the three-dimensional point cloud map with the sensor value during autonomous movement, i.e., the scanned point cloud, and estimates the deviation of the position and posture of the moving body relative to the origin of the three-dimensional point cloud map as the second position and posture of the node based on the position and posture of the moving body. The integration processing unit integrates the first position and posture and the second position and posture.

9. A method for estimating a position of a mobile body, wherein the mobile body can perform guided driving and autonomous driving, wherein the guided driving refers to the mobile body memorizing a position and a posture when the mobile body moves through a moving path according to manual operation of an operator, and the autonomous driving refers to the mobile body moving through the memorized position in the memorized posture, wherein the method for estimating a position is characterized by performing the following processing: information generation processing, corresponding each node on the movement path to a plurality of external sensors, and generating calculation information for calculating the deviation between the node passed during the guided driving and the location passed during the autonomous movement; Deviation calculation processing, for each of the external sensors, calculating the deviation according to the calculation information and the sensor value of the external sensor during the autonomous movement; Position and posture estimation processing, for each of the external sensors, estimating the obtained deviation as the position and posture of each node on the moving path based on the position and posture of the moving body; Integration processing, integrating the position and posture of each node on the moving path estimated by each of the external sensors; as well as, The autonomous movement processing autonomously moves through the movement path according to the position and posture of each node on the movement path integrated by the integration processing unit.

10. A computer program product, characterized in that The invention comprises computer instructions, which can enable a mobile body capable of guided driving that memorizes a position and a posture when moving through a moving path according to manual operation of an operator, and autonomously moving through the moving path to achieve the following functions: An information generating unit, for matching each node on the moving path with a plurality of external sensors, and generating calculation information for calculating the deviation between the node passed during the guided driving and the location passed during the autonomous driving; a deviation calculation unit, configured to calculate the deviation for each of the external sensors according to the calculation information and the sensor value of the external sensor during the autonomous movement; A position estimating unit for estimating the deviation obtained for each of the external sensors as a position and posture of each node on the moving path based on the position and posture of the moving body; An integration processing unit, used for integrating the position and posture of each node on the moving path estimated by each of the external sensors; as well as, The autonomous movement processing unit is used to autonomously move through the movement path according to the position and posture of each node on the movement path integrated by the integration processing unit.

11. A computer-readable storage medium, characterized in that: A computer program is stored therein which can enable the computer to realize the following functions: an information generating unit for corresponding each node on the moving path with a plurality of external sensors, and generating calculation information for calculating the deviation between the node passed during guided driving, wherein the guided driving makes the moving body memorize the position and posture when the moving body moves through the moving path according to the manual operation of the operator, and the autonomous driving is the moving body autonomously moving through the moving path; a deviation calculation unit, configured to calculate the deviation for each of the external sensors according to the calculation information and the sensor value of the external sensor during the autonomous movement; A position estimating unit for estimating the deviation obtained for each of the external sensors as a position and posture of each node on the moving path based on the position and posture of the moving body; An integration processing unit, used for integrating the position and posture of each node on the moving path estimated by each of the external sensors; as well as, The autonomous movement processing unit is used to autonomously move through the movement path according to the position and posture of each node on the movement path integrated by the integration processing unit.

12. A computer device comprising a memory storing a computer program and a processor, wherein: By executing the computer program by the processor, the computer device realizes the following functions: an information generating unit for corresponding each node on the moving path with a plurality of external sensors, and generating calculation information for calculating the deviation between the node passed during guided driving, wherein the guided driving makes the moving body memorize the position and posture when the moving body moves through the moving path according to the manual operation of the operator, and the autonomous driving is the moving body autonomously moving through the moving path; a deviation calculation unit, configured to calculate the deviation for each of the external sensors according to the calculation information and the sensor value of the external sensor during the autonomous movement; A position estimating unit for estimating the deviation obtained for each of the external sensors as a position and posture of each node on the moving path based on the position and posture of the moving body; An integration processing unit, used for integrating the position and posture of each node on the moving path estimated by each of the external sensors; as well as, The autonomous movement processing unit is used to autonomously move through the movement path according to the position and posture of each node on the movement path integrated by the integration processing unit.

Images