Mobile object control system and mobile object control method

Abstract

To provide a mobile body control system enabling high-performance self- position estimation and environmental map creation for a mobile body traveling on irregular ground.SOLUTION: A mobile body control system 100 comprises: a data association unit 112 which associates a point group of an acquired frame with an online environmental map 131 and calculates position posture information of a mobile body 200 and a frame index indicative of accuracy of the point group; a movement amount calculation unit 113 which calculates a movement amount of the mobile body 200; a key frame generation source detection unit 114 which detects a frame acquired at a prescribed timing as a key frame generation source; a frame selection unit 115 which selects a frame acquired past or before and after the timing and whose frame index meets a prescribed index value; and a movement amount correction unit 117 which corrects the movement amount of the mobile body 200 on the basis of a key frame generated by superposing the selected frames.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a mobile object control system and a mobile object control method that simultaneously estimates the self-position of a mobile object and creates an environmental map. [Background technology]

[0002] Mobile robots and automatic guided vehicles (AGVs) that move autonomously over sloping floors, uneven terrain with rubble, and large road undulations need to measure the surrounding environment in 3D, including road surface undulations, obstacles, etc. One known environmental measurement technology is SLAM (Simultaneous Localization and Mapping), which simultaneously creates an environmental map and estimates its own position based on measurement information of the surrounding environment.

[0003] For example, Non-Patent Document 1 proposes a SLAM technology that combines a ranging sensor and an image sensor. Meanwhile, Patent Document 1, which is a patent document, describes a computationally efficient SLAM technology that combines a ranging sensor and an image sensor. Patent Document 1 also describes a method for generating a new frame (integrated frame) by overlaying and integrating point clouds from multiple past frames when a certain amount of movement is detected or a certain amount of time has passed, in cases where the point clouds of data (frames) obtained from the sensor are sparse and inter-frame registration is difficult. This method constructs a point cloud for an integrated frame with high resolution, and aligns the point clouds of these integrated frames, making registration easier. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-117386 [Non-patent literature]

[0005] [Non-Patent Document 1] J. Zhang and S. Singh, "Visual-lidar Odometry and Mapping, Low-drift, Robust, and Fast," ICRA 2015, pp.2174-2181, 2015. Summary of the Invention [Problem to be solved by the invention]

[0006] However, because the position and orientation of a mobile vehicle traveling on rough terrain constantly changes, simply overlaying and integrating multiple past frames results in the point cloud of the integrated frame containing points with low registration accuracy. This leads to a deterioration in the registration accuracy when aligning the integrated frames, making it difficult to create an accurate environmental map. Furthermore, when traveling on rough terrain, a route must be planned taking into account the magnitude of the road surface undulations, and the vehicle must then travel based on that route, requiring recognition of three-dimensional, high-resolution road surface information. The present invention has been made in consideration of the above background, and aims to provide a mobile object control system and a mobile object control method that enable high-performance self-position estimation for a mobile object traveling on rough terrain. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems, the mobile object control system of the present invention includes a data acquisition unit that acquires frames that are point cloud data indicating objects in the environment surrounding a mobile object; an environmental map creation unit that generates an online environmental map that is a map of the surrounding environment based on the point cloud of the frames; a data association unit that associates the point cloud of the frames acquired by the data acquisition unit with the generated online environmental map and calculates position and orientation information of the mobile object in the surrounding environment and a frame index that indicates the accuracy of the point cloud; a movement amount calculation unit that calculates the movement amount of the mobile object based on the position and orientation information; a key frame generator detection unit that repeatedly detects frames acquired at a predetermined timing as key frame generators; a frame selection unit that selects frames that were acquired before the timing at which the key frame generator was detected or frames acquired before or after the detected timing, and whose frame index satisfies a predetermined index value; a key frame generation unit that generates a key frame by superimposing the frames selected by the frame selection unit; and a movement amount correction unit that corrects the movement amount of the mobile object based on the key frame. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide a mobile object control system and a mobile object control method that enable high-performance self-position estimation for a mobile object traveling on rough terrain. Problems, configurations, and effects other than those described above will become clear from the description of the following embodiments. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a configuration diagram of a moving body according to a first embodiment. [Figure 2] 1 is a configuration diagram of a moving body according to a first embodiment. [Figure 3] 1 is a functional block diagram of a mobile object control system provided in a mobile object according to a first embodiment. [Figure 4] FIG. 2 is a diagram for explaining a loop confinement correction technique according to the first embodiment. [Figure 5]3 is a flowchart of an online environment map creation process according to the first embodiment. [Figure 6] 4 is a flowchart of offline environment map creation processing according to the first embodiment. [Figure 7] FIG. 2 is a diagram showing a point cloud of an initial frame according to the first embodiment. [Figure 8] FIG. 4 is a diagram showing a point cloud generated by overlapping point clouds of multiple frames acquired at an initial position according to the first embodiment. [Figure 9] FIG. 10 is a functional block diagram of a mobile object control system provided in a mobile object according to a second embodiment. [Figure 10] FIG. 11 is a functional block diagram of a mobile object control system provided in a mobile object according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] <<Overview of the mobile control system>> A mobile object control system according to a mode (embodiment) for carrying out the present invention will be described below. The mobile object control system is mounted on a mobile object equipped with a ranging sensor and an actuator for movement. The mobile object control system periodically acquires point cloud data (also referred to as frames) of objects around the mobile object from the ranging sensor, matches point clouds acquired at different times to calculate the amount of movement, and creates an environmental map (an online environmental map, described later). Note that calculating the amount of movement of the mobile object is equivalent to performing self-localization estimation, which calculates (estimates) the position and orientation information of the mobile object. For this reason, the mobile object control system is a system that performs SLAM.

[0011] The mobile object control system also generates point cloud data (keyframes) by overlaying point clouds of frames acquired before a predetermined timing calculated based on the amount of movement and time, or before and after that timing, that satisfy a predetermined index. The index (frame index, described below) indicates the positional accuracy of the point cloud. An example of an index is the alignment score between the point cloud of a frame and an environmental map, and the mobile object control system generates keyframes by overlaying point clouds of frames whose score satisfies a predetermined value.

[0012] The mobile control system matches the point clouds of key frames to correct the amount of movement and then updates the environmental map. Since the accuracy of frames (point cloud data) that satisfy certain indicators is expected to be higher than that of other frames, the mobile control system can calculate the amount of movement with high accuracy (self-position estimation) and create a highly accurate environmental map.

[0013] <<Configuration of moving body>> FIG. 1 is a configuration diagram of a mobile body 200 according to the first embodiment. The mobile body 200 is a multi-legged robot that moves in a variety of environments (also referred to as the surrounding environment), including paved paths and roads, as well as uneven roads and rough terrain littered with objects, both indoors and outdoors. The means of movement of the mobile body 200 may be something other than legs. FIG. 2 is a configuration diagram of the mobile body 200 according to the first embodiment. As shown in FIG. 2, the means of movement may be caterpillars (endless tracks) or wheels. The mobile body 200 may move autonomously or may be moved by remote manual control.

[0014] The mobile object 200 further includes a distance measurement sensor 220, a communication unit 280 that transmits and receives communication data to and from external devices, and an actuator 270. The actuator 270 is intended for use as a drive unit for caterpillars, multi-legged vehicles, and drones and other flying objects. The mobile object 200 may be equipped with an arm to function as a work robot capable of performing work, and the structure of the mobile object can be changed depending on the purpose of the work.

[0015] The ranging sensor 220 can measure a three-dimensional point cloud of the surrounding environment, and is, for example, a LiDAR (Light Detection and Ranging) sensor, a TOF (Time of Flight) camera, a stereo camera, etc. Alternatively, the three-dimensional point cloud may be acquired by triangulation based on monocular movement using a method for detecting feature points based on an image acquired using an image sensor, such as the Lucas-Kanade method, and tracking the feature points using optical flow, or a feature point matching method based on the feature amounts of SIFT (Scale-Invariant Feature Transform) or ORB (Oriented FAST and Rotated BRIEF).

[0016] <Configuration of the mobile control system> 3 is a functional block diagram of a mobile object control system 100 provided in a mobile object 200 according to the first embodiment. The mobile object control system 100 is, for example, a computer, and includes a calculation unit 110 and a storage unit 130. The calculation unit 110 includes a microprocessor such as a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), or an application specific integrated circuit (ASIC). The calculation unit 110 includes a data acquisition unit 111, a data association unit 112, a movement amount calculation unit 113, a key frame generation source detection unit 114, a frame selection unit 115, a key frame generation unit 116, a movement amount correction unit 117, an environmental map creation unit 118, and a mobile object control unit 119. The moving object control unit 119 controls the actuator 270 to control the position and orientation of the moving object 200. Details of the functional units provided in the calculation unit 110 other than the moving object control unit 119 will be described later.

[0017] As described above, the mobile object control system 100 includes the mobile object control unit 119 that controls the position and attitude of the mobile object 200 .

[0018] ≪Storage section≫ The storage unit 130 includes storage devices such as a ROM (Read Only Memory), a RAM (Random Access Memory), a flash memory, etc. The storage unit 130 stores a frame information database 140, a key frame database 150, an online environment map 131, an offline environment map 132, and a program 138.

[0019] The frame information database 140 stores data relating to frames in association with the frames in addition to frames (point cloud data). The key frame database 150 stores data relating to key frames, which will be described later. The online environmental map 131 is a map showing objects (including obstacles) in the environment surrounding the moving body 200, and is created based on point cloud data. The offline environmental map 132 is a map showing objects in the environment surrounding the moving body 200, created based on key frames, which will be described later. The program 138 includes a description of the processing procedures of each functional unit that is a component of the calculation unit 110.

[0020] <<Calculation unit: data acquisition unit, data association unit, movement amount calculation unit, environmental map creation unit>> The data acquisition unit 111 periodically acquires point cloud data (frames) from the distance measurement sensor 220 and stores the data in the frame information database 140.

[0021] As described above, the mobile object control system 100 includes the data acquisition unit 111 that acquires frames that are point cloud data that indicate objects in the surrounding environment of the mobile object 200 .

[0022] The data association unit 112 associates the point cloud of the frame with the online environmental map 131. Examples of association (registration) techniques include ICP (Iterative Closest Point) and NDT (Normal Distributions Transform). The data association unit 112 associates a score (association score, registration score) indicating the quality of the association (registration) calculated in association with the frame and the position and orientation information of the moving object 200 with the frame and stores them in the frame information database 140. The association score is an example of a frame index.

[0023] ICP and NDT are just examples, and the data association unit 112 may use any other association method as long as it is a method that can calculate a score. By the association by the data association unit 112, the position (position and orientation information) of the moving object 200 on the online environmental map 131 can be calculated, and self-position estimation becomes possible.

[0024] As described above, the mobile object control system 100 includes a data matching unit 112 that matches the point cloud of the frame acquired by the data acquisition unit 111 with the online environmental map 131 that has already been generated (or is in the process of being generated), and calculates the position and orientation information of the mobile object 200 in the surrounding environment, as well as a frame index that indicates the accuracy of the point cloud.

[0025] The movement amount calculation unit 113 calculates the amount of change (movement amount, odometry) in the position and orientation of the moving object 200 based on the position and orientation information calculated by the data association unit 112, associates it with the frame, and stores it in the frame information database 140. The movement amount calculation unit 113 calculates the amount of movement from the time when the previous frame was acquired, for example.

[0026] As described above, the mobile object control system 100 includes the movement amount calculation unit 113 that calculates the movement amount of the mobile object 200 based on the position and orientation information.

[0027] The environment map creation unit 118 creates and updates the online environment map 131 based on the amount of movement calculated by the movement amount calculation unit 113. In detail, the environment map creation unit 118 performs 3D conversion processing on the point cloud indicated by the frame based on the amount of movement calculated for each frame, and overlays and integrates it onto the existing online environment map 131, thereby gradually creating the online environment map 131.

[0028] Correspondence (registration) techniques such as ICP and NDT generally require that the distances between the point clouds to be matched be close. The longer the frame acquisition interval, the greater the potential for movement, which can lead to errors in the correspondence, such as a local solution. For this reason, when creating the online environmental map 131, it is desirable to perform high-speed correspondence processing using a correspondence method that can be processed within the frame acquisition cycle of the ranging sensor 220. For example, if the frame rate of the ranging sensor 220 is 20 Hz, it is recommended to use a correspondence method that can be processed within 40 ms. Even if the point clouds of the acquired frames are coarse, registration is possible because the online environmental map 131 to be matched has high resolution. However, because the correspondence is performed sequentially, errors tend to accumulate.

[0029] As described above, the mobile object control system 100 includes an environmental map creation unit 118 that creates an online environmental map 131, which is a map of the surrounding environment, based on the point cloud of the frame.

[0030] <<Calculation section: key frame generator detection section, frame selection section, key frame generator section, movement amount correction section, environmental map creation section>> The key frame generator detection unit 114 repeatedly detects, as a key frame generator, a frame acquired when the cumulative value of the movement amount calculated by the movement amount calculation unit 113 reaches a predetermined value or when a predetermined elapsed time has passed, and outputs the detected frame to the frame selection unit 115. The cumulative value of the movement amount and the elapsed time are the cumulative value of the movement amount and the elapsed time since the movement amount calculation unit 113 previously detected a key frame. In other words, detecting a key frame generator means that a predetermined distance has been moved or a predetermined time has passed since the previous detection of the key frame generator. The key frame generator detection unit 114 may also detect key frames at other times.

[0031] As described above, the mobile object control system 100 includes the key frame generator detection unit 114 that repeatedly detects frames acquired at predetermined timing as key frame generators. The predetermined timing is the timing when the cumulative movement amount reaches a predetermined movement amount, or the timing when a predetermined time has elapsed.

[0032] The frame selection unit 115 selects a predetermined number of frames whose frame index satisfies a predetermined value from among frames acquired before or after the key frame, including the key frame generation source. The frame index is, for example, an association score (alignment score) calculated by the data association unit 112. As will be described later, the predetermined value (threshold, predetermined index value) is not limited to a fixed value and may vary. Note that frames whose frame index satisfies the predetermined value are expected to have high point cloud alignment accuracy and high point cloud position accuracy.

[0033] As described above, the mobile object control system 100 includes a frame selection unit 115 that selects frames acquired before the timing at which the key frame generator was detected, or frames acquired before or after that timing, and whose frame index satisfies a predetermined index value (predetermined value, threshold value). The frame index indicates the quality of the association between the point cloud of the frame and the online environmental map 131 performed by the data association unit 112.

[0034] The key frame generation unit 116 generates a key frame by superimposing the point clouds of the frames selected by the frame selection unit 115, and stores the key frame in the key frame database 150. The frames used for key frame generation are frames whose frame indexes satisfy a predetermined value, and therefore the position of the point clouds in the key frame is considered to be highly accurate.

[0035] As described above, the mobile object control system 100 includes the key frame generation unit 116 that generates a key frame by superimposing the frames selected by the frame selection unit 115.

[0036] The movement amount correction unit 117 corrects the movement amount based on the key frames in the key frame database 150. For example, the movement amount correction unit 117 uses a loop trapping correction method. More specifically, the movement amount correction unit 117 detects two key frames (key frames at both ends of a loop edge, hereinafter also referred to as fixed key frames) that have similar positions and orientations of the moving body 200 corresponding to the key frames. The movement amount correction unit 117 calculates relative position and orientation information by associating (aligning) point clouds of the fixed key frames, fixes one fixed key frame, corrects the other fixed key frame, and then corrects other key frames on the loop edge.

[0037] 4 is a diagram illustrating the loop trapping correction method according to the first embodiment. A solid circle indicates a position and orientation corresponding to a key frame before correction, and an arrow indicates the movement (amount of movement) of the moving body 200. The moving body 200 moves counterclockwise from a position indicated by a position and orientation 291 to a position indicated by a position and orientation 292. The positions and orientations 291 and 292 are close to each other, and the key frames corresponding to the positions and orientations 291 and 292 are fixed key frames. The dotted circle indicates the position and orientation corresponding to the corrected key frame. As the position and orientation 292 is corrected to the position and orientation 292A, the key frames between the key frames corresponding to the positions and orientations 291 and 292 are also corrected.

[0038] The number of points (number of fixed frames) used in the loop trapping correction method does not necessarily have to be two, but may be three or more. In recent years, a method using graph optimization technology has been proposed in which position and orientation information in key frames is used as a node and its relative position and orientation is used as an edge. The movement amount correction unit 117 may perform correction using such a technology. However, if the point clouds of the key frames at two points used for loop trapping are coarse, it becomes difficult to associate them. For this reason, it is desirable that not only are the multiple points close to each other, but also that the resolution of the point clouds at those points is high.

[0039] As described above, the moving object control system 100 includes the movement amount correction unit 117 that corrects the movement amount of the moving object 200 based on the key frame.

[0040] The environmental map creation unit 118 creates and updates the offline environmental map 132 based on the key frames corrected by the movement amount correction unit 117. In detail, the environmental map creation unit 118 performs three-dimensional conversion processing on the point cloud indicated by the key frames based on the corrected movement amount calculated for each key frame, and gradually creates the offline environmental map 132 by overlaying and integrating it onto the existing offline environmental map 132. Furthermore, the environmental map creation unit 118 creates and updates the online environmental map 131 based on the offline environmental map 132.

[0041] As described above, the environmental map creation unit 118 generates the offline environmental map 132 based on the corrected amount of movement, and updates the online environmental map 131 based on the offline environmental map 132 .

[0042] <Online environmental map creation processing> 5 is a flowchart of the online environment map creation process according to the first embodiment. The creation process of the online environment map 131 will be described with reference to FIG. In step S11, the environmental map creation unit 118 starts the process of repeating steps S12 to S15 until the measurement of the surrounding environment is completed. The determination of the end of measurement may be made in accordance with an instruction to end given by the operator of the mobile unit, for example, by referring to the created online environmental map 131 or offline environmental map 132.

[0043] In step S12, the data acquisition unit 111 acquires frames (point cloud data) from the distance measurement sensor 220 and stores them in the frame information database 140. In step S13, the data association unit 112 associates the point cloud of the frame acquired in step S12 with the online environmental map 131. Furthermore, the data association unit 112 stores the position and orientation information and the registration score calculated in association with the frame in the frame information database 140.

[0044] In step S14, the movement amount calculation unit 113 calculates the amount of change (movement amount, odometry) in the position and orientation of the moving object 200 based on the position and orientation information calculated in step S13. In step S15, the environmental map creation unit 118 updates and creates the online environmental map 131 based on the amount of movement calculated in step S14.

[0045] <<Offline environmental map creation processing>> Fig. 6 is a flowchart of the offline environment map creation process according to the first embodiment. The creation process of the offline environment map 132 will be described with reference to Fig. 6. Note that the online environment map creation process and the offline environment map creation process shown in Fig. 5 are processed simultaneously in parallel. In step S21, the environmental map creation unit 118 starts the process of repeating steps S22 to S28 until the measurement of the surrounding environment is completed.

[0046] In step S22, if the key frame generator detection unit 114 detects a key frame generator (step S22→YES), the process proceeds to step S23, and if not (step S22→NO), the process returns to step S22. In step S23, the frame selection unit 115 selects a predetermined number of frames whose frame indexes satisfy a predetermined value from among frames acquired before, before, or after the key frame generation source, including the key frame generation source. In step S24, the key frame generation unit 116 generates a key frame by superimposing the point clouds of the frames selected in step S23, and stores the key frame in the key frame database 150.

[0047] In step S25, if the movement amount correction unit 117 detects a fixed key frame (step S25→YES), the process proceeds to step S26, and if not (step S25→NO), the process returns to step S22. In step S26, the movement amount correction unit 117 corrects the key frame. In step S27, the environmental map creation unit 118 updates the offline environmental map 132 based on the key frames corrected in step S26. In step S28, the environmental map creation unit 118 updates the online environmental map 131 based on the offline environmental map 132 updated in step S27.

[0048] <Features of the mobile control system> The mobile object control system 100 repeatedly generates key frames by referring to the movement amount and elapsed time of the mobile object 200. The mobile object control system 100 also matches the point clouds of the key frames to correct the movement amount and update the offline environmental map 132. Since key frames are generated by overlapping frames whose frame indexes satisfy a predetermined value, high accuracy of the point cloud data can be expected, and a high-accuracy offline environmental map 132 is created. This in turn creates a high-resolution and high-accuracy online environmental map 131, enabling high-accuracy self-location estimation (calculation of position and orientation information of the mobile object 200).

[0049] <<Variation: Environment map at the starting point>> The data association unit 112 associates the point cloud of the frame with the online environmental map 131 to calculate the position and orientation information of the moving object 200 (self-location estimation), and it is necessary to have the online environmental map 131 in advance. When there is no online environmental map 131, such as in an unknown / new surrounding environment, the data association unit 112 sequentially associates the data with the initial frame (the frame first acquired at the initial position (measurement start position)). In other words, the initial frame is set as the initial online environmental map 131.

[0050] However, alignment is generally difficult when the resolution of the point cloud data acquired from the distance measurement sensor 220 is low (coarse) (see FIG. 7, which will be described later). For this reason, the data acquisition unit 111 instructs the mobile object control unit 119 to keep the mobile object 200 stationary or nearly stationary and acquire multiple frames at the initial position, and the environmental map creation unit 118 can overlay the point clouds of these multiple frames to create an online environmental map 131 with high resolution.

[0051] Fig. 7 is a diagram showing a point cloud of an initial frame according to the first embodiment. Fig. 8 is a diagram showing a point cloud generated by overlapping point clouds of multiple frames acquired at an initial position according to the first embodiment. By using the point cloud shown in Fig. 8 as the initial online environment map 131, it becomes possible to improve the accuracy of alignment of subsequent frames and calculation of movement amount.

[0052] As described above, the mobile object control unit 119 controls the mobile object 200 so that the data acquisition unit 111 acquires a plurality of frames at the initial position.

[0053] <<Variation: Frame Indicator>> In the above-described embodiment, the frame index in the first embodiment is the alignment score calculated by the data association unit 112. Alternatively, a frame index based on landmarks of the point cloud may be used. A landmark is a specific structure or object that serves as a landmark in the surrounding environment. For example, if the surrounding environment is inside a building, pillars, stairs, installed equipment, etc. may be landmarks. Other objects with unevenness may also be landmarks. Whether or not such landmarks are detected or the number of detected landmarks may also be the frame index.

[0054] Landmark detection uses point cloud-based object recognition techniques. Deep learning-based methods are mainstream in the field of object recognition technology, but landmark detection techniques using 3D object recognition and detection techniques that use 3D local features of point clouds, such as PFH (Point Feature Histogram), SHOT (Signature of Histograms of OrienTations), and PPF (Point Pair Feature), can also be used. If there is no prior information about objects specific to the surrounding environment, objects detected using object recognition and detection techniques can be added as landmarks on the spot.

[0055] When landmarks are used as frame indices, it is desirable to use landmarks detected from point cloud data as well as landmarks detected from images acquired by an image sensor in combination as frame indices. Furthermore, if the frame indices are simply based on the presence or absence of landmarks, the frame selection unit 115 will not select a frame in an environment where landmarks do not exist, so it is desirable to use frame indices based on landmarks in combination with the frame indices of the alignment scores.

[0056] As described above, the frame index is the presence or absence of landmarks in the surrounding environment detected from the point cloud of the frame, or the number of objects that can serve as landmarks.

[0057] <<Frame index threshold value: fixed value>> The predetermined value (threshold) used by the frame selector 115 to select a frame may be empirically set to a predetermined fixed value depending on the surrounding environment. Another threshold may be the average value of a frame index (e.g., alignment score) sequence calculated after matching the environmental map at the initial position with the point cloud acquired from the ranging sensor 220. The threshold may be a threshold with a buffer added to the average value (a value whose difference from the average value is within a predetermined value).

[0058] Furthermore, after moving around the surrounding environment and performing measurements for a certain period of time while matching point clouds, the average value of the series of frame indices up to that point or a threshold value obtained by adding a buffer to that average value may be used. However, since the frame indices may include outliers that deviate significantly from the average value, the median or a threshold value obtained by adding a buffer to that median, or the average value of the frame indices from which the outliers have been removed or a threshold value obtained by adding a buffer to that average value may be used.

[0059] <<Frame index specified value (threshold): moving average>> The threshold value may be varied in response to movement or changes in the surrounding environment. For example, for the threshold value immediately after measurement, an initial value of the frame index is set based on the initial position, and after measurement is performed by moving a certain number of frames, the average value of the frame index is calculated sequentially for a predetermined number of consecutive frames before or after the acquired frame, and the average value or a variable threshold value with a buffer added to the average value may be used.

[0060] As explained above, the initial predetermined index value is the frame index at the initial position of the moving body 200. The predetermined index value after a predetermined number of frames have been acquired is the average value of the frame indexes of frames acquired before the acquisition of the frame, a value whose difference from the average value is within a predetermined value, the average value of the frame indexes of frames acquired within a predetermined time before and after the acquisition of the frame, or a value whose difference from the average value is within a predetermined value.

[0061] <<Predetermined value (threshold) of frame index: score potential map>> Based on the result of self-localization, a potential map (correspondence table) of position and orientation information and frame indices (e.g., alignment scores) may be generated, and the alignment score corresponding to the position and orientation information calculated by the data association unit 112 or a value obtained by adding a buffer to the score may be used as the threshold. Such a threshold is effective when measuring the same surrounding environment multiple times. Each time the surrounding environment measurement is repeated, the accuracy of the potential score of the alignment score increases, and the frame selection unit 115 selects higher quality frames, allowing the key frame generation unit 116 to generate high-resolution and accurate key frames.

[0062] As described above, the mobile object control system 100 includes a memory unit 130 that stores a correspondence table (potential map) between position and orientation information and frame indices of frames previously acquired at a location corresponding to the position and orientation information, and the predetermined index value is the average value of the frame indices corresponding to the location (position and orientation information) at which the frame was acquired in the correspondence table, or a value whose difference from the average value is within a predetermined value.

[0063] <<Frame index threshold value: Machine learning>> The predetermined value (threshold) of the frame index may be determined using machine learning technology. For example, the predetermined value (threshold) may be calculated using a machine learning model that uses point cloud and position and orientation information as explanatory variables and frame indexes of frames acquired at points indicated by the position and orientation information as objective variables. The predetermined value may be a value obtained by adding a buffer to the value calculated using the machine learning model. Such a threshold is used when measuring the same surrounding environment multiple times, and learning (generation of a machine learning model) is repeated. The frame selection unit 115 calculates the predetermined value (threshold) using the latest machine learning model and selects a frame.

[0064] As explained above, the specified index value is a frame index obtained by inputting a frame (point cloud data) as an explanatory variable into a machine learning model generated using training data in which the explanatory variable is point cloud data and the objective variable is the frame index of the point cloud data, or a value whose difference from the frame index is within a specified value.

[0065] <<Variation: Measurement environment and conditions>> When aligning point clouds, it is desirable to use frames measured in a surrounding environment with distinctive three-dimensional shapes, and it is desirable to acquire multiple frames at locations where the frames contain one or more structures or objects with three-dimensional shape characteristics, such as unevenness. Furthermore, since the online environmental map 131 is often based on an initial position, it is advisable to measure multiple frames at the initial position. Furthermore, when performing correction using the loop closure method, the movement amount correction unit 117 performs correction based on the initial position or landmarks in the surrounding environment. Therefore, it is desirable for the mobile object control unit 119 to control the actuator 270 so that the mobile object 200 visits the initial position or landmark points multiple times when measuring the surrounding environment.

[0066] As described above, the mobile object control unit 119 controls the mobile object 200 so that the mobile object 200 visits a point where a feature point of a three-dimensional shape is detected in the point cloud of the frame multiple times.

[0067] <<Variation: Creating keyframes by controlling moving objects>> When measurements are taken while the moving object 200 is moving, a matching process is required, making it difficult to completely eliminate errors in the online environmental map 131. High-quality key frames can be generated by controlling the movement of the moving object 200 using the moving object control unit 119. For example, the key frame generation source detection unit 114 may instruct the moving object control unit 119 to stop or nearly stop the moving object 200 at the point where it detects the key frame generation source, and measure multiple frames at that position and orientation. The key frame generation unit 116 can generate high-resolution and accurate key frames by overlaying the point clouds of the multiple frames.

[0068] Including multiple such key frames can further improve the accuracy of the offline environmental map 132, and ultimately improve the accuracy and resolution of the online environmental map 131. The frame from which the key frame is generated may be acquired at any position, but as described above, it is desirable to measure in a surrounding environment with three-dimensional characteristics. Note that the mobile object control unit 119 is not limited to being stationary, and may also control the mobile object 200 so that the rate of change in position and orientation is smaller than a predetermined value (to be substantially stationary).

[0069] As described above, the moving object control unit 119 controls the moving object 200 so that it is stationary or nearly stationary at the point where the key frame generation source detection unit 114 detects the key frame generation source. The frame selection unit 115 selects frames acquired while the moving object is stationary or nearly stationary. The key frame generation unit 116 generates a key frame by superimposing the selected frames.

[0070] Second Embodiment In the first embodiment, the frame index is the alignment score of the point cloud included in the point cloud data (frame) measured by the distance measurement sensor 220. A frame index based on data measured by another sensor may also be used.

[0071] Distance measurement sensors 220 generally have lower vertical resolution than horizontal resolution and a narrow field of view. For example, LiDAR sensors can measure a wide range, with some capable of 360-degree horizontal measurement, but most sensors only cover a few tens of degrees vertically. Furthermore, the widely used multi-layer LiDAR lacks information in the vertical direction. Therefore, the accuracy of point cloud matching is likely to deteriorate with respect to vertical translation, pitch rotation, and roll rotation when the position and orientation of a moving object changes. Furthermore, because the frame rate is only a few tens of Hz, sudden changes in position and orientation, which often occur when moving over uneven terrain, can cause the distance between the matching points to increase during the point cloud matching process, potentially resulting in a failed registration.

[0072] In the second embodiment, by using a motion sensor in combination, point clouds of sudden position and posture changes and posture angles, which tend to be difficult to associate, are not used in generating key frames. As a result, accurate key frames can be generated. In addition, sensor fusion using a motion sensor compensates for the lack of information volume and utilizes features that are generally high-frequency (high-frequency of measurement) to reduce accuracy degradation of the online environmental map 131 and self-location estimation.

[0073] 9 is a functional block diagram of a mobile object control system 100A provided in a mobile object 200A according to the second embodiment. The motion sensor 230 provided in the mobile object 200A is an inertial sensor that is a six-axis sensor consisting of a three-axis acceleration sensor and a three-axis gyro sensor. The motion sensor 230 may also be a nine-axis sensor that further combines a three-axis geomagnetic sensor. It is assumed that calibration has been performed regarding the relative position and orientation between the distance measurement sensor 220 and the motion sensor 230, and that the positions and orientations can be converted between them.

[0074] The data acquisition unit 111A acquires the measurement results of the motion sensor 230 while acquiring frames (point cloud data) from the distance measurement sensor 220, and stores the speed, angular velocity, etc. of the moving object 200 in association with the frame information database 140A. Since the motion sensor 230 has a higher frequency than the distance measurement sensor 220, the measurement results of multiple motion sensors 230 acquired within several milliseconds before and after one frame (point cloud data) are stored in association with the frame. Frame selector 115A selects frames based on frame indicators that are based on measurements from motion sensor 230, which will be described below.

[0075] <Frame index: velocity, angular velocity> Frame indices based on the measurement values ​​of the motion sensor 230 include velocity and angular velocity (motion value). If the velocity and angular velocity at the time of frame acquisition are within a predetermined value, the frame selection unit 115A selects that frame. The predetermined value may be determined according to the frame rate, measurable range, and resolution of the distance measurement sensor 220. The velocity and angular velocity may be calculated by the data acquisition unit 111A and stored in the frame information database 140A, or may be calculated by the frame selection unit 115A.

[0076] <Frame index: angle (attitude angle)> Angle is a frame index based on the measurement value of the motion sensor 230. If the angle (attitude angle of the moving object 200, motion value) is within a predetermined value, the frame selection unit 115A selects the frame. The predetermined value may be determined according to the measurable range and resolution of the distance measurement sensor 220 used. Note that the angle may be calculated by the data acquisition unit 111A and stored in the frame information database 140A, or may be calculated by the frame selection unit 115A.

[0077] As described above, the data acquisition unit 111A acquires a motion value that indicates the speed of change in the position and posture or the posture (angle) of the moving object 200. The frame index of a frame is the motion value at the time the frame is acquired. A frame whose frame index satisfies a predetermined index value is a frame whose frame index is within the predetermined index value.

[0078] <Features of the second embodiment> By using the frame index using the motion sensor 230, an accurate key frame can be generated by selecting a frame at a flat point where there are no steps or slopes and where there is little change in the position and orientation of the moving body 200. In other words, an accurate key frame can be generated because frames during position and orientation changes, which are likely to result in inaccurate self-position estimation when using only the distance measurement sensor 220, are not included.

[0079] <<Variation: Creating keyframes by controlling moving objects>> The surrounding environment may be measured at a location where the fluctuation in the position and orientation of the moving object 200 is small, and multiple frames may be acquired. Such a location may be, for example, a flat location without steps or slopes. If the calculated fluctuation in the position and orientation is within a predetermined value, the data acquisition unit 111A may instruct the moving object control unit 119 to stop the moving object 200 and measure multiple frames at that position and orientation. The key frame generation unit 116 can generate high-resolution and accurate key frames by overlaying the point clouds of the multiple frames. Including multiple such key frames can further improve the accuracy of the offline environmental map 132, and ultimately the accuracy of the online environmental map 131. Note that the moving object control unit 119 is not limited to stopping the moving object 200 at a stationary position; it may also control the moving object 200 to reduce the rate of change in its position and orientation.

[0080] <<Variation: Sensor Fusion Processing>> The mobile object control system 100A may include a fusion processing unit 120A (see FIG. 9) that performs sensor fusion processing of the ranging sensor 220 and the motion sensor 230. By performing sensor fusion processing, it is possible to track position and orientation variations that cannot be captured at the frame rate of the ranging sensor 220, improving the accuracy of self-localization estimation. This allows the threshold value of the frame index to be relaxed compared to when sensor fusion is not used. As a method of sensor fusion processing, a Bayes filter such as an extended Kalman filter or a particle filter, a factor graph, etc. can be used.

[0081] Third Embodiment The sensor may be a combination of a distance measurement sensor 220 and an image sensor. FIG. 10 is a functional block diagram of a mobile object control system 100B provided in a mobile object 200B according to the third embodiment. The image sensor 240 provided in the mobile object 200B is a general RGB camera capable of detecting the brightness of RGB information for each pixel to acquire a two-dimensional image. However, the camera does not necessarily have to be capable of acquiring RGB information; it may be a monochrome camera. The image sensor 240 may also be a camera that detects light other than visible light, such as an infrared camera. Furthermore, the image sensor 240 may be a sensor integrated with the distance measurement sensor 220. The acquired image may be a color image or a monochrome image. The frame rate of the image sensor 240 is preferably equal to or higher than the frame rate of the distance measurement sensor 220, and the cycle may be determined according to the speed of the mobile object 200B.

[0082] The image sensor 240 is attached to the moving body 200B so as to include part or all of the imaging range of the ranging sensor 220. It is assumed that the ranging sensor 220 and the image sensor 240 have been calibrated, and that a correspondence relationship between the points of the point cloud acquired by the ranging sensor 220 and the pixels of the image sensor 240 has been established.

[0083] The timing of the frames acquired by the distance measurement sensor 220 and the timing of the images captured by the image sensor 240 do not necessarily need to be synchronized, but they should correspond in time series so that it is possible to determine which image corresponds to the corresponding frame of the distance measurement sensor 220. Specifically, it is advisable to associate the measurement results of multiple image sensors 240 acquired within several milliseconds before and after one frame (point cloud data).

[0084] The data acquisition unit 111B acquires images captured by the image sensor 240 while acquiring frames (point cloud data) from the distance measurement sensor 220, and stores the images in association with each other in the frame information database 140B. In other words, the data acquisition unit 111B associates several images acquired within several milliseconds before and after a frame of the distance measurement sensor 220 and stores them in the frame information database 140B. Frame selector 115B selects frames based on a frame index that is based on measurements by image sensor 240, which will be described below.

[0085] <Frame Indicators: Landmarks> A method in the field of object recognition technology is used to detect landmark objects and structures (landmarks) included in the images captured by the image sensor 240. Deep learning-based methods are mainstream in the field of object recognition technology, and object recognition and detection technology using deep learning is applied. For example, object recognition and detection technology such as R-CNN (Region Based Convolutional Neural Networks) or YOLO (You Only Look Once) may be used. Alternatively, rule-based object recognition and detection technology that follows if-then rules may be applied.

[0086] In a 2D image, an object is detected as a 2D rectangle, which is the bounding box of the object. A method for extracting the 2D shape of the object in more detail may also be used. Landmark objects and structures may be changed as appropriate depending on the surrounding environment of the application. If there is no prior information on objects specific to the environment, objects detected using object recognition and detection technology may be added as landmark objects and structures on the spot.

[0087] When a landmark object or structure is used as the frame index, the frame index may be determined based on whether or not the landmark object or structure is detected in the image acquired by the image sensor 240, or the number of detected objects or structures may be used as the index.

[0088] If the frame index is simply the presence or absence of landmark objects or structures in the image captured by the image sensor 240, no frame will be selected in an environment where there are no landmark objects or structures. For this reason, it is advisable to use a frame index based on landmark objects or structures in the image when a frame is not selected based on the frame index of the distance measurement sensor 220. This reduces the number of frames that should have been selected but were not.

[0089] <Frame index: feature point> Image feature points are the center points of textured areas in a 2D image or areas with large local variations in brightness, such as corners. If there is no prior information on feature points, they can be added on the fly using feature point detection techniques such as SIFT or ORB. When feature points are used as frame indices, the matching score of the matched feature points or the number of feature points is used as the frame indices. In the case of the matching score, the threshold value for the frame indices can be determined according to the performance of the image sensor 240 used. As for the number of feature points, the number of feature points that can be acquired differs depending on the environment as well as the performance of the image sensor, so the threshold value is determined according to the performance of the image sensor 240 and the surrounding environment.

[0090] <Frame index:Texture plane> A texture plane is a planar area with some kind of pattern drawn on it in a structure that includes flat surfaces, such as a wall in the surrounding environment. Texture planes can be detected using, for example, object recognition technology, template matching, or planar area detection using feature point detection. Texture planes can be changed appropriately depending on the environment to which they are applied. If there is no prior information about surfaces specific to the surrounding environment, a 2D rectangle in an area where many feature points are detected and which is recognized as a plane can be added on the spot as a texture plane.

[0091] When a texture plane is used as a frame index, the frame index is determined by whether or not a texture plane is detected in the image acquired by the image sensor 240, just as when a landmark is used as a frame index. However, if the frame index is simply determined by whether or not a texture plane is detected, no frame will be selected in an environment where a texture plane does not exist. For this reason, it is advisable to use a frame index based on a texture plane when a frame is not selected by the frame index of the ranging sensor 220. Although the frame indices relating to the image have been described as being related to landmarks, feature points, and texture planes, an index combining these may also be used.

[0092] As described above, data acquisition unit 111B acquires an image of the surrounding environment of moving body 200. The frame index of a frame is calculated based on at least one of the number of landmark objects in the surrounding environment detected from the image at the time the frame was acquired, the detection score of feature points detected from the image at the time the frame was acquired, the number of feature points detected from the image at the time the frame was acquired, and the presence or absence of texture detected from the image at the time the frame was acquired.

[0093] <Features of the third embodiment> By using frame indices using the image sensor 240, accurate key frames can be generated by selecting frames captured in an environment with a distinctive texture, a cluttered environment, or a location where many landmark objects or structures are captured. In other words, frames containing landmarks with high registration reliability are selected, and the point cloud of the selected frame can be used to generate key frames, allowing accurate key frames to be generated.

[0094] <<Variation: Sensor Fusion Processing>> The mobile object control system 100B may include a fusion processing unit 120B (see FIG. 10) that performs fusion processing of the self-localization estimation using the ranging sensor 220 and the self-localization estimation using the image sensor 240. By performing sensor fusion processing, it is possible to track changes in position and orientation that cannot be captured at the frame rate of the ranging sensor 220 and reduce the occurrence of self-localization loss. As a result, the accuracy of the self-localization estimation and the online environmental map 131 is improved. As a method of sensor fusion processing, a Bayes filter such as an extended Kalman filter or a particle filter, a factor graph, etc. can be used.

[0095] For self-localization using the image sensor 240 (and creating the online environmental map 131), it is recommended to use object SLAM or visual SLAM. Object SLAM is a method of estimating the 3D position and orientation of an object using a rectangular parallelepiped or sphere based on the appearance of the object in the image, and then estimating the self-localization from the relative position and orientation. On the other hand, visual SLAM is a method of estimating the self-localization by obtaining visual odometry with a single eye based on deep learning and feature points.

[0096] <<Variation: Multiple Sensors>> The moving body 200A of the second embodiment includes a distance measurement sensor 220 and a motion sensor 230, while the moving body 200B of the third embodiment includes a distance measurement sensor 220 and an image sensor 240. A moving body may also include a distance measurement sensor 220, a motion sensor 230, and an image sensor 240. When there are multiple sensors, the frame selection unit may select a frame by combining frame indices associated with the respective sensors. For example, a frame may be selected in which the frame indices associated with any one of the sensors satisfy a predetermined value, or a frame in which the frame indices associated with the distance measurement sensor 220 satisfy a predetermined value and another sensor satisfy a predetermined value may be selected. Alternatively, if there are no (or few) frames in which the frame indices associated with the distance measurement sensor 220 satisfy a predetermined value, a frame in which the frame indices associated with the other sensors satisfy a predetermined value may be selected.

[0097] A frame in which the frame index of the ranging sensor 220 satisfies a predetermined value and the frame index of the image sensor 240 also satisfies a predetermined value may be stored in the frame information database as a frame satisfying multiple frame indexes and as a point cloud of a frame with high matching reliability. When generating key frames, the key frame generation unit may prioritize overlaying point clouds of frames with high matching reliability.

[0098] At the timing (point) when a frame index related to any sensor satisfies a predetermined value or when multiple frame indexes satisfy predetermined values, the mobile object control unit 119 controls the mobile object 200 to be stationary or nearly stationary, and the data acquisition unit 111 acquires multiple frames. The frame selection unit 115 selects the multiple frames, and the key frame generation unit 116 can generate high-resolution and accurate key frames by superimposing point clouds of the multiple frames.

[0099] Furthermore, the key frame generation source detection unit 114 may determine, as a key frame generation source, a timing (point) at which a frame index related to any sensor satisfies a predetermined value, or a frame at which multiple frame indexes satisfy predetermined values, thereby generating high-resolution and accurate key frames.

[0100] As described above, the mobile object control unit 119 controls the mobile object 200 so that it is stationary or nearly stationary at the point where a frame satisfying a predetermined index value is acquired. The frame selection unit 115 selects frames acquired while the mobile object is stationary or nearly stationary. The key frame generation unit 116 generates a key frame by superimposing the selected frames.

[0101] <<Variation: Multiple Moving Objects>> In the above-described embodiment, the mobile object control systems 100, 100A, and 100B are installed in a single mobile object 200, 200A, and 200B, respectively, but this is not limited thereto. For example, the online environmental map 131 and the offline environmental map 132 may be stored on a server and shared by multiple mobile objects. In this case, the environmental map creation unit 118 may be located on the server. Furthermore, the key frame database 150 may also be shared by multiple mobile objects 200, 200A, and 200B, and the movement amount correction unit 117 may operate on the server. In this manner, the multiple mobile objects 200, 200A, and 200B can cooperate to generate a highly accurate online environmental map 131 and offline environmental map 132 in a short period of time. Furthermore, even if there is only one mobile object, some of the functional units included in the calculation unit 110 and some of the data in the storage unit 130 may be stored in a server (another device).

[0102] Other variations Although several embodiments of the present invention have been described above, these embodiments are merely illustrative and do not limit the technical scope of the present invention. The present invention can take on various other embodiments, and various modifications such as omissions and substitutions can be made without departing from the spirit of the present invention. These embodiments and their modifications are included within the scope and spirit of the invention described in this specification, etc., and are also included in the scope of the invention described in the claims and their equivalents. [Explanation of symbols]

[0103] 100, 100A, 100B Mobile Control System 111, 111A, 111B Data acquisition section 112 Data mapping section 113 Movement amount calculation unit 114 Keyframe generator detection unit 115, 115A, 115B Frame selection section 116 Keyframe Generation Unit 117 Movement amount correction unit 118 Environmental Mapping Department 119 Mobile Control Unit 120A, 120B Fusion processing section 131 Online Environmental Map 132 Offline Environment Maps 138 Programs 140, 140A, 140B Frame Information Database 150 keyframe database 200,200A,200B Mobile object 220 Distance Sensor 230 Motion Sensor 240 Image Sensor

Claims

1. a data acquisition unit that acquires frames of point cloud data representing objects in the surrounding environment of the moving object; an environmental map creation unit that creates an online environmental map, which is a map of the surrounding environment, based on the point cloud of the frame; a data association unit that associates the point cloud of the frame acquired by the data acquisition unit with the generated online environmental map, and calculates position and orientation information of the moving object in the surrounding environment and a frame index that indicates accuracy of the point cloud; a movement amount calculation unit that calculates a movement amount of the moving object based on the position and orientation information; a key frame generator detection unit that repeatedly detects frames acquired at predetermined timing as key frame generators; a frame selection unit that selects a frame acquired before the timing at which the key frame generation source is detected or a frame acquired before or after the timing at which the key frame generation source is detected, and the frame index satisfies a predetermined index value; a key frame generation unit that generates a key frame by superimposing the frames selected by the frame selection unit; a movement amount correction unit that corrects the movement amount of the moving body based on the key frame. Mobile control system.

2. The environmental map creation unit An offline environmental map is generated based on the corrected amount of movement, and the online environmental map is updated based on the offline environmental map. The mobile object control system according to claim 1 .

3. The frame index is The degree of quality of the correspondence between the point cloud of the frame and the online environmental map performed by the data correspondence unit. The mobile object control system according to claim 1 .

4. The frame index is The presence or absence of landmarks in the surrounding environment detected from the point cloud of the frame, or the number of objects that can be landmarks The mobile object control system according to claim 1 .

5. The data acquisition unit A motion value indicating a speed of change in the position and attitude of the moving object or the attitude is acquired; The frame index of the frame is the motion value at the time the frame was captured, The frame whose frame index satisfies the predetermined index value is The frame index is within the predetermined index value. The mobile object control system according to claim 1 .

6. The data acquisition unit acquiring an image of the surrounding environment of the moving object; The frame index of the frame is the number of landmark objects in the surrounding environment detected from the image at the time the frame was acquired; a detection score of the feature points detected from the image at the time the frame was acquired; The number of feature points detected from the image at the time the frame was acquired, and The calculation is based on at least one of the presence or absence of texture detected from the image at the time the frame is acquired. The mobile object control system according to claim 1 .

7. The initial predetermined index value is a frame index at the initial position of the moving object; The predetermined index value after a predetermined number of the frames have been acquired is: an average value of frame indices of frames acquired prior to the acquisition of said frame; A value whose difference from the average value is within a predetermined value, the average value of the frame indices of frames acquired within a predetermined time period around the acquisition time of the frame, or The difference from the average value is within a predetermined value The mobile object control system according to claim 1 .

8. The predetermined index value is A frame index obtained by inputting the point cloud data as an explanatory variable into a machine learning model generated using training data in which the explanatory variable is the point cloud data and the objective variable is a frame index of the point cloud data, or The difference from the frame index is within a predetermined value. The mobile object control system according to claim 1 .

9. a storage unit that stores a correspondence table between the position and orientation information and frame indices of frames previously acquired at a point corresponding to the position and orientation information; The predetermined index value is the average value of the frame indexes corresponding to the points at which the frames were acquired in the correspondence table, or The difference from the average value is within a predetermined value The mobile object control system according to claim 1 .

10. a moving body control unit that controls the position and attitude of the moving body; The moving body control unit controlling the moving object so that it becomes stationary or substantially stationary at a point where the key frame generating source detection unit detects the key frame generating source; The frame selection unit Selecting frames acquired during stationary or nearly stationary periods; The key frame generation unit Generate keyframes by overlapping selected frames The mobile object control system according to claim 1 .

11. a moving body control unit that controls the position and attitude of the moving body; The moving body control unit controlling the moving object so that it is stationary or substantially stationary at a point where a frame satisfying the predetermined index value is acquired; The frame selection unit Selecting frames acquired during stationary or nearly stationary periods; The key frame generation unit Generate keyframes by overlapping selected frames The mobile object control system according to claim 1 .

12. a moving body control unit that controls the position and attitude of the moving body; The moving body control unit At an initial position, the moving body is controlled so that the data acquisition unit acquires a plurality of the frames. The mobile object control system according to claim 1 .

13. a moving body control unit that controls the position and attitude of the moving body; The moving body control unit The moving object is controlled so as to visit a point where a feature point of a three-dimensional shape is detected in the point cloud of the frame a plurality of times. The mobile object control system according to claim 1 .

14. The predetermined timing is The timing when the cumulative movement amount reaches a predetermined movement amount or when a predetermined time has elapsed. The mobile object control system according to claim 1 .

15. A mobile control system that controls a mobile object acquiring a frame of point cloud data representing objects in the surrounding environment of the moving object; generating an online environment map, which is a map of the surrounding environment, based on the point cloud of the frame; a step of associating the acquired frame point cloud with the generated online environmental map, and calculating position and orientation information of the moving object in the surrounding environment and a frame index indicating accuracy of the point cloud; calculating a movement amount of the moving object based on the position and orientation information; repeatedly detecting frames acquired at predetermined timing as key frame generation sources; selecting a frame acquired before the timing at which the key frame generator was detected, or a frame acquired before or after the timing at which the key frame generator was detected, and the frame index satisfies a predetermined index value; overlapping the selected frames to generate a key frame; correcting the amount of movement of the moving object based on the key frame; A mobile object control method.

Images