Method for controlling a mobile object, control device, mobile object, program, and storage medium

Abstract

This invention provides a technology that can successfully re-evaluate a user's position using an image if the initial self-position estimation fails. [Solution] The method for controlling a moving object according to this disclosure includes: recognizing feature quantities of the environment surrounding the moving object from images taken of the area surrounding the moving object; estimating the self-position of the moving object on a map based on the feature quantities of the environment surrounding the moving object and map information corresponding to the environment surrounding the moving object; and controlling the movement of the moving object while repeatedly estimating its own position. In this case, controlling the movement of the moving object includes, if the moving object loses its own position due to a failure in estimating its own position on the map, causing it to perform a specific movement to successfully estimate its own position on the map again using feature quantities from the captured images.

Description

Technical Field

[0001] The present invention relates to a moving body control method, a control device, a moving body, a program, and a storage medium.

Background Art

[0002] In moving bodies such as robots, a technique for estimating the self-position in an environment is used to autonomously travel within the environment. Patent Document 1 discloses a technique for performing first self-position estimation based on a detection result regarding the state of the environment around the moving body and map data. In this technique, when the first self-position estimation fails, second self-position estimation based on travel information of the moving body is further performed, so that even in a situation where the first self-position estimation cannot be performed, autonomous travel based on the second self-position estimation is enabled.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problem to be Solved by the Invention

[0004] In the above-described conventional technique, autonomous travel can be continued by the second self-position estimation, but when self-position estimation based on the travel information of the moving body is continued, the accuracy of self-position estimation may decrease due to accumulation of errors or the like.

[0005] Meanwhile, when self-position estimation is performed by recognizing feature quantities of the surrounding environment from an image, even if the self-position estimation fails, if the self-position estimation using the image can be made successful again (restored), more accurate autonomous travel becomes possible. The conventional technique has not taken into account performing self-position estimation by recognizing feature quantities of the surrounding environment from an image, and making the self-position estimation using an image successful again.

[0006] The present invention has been made in view of the above problems, and its object is to provide a technology that makes it possible to successfully estimate one's own position again when the estimation of one's own position using an image has failed. [Means for solving the problem]

[0007] According to the present invention, A method for controlling a moving object, Recognizing characteristic features of the environment surrounding the moving object from images taken around the moving object, Based on the characteristic features of the environment surrounding the moving object and map information corresponding to the environment surrounding the moving object, the self-position of the moving object on the map is estimated. This includes controlling the movement of the moving body while repeatedly estimating its own position, A control method is provided for controlling the movement of the moving object, which includes, if the moving object loses its position due to a failure to estimate its own position on the map, causing it to perform a specific movement to allow the moving object to successfully estimate its own position on the map again using the features from the captured image. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide a technique that makes it possible to successfully estimate one's own position again when the estimation of one's own position using an image has failed. [Brief explanation of the drawing]

[0009] [Figure 1] A perspective view showing an example of the overall configuration of the mobile body of this embodiment. [Figure 2] A schematic plan view illustrating an example of a mobile unit for the mobile body of this embodiment. [Figure 3] A block diagram showing an example of the control system for the mobile body of this embodiment. [Figure 4] A block diagram showing an example of the functional configuration of the control unit in this embodiment. [Figure 5A]Figure (1) illustrates the outline of the recovery process in this embodiment. [Figure 5B] Figure (2) illustrates the outline of the recovery process in this embodiment. [Figure 6A] A flowchart illustrating the series of operations for the recovery process in this embodiment. [Figure 6B] A schematic diagram showing an example of a local map of this embodiment. [Figure 7] A flowchart illustrating the sequence of operations for the retry process in this embodiment. [Figure 8] A flowchart illustrating the series of operations in the search process of this embodiment. [Figure 9] A diagram showing an example of a path generated by the search process of this embodiment. [Figure 10] A diagram showing an example of a path generated by the search process of this embodiment. [Figure 11] A diagram showing an example of a path generated by the retry process in this embodiment. [Modes for carrying out the invention]

[0010] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims, and not all combinations of features described in the embodiments are essential to the invention. Two or more features from the multiple features described in the embodiments may be combined arbitrarily. Furthermore, identical or similar configurations will be given the same reference numeral, and redundant descriptions will be omitted.

[0011] <Embodiment> In the following embodiments, a case where micro-mobility as an example of a mobile object is used will be described as an example. The micro-mobility is, for example, a small electric vehicle that can carry cargo and follows a user or leads or guides the user. However, the present embodiment may be applied not only to cargo but also to an electric vehicle on which a person can ride. Further, the present embodiment is not limited to the example where the mobile object is an electric vehicle, and can also be applied to mobile objects other than electric vehicles. Furthermore, in the following description, a mobile object having one driven wheel will be described as an example, but the mobile object does not necessarily have to have a driven wheel, and the number of driven wheels is not limited to one, and may be two or more. The present embodiment can also be applied to a robot capable of autonomous walking using legs without having wheels.

[0012] <Configuration of Mobile Object> The configuration of the mobile object 100 according to the embodiment will be described with reference to Figure 1. Figure 1 is a perspective view showing the overall configuration of the mobile object. As shown in Figure 1, the mobile object 100 of the embodiment includes a lower housing 111, an upper housing 112, a front display unit 113, a plurality of detection units 114, an audio output unit 115, a light emitting unit 116, two drive wheels 120, and a driven wheel 121. The side on which the drive wheels 120 are provided is defined as the front side.

[0013] The lower housing 111 rotatably holds the drive wheels 120 and the driven wheel 121, and accommodates a traveling unit including the drive wheels 120 and the driven wheel 121. The upper portion of the lower housing 111 has a loading space 118 on which a user's cargo can be loaded. The lower housing 111 accommodates the light emitting unit 116. The region of the lower housing 111 that accommodates the light emitting unit 116 is made of a material that can transmit light.

[0014] The upper housing 112 is configured to extend upward from the rear portion of the lower housing 111. The upper housing 112 supports the front display unit 113, the detection units 114, and the audio output unit 115 at its upper end portion.

[0015] The front display unit 113 notifies the user of information through images. The front display unit 113 may be an image display device such as a liquid crystal display device or an organic EL (Electro-Luminescence) display device. The front display unit 113 is located on the front side of the upper end of the upper housing 112. The front display unit 113 is oriented forward. The front display unit 113 notifies the user of information by displaying images that mimic human eyes or faces, as well as characters.

[0016] Multiple detection units 114 detect targets such as people and objects around the moving body 100. The detection units 114 may be imaging devices having optical systems such as lenses and image sensors. The detection units 114 capture images of the surroundings and output image data. The detection units 114 may have devices capable of detecting objects, such as radar and lidar (Light Detection and Ranging), instead of or in addition to imaging devices. Multiple detection units 114 are provided, for example, on the front, back, left, and right sides of the upper end of the upper housing 112. This allows the multiple detection units 114 to capture images in all directions (i.e., 360° around) of the moving body 100 and generate images. The installation locations and number of detection units 114 are not particularly limited and may be changed as appropriate.

[0017] The audio output unit 115 notifies the user or other people in the vicinity of the device via voice. The audio output unit 115 may be an audio output device such as a speaker.

[0018] The light-emitting unit 116 notifies the user or other people in the vicinity of information using light. The light-emitting unit 116 has, for example, one or more light-emitting elements such as LEDs (Light Emitting Diodes). The light-emitting unit 116 is provided so as to surround the lower housing 111. The light-emitting unit 116 may emit light of multiple colors. The light-emitting unit 116 can emit light of different colors for each region or direction. That is, the light-emitting unit 116 may emit light of different colors in the front region 116F, the left region 116L, and the right region, for example. In addition, the light-emitting unit 116 can switch between illuminating and turning off for each region. Note that the illumination may be controlled in finer regions.

[0019] The two drive wheels 120 are located on the left and right sides of the front and lower part of the lower housing 111. The drive wheels 120 rotate around the axle when driving force is transmitted from the running unit.

[0020] The driven wheel 121 is located at the rear and center of the lower housing 111. The driven wheel 121 is held so as to be able to pivot around a vertical axis that extends in the vertical direction.

[0021] Figure 2 is a schematic plan view illustrating the travel unit 125 of the mobile body 100. The travel unit 125 of the mobile body 100 will be described with reference to Figure 2.

[0022] The travel unit 125 is located at the bottom of the lower housing 111. The travel unit 125 has two drive wheels 120, a driven wheel 121, a drive mechanism 122, a motor 123a, a motor 123b, and a battery 124.

[0023] When power is supplied from the battery 124, motors 123a and 123b function as a drive source and supply driving force to the drive mechanism 122. The drive mechanism 122 transmits the driving force supplied from motors 123a and 123b to the drive wheels 120. This causes the drive mechanism 122 to rotate the drive wheels 120, moving the mobile body 100 forward or backward. The drive mechanism 122 causes the direction of travel of the mobile body 100 to be either left or right by creating a rotational difference between motors 123a and 123b. By changing the magnitude of the rotational difference, the drive mechanism 122 enables various turns of the mobile body 100, such as pivot turns, slow turns, and super-pivot turns.

[0024] Figure 3 is a block diagram of the control system of the mobile unit 100. The control system of the mobile unit 100 will be described with reference to Figure 3. As shown in Figure 3, the mobile unit 100 further comprises a control unit 130, an operation unit 132, a top display unit 131, an audio input unit 133, a GNSS sensor 134, a storage device 135, and a communication unit 136.

[0025] The control unit 130 is an example of a control device, also called an ECU (Electronic Control Unit), and may be a computer. The control unit 130 is responsible for the overall control of the mobile body 100. The control unit 130 includes one or more processors, such as a CPU (Central Processing Unit), memory devices such as semiconductor memory, and interfaces with external devices.

[0026] The processor may be an MPU (Micro Processing Unit), GPU (Graphics Processing Unit), NPU (Neural Processing Unit), or QPU (Quantum Processing Unit) in addition to or instead of a CPU. The processor realizes various functions of the control unit 130 by reading and executing computer programs stored in memory devices or the like.

[0027] Some or all of the functions of the control unit 130 may be implemented by one or more circuits, such as an ASIC (Application Specific Integrated Circuit) and a PLD (Programmable Logic Device) including an FPGA (Field Programmable Gate Array).

[0028] The memory device may be, for example, RAM (Random Access Memory). The memory device stores programs executed by the processor and data used by the processor for processing. Multiple sets of processors, memory devices, and interfaces may be provided according to the function of the mobile unit 100, and each set may be configured to communicate with one another.

[0029] The control unit 130 acquires output such as image information output by the detection unit 114, user input information acquired by the operation unit 132, and voice information output by the voice input unit 133, and performs processing according to each piece of information. The control unit 130 performs behavioral control, including the movement of the mobile body 100, by controlling the motors 123a and 123b of the driving unit 125. The control unit 130 notifies the user or people in the vicinity of information. For example, the control unit 130 notifies the user of information by controlling the display of images displayed on the front display unit 113 and the top display unit 131. The control unit 130 may also notify the user of information by emitting light from the light-emitting unit 116. The control unit 130 may also notify the user of information by outputting voice from the voice output unit 115. The control unit 130 may perform processing on the image information from the detection unit 114 using a machine learning model for image recognition (e.g., a deep neural network). Furthermore, the control unit 130 may perform processing on the audio information from the audio input unit 133 using a machine learning model for speech recognition (e.g., a deep neural network).

[0030] The control unit 130 can use a machine learning model to recognize feature points in images captured by the detection unit 114. Furthermore, the control unit 130 can use the recognized feature points and map information pre-stored in the memory device 135 to estimate the self-position of the moving object 100, and can perform a series of operations, such as the recovery process described later.

[0031] The top display unit 131 may be an image display device. The top display unit 131 is provided on the upper surface of the upper end of the upper housing 112. Based on image information acquired from the control unit 130, the top display unit 131 displays images and characters related to information to be notified to the user. For example, the top display unit 131 displays an operation screen for accepting user input.

[0032] The operation unit 132 may be an input device such as a touch panel or keyboard. The operation unit 132 is, for example, provided on the display surface of the top display unit 131. The operation unit 132 receives input from the user and outputs it to the control unit 130.

[0033] The audio input unit 133 may be a microphone or the like. The audio input unit 133 picks up the voice of the user and the surroundings of the mobile unit 100 and outputs it to the control unit 130.

[0034] The GNSS sensor 134 receives GNSS signals from satellites to detect the current position of the mobile object 100. GNSS stands for Global Navigation Satellite System.

[0035] The storage device 135 includes a non-volatile recording medium for storing various data. The storage device 135 may be an HDD (Hard Disk Drive) or an SSD (Solid State Drive). The storage device 135 may store programs executed by the processor, parameters necessary for program execution, and data to be processed by the processor. The storage device 135 may also store various parameters of machine learning models for speech recognition and image recognition executed by the control unit 130 (e.g., trained parameters and hyperparameters of a deep neural network). The storage device 135 can store map information of the space in which the mobile body 100 travels. The map information may be map information generated while the mobile body 100 is traveling, or it may be map information provided in advance. The map information may include, for example, positional information of areas in which the mobile body 100 can travel and areas of obstacles.

[0036] The communication unit 136 communicates with external devices such as the user's communication terminal 140 via wireless communication, such as Wi-Fi and fifth-generation mobile communication.

[0037] Figure 4 is a functional block diagram showing the functions of the control unit 130. The functions of the control unit 130 will be explained with reference to Figure 4.

[0038] The control unit 130 includes a recognition unit 401, a self-position estimation unit 402, a local map generation unit 403, a path generation unit 404, and a drive control unit 405. The control unit 130 may, for example, realize the recognition unit 401, the self-position estimation unit 402, the local map generation unit 403, the path generation unit 404, and the drive control unit 405 by reading a recovery processing program from the storage device 135 and executing it.

[0039] The recognition unit 401 recognizes features of the environment surrounding the moving object 100 from images taken by the detection unit 114 that capture the area around the moving object 100. The recognition unit 401 may use one or more trained machine learning models to recognize features of the surrounding environment from multiple images taken from various directions of the moving object 100. Features of the surrounding environment include, for example, features in the images that can be associated with roads, pathways, buildings, walls, shops, signs, markers, obstacles, traffic participants, etc.

[0040] The self-position estimation unit 402 estimates the self-position of the moving object 100 on the map based on the features of the surrounding environment obtained from the image and map information corresponding to the surrounding environment. Known self-position estimation techniques can be used to estimate the self-position of the moving object on the map. For example, the scenery of the environment can be photographed in advance, and the photographic results or feature points recognized from the photographic results can be associated with a map to create map information. The self-position estimation unit 402 can estimate its own position by determining the agreement between feature points in the image taken during movement and feature points in the map information. Such self-position estimation is also called Visual Relocalizer, and the absolute position of the moving object 100 can be obtained. The self-position estimation unit 402 may also estimate the relative self-position (movement of the moving object 100) by estimating the distance to the feature points of a specific object using multiple images taken while moving. Such self-position estimation is also called Visual SLAM. These self-position estimation techniques are illustrative, and this embodiment is not limited to them.

[0041] The self-position estimation unit 402 can perform self-position estimation and determine if self-position estimation has failed. Furthermore, if self-position estimation fails and certain conditions (e.g., traveling a predetermined distance) are met, the self-position estimation unit 402 can determine that the vehicle has lost its position. These determinations will be described later.

[0042] The local map generation unit 403 generates a local map with a narrower range than the map information pre-stored in the storage device 135, if it succeeds in estimating its own position. For example, the local map generation unit 403 can use one or more images captured by the detection unit 114 to measure the distance from the moving object 100 to surrounding obstacles and generate a map that can distinguish between the drivable area and the area with obstacles.

[0043] The route generation unit 404 generates a route for the mobile body 100 to travel. If a user is present in the travel area, the route generation unit 404 generates a route to follow or lead the user. If the route generation unit 404 loses its own position on the map, it generates a specific route to successfully estimate its own position again in the recovery process described later.

[0044] The drive control unit 405 causes the mobile body 100 to travel along the path generated by the path generation unit 404 by controlling, for example, motor 123a and motor 123b. The drive control unit 405 may also cause motor 123a and motor 123b to rotate in opposite directions to perform a pivot turn or a pivot turn (also called a turn in place) without moving the mobile body 100. The drive control unit 405 may also turn the mobile body 100 by a turn other than a pivot turn or a pivot turn.

[0045] <Overview of recovery process> Next, an overview of the recovery process according to this embodiment will be described with reference to Figures 5A and 5B. Figure 5A schematically displays the path of the mobile body 100 on the map information 500 used by the mobile body 100. Area 501 is an area where obstacles such as walls exist, and the mobile body 100 cannot pass through area 501. That is, the mobile body 100 travels in areas other than area 501 so as not to come into contact with area 501.

[0046] For example, suppose that the mobile object 100 is traveling along route 503 while the self-position estimation unit 402 estimates its own position, and that the self-position estimation is successful while traveling along route 503. The self-position estimation unit 402 calculates, for example, the number of feature points in the image taken during travel that match the feature points in the map information, and determines that the self-position estimation is successful if that number is greater than or equal to a predetermined threshold. Once the mobile object 100 has successfully estimated its own position, it can identify its own position on the map, and therefore generates a route based on its own position and the map information, and continues traveling. For example, if the self-position estimation unit 402 determines that it has successfully estimated its own position at position 502, the route generation unit 404 generates a route 504 that passes through position 502 and heads towards a specific position.

[0047] Consider the case where, immediately after the moving object 100 passes position 502, the self-position estimation unit 402 determines that the number of feature points in the image that match the feature points in the map information is less than a predetermined threshold. In this case, the self-position estimation unit 402 determines that it has failed to estimate its own position. In this case, the moving object 100 travels a predetermined distance along the path 504 while the self-position estimation unit 402 repeatedly attempts to estimate its own position. For example, if the moving object 100 has traveled a predetermined distance after failing to estimate its own position (for example, the moving object 100 is located at position 505), the self-position estimation unit 402 determines that it has lost its own position. In this case, the image taken at position 505 may contain an object with few features (for example, a wall without texture), or it may contain features that are not included in the map information. In any case, this means that a sufficient number of features for estimating the self-position cannot be obtained from the image taken on the path 506 or at position 505.

[0048] In this embodiment, an example was described in which the self-position estimation unit 402 determines that the mobile body 100 has lost its position after failing to estimate its own position and traveling a predetermined distance. However, the system is not limited to this condition. The self-position estimation unit 402 may also determine that the mobile body has lost its position if it has failed to estimate its own position more than a predetermined number of times. Alternatively, the self-position estimation unit 402 may determine that the mobile body has lost its position when it fails to estimate its own position.

[0049] If the self-position estimation unit 402 determines that it has lost its own position, the route generation unit 404 generates a specific route, as shown in Figure 5B, to successfully estimate the self-position of the moving object 100 on the map again. Such a specific route may be, for example, a route 506 that returns to the position (position 502) where the self-position estimation was last successful. If the moving object 100 can return to position 502, the probability of successfully estimating its own position increases. If the self-position estimation is successful at position 502, the route generation unit 404 can generate, for example, a route 507 (a different route from route 504, which traveled from position 502 to position 505). In this way, by traveling on a different route from the one where the self-position estimation failed, the probability of obtaining more features that match the map information can be increased.

[0050] The path generation unit 404 may generate a specific path as described above, for example, a path like path 508. Path 508 is an example of a predetermined path for obtaining feature quantities different from those of the image captured at the location (location 505) where the user has lost their own position. Path 508 is, for example, a path that includes straight movement and a change of direction of travel near an obstacle. In this way, the moving object 100 can travel around its surroundings while changing its orientation along a path generated with low processing load, and the possibility of obtaining more feature quantities that match the map information can be increased. The path generation unit 404 is not limited to such paths and may generate other paths for obtaining feature quantities different from those of the image captured at location 505.

[0051] Referring to Figure 6A, a series of operations for the recovery process performed by the control unit 130 will be described. The series of operations for the recovery process is realized by the control unit 130 executing a program stored in the memory device 135, and each part within the control unit 130 executing its own processes. Furthermore, the series of operations for the recovery process may be performed when the mobile body 100 is autonomously moving. Immediately before the start of the recovery process, it is assumed that the self-position estimation described later has been successful at least once, and a local map of the location immediately before that has been generated.

[0052] In S601, the recognition unit 401 recognizes the characteristic features of the environment surrounding the mobile object 100 from the image taken of the area around the mobile object 100. In S602, the self-position estimation unit 402 estimates the self-position on the map based on the characteristic features of the environment surrounding the mobile object 100 and map information.

[0053] In S603, the self-position estimation unit 402 determines whether it has succeeded in estimating its own position. For example, the self-position estimation unit 402 calculates the number of feature points in the image acquired in S601 that match the feature points in the map information, and determines that it has succeeded in estimating its own position if that number is greater than or equal to a predetermined threshold. If the self-position estimation unit 402 determines that it has succeeded in estimating its own position, it proceeds to S604; otherwise, it proceeds to S605.

[0054] In S604, the local map generation unit 403 generates a local map of the location where its own position has been successfully estimated (for example, position 502). For example, as shown in Figure 6B, the local map 610 has a narrower range than the map information 500 that is pre-stored in the storage device 135. The local map generation unit 403 generates the local map 610 by estimating the distance to the feature points of a specific object using multiple captured images, for example, Visual SLAM (VSLAM) technology. If a specific object is above a predetermined height from the ground, the specific object may be set as an obstacle. The local map 610 shows that the moving object 100 is located at position 502, and that an obstacle 601, such as a wall, exists at a relative distance from the moving object 100.

[0055] In S605, the self-position estimation unit 402 determines whether the moving body 100 has traveled a predetermined distance without being able to estimate its own position. The self-position estimation unit 402 determines whether the moving body 100 has traveled a predetermined distance based, for example, on its relative positional relationship with the obstacle 601 in the generated local map 610. The self-position estimation unit 402 may also determine whether the moving body 100 has traveled a predetermined distance based, for example, on three-dimensional inertial motion (translational motion and rotational motion in the orthogonal three-axis direction) measured by an inertial measurement unit (IMU), or based on the amount of movement of the moving body 100 based on the amount of rotation of the two drive wheels 120.

[0056] In S606, if the self-position estimation unit 402 determines that the moving object 100 has traveled a predetermined distance without being able to estimate its own position, it determines that the moving object 100 has lost its own position at its current location.

[0057] In S607, the path generation unit 404 determines whether it is possible to travel to the point (position 502) where it last successfully estimated its own position. For example, if the number of times the translational acceleration and angular acceleration in the three-dimensional inertial motion measured by the inertial measuring device exceed their respective thresholds is greater than or equal to a predetermined value, the path generation unit 404 determines that the reliability of the wheel odometry of the moving body 100 is low. If the reliability of the wheel odometry of the moving body 100 is low, the path generation unit 404 determines that it is not possible to travel to position 502. In addition to the above determination, or alternatively, the path generation unit 404 may also determine that the reliability of the wheel odometry of the moving body 100 is low if the average values ​​of the translational acceleration and angular acceleration are greater than or equal to a threshold. If the path generation unit 404 determines that it is not possible to travel to the point (position 502) where it last successfully estimated its own position, it proceeds to S609; otherwise, it proceeds to S608.

[0058] The route generation unit 404 may determine, by using VSLAM, whether it is possible to travel to the point (position 502) where it last successfully estimated its own position. The route generation unit 404 may also determine that it is not possible to travel to the point (position 502) where it last successfully estimated its own position if the number of feature points in the image recognized by the recognition unit 401 is below a threshold. In addition to or as an alternative to the above determination method, the route generation unit 404 may also determine that it is not possible to travel to the point (position 502) where it last successfully estimated its own position if the number of features in the image that match the map information is below a predetermined threshold.

[0059] In S608, the path generation unit 404 generates a path that returns to the point (position 502) where the self-position estimation was last successful, and the drive control unit 405 drives the mobile body 100 along this path. The path generation unit 404 also generates a path different from the previously driven path from position 502 and drives the mobile body along this path. The process for returning to the point where the self-position estimation was successful and driving a different path from the previously driven path (retry process) will be described later with reference to Figure 7.

[0060] In S609, the path generation unit 404 generates a predetermined path that does not return to the point (position 502) where the self-position estimation was last successful, and the drive control unit 405 drives the mobile body 100 along this path. The process for traveling along this predetermined path (search process) will be explained with reference to Figure 8.

[0061] In this way, depending on the reliability of the movement up to the point where the self-position is lost, it is possible to adaptively control whether to return to the position where the self-position was successfully estimated and restart path generation, or to perform exploratory path generation to acquire new feature points.

[0062] Referring to Figure 7, the series of operations of the retry process performed by the control unit 130 will be described. The series of operations of the retry process is realized when the control unit 130 executes a program stored in the memory device 135, and each part within the control unit 130 executes its own processing.

[0063] In S701, the route generation unit 404 switches the map information to local map information. That is, the route generation unit 404 switches the map information to be used from map information 500 to local map information 610. In S702, the route generation unit 404 uses the local map 610 to generate a route to the point (position 502) where its own position has been successfully estimated. For example, the route generation unit 404 uses the feature points of obstacles 601 in the local map 610 to generate a route such that the relative amount of movement of the moving object 100 is from position 505 to position 502. In S703, the drive control unit 405 drives the moving object 100 to the point (position 502) where its own position has been successfully estimated, using the route generated by the route generation unit 404.

[0064] In S704, the path generation unit 404 generates a path different from the previously traveled path (i.e., the path from position 502 to position 505), and the drive control unit 405 travels from position 502 to position 505. The path different from the previously traveled path includes, for example, path 1102 shown in Figure 11. The path generation unit 404 generates a path different from the previously traveled path by applying a cost function that maximizes the cost on the previously traveled path (i.e., the path from position 502 to position 505). For example, the cost function is a cost function that follows a normal distribution with a domain extending in the direction perpendicular to the path. By setting cost 1101, which maximizes the cost on the previously traveled path, the movement of the mobile body can be controlled to avoid traveling on paths where self-position estimation failed as much as possible. If the cost function in this embodiment follows a normal distribution as described above, by increasing the magnitude of the variance in the domain direction, for example, a path that is even further away from path 504 than path 1102 can be generated. Alternatively, by reducing the magnitude of the variance in the domain direction of the normal distribution, it is possible to generate a path that is closer to path 504 than to path 1102, for example.

[0065] The route generation unit 404 may set the magnitude of the variance of the normal distribution according to the surrounding circumstances. For example, the route generation unit 404 reduces the magnitude of the variance as the number of features recognized (or the number of features that match the map information) at a point (position 505) where the vehicle has lost its own position increases. In this way, if a relatively large number of features can be obtained near the previously traveled route, the vehicle's movement can be controlled so as not to travel a route that differs significantly from the previous route.

[0066] The path generation unit 404 may change the magnitude of the variance of the normal distribution according to the number of dynamic obstacles (autonomously moving obstacles, such as traffic participants) present around the moving object 100. For example, the path generation unit 404 increases the variance as the number of dynamic obstacles increases. That is, the more dynamic obstacles there are, the further the path will deviate from the previous path.

[0067] In S705, the drive control unit 405 controls the movement from position 502 to position 505 along the path generated by the path generation unit 404.

[0068] This retry process increases the likelihood of successfully estimating the self-position again from feature points in the image by returning to the point where the self-position estimation was last successful. Furthermore, by traveling a different route than the one where the self-position estimation failed, the likelihood of obtaining more features that match the map information can be increased.

[0069] Referring to Figure 8, a series of operations of the search process performed by the control unit 130 will be described. The series of operations of the search process is realized when the control unit 130 executes a program stored in the memory device 135, and each part within the control unit 130 executes its own processing.

[0070] In S801, the path generation unit 404 generates a predetermined path by setting the curvature of the path. For example, Figure 9 shows that a helical path that circles around position 505 can be generated as the predetermined path. For example, the predetermined trajectory may be a curved path in which the radius of the arc-shaped path increases by a constant rate. Because the radius of the path increases by a constant rate, a uniform curve can be obtained. By setting the radius of the path to exceed the minimum turning radius of the moving body 100, the moving body 100 can move smoothly.

[0071] When the moving body 100 is traveling along the spiral path 901, the drive control unit 405 stops the moving body 100 on the path if a dynamic obstacle exists on the path. Alternatively, the path generation unit 404 may generate a path that avoids the obstacle if the maximum deviation from the path 901 to avoid the obstacle is within a predetermined threshold. On the other hand, if the maximum deviation from the path 901 to avoid the obstacle is greater than a predetermined threshold, the drive control unit 405 may stop on the path, or the path generation unit 404 may generate a new path of a different shape (for example, the path 1001 described later).

[0072] Furthermore, the predetermined path may be a path constructed using a clothoid curve. This predetermined path is a curve whose curvature increases at a constant rate. Because the curvature of such a path changes linearly, the change in acceleration becomes smoother. Such a path is suitable for situations where the moving object 100 is moving at high speed, or where sudden acceleration and deceleration should be avoided.

[0073] In S802, the drive control unit 405 causes the mobile body 100 to travel along a predetermined path from the point (position 505) where it lost its own position. In S803, the recognition unit 401 recognizes the characteristic features of the environment surrounding the mobile body 100 while it is traveling. In S804, the self-position estimation unit 402 estimates the self-position on the map based on the characteristic features of the environment surrounding the mobile body 100 and the map information 500. In S803 and S804, the recognition unit 401 and the self-position estimation unit 402 can perform the same processing as described in S601 and S602 above.

[0074] In S805, the self-position estimation unit 402 determines whether it has succeeded in estimating its own position. The self-position estimation unit 402 can make this determination in the same way as described in S603. If the self-position estimation unit 402 determines that it has succeeded in estimating its own position, it terminates the series of operations of the search process; otherwise, it returns to S803. When the process returns to S803, the mobile body 100 repeatedly acquires features of the surrounding environment and estimates its own position while moving.

[0075] In this way, by performing the search process, the mobile object 100 can change its orientation while traveling around the area, increasing the likelihood of acquiring more features that match the map information.

[0076] In the above-described search process, the example given was that the path generated by the path generation unit 404 is a curved path. However, the path generated in the search process is not limited to this example. For example, it could be a path 1001 that includes straight movement and changes in direction of travel near obstacles, as shown in Figure 10. By changing the orientation of the mobile body 100 while traveling around the surroundings using a path generated by such a relatively simple process, the possibility of acquiring more features that match the map information can be increased. Furthermore, in the search process, self-position return control may be performed to allow the mobile body 100 to continue acquiring surrounding features for a predetermined period of time by performing a super-pivot turn or a pivot turn.

[0077] Furthermore, the path generation unit 404 may generate a path that leads in a direction where more features are present in the captured image. The path generation unit 404 may also generate a path that proceeds to a region in the image recognized by the recognition unit 401 that has a high density of feature points or a large number of feature points. In this way, the possibility of obtaining more features that match the map information can be increased.

[0078] In the above-described embodiment, the control device was explained as being the control unit 130 of the mobile body 100. However, the control device may be included in, for example, an information processing server located outside the mobile body 100. In this case, the control device may perform some or all of the processing performed by the control unit 130 while communicating with the mobile body 100. In other words, this embodiment is also applicable when the mobile body 100 is remotely controlled from a device separate from the mobile body 100.

[0079] As described above, in the above embodiment, the control unit 130 recognizes feature quantities of the environment surrounding the mobile body 100 from images taken around the mobile body 100. Based on the feature quantities of the surrounding environment and map information 500 corresponding to the environment surrounding the mobile body 100, the control unit 130 estimates the mobile body's own position on the map and controls the movement of the mobile body while repeatedly estimating its own position. At this time, if the control unit 130 loses its own position due to a failure in estimating the mobile body's own position on the map, it causes the mobile body to perform a specific movement to successfully estimate its own position on the map again using feature quantities from the captured images. In this way, the possibility of obtaining more feature quantities that match the map information can be increased. In other words, if the estimation of the self-position using images fails, it becomes possible to successfully estimate the self-position again.

[0080] <Summary of Embodiments> (Composition 1) A method for controlling a moving object (for example, 100), Recognizing the characteristic features of the environment surrounding the moving object from images taken around the moving object (for example, S601), Estimating the self-position of the moving object on a map based on the characteristic quantities of the environment surrounding the moving object and map information corresponding to the environment surrounding the moving object (for example, S602), This includes controlling the movement of the moving body while repeatedly estimating its own position (for example, S608, S609), Controlling the movement of the moving object includes, if the moving object loses its own position due to a failure to estimate its own position on the map (e.g., S606), causing it to perform a specific movement (e.g., S608, S609) in order to successfully estimate its own position on the map again using the features from the captured image.

[0081] This configuration, for example, increases the likelihood of obtaining more features that match the map information, and makes it possible to successfully estimate the self-position again if the self-position estimation using the image fails.

[0082] (Configuration 2) The control method according to configuration 1, wherein the specific movement includes a first movement from a first point where the moving body has lost its own position to a second point where the moving body has successfully estimated its own position, based on information about the amount of movement recorded in the moving body.

[0083] With this configuration, for example, by returning to a point where self-position estimation was successful, the probability of successfully estimating self-position again from feature points in the image can be increased.

[0084] (Composition 3) The control method according to Configuration 1, wherein the specific drive includes a second drive from the first point where the state of losing its own position occurred, along a predetermined path for obtaining a feature quantity different from the feature quantity of an image taken at the first point where the state of losing its own position occurred.

[0085] This configuration allows, for example, the mobile object 100 to change its orientation while traveling around, increasing the likelihood of acquiring more features that match the map information. (Composition 4) The aforementioned specific driving is Based on the information regarding the amount of movement recorded in the moving body, a first journey is made from the first point where the body lost its own position to the second point where the body's own position was successfully estimated. The control method according to configuration 1, comprising a second drive from the first point where the self-position has been lost, along a predetermined path for obtaining a feature quantity different from the feature quantity of an image taken at the first point where the self-position has been lost.

[0086] With this configuration, for example, by returning to a point where self-position estimation was successful, the probability of successfully estimating self-position again from feature points in the image can be increased, or by driving around while changing the orientation of the moving object 100, the probability of acquiring more features that match the map information can be increased.

[0087] (Composition 5) Further including determining whether the first run is possible, Controlling the movement of the moving body is the control method according to configuration 4, wherein if it is determined that the first movement is not possible, the second movement is performed.

[0088] This configuration allows for adaptive control over whether to return to a position where self-position estimation was successful and restart path generation, or to perform exploratory path generation to acquire new feature points.

[0089] (Composition 6) The control method according to configuration 2, wherein controlling the movement of the moving body includes, after returning to the second point, traveling along a second path different from the first path traveled from the second point to the first point.

[0090] This configuration allows for, for example, increasing the likelihood of obtaining more features that match the map information by traveling a different route than the one in which self-position estimation failed.

[0091] (Composition 7) The control method according to configuration 6, wherein the moving object travels along a second path that is different from the first path traveled from the second point to the first point, by applying a cost function that results in the highest cost along the first path.

[0092] This configuration allows for controlling the movement of a mobile object to avoid traveling along paths where self-position estimation failed.

[0093] (Composition 8) The cost function that results in the highest cost along the first path is a cost function that follows a normal distribution. The control method according to configuration 7, wherein controlling the movement of the moving body generates different second paths by changing the magnitude of the variance of the normal distribution.

[0094] This configuration allows, for example, to control the magnitude of the difference from the previous route.

[0095] (Composition 9) The control method according to configuration 3, wherein the predetermined path for obtaining a feature quantity different from the feature quantity of the image captured at the first location is a path that includes straight movement and a change in the direction of travel in the vicinity of an obstacle.

[0096] With this configuration, for example, by changing the orientation of the moving object 100 while traveling around the area using a path generated by a relatively simple process, it becomes possible to increase the likelihood of obtaining more features that match the map information.

[0097] (Composition 10) The control method according to configuration 3, wherein the predetermined path for obtaining a feature quantity different from the feature quantity of the image captured at the first location is a curved path in which the radius of the path increases by a constant rate.

[0098] This configuration allows, for example, the mobile object 100 to change its orientation while traveling around, increasing the likelihood of acquiring more features that match the map information. Furthermore, the path becomes a uniform curve. By setting the radius of the path to exceed the minimum turning radius of the mobile object 100, the mobile object 100 can move smoothly.

[0099] (Composition 11) The control method according to configuration 3, wherein the predetermined path for obtaining a feature quantity different from the feature quantity of the image captured at the first location is a curved path whose curvature increases at a constant rate.

[0100] This configuration allows, for example, the mobile object 100 to change its orientation while traveling around, increasing the likelihood of acquiring more features that match the map information. The changes in acceleration while traveling along the path become smoother.

[0101] (Composition 12) The control method according to configuration 3, wherein the predetermined path for obtaining a feature quantity different from the feature quantity of the image captured at the first location is a path that moves in a direction in which more feature quantities exist in the captured image.

[0102] This configuration, for example, increases the possibility of obtaining more features that match the map information.

[0103] (Composition 13) A control device (e.g., 130) for a mobile body (e.g., 100), A recognition means (e.g., 401) that recognizes characteristic quantities of the environment surrounding the moving object from an image taken around the moving object, Estimation means (e.g., 402) for estimating the self-position of the moving object on a map based on the characteristic quantities of the environment surrounding the moving object and map information corresponding to the environment surrounding the moving object, The system includes control means (e.g., 404, 405) that control the movement of the moving body while repeatedly estimating its own position, The control means is a control device that, when the moving object loses its own position due to a failure to estimate its own position on the map, causes the moving object to perform a specific movement in order to successfully estimate its own position on the map again using the feature quantities from the captured image.

[0104] This configuration, for example, increases the likelihood of obtaining more features that match the map information, and makes it possible to successfully estimate the self-position again if the self-position estimation using the image fails.

[0105] (Composition 14) A moving object (for example, 100), A recognition means (e.g., 401) that recognizes characteristic quantities of the environment surrounding the moving object from an image taken around the moving object, Estimation means (e.g., 402) for estimating the self-position of the moving object on a map based on the characteristic quantities of the environment surrounding the moving object and map information corresponding to the environment surrounding the moving object, The system includes control means (e.g., 404, 405) that control the movement of the moving body while repeatedly estimating its own position, The control means, in the event that the mobile body loses its own position due to a failure to estimate its own position on the map, causes the mobile body to perform a specific movement to successfully estimate its own position on the map again using the feature quantities from the captured image.

[0106] This configuration, for example, increases the likelihood of obtaining more features that match the map information, and makes it possible to successfully estimate the self-position again if the self-position estimation using the image fails.

[0107] The invention is not limited to the embodiments described above, and various modifications and changes are possible within the scope of the gist of the invention. [Explanation of symbols]

[0108] 100 ···Mobile unit, 113 ···Front display unit, 115 ···Audio output unit, 116 ···Light-emitting unit, 130 ···Control unit, 140 ···Communication terminal, 401 ···Recognition unit, 402 ···Self-position estimation unit, 403 ···Local map generation unit, 404 ···Route generation unit, 405 ···Drive control unit

Claims

1. A method for controlling a moving object, Recognizing characteristic features of the environment surrounding the moving object from images taken around the moving object, Based on the characteristic features of the environment surrounding the moving object and map information corresponding to the environment surrounding the moving object, the self-position of the moving object on the map is estimated. This includes controlling the movement of the moving body while repeatedly estimating its own position, Controlling the movement of the moving object includes, if the moving object loses its own position due to a failure to estimate its own position on the map, causing it to perform a specific movement to enable the estimation of its own position on the map again using the features from the captured image.

2. The control method according to claim 1, wherein the specific travel includes a first travel from a first point where the mobile body has lost its own position to a second point where the mobile body has successfully estimated its own position, based on information about the amount of travel recorded in the mobile body.

3. The control method according to claim 1, wherein the specific driving includes a second driving from the first point where the state of losing its own position occurred, along a predetermined path for obtaining a feature quantity different from the feature quantity of an image taken at the first point where the state of losing its own position occurred.

4. The aforementioned specific driving is Based on the information regarding the amount of movement recorded in the moving body, a first run is performed from the first point where the body lost its own position to the second point where the body's own position was successfully estimated. The control method according to claim 1, further comprising a second drive from the first point where the self-position has been lost, along a predetermined path for obtaining a feature quantity different from the feature quantity of an image taken at the first point where the self-position has been lost.

5. Further including determining whether the first run is possible, The control method according to claim 4, wherein controlling the movement of the moving body is to cause the second movement to occur if it is determined that the first movement is not possible.

6. The control method according to claim 2, wherein controlling the movement of the moving body includes, after returning to the second point, traveling along a second path different from the first path traveled from the second point to the first point.

7. The control method according to claim 6, wherein the moving object is controlled to travel along a second path that is different from the first path traveled from the second point to the first point by applying a cost function that results in the highest cost along the first path.

8. The cost function that results in the highest cost along the first path is a cost function that follows a normal distribution. The control method according to claim 7, wherein controlling the movement of the moving body generates different second paths by changing the magnitude of the variance of the normal distribution.

9. The control method according to claim 3, wherein the predetermined path for obtaining a feature quantity different from the feature quantity of the image captured at the first location is a path that includes straight movement and a change in the direction of travel in the vicinity of an obstacle.

10. The control method according to claim 3, wherein the predetermined path for obtaining a feature quantity different from the feature quantity of the image captured at the first location is a curved path in which the radius of the path increases by a constant rate.

11. The control method according to claim 3, wherein the predetermined path for obtaining a feature quantity different from the feature quantity of the image captured at the first location is a curved path in which the curvature of the path increases at a constant rate.

12. The control method according to claim 3, wherein the predetermined path for obtaining a feature quantity different from the feature quantity of the image captured at the first location is a path that moves in a direction in which there are more feature quantities in the captured image.

13. A control device for a mobile body, A recognition means for recognizing characteristic quantities of the environment surrounding the moving object from an image taken of the area surrounding the moving object, An estimation means for estimating the self-position of the moving object on a map based on the characteristic quantities of the environment surrounding the moving object and map information corresponding to the environment surrounding the moving object, Includes control means for controlling the movement of the moving body while repeatedly estimating its own position, The control means is a control device that, when the moving object loses its own position due to a failure to estimate its own position on the map, causes the moving object to perform a specific movement in order to successfully estimate its own position on the map again using the feature quantities from the captured image.

14. It is a mobile object, A recognition means for recognizing characteristic quantities of the environment surrounding the moving object from an image taken of the area surrounding the moving object, An estimation means for estimating the self-position of the moving object on a map based on the characteristic quantities of the environment surrounding the moving object and map information corresponding to the environment surrounding the moving object, Includes control means for controlling the movement of the moving body while repeatedly estimating its own position, The control means, in the event that the mobile body loses its own position due to a failure to estimate its own position on the map, causes the mobile body to perform a specific movement to successfully estimate its own position on the map again using the feature quantities from the captured image.

15. A program for causing a computer to perform each step of the control method described in any one of claims 1 to 12.

16. A storage medium storing a program for causing a computer to perform each step of the control method described in any one of claims 1 to 12.

Images