Information processing device, information processing method, and computer-readable medium

Abstract

The present invention discloses an information processing device, an information processing method, and a computer-readable medium. An information processing device is provided. An acquisition unit acquires new positions of feature points included in the map information based on map information and information obtained by a sensor. An update unit limits updating of the map information when a distance between a first position of a vehicle estimated from the positions of the feature points included in the map information and a second position of the vehicle estimated based on the acquired new positions of the feature points is greater than a threshold value, and updates the map information between when the vehicle, traveling near the first position, reaches a predetermined distance or more from the first position and when the vehicle returns to the vicinity of the second position when the vehicle continues to travel along a closed route.

Description

Technical Field

[0001] The present invention relates to an information processing device, an information processing method and a computer-readable medium, and more particularly, to a technology for updating map information based on an image. Background Art

[0002] Until now, there has been a technology for generating a map for calculating the position and orientation of sensors mounted on autonomous vehicles, calculating the sensor's position and orientation by referring to the generated map, and controlling the vehicle. This technology includes updating the map multiple times based on sensor values ​​to improve map accuracy, and adding (updating) the map at any time based on the time series of sensor values ​​to calculate the position and orientation in unmapped areas. Japanese Patent No. 5444952 discusses a method for automatically generating and updating a map while a vehicle moves autonomously. Summary of the Invention

[0003] According to one aspect of the present invention, an information processing device is provided, including: an acquisition unit configured to: acquire new positions of feature points included in map information based on map information and sensor-acquired information, the map information including positions of feature points pre-measured in an environment including a closed route along which a vehicle travels, the sensor-acquired information including feature points measured by sensors installed on a vehicle when the vehicle travels along the closed route; and an update unit configured to: limit the update of the map information when a difference between a first position of the vehicle estimated from the positions of the feature points included in the map information and a second position of the vehicle estimated based on the acquired new positions of the feature points is greater than a threshold value, and, when a vehicle traveling near the first position continues to travel along the closed route, update the map information between when the vehicle reaches a predetermined distance or more from the first position and when the vehicle returns to the vicinity of the second position.

[0004] According to another aspect of the present invention, an information processing method is provided, comprising: acquiring new positions of feature points included in map information based on map information and sensor-acquired information, the map information including positions of feature points pre-measured in an environment including a closed route along which a vehicle travels, the sensor-acquired information including feature points measured by sensors mounted on a vehicle as the vehicle travels along the closed route; limiting the update of the map information in a case where a difference between a first position of the vehicle estimated from the positions of the feature points included in the map information and a second position of the vehicle estimated based on the acquired new positions of the feature points is greater than a threshold value, and updating the map information between a time when the vehicle reaches a predetermined distance or more from the first position and a time when the vehicle returns to the vicinity of the second position when the vehicle traveling near the first position continues to travel along the closed route.

[0005] Further features of the present invention will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 is a block diagram showing a functional configuration example of an information processing apparatus.

[0007] Figure 2 is a flowchart showing a processing procedure performed by the information processing apparatus.

[0008] Figure 3 is a diagram showing a hardware configuration example of an information processing apparatus.

[0009] Figure 4 is a diagram illustrating an example of a graphical user interface (GUI).

[0010] Figure 5 is a flowchart showing the details of the processing.

[0011] Figure 6 is a flowchart showing the details of the processing.

[0012] Figure 7 is a block diagram illustrating a functional configuration example of an information processing apparatus.

[0013] Figure 8 is a block diagram showing a functional configuration example of an information processing apparatus.

[0014] Figure 9 is a flowchart showing a processing procedure performed by the information processing apparatus.

[0015] 10A to 10D : is a diagram showing an example of the position estimation results of a vehicle before and after map update.

[0016] Figure 11: is a diagram showing an example of timing when a map is updated. DETAILED DESCRIPTION

[0017] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the claimed invention. While various features are described in the embodiments, this does not limit the invention to all such features, and multiple such features may be appropriately combined. Furthermore, in the accompanying drawings, identical reference numerals are assigned to identical or similar configurations, and repeated description thereof will be omitted.

[0018] For example, according to the technology discussed in Japanese Patent No. 5444952, a map is updated whenever new information about the position of a feature point is input. Consequently, the estimated position and orientation of a sensor or vehicle changes based on the updated map. In particular, if the difference between the original estimated position and orientation of a vehicle and its updated estimated results is large, the vehicle's movement speed and / or direction may change dramatically.

[0019] The first exemplary embodiment of the present invention can reduce abrupt changes occurring in estimation results of the position and orientation of an object whose position is estimated by using map information when the map information is updated.

[0020] This exemplary embodiment deals with a case where the information processing device 1 according to this exemplary embodiment is applied to an information processing system 100. An example of the information processing system 100 is a system including a vehicle, a sensor (measuring device), and the information processing device 1. The position and posture of the sensor mounted on the vehicle are calculated based on the sensor information measured by the sensor. Based on the calculated position and posture, a control value for moving the vehicle to the destination is calculated. The control device 11 moves the vehicle to the destination based on the control value. The system configuration is not limited to this. For example, the information processing system 100 may include a server that generates a control value, a vehicle that can communicate with the server, and an external sensor that observes the vehicle.

[0021] In this exemplary embodiment, the position and posture of the sensor are measured using simultaneous localization and mapping (SLAM) technology, a technique for calculating position and posture while generating and updating a map based on sensor information. Using SLAM technology, generating and updating a map changes the map elements used for position and posture estimation. Therefore, the estimated position and posture of the vehicle may change before and after the map is updated. Such changes may cause the vehicle's speed and / or direction to change dramatically.

[0022] Will refer to Figures 10A to 10D An example of this situation is described. In the first example (see Figure 10A and 10B), the position of the feature point is updated due to the increase in the observation viewpoint. For example, as the vehicle (camera) moves, the same feature point is observed from multiple viewpoints. This improves the depth accuracy of motion stereo and the three-dimensional position of the feature point can be calculated more accurately. If the map is rewritten with the calculated position of the feature point as the update value, the position of the feature point used for position and posture estimation changes (from white point 1000 to black point 2000), and the estimation result of the position and posture of the camera changes from estimation result 1010 to estimation result 2010. In the second example (see Figure 10C and 10D ), the positions of the feature points are updated by closed-loop detection. Closed-loop detection refers to a process in which, for example, when a vehicle has passed through a cyclical travel environment and returned to the starting point, the initially estimated starting point is matched with the estimated position of the starting point at the time of return. By comparing the image initially taken at the starting point with the image taken when the vehicle returns to the same point as the starting point or near the starting point, it is determined whether the two points are the same. If the similarity between the image features is higher than a predetermined value, the two points are determined to be the same. If the two points are determined to be the same, the three-dimensional positions (feature points) of the map are recalculated so that the world coordinates of the feature points contained in the images taken at the same point are consistent. This changes the three-dimensional positions of the feature points used for position and posture estimation and changes the estimation results of the position and posture of the camera. Examples of situations where closed-loop processing is required or where the feature points used for position and posture estimation are increased or decreased or where the positions of feature points are changed include situations where the position estimation error of the feature points exceeds a predetermined threshold or such errors accumulate, and situations where the environment changes. In the former case, the influence of the error varies greatly depending on the measurement accuracy of the sensor and the performance of the device performing the estimation processing. In the latter case, examples of environmental changes include changes in the layout of the travel environment, changes due to the movement of people or objects, and changes in lighting conditions. Such environmental changes may increase or decrease the number of feature points in the pre-prepared map, or shift initially observed feature points. Therefore, it is desirable to perform self-position and pose estimation in conjunction with map updates.

[0023] In this exemplary embodiment, a determination is made as to whether the vehicle is within a predetermined range from an area where the calculated position and posture of sensor 10 have significantly changed due to a map update (equivalent to an update area where elements have been rewritten due to a map update). For example, feature points whose coordinates have been rewritten are extracted, and the convex hull area surrounding the feature points is assumed to be the update area. If the vehicle is outside the predetermined range from the update area, the map is updated. This is because if the distance between the map update area and the vehicle's position is large, the vehicle's movement is less likely to be affected by the map update.

[0024] like Figure 1As shown, the information processing system 100 according to this exemplary embodiment includes a sensor 10, an information processing device 1, and a control device 11. The information processing device 1 includes an update information input unit 110, a difference acquisition unit 120, and an update behavior determination unit 130. The information processing device 1 according to this exemplary embodiment also includes a sensor information input unit 12, a position and posture estimation unit 13, a map storage unit 14, an update information generation unit 15, a map updating unit 16, and a control value determination unit 17. The sensor information input unit 12, the position and posture estimation unit 13, the map storage unit 14, the update information generation unit 15, the map updating unit 16, and the control value determination unit 17 may be included in an external device outside the information processing device 1.

[0025] The sensor 10 is a sensor (measuring device) for acquiring information about its external environment as sensor information. In the present exemplary embodiment, the sensor 10 is a stereo camera and acquires stereo images as sensor information. The sensor 10 according to the present exemplary embodiment is a camera mounted on a vehicle and therefore captures images of the environment in which the vehicle is traveling.

[0026] The control device 11 is a control device or actuator used to control the vehicle based on the control value calculated by the control value determination unit 17. Examples include motors, power cylinders, and power converters for driving wheels or steering devices. In this exemplary embodiment, the control device 11 includes the wheel motors of a two-wheel drive vehicle. The control value specifically refers to the number of revolutions of the two drive wheels (left and right). The vehicle is controlled by changing the control value to achieve straight driving and turning.

[0027] The sensor information input unit 12 receives input regarding the distance between the sensor 10 (the vehicle) and the surrounding environment as sensor information. In this exemplary embodiment, the sensor information input unit 12 receives input of an image obtained by the sensor 10 as sensor information and outputs the sensor information to the position and posture estimation unit 13. Specifically, an image of the vehicle's surroundings captured by a camera mounted on the vehicle is used as sensor information. The relative distance between the vehicle (the measuring device) and nearby objects can be estimated from the sensor information.

[0028] The position and posture estimation unit 13 calculates the position and posture of the sensor 10 by using the sensor information input by the sensor information input unit 12 and the map information stored in the map storage unit 14. The position and posture estimation unit 13 outputs the calculated position and posture values ​​to the control value determination unit 17. The map information according to this exemplary embodiment includes key frame group information, which includes one or more key frame information indicating a specific object in the real space. Using the map information, the position and posture estimation unit 13 can compare the features of the feature points obtained from the sensor information with the features of the feature points measured in advance, and measure the position and posture of the sensor 10 in the map coordinate system. In the position and posture estimation, as is typical in SLAM technology, the position and posture of the sensor 10 are calculated so that the coordinates of the indicator extracted from the sensor information measured by the sensor 10 coincide with the coordinates of the indicator registered in the map information.

[0029] The control value determination unit 17 receives input regarding the position and posture calculated by the position and posture estimation unit 13 and calculates a control value for driving the control device 11. The control value refers to the number of revolutions of the left and right wheels of a two-wheel drive vehicle. The control value determination unit 17 calculates the control value for controlling the control device 11 so that the position and posture of the vehicle at the target point of the vehicle, input from an input unit (not shown), match the position and posture calculated by the position and posture estimation unit 13. The control value determination unit 17 outputs the calculated control value to the control device 11.

[0030] The map storage unit 14 stores map information used by the position and posture estimation unit 13 to calculate the position and posture of the sensor 10. The map storage unit 14 outputs the map information to the update information generation unit 15. The map storage unit 14 can be located outside the information processing device 1. As used herein, map information refers to a three-dimensional map representing the structure or space of an environment, such as a three-dimensional model of the environment. Specifically, the map information is three-dimensional point group data (or a combination of three-dimensional point group data and color information) or a keyframe data set. The details of the map information and keyframes will be described below.

[0031] The update information generation unit 15 generates update information (map information) for updating the map based on the positions of feature points included in the map information stored in the map storage unit 14 and the new positions of the feature points obtained from the sensor information. Here, update information refers to information indicating the difference between the positions of the feature points in the map information and the positions of the feature points obtained from the sensor information when the feature points included in the map information are observed by the sensor 10. The update information generation unit 15 outputs the generated update information about the map to the update information input unit 110. The method for updating the map information will be described below. Specifically, the update information includes the following information. First, the update information includes information for correcting the same predetermined point based on the known positions of the feature points included in the map information and the new positions of the feature points to reconcile their different positions in the map coordinate system. The same point is determined based on whether the coordinates of keyframes calculated by two methods have the same coordinate values ​​(or similar coordinate values, given calculation errors). For example, the update information generation unit 15 compares a predetermined keyframe viewed from another keyframe in the global coordinate system with a predetermined keyframe in the global coordinate system. If the difference between the two points falls within a predetermined range, the update information generation unit 15 determines that the two points are the same. Secondly, the update information includes information containing new feature point coordinates determined based on the positions of the feature points on the images observed from multiple viewpoints. Thirdly, the update information includes (keyframe coordinate) information including the new positions of the feature points obtained from the positions of the feature points on the images observed from multiple viewpoints, as well as the new position of the camera that has observed the feature points. In other words, the update information is information used to update the map information so that the positions of the feature points in the environment, when observed by the sensor 10, are associated with and match the positions of the feature points in the map information.

[0032] The update information input unit 110 receives input of update information on a map generated by the update information generation unit 15. The update information input unit 110 outputs the input update information to the difference acquisition unit 120.

[0033] The difference acquisition unit 120 acquires the difference between a first position of the vehicle estimated from the sensor information and the map information and a second position of the vehicle estimated based on the new positions of the feature points obtained from the sensor information. The sensor information represents the three-dimensional positions of the feature points obtained by observing the environment surrounding the vehicle. The map information includes information regarding the three-dimensional positions of the observed feature points. Based on the update information input by the update information input unit 110 and the map information stored in the map storage unit 14, the difference acquisition unit 120 calculates the difference in the position and posture of the sensor 10 calculated by the position and posture estimation unit 13 due to the map update. The difference refers to a value corresponding to the magnitude of the change in the position and posture of the vehicle calculated by the position and posture estimation unit 13 before and after the map update. The difference acquisition unit 120 outputs the calculated difference to the update behavior determination unit 130. The detailed processing will be described below.

[0034] The update behavior determination unit 130 determines the update behavior of the map information based on the difference calculated by the difference acquisition unit 120. As used in this exemplary embodiment, the update behavior refers to a value normalized to the range of 0 to 1 used by the map update unit 16 to determine whether to update the map. The update behavior determination unit 130 outputs the calculated update behavior to the map update unit 16. A larger value (closer to 1) indicates a higher probability of determining that the map will be updated. A smaller value (closer to 0) indicates a higher probability of suppressing updates.

[0035] The map update unit 16 rewrites the map information stored in the map storage unit 14 based on the update behavior. The map update unit 16 updates the map information based on the update information. The map update unit 16 may update the map information once or multiple times depending on the update behavior. If the difference obtained is small, the map update unit 16 limits the map update at that stage.

[0036] The map information according to this exemplary embodiment includes key frame group information, which includes one or more key frame information. Map information is generated by moving a vehicle with a sensor 10 mounted thereon in an environment in advance before the actual operation of the vehicle. For example, while the vehicle is operated by a remote control or manually, a map (map information) is generated by combining sensor information obtained by the sensor 10 mounted on the vehicle. Alternatively, by using SLAM technology, the vehicle can move autonomously in the environment to generate map information. SLAM refers to a technology that accurately estimates the position and posture of a device while identifying the surrounding environment through sensors. If key frame data is used as map information, the key frame that is closest to the estimated position and posture in the previous frame is first selected. Based on the depth map, the pixels in the current frame are projected onto the key frame using the position and posture in the previous frame. Next, the pixel values ​​of the projected frame and the pixel values ​​of the key frame are matched to estimate the position and posture, thereby minimizing the brightness difference. The position and posture of the key frame can also be updated through graphics optimization at a predetermined timing (e.g., loop closure).

[0037] A keyframe is the smallest unit of map elements used by the position and posture estimation unit 13 to calculate the position and posture of the sensor 10. Keyframe information (described below) obtained from image information about an image captured by a camera at a certain point is stored in association with the camera's position and posture at that point in time. During position and posture estimation, the camera's position and posture relative to the keyframe are calculated based on the image captured by the camera and the keyframe information. The calculated camera position and posture are then multiplied by the keyframe's position and posture to calculate the sensor 10's position and posture in the global coordinate system.

[0038] Keyframe information refers to the data structure of a keyframe. Keyframe information is a combination of a depth map acquired at regular intervals and color information acquired from an image. A depth map is a quantized color gradient representation of depth information as seen from a certain viewpoint in an environment. A keyframe can be any information used to estimate position and posture. Examples include a combination of depth information and image information (color information) regarding feature points in an image. Keyframe information includes an identifier (ID) for uniquely identifying the keyframe, image information related to the image captured by the sensor 10, feature point information (e.g., corners detected from the image), and a matrix representing the position and posture of the keyframe (i.e., the position and posture of the sensor 10 at the time the image was captured). Feature point information includes the two-dimensional coordinates u and v of the feature points detected in the image (hereinafter referred to as the two-dimensional position of the feature points or the position of the feature points), feature quantities (small cropped image patches around the feature points), and three-dimensional coordinates X, Y, and Z representing the three-dimensional position in space. The position and posture values ​​are the matrix values ​​of a 4×4 matrix M representing six parameters: three parameters representing the position of the camera in a global coordinate system defined in real space, and three parameters representing the camera's posture.

[0039] According to this exemplary embodiment, the map update information refers to the updated values ​​of the key frame information to be updated in the key frame group information included in the map information (key frame update information). The key frame update information includes an ID for identifying each key frame to be updated included in the key frame group, updated values ​​for the position and posture values ​​of the key frame to be updated, and updated values ​​for the three-dimensional positions of the feature points included in the key frame information to be updated. Multiple key frame update information is collectively referred to as key frame group update information. The key frame group update information includes key frame update information.

[0040] Figure 3 is a diagram showing the hardware configuration of the information processing apparatus 1. The central processing unit (CPU) H11 controls various devices connected to the system bus H21. The read-only memory (ROM) H12 stores the basic input / output system (BIOS) program and the startup program. The random access memory (RAM) H13 is used as the main storage device of the CPU H11. The external memory H14 stores the program to be executed by the information processing apparatus 1. The input unit H15 includes a keyboard, a mouse, buttons and / or switches, and performs processing related to information input. The display unit H16 outputs the calculation result to the display device based on the instruction from the CPU H11. The display device can be of any type. Examples include a liquid crystal display, a projector, and a light emitting diode (LED) indicator. The communication interface (I / F) H17 performs information communication via a network. The communication interface H17 can be an Ethernet ( ) interface. The communication interface H17 can be of any type. Examples include a universal serial bus (USB), serial communication, and wireless communication I / F. In this exemplary embodiment, the target location of the vehicle is input via the communication I / F H17. In this exemplary embodiment, when the update information input unit 110 receives input of update information from the update information generation unit 15 and when the update behavior calculated by the update behavior determination unit 130 is output to the map update unit 16, the input / output (I / O) unit H18 is used.

[0041] Next, the processing procedure according to the present exemplary embodiment will be described. Figure 2 1 is a flowchart showing a processing procedure of the information processing system 100 including the information processing device 1 according to the present exemplary embodiment. The processing procedure includes the following processing steps: an initialization step (step S11), a sensor information input step (step S12), a position and orientation estimation step (step S13), a control value determination step (step S14), an update information generation step (step S15), a map update step (step S16), a termination determination step (step S17), an update information input step (step S110), a difference acquisition step (step S120), and an update behavior determination step (step S130).

[0042] In step S11, the information processing device 1 initializes the information processing system 100. Specifically, the CPU H11 reads the program and map information from the external memory H14 into the RAM H13. The CPU H11 also obtains the following values ​​used by the position and posture estimation unit 13 to estimate the position and posture: the internal parameters of the sensor 10 (camera) (focal length fx (in the horizontal direction of the image) and fy (in the vertical direction of the image), the image center position cx (in the horizontal direction of the image) and cy (in the vertical direction of the image), and the lens distortion parameters). The CPU H11 also reads various parameters including the parameters of the control device 11 (such as the size of the vehicle, the wheel diameter, and the motor torque value) from the external memory H14 into the RAM H13. The CPU H11 activates the sensor 10 and the control device 11, and makes the information processing system 100 including the information processing device 1 operable and controllable. After the initialization is completed, the processing enters step S12 (sensor information input step).

[0043] In step S12, the sensor information input unit 12 receives input of sensor information obtained by measuring the surroundings of the measuring device (vehicle). Here, the sensor information input unit 12 receives input of an image captured by the camera serving as the sensor 10. The input image is converted into a corrected image in which the influence of lens distortion is eliminated by using internal parameters. The sensor information input unit 12 outputs the corrected image to the position and posture estimation unit 13. Hereinafter, the corrected image will be referred to simply as an image. Then, the processing proceeds to step S13 (position and posture estimation step).

[0044] In step S13, the position and posture estimation unit 13 estimates the position and posture of the vehicle (a first position and posture) based on the positions of feature points obtained from sensor information obtained by measuring the surroundings of the measurement device and map information containing the positions of feature points present in the environment. In other words, the position and posture estimation unit 13 calculates the position and posture of the sensor 10 using the image input by the sensor information input unit 12 and the map information stored in the map storage unit 14. The position and posture of the sensor 10 is estimated using the method of Raul et al. (Raul Mur-Artal et al., "ORB-SLAM: A Versatile and Accurate Monocular SLAM System," IEEE Journal of Robotics). First, the position and posture estimation unit 13 matches the feature points detected from the image serving as sensor information with the feature points included in the keyframe information. Next, the position and posture estimation unit 13 calculates the position and posture of the sensor 10 by solving the PnP (Perspective-n-Point) problem based on the correspondence between the three-dimensional coordinates of the matched feature points and the two-dimensional positions of the detected feature points on the image. The position and posture estimation unit 13 outputs the calculated position and posture to the control value determination unit 17. Then, the processing proceeds to step S14.

[0045] In step S14, the control value determination unit 17 determines a control value based on the position and direction of the vehicle so that the vehicle approaches a preset target point. In other words, the vehicle is moved based on the map information indicating the position and posture of the vehicle at the target point (or route) and the position and posture of the sensor 10 estimated by the position and posture estimation unit 13. In order to calculate the control value, the control value determination unit 17 calculates all possible variations of the control value that reduce the Euclidean distance between the position and posture of the vehicle and the position and posture of the target point in the map information as control value candidates. The control value determination unit 17 selects the optimal control value from the control value candidates using a dynamic window method (DWA). The control value determination unit 17 outputs the calculated control value to the control device 11. After outputting the control value, the processing enters step S15 (update information generation step).

[0046] In step S15, the update information generation unit 15 generates update information. The update information is generated by using the loop detection and pose graph optimization discussed in the method of Raul et al. (Raul Mur-Artal et al., "ORB-SLAM: A Versatile and Accurate Monocular SLAM System IEEE, Journal of Robotics"). First, the update information generation unit 15 performs loop detection to select two key frame information that calculate different positions and postures for the same physical point from the key frame group information included in the map information. Next, the update information generation unit 15 performs pose graph optimization to calculate update values ​​for the position and posture values ​​of the key frame information of the nearby key frames, as well as update values ​​for the three-dimensional coordinates of the feature points, so that the positions and postures included in the two key frame information match each other. This process will be described in more detail. First, the update information generation unit 15 matches the feature points of key frame A with the feature points of key frame B. Using the three-dimensional (3D) coordinates of the matching feature points in key frame A and the two-dimensional (2D) coordinates of the feature points on the image of key frame B, the update information generation unit 15 performs 3D-2D position and orientation calculation (thereby finding the coordinates in key frame B as seen from key frame A). Using the calculated position and orientation from A to B, the update information generation unit 15 performs 2D projection processing from the 3D coordinates of the feature points in key frame A onto key frame B, and checks whether the feature points are projected at the predetermined positions. The update information generation unit 15 performs global optimization so that the coordinates of key frame B as seen from key frame A match the coordinates of key frame B. Therefore, the update information generation unit 15 inputs the calculated plurality of key frame update information as key frame group update information (i.e., update information) into the update information input unit 110. The process then proceeds to step S110 (update information input step).

[0047] In step S110, the update information input unit 110 inputs the update information generated by the update information generation unit 15 into the difference acquisition unit 120. Then, the process proceeds to step S120.

[0048] In step S120, the difference acquisition unit 120 uses the change between the position and orientation of sensor 10 in the original map information and the position and orientation of sensor 10 in the update information to acquire a difference representing the magnitude of the change in the position and orientation estimated by the position and orientation estimation unit 13 before and after the map update. First, the difference acquisition unit 120 calculates the distance between the key frame to be updated and the sensor 10. Specifically, based on the updated position values ​​of the key frame to be updated included in each key frame update information in the key frame group update information and the position of sensor 10, the difference acquisition unit 120 acquires the 3D Euclidean distance between the coordinates of the key frame to be updated and the coordinates of sensor 10. Next, the difference acquisition unit 120 determines the minimum Euclidean distance among the calculated Euclidean distances for each key frame to be updated. In this exemplary embodiment, the reciprocal of the minimum Euclidean distance is used as the difference value (difference). In other words, the larger the distance (the less the position and orientation measurements are affected), the greater the likelihood of a map update. The difference acquisition unit 120 outputs the difference value calculated from the position and orientation measurements to the update behavior determination unit 130. Processing then proceeds to step S130.

[0049] In step S130, the update behavior determination unit 130 determines the update behavior used by the map update unit 16 to determine whether to update the map based on the difference calculated by the difference acquisition unit 120. An example of an update behavior is a real number obtained by normalizing a value indicating the ease of updating to a range of 0 to 1. The smaller the difference, the closer the update behavior is to 0. The larger the difference, the closer the update behavior is to 1. In this exemplary embodiment, an exponential function is used for normalization, using a value obtained by multiplying the difference calculated by the difference acquisition unit 120 in step S120 by -1 as an exponent. Based on the value indicating the ease of updating determined in this manner, if the value indicating the ease of updating is greater than or equal to a predetermined threshold, the update behavior determination unit 130 determines to execute an update method. The update behavior determination unit 130 outputs the update behavior for updating the map to the map update unit 16. Processing then proceeds to step S16. Alternatively, the update behavior determination unit 130 may be configured to change the update behavior based on the value indicating the ease of updating or the update information. If the value indicating the ease of updating is less than the threshold, the map is not updated. In this case, the updating behavior determination unit 130 inputs information indicating that there is no update to the map updating unit 16 .

[0050] In step S16, the map update unit 16 determines whether to update the map information stored in the map storage unit 14 based on the update behavior determined by the update behavior determination unit 130 in step S130. If it is determined that the map information is to be updated, the map update unit 16 updates the map. In this exemplary embodiment, the map update unit 16 rewrites the map information stored in the map storage unit 14 based on the updated information about the map. The process then proceeds to step S17. If the value indicating the ease of updating is less than a threshold, the map update unit 16 restricts map updates at this stage, and the process proceeds to step S17. In this case, for example, the map is updated after a predetermined time, or the map is updated step by step multiple times based on the update behavior. For example, if the area of ​​the map to be updated is far from the sensor 10, the map update unit 16 updates the map. If the area of ​​the map to be updated is close to the sensor 10, the map update unit 16 restricts map updates.

[0051] In step S17, CPU H11 determines whether to terminate information processing system 100. If the user inputs a command to terminate information processing system 100 via input unit H15 such as a mouse and keyboard ("Yes" in step S17), CPU H11 terminates information processing system 100. Otherwise ("No" in step S17), the process returns to step S12 to continue position and orientation estimation and map update processing.

[0052] In the first exemplary embodiment, the amount of change in position measurement values ​​due to updates can be reduced by enabling map updates when the sensor 10 (ie, the vehicle) is away from the update area. This can reduce sudden changes in the speed and direction of the vehicle.

[0053] In this exemplary embodiment, the difference acquisition unit 120 calculates the difference based on the distance between each key frame to be updated and the sensor 10. However, the exemplary embodiment is not limited to the above. The difference acquisition unit 120 may calculate any difference value calculated based on the key frame update information that reduces the change in the position and posture of the sensor 10 due to the map update, as calculated by the position and posture estimation unit 13. For example, the difference may be the posture difference between each key frame to be updated and the sensor 10. The posture difference refers to the angular difference between the updated posture value of the key frame to be updated, included in each key frame update information in the key frame group update information, and the posture value of the sensor 10 calculated by the position and posture estimation unit 13. Specifically, when a vehicle travels in the opposite direction of a circular route or returns to a round-trip route, the map update may be switched based on the travel direction. Alternatively, the difference acquisition unit 120 may calculate a convex hull space circumscribing the key frame group to be updated and calculate the difference based on the distance between the surface of the convex hull space and the sensor 10. Alternatively, the difference acquisition unit 120 may calculate a binary value indicating whether the sensor 10 is located in the convex hull space. Specifically, the difference acquiring unit 120 determines the difference as 1 if the sensor 10 is located in the convex hull space, and determines the difference as 0 if the sensor 10 is not located in the convex hull space.

[0054] In this exemplary embodiment, the difference is calculated as the reciprocal of the distance between the sensor 10 and the nearby object. However, the difference is not limited to the reciprocal of the distance, and any setting method can be used as long as the difference decreases as the distance increases. For example, to calculate the difference from the distance between the sensor 10 and the nearby object, an inverse cotangent function can be used. An exponential function with negative distance values ​​as exponents can also be used. A sigmoid function can be used, which substitutes the distance value, multiplies by -1, and adds 1.

[0055] The difference acquisition unit 120 can be configured to calculate the difference based on the distance between the 3D coordinates of the feature points in each key frame update information included in the key frame group update information and the sensor 10. According to this configuration, the difference acquisition unit 120 determines the feature point information of the feature point with the smallest Euclidean distance from the sensor 10 from the feature point information included in the key frame group update information. Here, the difference acquisition unit 120 uses the reciprocal of the distance value from the feature point with the smallest Euclidean distance as the difference value. The feature point with the smallest Euclidean distance is not limited, and any feature point can be used as long as the distance from the update range can be measured. Specifically, a predetermined number of feature points with a small distance from the sensor 10 can be selected, and the average or median value of the distance between the feature points and the sensor 10 can be used. The number of feature points within the predetermined distance from the sensor 10 can be calculated as the difference value.

[0056] The difference can also be calculated based on the size of the range included in the update information (i.e., the size of the area including the key frame group). Specifically, the wider the update area, the larger the calculated difference. More specifically, the difference acquisition unit 120 calculates the volume of the aforementioned convex hull space of the key frame group to be updated, and calculates the reciprocal of the volume value as the difference. Alternatively, the difference can be the reciprocal of the surface area of ​​the convex hull space. The reciprocal of the volume of the ellipse circumscribed or inscribed in the convex hull space, or the reciprocal of the length of the major axis of the ellipse can be used. Although the convex hull space is calculated to include the key frame group, the convex hull space can be calculated to include the feature points included in the update information.

[0057] The update behavior is not limited to that described in this exemplary embodiment, as long as changes in map update timing are observed. For example, the difference acquisition unit 120 can be configured to calculate differences for each keyframe to be updated, and to calculate the update behavior for each keyframe to be updated. With this configuration, updates are controlled based on the keyframes. As a result, the map update unit 16 updates keyframes with small differences earlier, while omitting updates for keyframes with large differences. This allows for stable map updates while reducing sudden changes in calculated position and posture.

[0058] The greater the difference between the estimated positions of the vehicle before and after the update, the longer it takes for the map update unit 16 to gradually update the positions and postures of the key frames or feature points to their updated values. Specifically, the map update unit 16 updates the key frames stored in the map storage unit 14 using the combined positions and postures obtained by combining the positions and postures of the key frames to be updated included in the key frame group update information and the positions and postures of the key frames stored in the map storage unit 14 as a weighted sum. Figure 11 As shown, if the update behavior of the map (here, the change in the position of the feature points before and after the update) is large, the update amount applied to the map is suppressed to a predetermined threshold, and the position of the feature points is corrected step by step multiple times to prevent the camera position from changing drastically. In particular, the larger the update behavior of the map, the more gradually the camera position can be changed by increasing the number of corrections. Each time the position and posture are measured, the weights of the position and posture of the keyframes included in the keyframe group are increased. The larger the update behavior, the smaller the value of the increase rate. Therefore, the map is updated more slowly when the influence on the position and posture estimation is greater, the amount of change in the position measurement value due to the update is reduced, and the drastic changes in the speed and direction of the vehicle are reduced.

[0059] In this exemplary embodiment, updating a map means updating the values ​​of map components, namely, updating the positions and postures of keyframes and the 3D positions of feature points. The map updating method according to this exemplary embodiment can also be used when switching maps, namely, replacing some keyframe groups included in the map information with new keyframe groups stored in a storage unit (not shown). In this configuration, the difference acquisition unit 120 calculates the difference using the new keyframe group information as keyframe group update information. This reduces the amount of change in the calculated position measurement value when switching maps and reduces the possibility of sudden changes in the vehicle's speed and direction.

[0060] Furthermore, the map updating method according to this exemplary embodiment is not limited to switching maps and can also be applied to loading a global map stored in an unillustrated storage unit into map storage unit 14 as a local map. For example, when loading a map generated by another vehicle to update its own map (adding it to the map stored in map storage unit 14), this can reduce sudden changes in the vehicle's speed and direction. Similarly, when loading a portion of a map stored in a cloud server as a local map and estimating position and posture, for example, sudden changes in the vehicle's speed and direction when loading a map (adding it to the map stored in map storage unit 14) can be reduced.

[0061] In this exemplary embodiment, map information refers to a keyframe group. However, map information is not limited to this and may include any indicator used to calculate the position and posture of sensor 10. For example, assume that the position and posture are estimated by measuring the position of an object as an indicator, that is, the position and posture are calculated based on the 3D coordinates of the object captured by a camera. In such a configuration, the object and 3D coordinates serve as map information. With such a configuration, if the updated values ​​of the object's position and posture are processed as keyframe update information according to this exemplary embodiment, the difference can be calculated using the method according to this exemplary embodiment.

[0062] The map information may be configured to not include key frames, that is, to include only feature points. Even in such a configuration, the updated values ​​of the 3D positions of the feature points may be used as the key frame update information according to the present exemplary embodiment to calculate the difference according to the present exemplary embodiment.

[0063] As a variation, if a light detection and ranging (LiDAR) sensor is used as the sensor 10, a scanning point group measured by the LiDAR or an occupancy grid map (a map having a data structure in which passable areas and impassable areas such as walls can be drawn on a grid) can be used. If the grid cells are regarded as feature points according to the present exemplary embodiment, the present exemplary embodiment can be applied to such a configuration using a LiDAR sensor. Specifically, based on the distance values ​​from each grid cell, the difference is calculated by the method used in the present exemplary embodiment. Such a method is not restrictive. If the grid cell to be updated is regarded as an update area, the distance from the update area to the sensor 10 can be determined. It can also be determined whether the sensor 10 is located in the update area based on whether the grid cell where the sensor 10 is located is to be updated. Therefore, the present exemplary embodiment can be applied to a configuration using a LiDAR sensor.

[0064] In this exemplary embodiment, the update information refers to the key frame update information included in the key frame group update information, or equivalently, the updated values ​​of the key frames to be updated or the updated values ​​of the feature points. The update information is not limited to this and can be any information that can update the map. Specifically, the update information can be correction values ​​(differences) for elements of the map stored in the map storage unit 14.

[0065] In this exemplary embodiment, the update information is described as being stored as keyframe update information. However, the configuration for storing update values ​​for rewriting elements of the map stored in map storage unit 14 is not limited to storing update values ​​as keyframe update information. Specifically, only the value of the ID used to identify the keyframe, the updated position and posture, and the updated 3D position of the feature point may be stored as update information. If the occupancy grid map described in the aforementioned variation is used as map information, the coordinates of the grid cells and the rewritten values ​​may be stored as update information.

[0066] In this exemplary embodiment, update information is generated by detecting cycles and determining update values ​​of the position and posture of the key frame to be updated and update values ​​of the 3D position of the feature point by pose graph optimization. However, the generation of update information is not limited to the aforementioned method. For example, instead of pose graph optimization, the bundle adjustment described in the method of Raul et al. (Raul Mur-Artal et al., ORB-SLAM: A Versatile and Accurate Monocular SLAM System, IEEE Journal of Robotics) can be used. The update values ​​of the position and posture of the key frame can be determined based on the position and posture calculated by an inertial measurement unit (IMU) or by a Kalman filter for establishing consistency with odometer measurements determined from integer values ​​of the number of revolutions of the wheels of the vehicle.

[0067] In addition, this exemplary embodiment can also be applied to the situation of adding a map, that is, adding new key frame information to the key frame group information included in the map information. In this case, the method according to this exemplary embodiment can be applied by using the newly added key frame information as key frame update information.

[0068] In this exemplary embodiment, the sensor 10 is a camera. Any camera capable of obtaining images that can be used for position and posture measurement can be used as the camera. Examples include monochrome cameras, color cameras, stereo cameras, and depth cameras. Depending on the camera, any sensor information capable of position and posture estimation can be used. Examples of sensor information include monochrome images, red, green, and blue (RGB) images, and depth maps.

[0069] Furthermore, sensor 10 may be any sensor capable of obtaining sensor information useful for position measurement, including a LiDAR sensor. Examples include 2D LiDAR sensors and 3D LiDAR sensors. Depending on the sensor type, any sensor information capable of estimating position and orientation may be used. Examples of sensor information include 2D point groups and 3D point groups.

[0070] The sensor 10 is not limited to the foregoing, and any sensor capable of calculating a position and a posture based on map information and sensor values ​​measured by the sensor 10 may be used.

[0071] The target control device 11 is not limited to a single control wheel, and may be any control unit capable of changing the speed, acceleration, angle, angular velocity, and / or angular acceleration of the vehicle. For example, the control device 11 may control the steering wheel, and if the vehicle is a drone, may control the rotation of the propellers.

[0072] The information processing device 1 can be applied to any configuration that calculates the position and posture by using an updated map. For example, the vehicle can be a mobile robot, an automatic guided vehicle (AGV) / an autonomous mobile robot (AMR), an unmanned ground vehicle (UGV), an autonomous underwater vehicle (AUV), an unmanned guided vehicle, an autonomous vehicle, or an unmanned aerial vehicle (UAV) such as a drone. The movement control described in this exemplary embodiment can be applied thereto. The movement control described in this exemplary embodiment can also be applied to vehicles that fly and move in the air, vehicles that move on water, and vehicles that are immersed and move in water, in addition to vehicles that walk or run on the ground.

[0073] In this exemplary embodiment, the information processing device 1 is described as being included in the information processing system 100. However, the information processing device 1 can be located anywhere as long as it can calculate map updates. For example, the information processing device 1 can be located on a cloud server separate from the information processing system 100, or in another information processing system. The timing of its operation can be freely set. Although the processing steps are described as being performed continuously in this exemplary embodiment, certain processing steps, such as steps S15 and S16, can be executed in parallel as a separate thread from the thread of steps S12 to S14.

[0074] In this exemplary embodiment, it is described that whether to update the map is determined each time an image is input. However, this configuration for determining whether to update the map each time an image is input is not restrictive, and whether to update the map may be determined at a predetermined timing.

[0075] Specifically, step S15 (update information generation step) can be configured to proceed to step S110 and subsequent steps each time a keyframe is added. Alternatively, step S110 and subsequent steps can be executed only when a loop is detected by matching the newly added keyframe with the keyframes stored in map storage unit 14. In this configuration, step S110 and subsequent steps are executed only when the map can be updated. Therefore, it is possible to determine whether to update the map while reducing the processing load.

[0076] The information processing device 1 may include a presentation unit not shown. For example, a three-color light of red, yellow and green can be used as the presentation unit. When the map is not updated, the light is lit in green, and when the map is updated, the light is lit in yellow. This can visualize to the user that the map is being updated. The indication color can be set freely as long as the visualization of the map to be updated can be presented to the user. The presentation unit can be any component as long as information about the map update status can be presented to the user. Examples include LED lights and liquid crystal displays. The presentation unit can be a speaker, which can be configured to emit a specific alarm sound or play a specific melody based on the map update status.

[0077] The presentation unit may present the difference, the update behavior and / or the update range as the information about the map update. Figure 4 An example is shown in which the presentation unit includes a display and displays a graphical user interface (GUI) GUI101 for presenting map information, differences, and update ranges.

[0078] Presentation area G110 presents map information and differences. Indicator G111 indicates the extent and difference of the map update. In a variation, the greater the difference, the darker the map update area. Presentation area G110 presents the vehicle's movement route G112 and the vehicle's current position G113.

[0079] GUI G120 is designed to allow the user to adjust map update parameters. Mouse cursor G121 can be used to vertically move threshold adjustment bar G122 to determine the degree of difference allowed for map update. Indicator G123 represents the size of the difference.

[0080] In this way, the user can be more easily presented with the areas of the map update, and the user can adjust the threshold for allowing the map update. This allows the user to intuitively understand the differences caused by the map update.

[0081] Although the display is described as being used as the presentation unit, the presentation unit may be a speaker. The presentation unit may be configured to change the volume or tone or type of melody of the warning sound based on differences in the area through which the vehicle passes.

[0082] In the first exemplary embodiment, the difference acquisition unit 120 was described as acquiring the difference using the distance from the update region. In the second exemplary embodiment, the difference is calculated based on the amount of change in the update element. In this exemplary embodiment, the amount of change refers to the magnitude of the position change resulting from the update of the key frame to be updated, which is included in the key frame group update information. The greater the position change of the key frame, the greater the calculated amount of change in the position and posture of the sensor 10. Therefore, a configuration for calculating a larger difference as the amount of change in the position of the key frame increases will be described.

[0083] The configuration of the information processing apparatus according to the second exemplary embodiment is similar to that of the information processing apparatus 1 described in the first exemplary embodiment. Figure 1 The configuration of is different in that the map information stored in the map storage unit 14 is input by the difference acquisition unit 120. Similar to the first exemplary embodiment, the information processing apparatus has Figure 3 The hardware configuration shown.

[0084] The processing according to the second exemplary embodiment is similar to Figure 2 , which shows the processing of the information processing apparatus 1 described in the first exemplary embodiment. Therefore, its description will be omitted. The difference from the first exemplary embodiment lies in the details of the processing of step S120 (difference acquisition step).

[0085] In step S120 (difference acquisition step), the difference acquisition unit 120 calculates the amount of change in the position of the key frame to be updated, included in the key frame update information included in the key frame group update information. In this exemplary embodiment, the magnitude of the position change refers to the Euclidean distance between the coordinates of the key frame before the update and the coordinates of the key frame after the update. The difference acquisition unit 120 outputs the maximum value of the calculated distance values ​​for each key frame to be updated, included in the key frame group update information, as a difference to the update behavior determination unit 130. The process then proceeds to step S130.

[0086] In the second exemplary embodiment, the map update behavior is calculated based on the amount of change in the update element. Specifically, if the change in the map update element is less than or equal to a predetermined value, the map is updated. If the update behavior exceeds a predetermined level, the map is not updated. This configuration prevents the amount of change in position measurement values ​​due to updates from reaching or exceeding the predetermined level. This reduces abrupt changes in vehicle speed and direction.

[0087] In the second exemplary embodiment, the difference is calculated based on the Euclidean distance of the coordinates of the key frame before and after the map is updated. However, the difference can be calculated by using any value of the change in the update element based on the map. Specifically, the change in the posture of the key frame can be used. The change in the 3D position of the feature point can be used. In addition, the change is not limited to the geometric change in the position or posture of the component of the map, and can be any change in the map element used for position and posture measurement. Specifically, if the feature point stores the histogram of the gradient direction in the local area of ​​the smoothed image as a feature value (such as the scale-invariant feature transform (SIFT) feature value and the orientation FAST and rotation BRIED (ORB) feature value), the update amount of the feature value can be used. The update of the feature value refers to the process of averaging the feature values ​​of feature points that have similar 3D coordinates and feature values ​​and are considered to be the same in the feature point information included in multiple key frame information, or replacing the feature value with a representative value. For example, the update amount of the feature value is determined based on the norm or cosine similarity value between the ORB or SIFT feature value vectors before and after the update. If not only the descriptors but also small patches of feature point regions are smoothed for multiple viewpoints for updating, the total value, average value, maximum value, etc. of the change in brightness of the patches before and after the update can be used.

[0088] The ratio of the map indices used by the position and posture estimation unit 13 to estimate the position and posture of the sensor 10 can be used as the amount of change. For example, the ratio of feature points used for position and posture estimation whose 3D positions have shifted by a predetermined distance or more before and after a map update can be used as the amount of change. Not only position changes but also changes in the number of map elements can be used as the amount of change. Specifically, changes in the number of feature points used for position and posture estimation can be used. Furthermore, changes in the number of referenced key frames or their ratios can be used.

[0089] If a LiDAR sensor is used as sensor 10, it is possible to calculate the grid cell to which a certain grid element of the occupancy grid map has moved due to a map update, and the Euclidean distance between the two grid cells before and after the update can be used as the difference. Furthermore, the change in the value of each grid cell of the occupancy grid map due to the map update can be used as the difference. As used herein, the value of each grid cell refers to a probability value between 0 and 1 indicating the presence or absence of an object in the occupancy grid map. In other words, the change in the probability value of a particular grid cell before and after the map update can be used as the difference.

[0090] In the second exemplary embodiment, the difference is calculated based on the amount of change in map elements due to the update. In the third exemplary embodiment, the difference is calculated not based on map elements but based on the amount of change in the position and orientation of sensor 10 calculated by position and orientation estimation unit 13 before and after the map update. Specifically, a method for calculating a difference that increases as the amount of change in the calculated position and orientation due to the map update increases will be described.

[0091] An information processing apparatus according to a third exemplary embodiment has Figure 1 Similar configuration, Figure 1 FIG1 shows the configuration of the information processing apparatus 1 described in the first exemplary embodiment. Therefore, its description will be omitted. Similar to the first exemplary embodiment, the information processing apparatus has Figure 3 The hardware configuration shown is different from the first exemplary embodiment in the processing steps in which the difference acquisition unit 120 calculates the difference.

[0092] The overall processing procedure according to the third exemplary embodiment is similar to Figure 2 resemblance, Figure 2 The processing procedure of the information processing device 1 described in the first exemplary embodiment is shown. Therefore, its description will be omitted. The difference from the first exemplary embodiment is that in step S120 (difference acquisition step), the difference acquisition unit 120 estimates the position and orientation of the sensor 10 calculated by the position and orientation estimation unit 13 after the map is updated.

[0093] Figure 5 This is a flowchart showing the details of step S120 performed by the difference acquisition unit 120 according to the third exemplary embodiment. In step S1210, the difference acquisition unit 120 selects map elements near the sensor 10. In this exemplary embodiment, based on the position and orientation calculated by the position and orientation estimation unit 13, the difference acquisition unit 120 selects a predetermined number of key frame information items related to key frames located within a predetermined distance from the sensor 10 from the key frames stored in the map storage unit 14. The process then proceeds to step S1220. In this exemplary embodiment, the selected key frame information is referred to as neighboring key frame information.

[0094] In step S1220, the difference acquisition unit 120 selects key frame information whose ID matches the ID included in the adjacent key frame information from the key frame update information included in the key frame group update information input by the update information input unit 110. The process then proceeds to step S1230. The selected key frame update information is referred to as the adjacent key frame update information.

[0095] In step S1230, the difference acquisition unit 120 uses the neighboring keyframe update information obtained in step S1220 and the image input by the sensor 10 to calculate the position and orientation of the sensor 10 or camera after the map update (hereinafter referred to as the updated position and orientation). Although the updated position and orientation are calculated using the method described in step S13, the difference is that the updated information is used instead of the map information. After the updated position and orientation are calculated, the process proceeds to step S1240.

[0096] In step S1240, the difference acquisition unit 120 calculates the distance between the updated position and orientation calculated in step S1230 and the position and orientation of the sensor 10 calculated by the position and orientation estimation unit 13. As used herein, the distance between the position and orientation refers to the Euclidean distance between the coordinates of the sensor 10 calculated in step S1230 and the coordinates of the sensor 10 calculated by the position and orientation estimation unit 13. The inverse of the Euclidean distance thus calculated is used as the difference. The process then proceeds to step S130.

[0097] In the third exemplary embodiment, the position and orientation of sensor 10 are calculated using the updated elements in the map. The greater the change in the position and orientation of sensor 10 before and after the update, the greater the calculated difference. This reduces sudden changes in the speed and direction of the vehicle.

[0098] In this exemplary embodiment, the update information is used to calculate the position and posture of sensor 10 before and after a map update. However, this method is not limiting, and any method capable of estimating the amount of change in the position and posture of sensor 10 before and after a map update may be used. For example, the map elements used by position and posture estimation unit 13 when calculating the position and posture of sensor 10 may be used instead of the neighboring keyframe information about the surroundings of sensor 10. Specifically, position and posture estimation unit 13 pre-identifies the feature points used. Difference acquisition unit 120 can then estimate the updated position and posture based on the updated values ​​of the feature point information that match the identified feature points included in the update information.

[0099] In this exemplary embodiment, the position and orientation of sensor 10 are calculated using update information. However, without performing position and orientation estimation, the change in sensor 10's position and orientation can be estimated from the change in nearby map elements. Specifically, a rigid transformation is determined that minimizes the distance between the coordinates of feature points before and after the update, and this rigid transformation is considered the change in sensor 10's position and orientation. Alternatively, a rigid transformation can be determined that minimizes the distance between the coordinates of key frames before and after the update.

[0100] Although the difference is calculated by using the inverse of the distance between the coordinates of the sensor 10 calculated before and after the update, any setting method can be used as long as the difference decreases as the distance increases. An inverse cotangent function can be used. An exponential function with negative distance values ​​as exponents can be used. A sigmoid function can be used by substituting the distance value, multiplying by -1, and adding 1.

[0101] In the third exemplary embodiment, the difference is calculated based on the amount of change in the position and orientation of sensor 10 calculated before and after a map update. In the fourth exemplary embodiment, the difference is calculated based on the amount of change in the vehicle's control value calculated before and after a map update. Specifically, a method for calculating the impact of a significant change in the vehicle's direction due to a map update will be described.

[0102] The diagram showing the configuration of the information processing apparatus according to the fourth exemplary embodiment is different from the diagram showing the configuration of the information processing apparatus 1 described in the first exemplary embodiment. Figure 1 Therefore, the figure will be omitted. The difference from the first exemplary embodiment lies in the processing steps in which the difference acquisition unit 120 calculates the difference.

[0103] The overall processing procedure according to the fourth exemplary embodiment is similar to the process procedure of the information processing apparatus 1 described in the first exemplary embodiment. Figure 2 Similar. Therefore, its description will be omitted. Unlike the first exemplary embodiment, in step S120 (difference acquisition step), the difference acquisition unit 120 estimates not only the position and orientation of the sensor 10 calculated by the position and orientation estimation unit 13 after the map update, but also the control value calculated by the control value determination unit 17 after the map update. The difference acquisition unit 120 also calculates the amount of change in the control value before and after the map update.

[0104] Figure 6 is a flowchart illustrating the details of the processing performed by the difference acquisition unit 120 according to this fourth exemplary embodiment. Steps S1210 to S1230 are similar to the steps of the processing procedure described in the third exemplary embodiment. The difference from the third exemplary embodiment is that, in step S1310, based on the position and posture of the sensor 10 calculated after the map update in step S1230, the control value after the map update and the change in the control value before and after the map update are further calculated. In this exemplary embodiment, the control value refers to the number of revolutions of the two drive wheels (left and right wheels). The change in the control value refers to the difference in the number of revolutions of each wheel before and after the map update.

[0105] In step S1310, the difference acquisition unit 120 calculates the change in the control value before and after the map update. First, the difference acquisition unit 120 calculates the control value based on the position and posture of the sensor 10 after the map update, which was calculated by the difference acquisition unit 120 in step S1230. The control value is calculated using the DWA, which reduces the Euclidean distance between the coordinates of the sensor 10 after the map update and the target position of the vehicle. In this exemplary embodiment, the target position of the vehicle is input to the difference acquisition unit 120 via an input unit (not shown). Next, the difference acquisition unit 120 calculates the difference between the control value calculated after the map update and the control value calculated by the control value determination unit 17. In this exemplary embodiment, the control value refers to the number of revolutions of the two drive wheels. The difference acquisition unit 120 calculates the average of the differences in the number of revolutions of the two drive wheels before and after the map update as the difference.

[0106] In the fourth exemplary embodiment, the greater the amount of change in the control value of the vehicle calculated before and after the map update, the greater the calculated difference. This reduces sudden changes in the speed and direction of the vehicle.

[0107] In the fourth exemplary embodiment, the difference is the average value of the difference in the number of revolutions of the two drive wheels before and after the map update. However, any value may be used as long as the difference in the control values ​​before and after the map update is reduced. If two drive wheels are used as in the present exemplary embodiment, the larger value of the difference in the number of revolutions before and after the map update may be used as the difference. Alternatively, the smaller value of the difference may be used as the difference. The difference between the weighted numbers of revolutions to which a predetermined weight is added may be used as the difference. Although the present exemplary embodiment describes a case in which the control device 11 controls the two drive wheels, the configuration of the control device 11 is not limited as long as a larger difference can be calculated as the difference in the control values ​​as the difference in the control values ​​becomes larger due to the map update.

[0108] In the fourth exemplary embodiment, DWA is used to calculate the control value. However, DWA is not restrictive, and any method that can calculate the control value that can be compared before and after the map update can be used. A graph search method can be used, which is a technique for calculating the change of the control value not at a certain point in time, but at multiple times in the future, calculating the position and posture (predicted trajectory) at multiple times, and selecting the control value for tracking the predicted trajectory. Other techniques can also be used. Examples include a random method for selecting a predicted trajectory, which is generated by randomly repeatedly sampling a certain obstacle-free area in 2D space based on the current position and further randomly sampling the nearby space with the point as a node.

[0109] In a fourth exemplary embodiment, the control values ​​are calculated and compared directly. However, in the case where the control values ​​are not calculated, the amount of change of the elements used to calculate the control values ​​can be used as a difference. Specifically, such an amount of change refers to the amount of change in the relative target position. As used herein, the amount of change in the relative target position refers to the Euclidean distance between the coordinates of the target position relative to the position and posture of the sensor 10 before the map update and the coordinates of the target position relative to the position and posture of the sensor 10 after the map update. The calculated Euclidean distance is used as the difference. The Euclidean distance is not limited and any applicable value can be used. Examples include the difference in the direction of the target position as seen from the sensor 10 before and after the map update and the norm of the difference vector between the two target positions.

[0110] In the first to fourth exemplary embodiments, the difference is calculated based on the internal state of information processing system 100 (e.g., changes in the position and posture of sensor 10 or vehicle control values ​​due to map updates). In the fifth exemplary embodiment, the difference is calculated based on the situation surrounding the vehicle. In this exemplary embodiment, the situation surrounding the vehicle refers to the arrangement of nearby objects. A method for implementing a configuration in which the map is not updated when the vehicle approaches an object, but is updated when the vehicle moves away from the object will be described.

[0111] The diagram showing the configuration of the information processing apparatus according to the fifth exemplary embodiment is different from the diagram showing the configuration of the information processing apparatus 1 described in the first exemplary embodiment. Figure 1 Therefore, its figure will be omitted. The difference from the first exemplary embodiment lies in the processing content of the processing step of calculating the difference by the difference acquisition unit 120. Similar to the first exemplary embodiment, using Figure 3 The hardware configuration shown.

[0112] The overall processing procedure according to the fifth exemplary embodiment is different from the one showing the processing procedure of the information processing apparatus 1 described in the first exemplary embodiment. Figure 2 Therefore, the description thereof will be omitted. The difference from the first exemplary embodiment is that the difference acquisition unit 120 calculates the difference based on the distance value between the feature point as the situation around the vehicle and the sensor 10 in step S120 (difference acquisition step).

[0113] In step S120, the difference acquisition unit 120 calculates the Euclidean distance between the 3D positions of the feature points included in all the key frame information included in the map information stored in the map storage unit 14 and the positions calculated by the position and posture estimation unit 13. The difference acquisition unit 120 calculates the reciprocal of the minimum value of the calculated Euclidean distances as the difference.

[0114] In the fifth exemplary embodiment, the difference is calculated based on the surrounding conditions (i.e., the distance to nearby objects). The smaller the distance to nearby objects, the larger the difference. Therefore, the smaller the distance between the vehicle and nearby objects, the smaller the sudden changes in speed and direction.

[0115] In the present exemplary embodiment, the surrounding situation refers to the arrangement of nearby objects, and the difference is calculated based on the distance between the sensor 10 and the feature point. However, any method capable of measuring the distance between the sensor 10 and the nearby object may be used.

[0116] For example, if the sensor 10 includes a depth camera or a stereo camera capable of obtaining a depth value, the depth value obtained by such a camera may be used as the distance to a nearby object.

[0117] Alternatively, depth values ​​obtained by using a convolutional neural network (CNN) trained to estimate a depth map (a data structure storing depth values ​​in each pixel of an image) from an input image may be used as the distance to a nearby object.

[0118] If the sensor 10 is a LiDAR sensor, the distance to a nearby measurement point may be used as the distance to the nearby object.

[0119] If a vehicle is configured to include a distance sensor such as an infrared sensor and an ultrasonic sensor, the measurement value of such a distance sensor may be used as the distance to a nearby object.

[0120] If a distance sensor is not installed on the vehicle, the distance between the vehicle and nearby objects can be measured using a sensor installed on an object other than the vehicle. Examples of such sensors include surveillance cameras and proximity sensors. If a layout map of the vehicle's surroundings is stored in a storage unit (not shown), the distance to nearby objects can be calculated based on the position and orientation of sensor 10 calculated by position and orientation estimation unit 13 and the layout map.

[0121] In this exemplary embodiment, the difference is calculated using the distance between sensor 10 and nearby objects. Since vehicles occupy a certain volume in space, a model of the vehicle's shape can be stored, and the shortest distance from the vehicle's surface to an object can be used as the distance value according to this exemplary embodiment. This enables more accurate measurement of the distance between nearby objects and the vehicle, and more precise calculation of the difference. Consequently, the vehicle can be operated more stably and with greater safety.

[0122] Although the difference is calculated using the reciprocal of the minimum distance between the sensor 10 and the nearby feature points, any setting method that reduces the difference value as the distance increases can be used. An inverse cotangent function can be used. An exponential function whose exponent is the negative value of the distance value can be used. A sigmoid function that substitutes the distance value, multiplies by -1, and adds 1 can be used. Although the difference is calculated using the minimum distance between the sensor 10 and the nearby feature points, the average or median value of the distances to the feature points located within a predetermined distance from the sensor 10 can be used.

[0123] In this exemplary embodiment, the difference is calculated based on the distance to the object. However, the object's attributes can be used as the surrounding conditions. If a camera is used as sensor 10, a CNN pre-trained to determine the type of object from video images captured by the camera can be used to detect the object and calculate the difference based on the type and number of objects. For example, the difference may increase if there are people or other vehicles nearby. Increasing the difference reduces the risk of the vehicle suddenly changing its direction or position due to a map update. The more people or other vehicles there are, the greater the difference may be. The difference may also increase near doors or stairs. Fragile objects can be identified to increase the difference. The difference may increase as the weight of the nearby object increases. Conversely, in some applications, a larger difference can be calculated as the weight decreases. The difference may increase as the speed of the nearby object increases. Any measurement method capable of measuring the speed of the nearby object can be used. The position change of the nearby moving object can be measured based on a time series of images. If the nearby moving object is another moving object, the speed of the other moving object can be received from a mobile object management system (not shown).

[0124] The aforementioned measures make it less likely that a sudden change in the vehicle's speed or direction due to a map update will occur if there are people, other vehicles, doors, stairs, or expensive objects nearby, or if nearby moving objects are moving at high speeds. This allows for more stable vehicle operation with greater safety.

[0125] The difference acquisition unit 120 can be configured to calculate the difference using location information as a guide to surrounding conditions, thereby increasing the difference at locations where sudden changes in the vehicle's speed or direction are undesirable. Specifically, for example, the difference can be increased at corners and intersections. If specific objects such as doors and automated machinery are registered on a map or along the travel route, the difference acquisition unit 120 can be increased at these locations. The difference acquisition unit 120 can also be configured to increase the difference on narrow routes. Furthermore, based on the travel route, if the vehicle is traveling on a route where another vehicle may be traveling nearby, the difference acquisition unit 120 can increase the difference value.

[0126] Regarding such spot information, its area can be detected based on a map or route information stored in the map storage unit 14. The spot information can be input from a not-shown vehicle management system via the communication interface H17.

[0127] The difference acquisition unit 120 may be configured to increase the difference when the sensor 10 is located in a specific setting area that is previously set by the user from an input unit (not shown). Examples of the setting area include an intrusion detection area and a map update prohibition area.

[0128] In the fifth exemplary embodiment, the difference is calculated based on information about objects around the vehicle (i.e., the distance to the objects). In the sixth exemplary embodiment, the difference is calculated based on the state of the vehicle. As employed in this exemplary embodiment, the state of the vehicle refers to its mass. The greater the mass of the vehicle, the greater the kinetic energy of the sudden change in the vehicle's speed and direction. In this case, a configuration will be described in which a larger difference is calculated as the mass of the vehicle increases.

[0129] The diagram showing the configuration of the information processing apparatus according to the sixth exemplary embodiment is different from the diagram showing the configuration of the information processing apparatus 1 described in the first exemplary embodiment. Figure 1 Therefore, its figure will be omitted. The difference from the first exemplary embodiment lies in the processing steps in which the difference acquisition unit 120 calculates the difference. Similar to the first exemplary embodiment, using Figure 3 The hardware configuration shown.

[0130] The overall processing procedure according to the sixth exemplary embodiment is similar to the procedure showing the processing procedure of the information processing apparatus 1 described in the first exemplary embodiment. Figure 2 Therefore, description thereof will be omitted. A difference from the first exemplary embodiment is that the difference acquisition unit 120 calculates the difference based on the state of the vehicle in step S120 (difference acquisition step).

[0131] In step S120, the difference acquisition unit 120 acquires the quality information on the vehicle stored in the storage unit not shown. In the present exemplary embodiment, the quality of the vehicle is used as the difference value.

[0132] In the sixth exemplary embodiment, the difference increases as the state of the vehicle (ie, the mass of the vehicle) increases. Therefore, the greater the mass of the vehicle, the smaller the drastic change in the speed and direction of the vehicle due to the map update.

[0133] In the present exemplary embodiment, the difference is calculated based on the mass of the vehicle as the state of the vehicle. However, any method that calculates a larger difference as the kinetic energy or potential energy of the vehicle becomes higher may be used.

[0134] If a vehicle transports an object, the mass of the vehicle may include the mass of the object being transported. There are no particular limitations on the method for inputting the mass of the object to be transported. The mass of the object to be transported may be obtained from a vehicle management system (not shown) via the communication interface H17, or input by a user through a GUI (not shown). The information processing device 1 may be configured to input a measurement value of a weight measuring device (not shown) mounted on the vehicle via the I / O unit H18.

[0135] Alternatively, as the moving speed or angular velocity of the vehicle becomes higher, a larger difference may be calculated.

[0136] If the vehicle is an aircraft such as a drone, a larger difference can be calculated as the altitude becomes higher.

[0137] The status of a vehicle can be determined based on the priority or importance of the vehicle's task. For example, operational information about the vehicle can be used as the status of the vehicle. Examples of operational information include moving, stationary, transporting a load, charging, and loading a load. The user inputs the difference based on each piece of operational information in advance using an input unit not shown. This makes it possible to update the map according to the operational status of the vehicle. For example, map updates can be disabled when transporting a load, while map updates can be enabled when position measurement is not used (for example, when the vehicle is stationary and when charging the vehicle).

[0138] The difference can be calculated based on vehicle attributes as the vehicle state. For example, the difference acquisition unit 120 can calculate a higher difference as the vehicle's rigidity increases. The difference acquisition unit 120 can also be configured to calculate a higher difference as the vehicle's braking distance increases. The difference acquisition unit 120 can also calculate a higher difference as the obstacle sensor's detection distance setting parameter decreases. The price of the vehicle itself can also be used as a vehicle attribute.

[0139] The difference can be calculated based on the properties of the objects to be transported mounted on the vehicle as the state of the vehicle. For example, the difference acquisition unit 120 can obtain the price of the objects to be transported from an unillustrated vehicle management system and calculate a larger difference as the price increases. The difference acquisition unit 120 can calculate a larger difference as the rigidity of the objects to be transported increases or the weight increases. The difference acquisition unit 120 can be configured to calculate a larger difference as the number of objects to be transported increases. The difference acquisition unit 120 can be configured to calculate a larger difference if the objects to be transported are not stacked smoothly. A load that is stacked smoothly means that the load does not tilt too much. Whether the load is stacked smoothly is determined by taking an image of the load with an unillustrated camera and measuring the tilt of each load. In this way, sudden changes in the speed and direction of the vehicle due to map updates can be reduced based on the properties of the objects to be transported.

[0140] Furthermore, the differences described in the first through sixth exemplary embodiments can be combined, and the difference acquisition unit 120 can calculate such a combined difference. The differences can be combined in any manner, as long as the combined difference increases as the change in the position and direction of the sensor 10, or the change in the vehicle, or the change in the vehicle's control values ​​calculated before and after the map update increases. As described in the fifth and sixth exemplary embodiments, any method that can reduce the drastic changes in the vehicle's speed and direction due to a map update based on the vehicle's state or the properties of nearby objects can be used. Specifically, a weighted sum or weighted product of the various differences calculated using the methods described in the first through sixth exemplary embodiments can be used. Furthermore, only some of the exemplary embodiments' methods can be combined to calculate the combined difference. The update behavior determination unit 130 can determine the update behavior by using the combined differences. This enables a map update that satisfies the various conditions described in the first through sixth exemplary embodiments. Consequently, the vehicle can be operated stably with greater safety.

[0141] The first to sixth exemplary embodiments are described as being applied to a vehicle. However, the exemplary embodiments can be applied to any configuration that can estimate a position and a posture based on sensors and map information and update a map, and is not limited to a configuration installed on a vehicle. The seventh exemplary embodiment deals with one of such application examples. The information processing device according to this exemplary embodiment is installed on a device that implements mixed reality (MR), augmented reality (AR), or virtual reality (VR). Examples of the device include a head-mounted display (HMD), a smartphone, and a tablet computer. The configuration of the information processing device will be described, which is installed on such a device and is used to calculate the rendering position of a computer graphics (CG) image.

[0142] Figure 7This figure shows an example of a hardware configuration in which an information processing device 1 according to this exemplary embodiment is applied to an MR system. The information processing device 1 includes a sensor 10, a display device 20, a sensor information input unit 12, a position and posture estimation unit 13, a map storage unit 14, an update information generation unit 15, a map update unit 16, and a display information generation unit 21. The information processing device 1 also includes an update information input unit 110, a difference acquisition unit 120, and an update behavior determination unit 130.

[0143] The sensor 10, sensor information input unit 12, position and posture estimation unit 13, map storage unit 14, update information generation unit 15, map update unit 16, update information input unit 110, difference acquisition unit 120, and update behavior determination unit 130 are similar to those described in the first exemplary embodiment. Therefore, their description will be omitted. The difference from the first exemplary embodiment lies in the addition of a display device 20 and a display information generation unit 21.

[0144] The display information generation unit 21 uses the position and orientation input by the position and orientation estimation unit 13 and the internal and external parameters of the camera stored in a storage unit (not shown) to render a CG image of the virtual object. The display information generation unit 21 superimposes the CG image on the input image input by the sensor information input unit 12 to generate a composite image. The display information generation unit 21 outputs the composite image to the display device 20.

[0145] The display device 20 is a display of the mobile terminal and displays the composite image generated by the display information generating unit 21 .

[0146] The processing procedure according to the seventh exemplary embodiment is different from the processing procedure of the information processing apparatus 1 described in the first exemplary embodiment. Figure 2 The processing of is similar. Therefore, its description will be omitted. The difference from the first exemplary embodiment is that the display information generating step is performed instead of step S14 (control value determining step).

[0147] In the display information generation step, the display information generation unit 21 uses the position and orientation of the sensor 10 calculated by the position and orientation estimation unit 13 in step S13 to render a CG image of the virtual object, superimposes the CG image on the input image, and combines the CG image with the input image to generate a composite image. The display information generation unit 21 inputs the composite image to the display device 20 for presentation.

[0148] In the seventh exemplary embodiment, the map is updated so that the change in position and posture calculated before and after the map update is reduced. This reduces the dramatic changes in position and posture caused by map updates near the space where the user is using MR, AR, or VR. As a result, CG images can be presented with less CG image displacement or loss, making them less noticeable to the user. This can improve the user's MR, AR, or VR experience.

[0149] In the present exemplary embodiment, the difference is described as being calculated by the method described in the first exemplary embodiment. However, the difference according to the present exemplary embodiment can be calculated by any of the methods described in the first to sixth exemplary embodiments, as long as the drastic changes in position and posture due to map updating are reduced. Similar to the second exemplary embodiment, the amount of change in the update element can be used. Similar to the third exemplary embodiment, the difference can be calculated based on the amount of change in the position and posture of the sensor 10 calculated before and after the map update. Unlike the fourth exemplary embodiment, no control value is calculated in the present exemplary embodiment. However, in the case where the target position is rewritten as the presentation position of the CG image, the difference can be calculated so as not to change the position of the CG image. Similar to the fifth exemplary embodiment, the difference can be calculated based on the distance to a nearby object. Similar to the sixth exemplary embodiment, the difference can be calculated based on the state of the MR system (i.e., the moving speed of the MR system), whether the operating state is CG image presentation and / or the weight of the MR system.

[0150] In addition, the difference can be calculated based on attributes such as the type and size of the CG content. Specifically, the difference is calculated based on attributes that the user is more likely to notice blur in the CG image. For example, the larger the size of the CG image, the larger the difference can be calculated. The higher the contrast, the larger the difference can be calculated. The closer the CG image is to the center of the field of view, the larger the difference can be calculated. In addition, an eye tracking device not shown can be included, and the difference can be increased as the eye movement decreases. Such a configuration can reduce the blur in the CG image that is perceptible to the user due to map updates, and the user can enjoy the MR, AR or VR experience more comfortably.

[0151] In the first to seventh exemplary embodiments, the map update behavior is determined based on the difference in the calculated position and posture of the sensor 10 or the control value of the vehicle due to the map update. In the eighth exemplary embodiment, a method for pre-controlling the vehicle to avoid a sudden change in the speed and direction of the vehicle due to the map update will be described. Similar to the first exemplary embodiment, using Figure 3 The hardware configuration shown. Figure 3The hardware configuration shown can be installed in a vehicle. The vehicle controls itself and updates the map by using the measurement values ​​obtained by measuring the surrounding environment using sensors. Alternatively, Figure 3 The hardware configuration example shown can be implemented in a host system that controls an entire factory or an intermediate system that controls vehicles, without having to install the hardware configuration on the vehicles. In this case, the map information is updated by exchanging sensor information and map information between each vehicle and the system via a communication interface.

[0152] Figure 8 is a diagram illustrating an example of the functional configuration of an information processing device 2 according to the present exemplary embodiment. The information processing system according to the present exemplary embodiment includes an information processing device 2, a sensor 10, and a control device 11. The information processing device 2 includes a sensor information input unit 12, a position and posture estimation unit 13, a map storage unit 14, an update information generation unit 15, and a map update unit 16. The information processing device 2 also includes an update information input unit 110, an updated position and posture estimation unit 210, and a control value determination unit 220. Among these units, the sensor 10, the control device 11, the sensor information input unit 12, the position and posture estimation unit 13, the map storage unit 14, the update information generation unit 15, the map update unit 16, and the update information input unit 110 are similar to those of the first exemplary embodiment. Therefore, their description will be omitted.

[0153] The difference from the first exemplary embodiment is that the difference acquisition unit 120 is removed, and an updated position and orientation estimation unit 210 and a control value determination unit 220 are newly added to the information processing apparatus 2 .

[0154] The updated position and posture estimation unit 210 predicts the position and posture of the sensor 10 to be calculated by the position and posture estimation unit 13 after the map update based on the update information input by the update information input unit 110. Specifically, the updated position and posture estimation unit 210 estimates the predicted position and posture (second position and posture) of the estimated vehicle based on the obtained new feature points. The updated position and posture estimation unit 210 outputs the calculated predicted position and posture to the control value determination unit 220.

[0155] Based on the predicted position and orientation calculated by the updated position and orientation estimation unit 210, the control value determination unit 220 calculates a control value so that the difference between the position and orientation calculated by the position and orientation estimation unit 13 and the predicted position and orientation is reduced. The control value determination unit 220 outputs the calculated control value to the control device 11.

[0156] Figure 9: This is a flowchart showing the processing procedure performed by the information processing device 2 according to the present exemplary embodiment. The processing procedure includes the following processing steps: an initialization step (step S11), a sensor information input step (step S12), a position and posture estimation step (step S13), an update information generation step (step S15), a map update step (step S16), a termination determination step (step S17), an update information input step (step S110), an updated position and posture estimation step (step S210), and a control value determination step (step S220).

[0157] The initialization step (step S11), the sensor information input step (step S12), the position and orientation estimation step (step S13), the update information generation step (step S15), the map update step (step S16), the termination determination step (step S17), and the update information input step (step S110) are similar to those in the first exemplary embodiment. Therefore, their description will be omitted.

[0158] In step S210 (updated position and orientation estimation step), based on the update information input by the update information input unit 110, the updated position and orientation estimation unit 210 estimates the position and orientation to be calculated by the position and orientation estimation unit 13 after the map update as the predicted position and orientation. Specifically, the updated position and orientation estimation unit 210 selects feature points within a predetermined distance from the sensor 10 from the feature point information included in each piece of key frame update information included in the key frame group update information included in the update information, and determines a rigid transformation based on the amount of movement of the feature points before and after the map update. A rigid transformation refers to a transformation matrix used to uniquely transform 3D coordinates into different 3D coordinates. In this exemplary embodiment, the rigid transformation is a 4×4 matrix. In this configuration, the rigid transformation is determined to minimize the distance between the positions of the feature points before and after the update. The updated position and orientation estimation unit 210 calculates the rigid transformation as the amount of change in the position and orientation of the sensor 10 and determines the predicted position and orientation by multiplying the position and orientation of the sensor 10 before the map update by the rigid transformation.

[0159] In step S220 (control value determination step), the control value determination unit 220 calculates a control value for controlling the vehicle to the predicted position and orientation estimated by the updated position and orientation estimation unit 210. Specifically, the control value determination unit 220 calculates a control value that reduces the Euclidean distance from the position and orientation of the sensor 10 calculated by the position and orientation estimation unit 13, targeting the predicted position and orientation. In this exemplary embodiment, the control value is calculated by DWA.

[0160] In the eighth exemplary embodiment, based on changes in elements caused by a map update, the position and orientation of the sensor 10 to be calculated after the map update are predicted in advance, and control values ​​are calculated to move the vehicle to the predicted position and orientation in advance. This reduces sudden changes in the vehicle's speed and direction.

[0161] In the eighth exemplary embodiment, the position and posture of sensor 10 after a map update are predicted based on the update amount of map elements near the input position and posture. As a method for calculating the position and posture of sensor 10 after a map update, the updated values ​​of map elements and sensor information can be used to calculate the position and posture. Specifically, updated position and posture estimation unit 210 can calculate the position and posture of sensor 10 after a map update using an image and key frame group update information included in the update information, using a method similar to the method used by position and posture estimation unit 13 to calculate the position and posture.

[0162] In the eighth exemplary embodiment, control values ​​are calculated so that the vehicle assumes the predicted position and posture estimated by the updated position and posture estimation unit 210. However, the method for calculating control values ​​is not limited to this, as long as it can reduce sudden changes in the vehicle's speed and direction. For example, to avoid sudden changes in the vehicle's direction of movement, control values ​​can be precalculated so that only the vehicle's posture matches the predicted posture. If the position and posture are predicted to change along the vehicle's direction of travel, control values ​​for deceleration can be precalculated. On the other hand, if the position and posture are predicted to change in the direction opposite to the direction of travel, control values ​​for acceleration can be precalculated.

[0163] In the eighth exemplary embodiment, control values ​​are calculated after a map update to achieve the predicted position and orientation of sensor 10. However, it is possible to calculate predicted control values ​​resulting from a map update rather than the predicted position and orientation of sensor 10, and to precalculate control values ​​to reduce the difference between the control values ​​before the map update and the predicted control values. Specifically, the control value is calculated as a weighted average of the control value calculated based on the predicted position and orientation of sensor 10 after the map update and the current control value. This can reduce sudden changes in the control value.

[0164] In the eighth exemplary embodiment, control values ​​are calculated so that the vehicle assumes the position and posture after the map update. Alternatively, the map update unit 16 can be configured to receive control progress from the vehicle and wait, without updating the map, until the vehicle reaches the position and posture after the map update. By updating the map after vehicle control is complete and the impact of changes in position and posture due to the map update has diminished, the vehicle can be operated more stably and with greater safety.

[0165] In the eighth exemplary embodiment, a control value is calculated so that the vehicle assumes the position and posture after the map update. Alternatively, the control value can be calculated based on the differences described in the first to sixth exemplary embodiments. In the ninth exemplary embodiment, a configuration for calculating a control value is described so that the differences described in the first exemplary embodiment are utilized to cause the vehicle to leave a range where the difference due to the map update is greater than a predetermined value.

[0166] The information processing apparatus according to this exemplary embodiment has a configuration similar to that of the eighth exemplary embodiment. Therefore, its description will be omitted. Similar to the first exemplary embodiment, the information processing apparatus according to the present exemplary embodiment has a configuration similar to that of the eighth exemplary embodiment. Figure 3 The hardware configuration shown. A difference from the eighth exemplary embodiment is that the control value determination unit 220 inputs the difference calculated by the difference acquisition unit 120 described in the first exemplary embodiment, rather than the difference calculated by the updated position and posture estimation unit 210 described in the eighth exemplary embodiment. Another difference is that the control value determination unit 220 calculates a control value for moving the vehicle to a location where the difference is reduced. In addition, the difference acquisition unit 120 receives movement route information stored in a storage unit not shown. The movement route information is a list of coordinates leading to the target point.

[0167] The processing steps according to this exemplary embodiment are similar to Figure 9 The processing steps described in the eighth exemplary embodiment shown in FIG. Therefore, their description will be omitted. The difference from the eighth exemplary embodiment lies in that step S210 (updating position and posture estimation step) is replaced with step S120 (difference acquisition step) described in the first exemplary embodiment, and in the processing content of step S220 (control value determination step).

[0168] In step S220 (control value determination step) according to this exemplary embodiment, the control value determination unit 220 calculates the control value of the vehicle, using the location outside the map area where the difference calculated by the difference acquisition unit 120 in step S120 (difference acquisition step) is greater than a predetermined value as the target location. The target location outside the map area is the location where the vehicle first arrives outside the map area, after tracking the movement route information starting from the current position and posture of the sensor 10. The control value for moving the vehicle to the target location is calculated using DWA.

[0169] In the ninth exemplary embodiment, the vehicle is controlled to move outside a map area where the position and posture of the sensor 10 or the vehicle control value due to map update calculated based on the difference has changed significantly. This reduces abrupt changes in the vehicle's speed and direction.

[0170] In the ninth exemplary embodiment, step S120 (difference acquisition step) may use any difference calculation method according to the first to sixth exemplary embodiments.

[0171] The control value determination unit 220 may use any method as long as a control value for reducing the difference can be calculated. Specific examples of such control values ​​include a control value for increasing the distance from the update range, a control value for moving the vehicle to a location where the amount of change in map components is small, and a control value for moving the vehicle to a location where the distance between the vehicle and a nearby object is increased.

[0172] Any method can be used to search for a location with a reduced difference as the target location. The control value determination unit 220 can divide the map area of ​​the vehicle into a grid, calculate the difference in each grid cell, and calculate the grid cell with the smallest difference as the target location. This allows map updates without requiring the vehicle to move significantly. Alternatively, the control value determination unit 220 can randomly select a target point on the movement route for the vehicle to travel, and reselect the target point until a location with a difference less than a predetermined value is selected. This allows the map to be updated when the vehicle reaches the predetermined target point without changing the vehicle's movement route.

[0173] In the ninth exemplary embodiment, a control value is calculated to move the vehicle to a location where the difference is less than a predetermined value. However, this method is not restrictive, and any method can be used as long as the control value is calculated so that the speed and direction of the vehicle do not change drastically due to map updates. For example, if the vehicle is at a location where the difference is greater than a predetermined value, the control value can be calculated so that the maximum amount of change in the control value of the vehicle falls at or below the predetermined value. A control value can be calculated to pre-limit the speed or angular velocity of the vehicle to a predetermined value or below. A control value can be calculated to temporarily stop the vehicle. A control value can be calculated to increase the distance to nearby objects.

[0174] In the ninth exemplary embodiment, a control value is calculated to move the vehicle to a location where the difference is less than a predetermined value. Conversely, a location where the amount of change in the map element used for position and posture estimation is less than a predetermined value can be determined from the difference, and a control value can be calculated to move the vehicle to that location.

[0175] Such a location can be calculated using any method. For example, the control value determination unit 220 may calculate the range of the sensor 10 that can be measured at a given location on the map, and determine a location within the range where the ratio of feature points whose movement due to map updates exceeds a predetermined value is lower than a predetermined level. The control value determination unit 220 may also calculate a location where the ratio of key frames whose movement due to map updates exceeds a predetermined value is lower than a predetermined level.

[0176] In the first exemplary embodiment, the amount of change in position measurement values ​​due to updates can be reduced by enabling map updates when the sensor 10 (ie, the vehicle) is away from the update area. This can reduce sudden changes in the speed and direction of the vehicle.

[0177] In the second exemplary embodiment, the map update behavior is calculated based on the amount of change in the update element. Specifically, if the change in the map update element is less than or equal to a predetermined value, the map is updated. If the update behavior exceeds the predetermined value, the map is not updated. This configuration prevents the amount of change in position measurement values ​​due to updates from reaching or exceeding a predetermined level. This reduces abrupt changes in vehicle speed and direction.

[0178] In the third exemplary embodiment, the position and orientation of sensor 10 are calculated using the updated elements in the map. The greater the change in the position and orientation of sensor 10 before and after the update, the greater the calculated difference. This reduces sudden changes in the speed and direction of the vehicle.

[0179] In the fourth exemplary embodiment, the greater the amount of change in the control value of the vehicle calculated before and after the map update, the greater the calculated difference. This reduces sudden changes in the speed and direction of the vehicle.

[0180] In the fifth exemplary embodiment, the difference is calculated based on the surrounding conditions (i.e., the distance to nearby objects). The smaller the distance to nearby objects, the larger the difference. Therefore, the smaller the distance between the vehicle and nearby objects, the smaller the sudden changes in speed and direction.

[0181] In the sixth exemplary embodiment, the difference increases as the state of the vehicle (ie, the mass of the vehicle) increases. Therefore, the greater the mass of the vehicle, the smaller the drastic change in the speed and direction of the vehicle due to the map update.

[0182] In the seventh exemplary embodiment, the map is updated so that the change in position and posture calculated before and after the map update is reduced. This reduces the dramatic changes in position and posture caused by map updates near the space where the user is using MR, AR, or VR. This allows CG images to be presented with less CG image displacement or loss, making them less noticeable to the user. This can improve the user's MR, AR, or VR experience.

[0183] In the eighth exemplary embodiment, the position and orientation of the sensor 10 to be calculated after the map update are predicted based on changes in elements due to the map update, and control values ​​are calculated to move the vehicle to the predicted position and orientation in advance. This reduces sudden changes in the vehicle's speed and direction.

[0184] In the ninth exemplary embodiment, the vehicle is controlled to move outside a map area where the position and posture of the sensor 10 or the control value of the vehicle due to map update based on the difference calculation is greatly changed. This reduces abrupt changes in the speed and direction of the vehicle.

[0185] The update information input unit 110 can have any configuration as long as update information about the map element is input. For example, if the map element is a key frame or a feature point, the update value of the position and posture of the key frame or the 3D coordinates of the feature point can be input. A key frame refers to the smallest unit of a map element used to calculate the position and posture of the sensor 10. A feature point is an indicator used for position and posture estimation. If the map element is an occupancy grid map (a map having a data structure in which passable areas and non-passable areas (such as walls) are drawn on a grid), an update value of the value stored in the grid cell can be input.

[0186] The difference acquisition unit 120 may have any configuration as long as a larger value is calculated as the difference as the amount of change in the position and orientation of the sensor 10 due to map update becomes larger.

[0187] For example, the difference may be calculated based on the updated area of ​​the map. Specifically, a larger value may be calculated as the updated area of ​​the map becomes wider and the distance between the updated area and the sensor 10 becomes closer. If the sensor 10 is located in the updated area, a larger difference value may be calculated than if it is not located in the updated area.

[0188] The difference can be calculated based on the amount of change in the updated elements of the map. Specific examples include the amount of movement of a map element due to a map update, and the ratio of map elements used to calculate the position and posture of the sensor 10 whose amount of change in the updated elements due to a map update is greater than a predetermined value.

[0189] As the amount of change in the position and orientation of the sensor 10 calculated before and after the map update increases, the difference value increases.

[0190] The greater the amount of change in the control value for controlling the vehicle due to the map update, the greater the difference that can be calculated.

[0191] The difference can be calculated based on the situation around the sensor 10. Specifically, the smaller the distance to the nearby object, the larger the difference can be calculated. The difference that can be calculated increases as the specific situation of the nearby object (e.g., the type, size, number, movement speed, weight, and price of the nearby objects) increases.

[0192] The difference can be calculated based on the state of the vehicle on which the sensor 10 is mounted. Specifically, the greater the weight of the vehicle or the object mounted thereon, the greater the calculated difference. In addition, the lower the stacking stability of the mounted object, the greater the calculated difference.

[0193] The update behavior determination unit 130 can have any configuration as long as the update behavior determination unit 130 calculates a value so that the map is less likely to be updated as the difference becomes larger. For example, the update behavior determination unit 130 can be configured so that a real number normalized to the range of 0 to 1 is used as the value of the update behavior, and the update behavior approaches 0 when the difference decreases and approaches 1 when the difference increases. Here, the difference can be normalized by any method. Specifically, an inverse cotangent function of the difference can be used. An exponential function with a negative value of the difference as an exponent can be used. An S-type function that substitutes the difference, multiplies by -1, and adds 1 can be used.

[0194] The control value determination unit may use any calculation method that reduces changes in the position and orientation of sensor 10 due to a map update, or reduces changes in the control value due to a map update. For example, the control value determination unit may calculate a control value for controlling the vehicle to the position and orientation after the map update. The control value determination unit may calculate the control value so that the control value is close to the control value to be calculated after the map update. The control value determination unit may calculate the control value so that the difference calculated by the difference acquisition unit 120 is reduced.

[0195] Instead of the position and posture estimation unit 13 and the update information generation unit 15 in the aforementioned processing unit, a training model generated by machine learning can be used to perform processing. In this case, for example, multiple combinations of input data to the processing unit and output data from the processing unit are prepared as training data. The following training model is generated: that is, the training model acquires knowledge from the training data through machine learning, and based on the acquired knowledge, outputs output data corresponding to the input data as a result. The training model can be configured as a neural network model, for example. The training model performs the processing of the processing unit by operating as a program for performing processing equivalent to the processing of the processing unit in collaboration with the CPU H11 or the graphics processing unit (GPU). After certain processing, the training model can be appropriately updated.

[0196] Other embodiments

[0197] The embodiments of the present invention can also be implemented by the following method, that is, providing software (program) that performs the functions of the above-mentioned embodiments to a system or device through a network or various storage media, and the computer or central processing unit (CPU) or microprocessing unit (MPU) of the system or device reads and executes the program.

[0198] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. An information processing device, comprising: an acquisition unit configured to acquire new positions of feature points included in the map information based on the map information and sensor-acquired information, the map information including positions of feature points pre-measured in an environment including a closed route along which a vehicle travels, the sensor-acquired information including feature points measured by sensors mounted on the vehicle while the vehicle travels along the closed route; as well as an updating unit configured to, when the vehicle traveling in the vicinity of the first position of the vehicle continues to travel along the closed route, update the map information between when the vehicle reaches a predetermined distance or more from the first position and when the vehicle returns to the vicinity of the first position of the vehicle, wherein first position information of the first position of the vehicle is estimated from positions of feature points included in the map information and second position information of the first position of the vehicle is estimated based on the obtained new positions of the feature points, Wherein, when the distance between the position indicated by the first position information and the position indicated by the second position information is greater than a threshold, the updating unit gradually updates the map information by the amount indicated by the update information through multiple update operations.

2. The information processing device according to claim 1, wherein The updating unit is further configured to update the map information based on the acquired new positions of the feature points when the first position of the vehicle is not included in an area of ​​feature points included in the measurable map information and the new positions of the feature points are acquired based on information acquired by the sensors, and to limit the updating of the map information based on the acquired new positions of the feature points when the first position of the vehicle is included in an area of ​​feature points included in the measurable map information and the new positions of the feature points are acquired based on information acquired by the sensors.

3. The information processing device according to claim 1, in, Map information includes information about the location of the destination point of the vehicle, and In which, the update unit is configured to: limit the update of map information when the change amplitude between a first control value determined based on the position of the vehicle indicated by the first position information and the position of the target point of the vehicle and a second control value determined based on the position of the vehicle indicated by the second position information and the position of the target point of the vehicle is greater than a threshold.

4. The information processing apparatus according to claim 1, wherein: The updating unit is configured to update the map information multiple times when a change between the position indicated by the first position information and the position indicated by the second position information is greater than a threshold.

5. The information processing device according to claim 1, in, The acquiring unit is configured to: further acquire an updated area including a feature point included in the map information, wherein a new position of the feature point is acquired based on information obtained by the sensor; and The updating unit is configured to update the map information based on the acquired new position of the feature point when the first position of the vehicle is not included in the update area. The information processing apparatus according to claim 5 , wherein: The updating unit is configured to, in a case where the first position of the vehicle is included in the update area, update the map information when the vehicle is a predetermined distance away from the update area.

7. The information processing device according to claim 1, in, The acquisition unit is configured to: further acquire information indicating a distance between surrounding objects near the vehicle and the vehicle based on the information obtained by the sensor; The updating unit is configured to update the map information when there are no surrounding objects within a predetermined range from the vehicle based on the acquired information, and to restrict the updating of the map information when there are surrounding objects within the predetermined range from the vehicle.

8. The information processing device according to claim 1, in, The acquisition unit is configured to: further acquire information on the weight of the load stacked on the vehicle or the weight of the vehicle, and The updating unit is configured to restrict the updating of the map information when the weight of the vehicle or the load is greater than a threshold.

9. The information processing apparatus according to any one of claims 1 to 8, wherein: The sensor-derived information is a measurement of the distance between the vehicle and its surroundings measured by a measuring device.

10. The information processing apparatus according to any one of claims 1 to 8, wherein: Map information is information that uses three-dimensional coordinates to represent the location of feature points in the environment. The information processing device also includes an estimation unit, which is configured to: estimate the first position and posture of the vehicle based on the three-dimensional positions of the feature points in the map information and the feature points obtained from the information obtained from the sensor, and estimate the second position and posture of the vehicle based on the three-dimensional positions of the feature points obtained from the information obtained from the sensor.

11. The information processing apparatus according to any one of claims 1 to 8, further comprising: The presenting unit is configured to present information indicating that the map information is being updated by the updating unit.

12. The information processing apparatus according to any one of claims 1 to 8, wherein: The sensor-acquired information is information indicating at least any one of the distance between the sensor and a nearby object and the type, size, number, moving speed, weight, and price of the nearby object.

13. The information processing apparatus according to any one of claims 1 to 8, further comprising: a map storage unit configured to store map information; The updating unit is configured to store the updated map information in the map storage unit.

14. An information processing method, comprising: acquiring new positions of feature points included in the map information based on the map information and sensor-acquired information, the map information including positions of feature points previously measured in an environment including a closed route along which the vehicle travels, the sensor-acquired information including feature points measured by sensors mounted on the vehicle as the vehicle travels along the closed route; as well as updating, when the vehicle traveling in the vicinity of the first position of the vehicle continues to travel along the closed route, map information between when the vehicle reaches a predetermined distance or more from the first position and when the vehicle returns to the vicinity of the first position of the vehicle, wherein first position information of the first position of the vehicle is estimated from positions of feature points included in the map information and second position information of the first position of the vehicle is estimated based on the obtained new positions of the feature points, Wherein, when the distance between the position indicated by the first position information and the position indicated by the second position information is greater than a threshold, the map information is gradually updated by the amount indicated by the update information through multiple update operations.

15. A computer-readable medium storing a program which, when executed by a computer, causes the computer to perform the method according to claim 14.

Images