Information processing device, mobile object, information processing method, and computer program
The integration of multiple sensors with state-dependent weight adjustment improves the accuracy of position and orientation estimation for moving objects by mitigating noise from inertial sensors during low acceleration.
Patent Information
- Application Number
- JP2021146513
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-09-08
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2041-09-08
AI Technical Summary
Existing position and orientation estimation methods for moving objects, such as AGVs, suffer from reduced accuracy due to noise in inertial measurement sensors when the acceleration is low.
An information processing device that integrates outputs from both an inertial measurement sensor and a separate position and orientation sensor, adjusting weights based on the moving state to minimize the influence of noise, particularly when the object is stationary.
This approach enhances the accuracy of position and orientation estimation by reducing the reliance on noisy inertial measurements during low acceleration periods.
Smart Images

Figure 0007581163000001 
Figure 0007581163000002 
Figure 0007581163000003
Abstract
Description
[Technical Field]
[0001] The present invention relates to an information processing device capable of estimating the position and orientation of a moving body, a moving body, an information processing method, a computer program, and the like. [Background technology]
[0002] Position and orientation calculation techniques have been proposed for automatically moving a mobile object such as a transport vehicle (e.g., an AGV (Automated Guided Vehicle)) within an environment such as a factory or a logistics warehouse. When a mobile object is made to travel autonomously, there is a method for improving the accuracy of position and orientation calculation by using multiple types of sensors to comprehensively calculate the position and orientation. For example, in Patent Document 1, a translation vector and a rotation matrix calculated using an inertial measurement sensor are corrected based on the error between the translation vector and the rotation matrix calculated using an imaging device. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2011-141221 Summary of the Invention [Problem to be solved by the invention]
[0004] However, the method of Patent Document 1 has a problem in that when the acceleration of the moving body is low, noise from the inertial measurement sensor becomes significant and the accuracy of position and orientation estimation decreases. The present invention has been made in view of the above-mentioned problems, and has an object to provide an information processing device that can reduce the decrease in accuracy of estimating the position and orientation of a moving object. [Means for solving the problem]
[0005] An information processing device according to one aspect of the present invention comprises: an inertial measurement sensor mounted on the moving body; a position and orientation sensor provided on the moving body separately from the inertial measurement sensor, the position and orientation sensor estimating the position and orientation of the moving body; a position and orientation estimation means for estimating the position and orientation of the moving body by integrating a first estimation result obtained by performing a first estimation process on the position and orientation of the moving body using the output of the inertial measurement sensor and a second estimation result obtained by performing a second estimation process on the position and orientation of the moving body using the output of the position and orientation sensor; a moving state acquisition means for acquiring a moving state of the moving body; The moving state acquired by the moving state acquisition means is considered to be in a stationary state. a first weight for integrating the first estimation result; The moving state is smaller than when the moving state is not regarded as a stationary state. and a first weight determining means for determining the weight. [Effects of the Invention]
[0006] According to the present invention, it is possible to realize an information processing device that can reduce a decrease in accuracy in estimating the position and orientation of a moving object. [Brief explanation of the drawings]
[0007] [Figure 1] 1 is a functional block diagram illustrating a configuration example of an information processing device and the like according to a first embodiment. [Figure 2] 1 is a block diagram illustrating an example of a hardware configuration of an information processing apparatus 100 according to a first embodiment. [Figure 3] 4 is a flowchart illustrating a process executed by the information processing device according to the first embodiment. [Figure 4] FIG. 10 is a functional block diagram illustrating a configuration example of an information processing device and the like according to a second embodiment. [Figure 5] 10 is a flowchart illustrating a process executed by an information processing device according to a second embodiment. [Figure 6] 11 is a flowchart illustrating a process executed by an information processing device according to a third embodiment. [Figure 7] 7 is a flowchart showing the process of step S301 in FIG. 6. [Figure 8]FIG. 10 is a diagram showing an example of a GUI displayed on a display unit 216 in the fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION
[0008] Hereinafter, preferred embodiments of the present invention will be described by way of example with reference to the accompanying drawings. In each drawing, the same members or elements are designated by the same reference numerals, and duplicated descriptions will be omitted or simplified. In addition, in the embodiments, the mobile body has the form of an AGV (Automated Guided Vehicle) or an AMR (Autonomous Mobile Robot), but it also includes mobile objects such as automobiles, trains, airplanes, motorcycles, drones, and robots. [Example]
[0009] In this embodiment, an example is described in which a first weight for the output information of an inertial measurement sensor is determined based on the moving state of the moving body, and the position and orientation of the moving body is estimated (calculated) based on an imaging device or a distance measurement device, the output information of the inertial measurement sensor, and the first weight. In particular, in this embodiment, when the moving body is stationary, the first weight is set low, thereby reducing the usage rate of the inertial measurement sensor and reducing the influence of noise from the inertial measurement sensor, thereby mitigating the decrease in accuracy of position and orientation estimation.
[0010] FIG. 1 is a functional block diagram illustrating a configuration example of an information processing apparatus and the like according to the first embodiment. 1 are realized by causing a computer (described later) included in the information processing device to execute a computer program stored in a memory serving as a storage medium. However, some or all of these functions may be realized by hardware. Examples of hardware that can be used include dedicated circuits (ASICs) and processors (reconfigurable processors, DSPs).
[0011] Furthermore, the functional blocks shown in FIG. 1 do not have to be housed in the same housing, and the information processing device may be composed of separate devices connected to each other via signal paths. The above description regarding FIG. 1 also applies to the functional block diagram of FIG. 1, a movement state determination unit 101 determines the movement state of a moving object based on output information from a movement measurement sensor for measuring the movement state of the moving object. The movement measurement sensor may be one or more. In this embodiment, an imaging device (imaging unit) such as a stereo camera is used as the movement measurement sensor. In this embodiment, the movement state of the moving object is determined as whether or not it is stationary.
[0012] The method for determining whether a moving object is stationary is to track the movement of the same feature points in past and present images of a time series acquired from an imaging device (imaging unit) such as a stereo camera. If the average movement amount of the same feature points is 0 or less than a predetermined threshold, i.e., if the number of feature points that have not moved from the past to the present is less than a predetermined threshold, the moving state of the moving object is determined to be stationary.
[0013] The moving state acquisition unit 102 is a functional component that acquires the moving state of the moving body determined by the moving state determination unit 101. Here, the moving state includes at least one of whether the movement of the moving body is equal to or less than a predetermined threshold, whether the acceleration of the moving body is equal to or less than a predetermined threshold, and whether the angular velocity of the moving body is equal to or less than a predetermined threshold. Alternatively, the values of the acceleration and angular velocity may be used as the current acceleration state and angular velocity state.
[0014] The first weight determination unit 103 is a functional component that determines a first weight based on the moving state of the moving object acquired by the moving state acquisition unit 102. The first weight is a weight that is multiplied by the output information of the inertial measurement sensor 104, and is expressed as a decimal number between 0 and 1. When the first weight is 0, the output information of the inertial measurement sensor 104 is not used for estimating the position and orientation, and when the first weight is 1, it is used 100%.
[0015] Furthermore, the first weight determination unit in the embodiment may determine the first weight to be 0 or a value lower than a normal value when the acceleration or angular velocity of the moving object is lower than a predetermined threshold value. Alternatively, the acceleration or angular velocity value may be used as is, and the larger the value of the acceleration or angular velocity, the larger the value of the first weight.
[0016] Note that the first weight determination unit 103 may determine the relative weight of the output of the inertial measurement sensor 104 with respect to other outputs such as the position and orientation sensor 105, rather than determining the weight to be directly applied to the output information of the inertial measurement sensor 104. That is, for example, instead of directly lowering the weight of the output information of the inertial measurement sensor 104, the weight of the output of the position and orientation sensor 105 may be increased, thereby relatively lowering the weight of the output of the inertial measurement sensor 104. In this way, the first weight determination unit includes a unit that relatively determines the weight of the output information of the inertial measurement sensor 104 with respect to other outputs such as the position and orientation sensor 105.
[0017] The inertial measurement sensor 104 is an inertial measurement unit (IMU) or the like fixed to a moving body to estimate the position and orientation of the moving body. Note that since the inertial measurement unit IMU moves and rotates in conjunction with the movement and rotation of the moving body, the position and orientation of the moving body correspond to the position and orientation of the inertial measurement unit.
[0018] The position and orientation sensor 105 is a position and orientation sensor provided separately from the inertial measurement sensor 104 in order to detect the position and orientation of the moving body, and may be one or more. In this embodiment, an imaging device (imaging unit) such as a stereo camera is used as the position and orientation sensor. The stereo camera is fixed to the moving body, and moves and rotates in conjunction with the movement and rotation of the moving body, so the position and orientation of the moving body corresponds to the position and orientation of the stereo camera. Note that the stereo camera may also be the same as the stereo camera used as a movement measurement sensor to measure the movement state of the moving body.
[0019] The sensor information acquisition unit 106 acquires output information from the inertial measurement sensor 104 and the position and orientation sensor 105 . The position and orientation estimation unit 107 estimates the position and orientation of the moving body based on the first weight determined by the first weight determination unit 103 and the output information of the inertial measurement sensor 104 and the position and orientation sensor 105 acquired by the sensor information acquisition unit 106.
[0020] The control unit 108 determines the direction and amount of movement that the mobile object should move next based on the position and orientation estimated by the position and orientation estimation unit 107, and drives the motors and engines connected to the wheels of the mobile object by the amount necessary for the movement. In this embodiment, the direction and amount of movement for proceeding along a preset movement path are determined based on the current position and orientation. The control unit 108 includes a CPU 211 as a computer, as will be described later, which controls the operation of each part of the entire apparatus based on a computer program stored in a memory as a storage medium.
[0021] Here, the information processing device 100 in this embodiment is configured with a moving state acquisition unit 102, a first weight determination unit 103, a sensor information acquisition unit 106, a position and orientation estimation unit 107, etc. The inertial measurement sensor 104, the position and orientation sensor 105, the moving state determination unit 101, and the control unit 108 are assumed to be mounted on a moving body. The moving state determination unit 101 and the control unit 108 may also be included in the information processing device 100. Alternatively, the information processing device 100 itself may be mounted on a moving body.
[0022] FIG. 2 is a block diagram illustrating an example of the hardware configuration of the information processing apparatus 100 according to the first embodiment. Reference numeral 211 denotes a CPU of the computer, which controls various devices connected to a system bus 220. Reference numeral 212 denotes a ROM, which stores a BIOS program and a boot program. Reference numeral 213 denotes a RAM, which is used as the main storage device of the CPU 211.
[0023] An external memory 214 stores computer programs to be processed by the information processing device 100. An input unit 215 is a keyboard, mouse, robot controller, etc., and performs processing related to input of information, etc. A display unit 216 outputs the results of calculations by the information processing device 100 to a display device in accordance with instructions from the CPU 211.
[0024] The display device can be of any type, such as a liquid crystal display device, projector, or LED indicator. 217 is an I / O that communicates information via a communication interface, and the communication interface can be Ethernet, USB, serial communication, wireless communication, or any other type. The I / O receives output information from various sensors or other information processing devices.
[0025] Fig. 3 is a flowchart showing the processing executed by the information processing device in Example 1. Note that the CPU 211 as a computer in the control unit 108 executes a computer program stored in the external memory 214 to perform the operation of each step in Fig. 3. Note that the same applies to each flowchart in Figs. 5 to 7 described later.
[0026] In step S101, the information processing device 100 is initialized. That is, a computer program is read from the external memory 214 to put the information processing device 100 into an operable state. In addition, various setting parameters, such as various thresholds described below and initial values of first weights, are read from the external memory 214 into the RAM 213 as necessary.
[0027] In step S102, the sensor information acquisition unit 106 acquires output information from the inertial measurement sensor 104 and the position and orientation sensor 105. In this embodiment, the inertial measurement sensor 104 and the position and orientation sensor 105 are fixed to the moving body, and each sensor moves and rotates in conjunction with the movement and rotation of the moving body, so the position and orientation of the moving body corresponds to the position and orientation of each sensor. In step S103 (movement state acquisition step), the movement state acquisition unit 102 acquires the movement state of the moving object determined by the movement state determination unit 101. In this embodiment, for example, it acquires whether the moving object is stationary (or whether its movement is equal to or less than a predetermined threshold).
[0028] In step S104 (first weight determination step), the first weight determination unit 103 determines a first weight based on the moving state of the moving object acquired by the moving state acquisition unit 102. The first weight is set as an initial value in step S101, but may be set to another value by a user operation. For example, the initial value is set to 1. If the first weight applied to the output information of the inertial measurement sensor 104 is 1, the output information of the inertial measurement sensor 104 is used 100% to estimate the position and orientation of the moving body.
[0029] In this embodiment, if the moving state of the moving object is not stationary, the first weight is set to the initial value 1 set in step S101 or a value set by a user operation. If the moving state of the moving object is stationary, the first weight is set to 0 or a value lower than the value when the moving object is not stationary. If the first weight is 0, the output information of the inertial measurement sensor 104 is multiplied by 0 and is not used in estimating the position and orientation of the moving object. If the first weight is a value lower than the value when the moving object is not stationary, the proportion of use of the output information of the inertial measurement sensor 104 in estimating the position and orientation of the moving object decreases.
[0030] In step S105, the position and orientation estimation unit 107 estimates the position and orientation of the moving object based on the first weight determined by the first weight determination unit 103 and the output information of the inertial measurement sensor 104 and the position and orientation sensor 105 acquired by the sensor information acquisition unit 106. Step S105 constitutes a position and orientation estimation process. In this embodiment, the position and orientation of the moving object is estimated based on the position and orientation information from the position and orientation sensor 105 and the output information from the inertial measurement sensor 104 multiplied by the first weight.
[0031] In this embodiment, the ORB-SLAM3 method is used as a method for estimating position and orientation using the output of the camera serving as the position and orientation sensor 105 and the output of the inertial measurement sensor 104. The specific method of ORB-SLAM3 is described in detail in the method of Campos et al. (Carlos Campos et al., ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertialand Multi-MapSLAM, Cornell University), and therefore will not be described here.
[0032] In step S106, it is determined whether or not to terminate the entire processing of the information processing device 100. For example, this processing is terminated when the moving object arrives at the destination or when an end instruction is received from the user via the input unit 215. If No in step S106, the processing returns to step S102 and continues. In this way, in this embodiment, when the moving object is stationary (or its movement is equal to or less than a predetermined threshold), the first weight is set low, thereby reducing the usage rate of the inertial measurement sensor 104 and reducing the influence of noise from the inertial measurement sensor 104. This makes it possible to reduce the degradation of accuracy in position and orientation estimation.
[0033] In the first embodiment, an imaging unit such as a stereo camera is used as a movement measurement sensor for measuring the movement state of a moving object, but this is not limiting. For example, a monocular camera, a camera with three or more lenses, a distance measurement device, an encoder that measures the amount of rotation of a motor that turns the wheels of the moving object, a GPS (Global Positioning System), etc. may be used. Alternatively, a drive command to a drive source such as a motor or engine connected to the wheels of the moving object may be used. Furthermore, a camera installed so as to overlook the space from outside the moving object, a camera mounted on a drone, etc. may be used, as long as it is at least one of the above-mentioned multiple sensors. Furthermore, they may be used in appropriate combination.
[0034] In the above embodiment, the position and orientation sensor 105 for detecting the position and orientation of the moving body uses an imaging device (imaging unit) that captures an image of a subject, such as a stereo camera, but is not limited to this. For example, a distance measurement device that measures the distance to the subject, a GPS, or the like may also be used. Sensing technologies such as LiDAR (Light Detection and Ranging) may also be used. Furthermore, a camera installed so as to overlook the space from outside the moving body, a camera mounted on a drone, or the like may also be used, as long as at least one of the above is used. The type is not important as long as it can obtain output information necessary to estimate the position and orientation of the moving body. Furthermore, a combination of the above sensors may also be used.
[0035] In the first embodiment, the moving state determination unit 101 determines whether a moving object is stationary (or its movement is equal to or less than a predetermined threshold) based on the magnitude of the movement of feature points between the past and present in time-series images acquired from a stereo camera. However, the present invention is not limited to this method. For example, if the movement of feature points in distance images acquired by a distance measurement device, rather than a stereo camera, is 0 or equal to or less than a predetermined threshold, the moving object may be determined to be stationary.
[0036] Alternatively, the moving object may be determined to be stationary if the amount of rotation acquired from an encoder that measures the amount of rotation of a motor that turns the wheels of the moving object is 0 or less than a predetermined threshold. Alternatively, the moving object may be determined to be stationary if a drive command is not issued to a motor connected to the wheels of the moving object. Furthermore, the moving object may be determined to be stationary if the position of the moving object measured by a GPS, a camera installed so as to overlook the space from outside the moving object, a camera mounted on a drone, or the like does not change over time or if the amount of change is less than a predetermined threshold.
[0037] In the first embodiment, the moving state of the moving body determined by the moving state determination unit 101 is a stationary state (or a state in which the movement is equal to or less than a predetermined threshold), but this is not limited to this. For example, the amount of movement per unit time of the moving body acquired from a sensor for measuring the moving state is differentiated twice by unit time to calculate acceleration, and if this acceleration is equal to or less than a preset threshold, it may be determined that the acceleration is low.
[0038] Similarly, the angular velocity may be calculated by differentiating the amount of movement of the moving body in the rotational direction once per unit time, and if this angular velocity is equal to or less than a preset threshold, the angular velocity may be determined to be low. Furthermore, the acceleration and angular velocity may not be subjected to threshold determination, and their values may be used as they are as the current acceleration and angular velocity states. Furthermore, the movement state of these moving bodies may be determined by appropriately combining the above methods.
[0039] Furthermore, in the first embodiment, when the moving state of the moving object is stationary (or the movement is below a predetermined threshold), the first weight is determined to be 0 or a value lower than the value when the moving object is not stationary. However, this is not limited to this. For example, when the acceleration or angular velocity of the moving object is lower than a threshold, the acceleration or angular velocity is low, the first weight may be determined to be 0 or a value lower than a normal value. Alternatively, instead of comparing the acceleration or angular velocity value with a threshold, the larger the value, the larger the first weight value may be.
[0040] In the first embodiment, the position and orientation of the moving body are estimated based on the output information of the position and orientation sensor 105 and the output information of the inertial measurement sensor 104 multiplied by the first weight, but this is not limitative. For example, if the first weight is equal to or smaller than a predetermined threshold, the output information of the inertial measurement sensor 104 may not be used.
[0041] Furthermore, the position and orientation estimated from the output information of the position and orientation sensor 105 may be combined with the position and orientation estimated from the output information of the inertial measurement sensor 104 multiplied by a first weight. Alternatively, the position and orientation estimated from the output information of the position and orientation sensor 105 may be combined with the position and orientation estimated based on the output information of the inertial measurement sensor 104 multiplied by a first weight. As a method for estimating the position and orientation using the position and orientation sensor 105 such as a camera, for example, bundle adjustment may be performed to estimate the position and orientation of the moving object.
[0042] Bundle adjustment is a process for measuring the camera position and orientation that minimizes the sum of the differences (residuals) between the projection points obtained by projecting the 3D positions of image features stored as a position and orientation measurement map onto a key frame and the image feature positions detected from image data captured during autonomous driving.Details of key frames and bundle adjustment are provided in the method by Raul et al. (Raul Mur-Artal et al., ORB-SLAM: A Versatile and Accurate Monocular SLAM System. IEEE Transactions on Robotics), so a detailed explanation will be omitted here.
[0043] The position and orientation estimation method using the inertial measurement sensor 104 involves calculating the amount of displacement by integrating the acceleration, which is output information from the inertial measurement sensor 104, twice per unit time, to estimate the position. Similarly, the angular velocity, which is output information from the inertial measurement sensor 104, is integrated once per unit time to calculate the angle, to estimate the orientation. Alternatively, a loose coupling method may be used in which the results of position and orientation measurement from the position and orientation sensor 105 such as a camera and the inertial measurement sensor 104 are integrated, or a tight coupling method may be used in which the data from two types of sensors are integrated to perform position and orientation measurement.
[0044] Specifically, during the above-mentioned bundle adjustment, the position and orientation are estimated by correcting the position and orientation using the output information of the inertial measurement sensor 104. Alternatively, a method based on measuring the radio wave intensity in wireless communication such as Wi-Fi, satellite positioning such as GPS, a magnetic tracking sensor, or a method using the time difference between sensing light scanning through space with multiple optical sensors may be used. [Example]
[0045] In the second embodiment, when an image capturing device (image capturing unit) or a distance measuring device (distance measuring unit) is used as the position and orientation sensor 105, the second weight is determined based on the reliability of the output information thereof. Then, the position and orientation of the moving object is estimated based on the image capturing device or the distance measuring device, the output information of the inertial measurement sensor 104, the first weight, and the second weight. In particular, by setting the second weight low when the reliability of the output information of the image capturing device or distance measuring device serving as the position and orientation sensor 105 is low, the proportion of inaccurate sensor output information used decreases, and the deterioration in accuracy of position and orientation estimation can be alleviated.
[0046] FIG. 4 is a functional block diagram illustrating a configuration example of an information processing device and the like according to the second embodiment. 1 in that a reliability calculation unit 201, a reliability acquisition unit 202, and a second weight determination unit 203 are provided, and in the processing in the position and orientation estimation unit 107. Only the parts that differ from the first embodiment will be described here.
[0047] A reliability calculation unit 201 calculates the reliability of output information of the position and orientation sensor 105 related to position and orientation measurement of a moving object. In the second embodiment, the reliability is calculated based on output information of an imaging device such as a stereo camera used as the position and orientation sensor 105. Specifically, for example, if the number of feature points in an image captured by an imaging device such as a stereo camera used as the position and orientation sensor 105 is small, the reliability value is set to a low value.
[0048] The reliability acquisition unit 202 acquires the reliability calculated by the reliability calculation unit 201 . The second weight determination unit 203 determines the second weight based on the reliability acquired by the reliability acquisition unit 202. For example, when the reliability is low, the second weight is determined to be low. The position and orientation estimation unit 107 estimates the position and orientation of the moving object based on the first weight, the second weight, and the output information of the inertial measurement sensor 104 and the position and orientation sensor 105 acquired by the sensor information acquisition unit 106.
[0049] Note that the second weight determination unit 203 does not determine a weight to be directly applied to the output information of the position and orientation sensor 105, but includes a unit that determines the relative weight of the output of the position and orientation sensor 105 with respect to the output of the inertial measurement sensor 104. In other words, the second weight determination unit includes a unit that relatively determines the weight of the output information of the position and orientation sensor 105 with respect to the output of the inertial measurement sensor 104.
[0050] Fig. 5 is a flowchart showing the processing executed by the information processing apparatus of Example 2. Explanation of the same steps as in the flowchart of Fig. 3 will be omitted, and only the processing different from the flowchart of Fig. 3 will be explained. In step S201 (reliability acquisition step), the reliability acquisition unit 202 acquires the reliability calculated by the reliability calculation unit 201.
[0051] In step S202 (second weight determination step), the second weight determination unit 203 determines the second weight based on the reliability acquired by the reliability acquisition unit 202. An initial value of the second weight is set in advance in step S101, but it may be set to another value by a user operation. For example, the initial value is set to 1. If the second weight applied to the output information of the position and orientation sensor 105 is 1, the output information of the position and orientation sensor 105 is used 100% to estimate the position and orientation of the moving object.
[0052] In this embodiment, when the reliability is equal to or less than a preset threshold, the second weight is set to 0, or to a value lower than the value when the reliability is not equal to or less than the threshold. When the second weight is 0, the output information of the position and orientation sensor 105 is multiplied by 0 and is not used in estimating the position and orientation of the moving body. When the second weight is a value lower than the value when the reliability is not equal to or less than the threshold, the usage rate of the output information of the position and orientation sensor 105 in estimating the position and orientation of the moving body is reduced.
[0053] In step S105, the position and orientation estimation unit 107 estimates the position and orientation of the moving body based on the first weight, the second weight, the output information of the inertial measurement sensor 104, and the output information of the position and orientation sensor 105. In this embodiment, the position and orientation of the moving body is estimated based on the output information of the stereo camera serving as the position and orientation sensor 105 multiplied by the second weight, and the output information of the inertial measurement sensor 104 multiplied by the first weight. The method of estimating the position and orientation is the same as in the first embodiment, and therefore will not be described again.
[0054] According to this embodiment, by setting the second weight low when the reliability of the output information of the image capture device, distance measurement device, or the like serving as the position and orientation sensor 105 is low, the proportion of inaccurate sensor output information used decreases, and it is possible to mitigate a decrease in the accuracy of position and orientation estimation.
[0055] In the second embodiment, the reliability calculation unit 201 calculates the reliability of the output information of the position and orientation sensor 105 by lowering the reliability when the number of feature points in an image captured by an imaging device such as a stereo camera serving as the position and orientation sensor 105 is small. However, this is not limiting. For example, the reliability may be lowered when there is a bias in the distribution of feature points in an image captured by an imaging device such as a stereo camera serving as the position and orientation sensor 105. Specifically, the screen is divided into multiple sections according to a division method specified by the user or preset by the system, and it is determined whether there are a number of feature points equal to or greater than a predetermined threshold in each section. Then, if the sum of the number of sections in which there are a number of feature points equal to or greater than the predetermined threshold is equal to or less than the predetermined threshold, it is determined that there is a bias.
[0056] Furthermore, the reliability may be lowered when the average brightness in the image or the brightness of the space around the moving object measured by an external sensor is too bright (blown-out highlights in the image) or, conversely, too dark (crushed blacks in the image). Specifically, if the average pixel value in the image is greater than a predetermined threshold, it is determined to be too bright, and if it is less than the predetermined threshold, it is determined to be too dark. Alternatively, an illuminance sensor may be used, and if the value of the illuminance sensor is greater than a threshold, it may be determined to be too bright, and if it is less than the predetermined threshold, it may be determined to be too dark. The threshold for determining that something is too bright is set to a value greater than the threshold for determining that something is too dark.
[0057] Alternatively, the reliability may be lowered when there is a large fluctuation in brightness, such as when there is a sudden change in brightness or when illumination fluctuations are repeated. Specifically, successive images are acquired in a time series, and if the average brightness in the images changes over time by more than a predetermined threshold, it is determined that there has been a sudden change in brightness, and if this state is repeated, it is determined that illumination fluctuations are repeated.
[0058] Alternatively, if a large number of moving objects are detected, the reliability may be lowered because accurate map information cannot be created. Specifically, object recognition is performed on time-series images using machine learning or the like, and if there are a large number of objects whose positions within the images are moving over time, the reliability may be lowered. Alternatively, moving objects may be detected using a motion detection technology using background subtraction or the like.
[0059] Alternatively, when object recognition using machine learning or the like is performed and it is determined that a mirror surface, a window that transmits light, or the like is present, the reliability may be lowered because feature points cannot be accurately acquired. Also, when similar scenes appear one after another, the reliability may be lowered because the current location cannot be determined. Also, when object recognition using machine learning or the like is performed and it is determined that a large door that occupies most of the field of view of the stereo camera is present, the reliability may be lowered because feature points change depending on whether the door is open or closed.
[0060] That is, the reliability may be calculated according to whether or not a predetermined subject is present in the image. Furthermore, instead of using a stereo camera, a distance measurement device capable of estimating the position and orientation using feature points may also be used. Furthermore, when using a GPS for the position and orientation sensor 105, the reliability may be lowered when the radio wave reception is poor, such as inside a building. It is sufficient to calculate the reliability based on at least one of these multiple reliability calculation methods. Of course, multiple methods may be combined.
[0061] In the second embodiment, the second weight determination unit 203 determines the second weight by setting the second weight to 0 when the reliability is equal to or less than a preset threshold, or to a value lower than the value when the reliability is not equal to or less than the threshold. However, the present invention is not limited to this. For example, the value of the second weight may be increased stepwise as the reliability value increases.
[0062] In the second embodiment, the position and orientation of the moving body is estimated based on the output information of the position and orientation sensor 105 multiplied by the second weight and the output information of the inertial measurement sensor 104 multiplied by the first weight, but this is not limitative. For example, if the second weight is equal to or less than a predetermined threshold, the output information of the position and orientation sensor 105 may not be used.
[0063] Alternatively, the position and orientation may be estimated by multiplying the output information of the position and orientation sensor 105 by the second weight and by multiplying the output information of the inertial measurement sensor 104 by the first weight, and the resulting position and orientation may be combined. Alternatively, the position and orientation estimated from the output information of the position and orientation sensor 105 multiplied by the second weight and the position and orientation estimated from the output information of the inertial measurement sensor 104 multiplied by the first weight may be combined.
[0064] The method of estimating the position and orientation using the position and orientation sensor 105 and the inertial measurement sensor 104, and the method of combining the same are the same as those in the first embodiment, and therefore a description thereof will be omitted. Furthermore, if both the first weight and the second weight are low, such as being equal to or less than a predetermined threshold, the output information from neither sensor is reliable, and the moving body may be stopped. Alternatively, the moving body may be decelerated. Alternatively, a warning may be displayed on the display unit 216 to alert the user. Alternatively, a warning sound may be sounded. [Example]
[0065] In the third embodiment, based on the moving state of the moving body and the position and orientation estimation results, the moving body is moved or its orientation is changed, such as by turning, so as to reduce the influence of noise in the output information of the inertial measurement sensor. In particular, we will explain an example in which, when the moving state of a moving body is stationary, the influence of noise is reduced and a decrease in the accuracy of position and orientation estimation is mitigated by changing the posture, such as by moving or turning, and increasing the value of the output information of the inertial measurement sensor.
[0066] The functional blocks of the third embodiment are the same as those shown in Fig. 1, and therefore a description thereof will be omitted. However, the processing in the control unit 108 differs from that in the first embodiment. That is, the control unit 108 of the third embodiment changes the movement and attitude of the moving body based on the movement state acquired by the movement state acquisition unit 102 and the position and attitude estimated by the position and attitude estimation unit 107 so as to reduce the influence of noise on the output information of the inertial measurement sensor 104.
[0067] Fig. 6 is a flowchart showing the processing executed by the information processing device in Example 3. Explanation of the same steps as in the flowchart in Fig. 3 will be omitted, and only the processing different from Example 1 will be explained. In step S301, the posture of the moving object, such as movement or rotation, is changed based on the moving state acquired by the moving state acquisition unit 102 and the position and posture estimated by the position and posture estimation unit 107 so as to reduce the influence of noise in the output information of the inertial measurement sensor 104. Details of the processing in step S301 will be described with reference to FIG.
[0068] FIG. 7 is a flowchart showing the process of step S301 in FIG. In step S401, it is determined whether the moving state acquired by the moving state acquisition unit 102 is a stationary state. If it is a stationary state, the process proceeds to step S402, and if it is not a stationary state, the flow in FIG. 7 is terminated and the process proceeds to step S106 in the flowchart in FIG.
[0069] In step S402, it is determined whether there is any problem in moving or changing the attitude of the moving body, taking into consideration the current position and attitude of the moving body estimated by the position and attitude estimation unit 107. Whether there is any problem in moving or changing the attitude of the moving body is determined to be no problem when there are no obstacles around the moving body, particularly in the direction of movement or turning, and the difference between the current position / orientation and the position / traveling direction of the route is equal to or less than a predetermined threshold.
[0070] If it is determined in step S402 that there is no problem with the movement or change of attitude, the process proceeds to step S403. In step S403, for example, the amount of movement and direction of travel of the moving body are determined within a range in which the position of the moving body does not deviate from the position and direction of travel of the path by more than a predetermined threshold, and the moving body is moved or its attitude is changed.
[0071] In this way, in the third embodiment, when the moving body is stationary, the moving body moves and changes its attitude if there is no problem in changing the movement or attitude of the moving body. This increases the value of the output information of the inertial measurement sensor 104, reduces the influence of noise from the inertial measurement sensor 104, and reduces the degradation of the accuracy of position and attitude estimation.
[0072] In the third embodiment, when the moving state of the moving body is stationary (or the movement is below a predetermined threshold), the moving body is allowed to move if there is no impediment to the movement of the moving body, but this is not limited to this. For example, when the acceleration of the moving body is below a predetermined threshold and it is determined that the acceleration is low, and it is determined that there is no impediment to the movement of the moving body as in step S402, the moving body may be accelerated or decelerated.
[0073] Alternatively, if the angular velocity of the moving body is equal to or less than a predetermined threshold, it is determined that the angular velocity is low, and it is determined that there is no problem with the movement of the moving body, as in step S402, the moving body may be moved or its posture may be changed.Movement may also be determined by combining a plurality of these conditions.
[0074] Furthermore, in the third embodiment, a process of moving the moving object may be added so as to improve the reliability of the output information of the position and orientation sensor 105 in the second embodiment. For example, when a stereo camera or a distance measurement device is used as the position and orientation sensor 105, if the number of feature points in an image is small or the distribution of feature points is uneven, the moving object may be moved so as to move toward a location in the acquired image where there are more feature points.
[0075] Furthermore, if an object that reduces reliability, such as a mirror surface, a light-transmitting window, or a large door, exists in the image as described in Example 2, the moving object may be moved so that the object is outside the angle of view of the stereo camera or distance measurement device. Furthermore, if GPS is used for the position and orientation sensor 105, when the radio wave conditions become poor because the moving object is inside a building or in the shadow of an obstacle, the moving object may be moved out of the shadow of the building or obstacle to a position where the radio wave conditions are good. [Example]
[0076] In the fourth embodiment, the first weight for the output information of the inertial measurement sensor 104 is visualized to show the user the locations where the sensor is poorly used, and encourage the user to change the route or perform a movement operation to reduce the noise impact of the inertial measurement sensor 104. The functional blocks of this embodiment are almost the same as those in Fig. 1, except for the presence of an image generation unit. Here, only the parts that differ from the first embodiment will be described.
[0077] The image generation unit generates an image displaying the first weight determined by the first weight determination unit 103, and transmits the image to the display unit 216, whereby the image is displayed on a screen or the like. FIG. 8 is a diagram showing an example of a GUI displayed on the display unit 216 in the fourth embodiment. G100 is the display content output by the information processing device 100 and displayed on the display unit 216, and may be displayed on the entire screen or on a part of the screen.
[0078] G110 shows the horizontal plane of the moving space when viewed from above, showing the moving path of the moving object. The black parts of this diagram are walls, etc., and the white parts are passageways. The moving object moves through the passageways, which are the white parts. The dotted arrows indicate the travel path, and G111 is an icon showing the moving object's current position.
[0079] G120 represents the first weighting for the output value of the inertial measurement sensor 104 as "utilization." However, the display method is not limited to this. In the example shown in this figure, the moving object is moving at a constant speed in a straight line, so the acceleration is almost 0. Therefore, the first weight is 0.1, and the output information of the inertial measurement sensor 104 is multiplied by 0.1. This allows the user to understand that the usage rate of the output information of the inertial measurement sensor 104 is low.
[0080] In this way, in the fourth embodiment, by visualizing the first weight for the output information of the inertial measurement sensor 104 together with the current position of the moving object, it is possible to show the user the location where the inertial measurement sensor 104 is poorly utilized. It is also possible to prompt the user to change the route or perform a movement operation to reduce the influence of noise from the inertial measurement sensor 104.
[0081] In the fourth embodiment, the first weight for the output information of the inertial measurement sensor 104 is visualized, but the second weight for the output information of the position and orientation sensor 105 may also be visualized. Also, both may be visualized simultaneously. In the fourth embodiment, the weight is displayed as a numerical value, but for example, the color or size of the icon G111 may be changed depending on the magnitude of the weight value.
[0082] Furthermore, when the weight falls below a predetermined threshold, the icon G111 may blink, change color or size, or display a warning message about a decrease in the utilization rate of the inertial measurement sensor 104. When weights are set for multiple sensors, the icon G111 may be color-coded to distinguish which sensor's weight has decreased.
[0083] As described above, in the first embodiment, by setting the first weight low when the moving body is stationary, the utilization rate of the inertial measurement sensor 104 is reduced, and the influence of noise from the inertial measurement sensor 104 is reduced, thereby reducing the decrease in accuracy of position and orientation estimation. In addition, in the second embodiment, by setting the second weight low when the reliability of the output information of the image capture device or the distance measurement device is low, the proportion of inaccurate sensor output information used decreases, and it is possible to mitigate a decrease in the accuracy of position and orientation estimation.
[0084] In addition, in the third embodiment, the moving body starts moving when it is in a stationary state, and the value of the output information of the inertial measurement sensor 104 is increased, thereby reducing the influence of noise and reducing the decrease in accuracy of position and orientation estimation. In the fourth embodiment, by visualizing the first weight for the output information of the inertial measurement sensor 104, it is possible to show the user the locations where the sensor is poorly used, and to encourage the user to change the route or perform a movement operation to reduce the noise impact of the inertial measurement sensor 104.
[0085] The position and orientation of a moving object may be estimated by appropriately combining the above-described embodiments 1 to 4. This allows the position and orientation of a moving object to be estimated more precisely. Furthermore, in the above embodiments, the position and orientation sensor 105 is provided in addition to the inertial measurement sensor 104, but the first, third, and fourth embodiments can also be applied to a configuration in which the position and orientation sensor 105 is not provided.
[0086] In the above embodiments, the mobile body is in the form of an AGV or an AMR (autonomous mobile robot), and includes a drive device such as a travel motor or engine for moving (traveling) the AGV or AMR, a movement direction control device for changing the movement direction of the AGV or AMR, and a movement control unit for controlling the drive amount of the drive device and the movement direction of the movement direction control device.
[0087] The movement control unit has a built-in CPU as a computer and a memory that stores a computer program, and communicates with other devices, thereby controlling, for example, the information processing device 100 and acquiring position and orientation information, travel route information, etc. from the information processing device 100. The AGVs and AMRs as moving bodies are configured so that the movement direction, movement amount, and movement route of the AGVs and AMRs are controlled by a movement control unit based on position and posture information, travel route information, etc. generated by the information processing device 100.
[0088] The present invention has been described in detail above based on its preferred embodiments, but the present invention is not limited to the above embodiments, and various modifications are possible based on the gist of the present invention, and these modifications are not excluded from the scope of the present invention. Note that a computer program that realizes part or all of the control in this embodiment and the functions of the above-described embodiment may be supplied to information processing or the like via a network or various storage media. A computer (or a CPU, MPU, or the like) in the information processing or the like may then read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]
[0089] 100: Information processing device 101: Movement state determination unit 102: Moving status acquisition unit 103: First weight determination unit 104: Inertial measurement sensor 105: Position and orientation sensor 106: Sensor information acquisition unit 107:Position and orientation estimation unit 108: Control unit
Claims
1. An inertial measurement sensor mounted on a moving object; a position and orientation sensor provided on the moving body separately from the inertial measurement sensor, the position and orientation sensor estimating a position and orientation of the moving body; a position and orientation estimation means for estimating a position and orientation of the moving body by integrating a first estimation result obtained by performing a first estimation process of the position and orientation of the moving body using an output of the inertial measurement sensor and a second estimation result obtained by performing a second estimation process of the position and orientation of the moving body using the output of the position and orientation sensor; A moving state acquisition means for acquiring a moving state of the moving body; an information processing device comprising: a first weight determination means for determining, when the moving state acquired by the moving state acquisition means is regarded as a stationary state, a first weight for integrating the first estimation result to be smaller than when the moving state is not regarded as a stationary state.
2. 2 . The information processing apparatus according to claim 1 , wherein the position and orientation sensor includes at least one of an imaging means for imaging an object, a distance measurement means for measuring a distance to the object, a GPS, and a LiDAR.
3. 3. The information processing apparatus according to claim 2, wherein the moving state acquisition means and the position and orientation sensor use the same image capture means.
4. 3. The information processing apparatus according to claim 1, wherein the moving state acquisition means includes at least one of an imaging means, a distance measurement device, an encoder, a GPS, and a drive command for a drive source.
5. The information processing device according to claim 1, characterized in that the moving state acquisition means acquires as the moving state at least one of whether or not the amount of movement of the moving body is equal to or less than a predetermined threshold, whether or not the acceleration of the moving body is equal to or less than a predetermined threshold, or whether or not the angular velocity of the moving body is equal to or less than a predetermined threshold.
6. The information processing device according to claim 1, characterized in that the first weight determination means sets the weight for the output information of the inertial measurement sensor lower than a predetermined value when at least one of the following conditions is satisfied: the amount of movement of the moving body is less than a predetermined threshold, the acceleration of the moving body is less than a predetermined threshold, and the angular velocity of the moving body is less than a predetermined threshold.
7. a reliability acquisition means for acquiring a reliability of the position and orientation measured by the position and orientation sensor; a second weight determining means for determining a second weight for the output information of the position and orientation sensor based on the reliability; 2 . The information processing apparatus according to claim 1 , wherein the position and orientation estimation means performs the estimation process based on an output of the position and orientation sensor, an output of the inertial measurement sensor, the first weight, and the second weight.
8. The information processing device according to claim 7, characterized in that the reliability acquisition means calculates the reliability based on at least one of the number or distribution of feature points detected from the image output by the position and orientation sensor, the brightness of the image, the variation in the brightness, the number of moving objects in the image, the presence or absence of a specified subject in the image, and a reception state of GPS radio waves.
9. a reliability acquisition unit for acquiring a reliability of the position and orientation measured by the position and orientation sensor; 2. The information processing apparatus according to claim 1, further comprising a control unit that controls the movement or attitude change of the moving object so as to improve the reliability when the reliability acquired by the reliability acquisition unit is equal to or lower than a predetermined threshold value.
10. 10. The information processing apparatus according to claim 1, further comprising a display means for displaying the weight determined by said first weight determination means together with the current position of said mobile object.
11. An information processing device according to any one of claims 1 to 10; a movement control unit that controls the movement of the moving body based on the position and orientation estimated by the position and orientation estimation unit.
12. a position and orientation step of estimating a position and orientation of the moving object using an output of a position and orientation sensor mounted on the moving object; a position and orientation estimation step of estimating the position and orientation of the moving body by integrating a first estimation result obtained by performing a first estimation process of the position and orientation of the moving body using an output from an inertial measurement sensor mounted on the moving body separately from the position and orientation sensor, and a second estimation result obtained by performing a second estimation process of the position and orientation of the moving body using the output from the position and orientation sensor; a moving state acquisition step of acquiring a moving state of the moving body; an information processing method comprising: a first weight determination step of, when the moving state acquired by the moving state acquisition step is regarded as a stationary state, determining a first weight for integrating the first estimation result to be smaller than when the moving state is not regarded as a stationary state.
13. 12. An information processing device according to claim 1, or a computer program for controlling each means of a moving body according to claim 11 by a computer.
Citation Information
Patent Citations
Navigation system and navigation method
JP2008076389A
Positioning device and positioning method
JP2011141221A
Control device for autonomous travel type automatic conveyance vehicle and conveyance system
JP2021096731A
Position determination device
JP2022105431A
Apparatus of controlling movement of mobile robot mounted with wide angle camera and method thereof
US20170153646A1
