Information processing device, control method and program for information processing device

Abstract

To provide an information processing apparatus capable of stably operating a moving body.SOLUTION: An information processing apparatus 110 determines control contents for a moving body and a peripheral device of the moving body on the basis of a result of position and attitude measurement using first sensor data and second sensor data. The information processing apparatus 110 comprises: dependency acquisition means 111 which acquires a degree of dependence on the second sensor data in the position and attitude measurement; reliability calculation means 112 which calculates a degree of reliability of the position and attitude measurement on the basis of the degree of dependence acquired by the dependency acquisition means 111; and control-content determination means 113 which determines control contents on the basis of the degree of reliability calculated by the reliability calculation means 112.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device, a control method and a program for the information processing device, and more particularly to the control of a mobile object and peripheral devices of the mobile object. [Background technology]

[0002] Technologies have been proposed to automatically move mobile objects such as automated guided vehicles (AGVs) within environments such as factories and logistics warehouses. When moving automatically, it is sometimes necessary to create a map of the real space and measure the vehicle's own position and orientation within the real space. Known techniques for this include the SLAM (Simultaneous Localization and Mapping) method. Sensor data for measuring position and orientation include image data captured by a camera and laser image data measured by a LiDAR (Light Detection and Ranging). Alternatively, inertial measurement data (such as acceleration and angular velocity) measured by an IMU (Inertial Measurement Unit) can be used.

[0003] Patent Document 1 proposes a method for stabilizing the movement of a moving object by measuring the position and orientation using two types of sensor data, and selecting the result of measuring the position and orientation using the second sensor data when the reliability of the position and orientation measurement using the first sensor data decreases. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2019 / 026761 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the method of Patent Document 1 has a problem in that if the reliability of position and orientation measurement using the selected second sensor data decreases over time or distance, the mobile object cannot travel stably.

[0006] The present invention has been made in view of the above problems, and has an object to provide an information processing device that can operate a mobile body stably. [Means for solving the problem]

[0007] In order to achieve the above object, the information processing device of the present invention includes: First position and orientation measurement using and the second sensor data and the second position and orientation measurement based on An information processing apparatus that determines control details of a moving object and peripheral devices of the moving object based on a result of position and orientation measurement, Indicates the degree of dependence on position and orientation measurement a dependency degree obtaining means for obtaining a dependency degree; and The magnitude of the accumulated value Based on , the above Position and orientation measurement Result a reliability calculation means for calculating a reliability; and a control content determination means for determining the control content based on the reliability calculated by the reliability calculation means. When the accuracy of the first position and orientation measurement is equal to or greater than a predetermined threshold, the second position and orientation measurement is not used, and when the accuracy of the first position and orientation measurement is less than the predetermined threshold, the second position and orientation measurement is also used; The second position and orientation measurement has a larger error accumulated over time or distance than the first position and orientation measurement, and the accumulated value related to the degree of dependency is a value accumulated since the second position and orientation measurement started to be used. It is characterized by: [Effects of the Invention]

[0008] According to the present invention, a moving body can be operated stably. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a mobile system including an information processing device according to a first embodiment. [Figure 2] FIG. 1 is a diagram showing the hardware configuration of an information processing device according to a first embodiment. [Figure 3] 4 is a flowchart showing a processing procedure of the information processing device in the first embodiment. [Figure 4]FIG. 10 is a diagram showing an example of the configuration of a mobile system including an information processing device according to a second embodiment. [Figure 5] 10 is a flowchart showing a processing procedure of an information processing device according to a second embodiment. [Figure 6] FIG. 11 is a diagram showing an example of the configuration of a mobile system including an information processing device according to a third embodiment. [Figure 7] 10 is a flowchart showing a processing procedure of an information processing device according to a third embodiment. [Figure 8] FIG. 10 is a diagram showing a display screen in the fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the following embodiments do not limit the scope of the present invention, and not all of the combinations of features described in the embodiments are necessarily essential to the solution of the present invention. Note that in the following embodiments, "improving" reliability, accuracy, etc. means "enhancing" reliability, accuracy, etc.

[0011] [First embodiment] In this embodiment, an example will be described in which the method of the present invention is applied to notification control in a mobile object that transports luggage. In this embodiment, the mobile object travels automatically repeatedly along a predetermined route. During the automatic travel of the mobile object, position and orientation measurement is performed using SLAM technology using image feature points detected from image data captured by a camera mounted on the mobile object and acceleration and angular velocity measured by an IMU. Hereinafter, the output data of the IMU will be collectively referred to as "inertial measurement data."

[0012] In general, in SLAM based on camera images, the reliability of position and orientation measurement can be reduced due to changes in illumination, such as when lights are turned on and off, or changes in the placement of the subject around the camera. In such cases, the dependence on inertial measurement data for position and orientation measurement increases. However, position and orientation measurement using inertial measurement data involves, for example, integrating acceleration twice per unit time to calculate the amount of displacement and estimate the position, so the error increases as time passes.

[0013] Therefore, in this embodiment, the degree of dependency, which is the degree of dependency on inertial measurement data in position and orientation measurement of a moving body, is obtained. An accumulated value related to the degree of dependency is also obtained. Furthermore, the reliability, which indicates the degree of error accumulation in the position and orientation measurement, is calculated based on the degree of dependency. If the reliability of the position and orientation measurement decreases, a warning is issued.

[0014] In this embodiment, the moving object moves on a plane. Therefore, the position and orientation of the moving object are three parameters: two parameters (X, Y) that represent an arbitrary position on a two-dimensional plane, and one parameter (θ) that represents the orientation of the moving object.

[0015] 1 is a diagram showing an example of the configuration of a mobile body system including an information processing device in the first embodiment. The mobile body system 100 is composed of a first sensor data acquisition means 101, a second sensor data acquisition means 102, a position and orientation measurement means 103, a notification means 104, a movement means 105, a display means 106, and an information processing device 110. The mobile body system 100 in this embodiment is an example of a mobile body controlled by the information processing device 110, and the notification means 104, the movement means 105, and the display means 106 are examples of peripheral devices of the mobile body.

[0016] The information processing device 110 comprises a dependency obtaining means 111 , a reliability calculating means 112 , and a control content determining means 113 .

[0017] The first sensor data acquiring means 101 acquires first sensor data from a first sensor. In this embodiment, the first sensor is a camera, and the first sensor data is image data in which a luminance value is recorded for each pixel.

[0018] The second sensor data acquiring means 102 acquires second sensor data from a second sensor. In this embodiment, the second sensor is an IMU, and the second sensor data is acceleration and angular velocity.

[0019] Based on the first sensor data and the second sensor data, the position and orientation measurement means 103 measures the position and orientation of the mobile system 100. Details of the position and orientation measurement will be described later.

[0020] The notification means 104 acquires the notification content determined by the control content determination means 113 from the control content determination means 113 and notifies the user of the notification content. The notification content here is mainly notification content indicating the magnitude of the reliability, but is not limited to this.

[0021] The moving means 105 acquires the moving content determined by the control content determining means 113 from the control content determining means 113 and controls the traveling of the mobile body system 100 .

[0022] The display means 106 acquires the display contents determined by the control content determination means 113 from the control content determination means 113 and displays them on the screen.

[0023] The dependency obtaining means 111 obtains the dependency of the position and orientation measurement by the position and orientation measurement means 103 on the second sensor data.

[0024] The reliability calculation means 112 calculates the reliability of the position and orientation measurement based on the degree of dependency acquired by the degree of dependency acquisition means 111 .

[0025] In this embodiment, the control content determination means 113 determines control content such as notification content related to the mobile body system 100. Specifically, it determines notification content for the notification means 104, movement content for the movement means 105, and display content for the display means 106. Furthermore, as the movement content for the movement means 105 in this embodiment, the control content determination means 113 determines at least one of the progress of the mobile body, the speed of the mobile body, the attitude of the mobile body, or the path of the mobile body. Furthermore, as the display content for the display means 106, the control content determination means 113 determines display content that allows the magnitude of reliability to be identified. Note that the control content of the control content determination means 113 is not limited to these.

[0026] FIG. 2 is a diagram showing the hardware configuration of the information processing device 110 in the first embodiment. The CPU 11 controls various devices connected to the system bus 20. The ROM 12 stores BIOS programs and boot programs. The RAM 13 is used as the main storage device of the CPU 11. The external memory 14 stores programs processed by the information processing device 110. The input unit 15 is a keyboard, mouse, or robot controller, and performs processing related to the input of information, etc. The display unit 16 outputs the results of calculations performed by the information processing device 110 to a display device in accordance with instructions from the CPU 11. The display device may be any type, such as a liquid crystal display device, a projector, or an LED indicator. An I / O 17 is also connected to the system bus 20.

[0027] FIG. 3 is a flowchart showing the processing procedure of the information processing device 110 in the first embodiment.

[0028] In step S301, the information processing device 110 is initialized. Specifically, the reliability calculation means 112 reads a threshold value related to the degree of dependency used when calculating the reliability. The control content determination means 113 also reads settings such as a list of multiple pairs of reliability and control content used when determining the control content.

[0029] In step S302, the dependency obtaining unit 111 obtains the dependency on the second sensor data in the position and orientation measurement at a certain point in time from the position and orientation measurement unit 103. The dependency is a value indicating the degree to which the position and orientation measurement performed by the position and orientation measurement unit 103 depends on the position and orientation measurement using inertial measurement data. Details of the obtained dependency will be described later.

[0030] In step S303, the reliability calculation means 112 calculates the reliability based on the degree of dependency acquired in step S302. The reliability is a value indicating the degree of error accumulation in the measurement results obtained by the position and orientation measurement means 103. The method for calculating the reliability will be described in detail later.

[0031] In step S304, the control content determination means 113 refers to the list made up of pairs of reliability and control content described above, based on the reliability calculated in step S303, and determines the control content for the mobile body system 100. In this embodiment, the control content determination means 113 refers to the list, and when the calculated reliability falls below a predetermined value, determines to notify a warning message indicating that the reliability of the position and orientation measurement has decreased.

[0032] In step S305, it is determined whether or not the entire processing of the information processing device 110 is to be terminated. Specifically, the processing is terminated when the mobile body system 100 reaches the destination and ends the automatic traveling. If not, the processing returns to step S302 and continues.

[0033] In this embodiment, the position and orientation measurement means 103 performs position and orientation measurement using image data and inertial measurement data. Here, the image data is first sensor data captured by a camera, which is a first sensor. The inertial measurement data is second sensor data measured by an IMU, which is a second sensor.

[0034] In this embodiment, the ORB-SLAM3 method is used as a method for measuring the position and orientation of image data and inertial measurement data. Specifically, the method is the following method disclosed by Carlos Campos et al. at Cornell University: ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial and Multi-Map SLAM.

[0035] In this method, image features detected from image data captured by a camera are matched with multiple image features stored as a position and orientation measurement map. Next, the position and orientation are measured to minimize the sum of the differences (residuals) between the image feature positions and the inertial measurement data. At this time, the number of image feature points detected from the image data is calculated as the degree of dependency on the inertial measurement data.

[0036] The position and orientation measurement map is a collection of captured image data, the position and orientation of the camera at the time of capture, image feature points detected from the image data, and the three-dimensional positions of the image feature points. Image feature information refers to the feature points of the image features and their two-dimensional coordinates on the image. Image features are detected as feature points that indicate geometric structures such as corners in the image. A set of image data, the position and orientation at the time of capture, image feature points detected from the captured image, and the three-dimensional positions of the features is called a key frame.

[0037] In this embodiment, the position and orientation measurement map is created by the position and orientation measurement means 103 before the mobile body system 100 starts automatic traveling. The position and orientation measurement map is held by the position and orientation measurement means 103.

[0038] In this embodiment, the camera, which is the first sensor, is fixed to the mobile body system 100. Because the camera moves and rotates in conjunction with the movement and rotation of the mobile body system 100, the position and orientation of the mobile body system 100 are linked to the position and orientation of the camera. Because captured image data depends on the position and orientation of the camera, a change in the position or orientation of the mobile body system 100 affects the results of position and orientation measurement using image data.

[0039] In this embodiment, while the mobile body system 100 is traveling autonomously, the first sensor data acquisition means 101 repeatedly acquires image data at predetermined time intervals. The second sensor data acquisition means 102 repeatedly acquires inertial measurement data at predetermined time intervals. The position and orientation measurement means 103 repeatedly performs position and orientation measurement at predetermined intervals (for example, for each key frame held as a map for position and orientation measurement) using the latest image data and inertial measurement data acquired at that time.

[0040] In this embodiment, the degree of dependency on the second sensor data acquired by the dependency acquisition means 111 from the position and orientation measurement means 103 is calculated based on the small number of image feature points. Specifically, the degree is the number of image feature points detected from image data acquired at a certain point in time when the position and orientation measurement means 103 measures the position and orientation at that time during autonomous driving of the mobile body system 100. The degree of dependency is calculated so that the greater the number of image feature points, the lower the degree of dependency, and the fewer the number of image feature points, the higher the degree of dependency. This is because the accuracy of position and orientation measurement related to image data depends on the position and orientation of the mobile body at the time the image data is measured. Specifically, the accuracy of position and orientation measurement using image data depends on the number of image feature points, and the fewer the number of image feature points, the lower the accuracy of position and orientation measurement using image data, resulting in dependence on position and orientation measurement using inertial measurement data.

[0041] In this embodiment, the reliability calculated by the reliability calculation means 112 is calculated so that the reliability decreases as the time that the dependency acquired by the dependency acquisition means 111 continues to exceed a predetermined threshold increases. Specifically, the lowest reliability value is 0.0, the highest reliability value is 1.0, and the initial reliability value is 1.0. Starting from the time that the dependency exceeds the predetermined threshold, the reliability is reduced by 0.1 every time a predetermined time interval elapses while that state continues. If the dependency falls below the predetermined threshold two consecutive times within the predetermined time interval, the reliability is reset to the initial value of 1.0.

[0042] When the reliability of position and orientation measurement is reduced by the method of the first embodiment, the information processing device is controlled to issue a warning to the user before the situation becomes such that the mobile body system 100 cannot operate stably. As a result, the mobile body can operate stably.

[0043] In this embodiment, the degree of dependence on the second sensor data is calculated using the number of image feature points detected from the image data, but this is not limiting and the spatial bias of the image feature points may also be used. The degree of dependence is calculated so that the degree of dependence is low when the image feature points are spatially distributed widely, since the accuracy of position and orientation measurement using the image data is high. On the other hand, when the image feature points are spatially distributed unevenly, since the accuracy of position and orientation measurement using the image data is low, the degree of dependence is calculated so that the degree of dependence is high.

[0044] Furthermore, the present invention is not limited to this, as long as the setting values ​​used during position and orientation measurement or intermediate calculation results can be acquired from the position and orientation measurement means 103. For example, a residual error related to image data calculated during bundle adjustment of position and orientation measurement may be used. As a method for calculating the degree of dependency, if the residual error related to image data is small, the accuracy of position and orientation measurement using image data will be high, so the degree of dependency is calculated to be low. On the other hand, if the residual error related to image data is large, the accuracy of position and orientation measurement using image data will be low, so the degree of dependency is calculated to be high.

[0045] In addition, in position and orientation measurement using the ORB-SLAM3 method used in this embodiment, the degree of dependency may be calculated using a setting value for adjusting the proportion of image data and the proportion of inertial measurement data. The method for calculating the degree of dependency is such that the larger the setting value indicating the proportion of inertial measurement data, the higher the degree of dependency, and the smaller the setting value, the lower the degree of dependency.

[0046] Furthermore, the results of bias correction for inertial measurement data calculated during position and orientation measurement may be used. Due to the characteristics of the IMU, the inertial measurement data measured by the IMU contains a bias, and the ORB-SLAM3 position and orientation measurement method used in this embodiment performs processing to correct this bias. However, if the accuracy of position and orientation measurement using image data decreases, the difference in correction values ​​increases. Therefore, the larger the difference in correction values, the greater the degree of dependence on the inertial measurement data, which is the second sensor data, is calculated.

[0047] In this embodiment, the reliability calculation means 112 calculates the reliability so that the larger the accumulated value related to the dependency, the lower the reliability. For example, the reliability calculation means 112 calculates the reliability so that the longer the state in which the dependency exceeds a predetermined threshold continues, the lower the reliability. However, this is not limited to this. In other words, other methods may be used as long as they can calculate the error accumulation, which is the accumulated value of the dependency. For example, the reliability calculation means 112 may calculate the reliability so that the larger the sum of the dependency, the lower the reliability. Specifically, the sum of the dependency is calculated starting from the point in time when the dependency exceeds a predetermined threshold, and the reliability is reduced by a certain percentage depending on the size of the sum.

[0048] Furthermore, the reliability calculation means 112 may calculate the reliability so that the greater the distance traveled while the degree of dependency exceeds a predetermined threshold, the lower the reliability becomes. Specifically, the position and orientation measurement means 103 holds key frames at every predetermined distance and sets the initial value of the reliability to 1.0. Then, starting from the position where the degree of dependency exceeds the predetermined threshold, the position and orientation measurement means 103 reduces the reliability by 0.1 for each key frame it references while that state continues. If the degree of dependency falls below the predetermined threshold for consecutive key frames, the reliability is reset to 1.0.

[0049] As described above, the reliability calculation means 112 reduces the reliability by a certain percentage depending on the accumulated value of the dependency, the passage of time, and the distance traveled, but this is not limiting, and the reliability may be reduced exponentially, for example. Furthermore, the reliability may be calculated so as to be the reciprocal of the accumulated value of the dependency.

[0050] Furthermore, the reliability calculation means 112 reduces the reliability by a certain percentage starting from the point when the dependency exceeds a predetermined threshold, but it is also possible to calculate the reliability so that it is reduced appropriately depending on the magnitude of the dependency value without setting a predetermined threshold.

[0051] In this embodiment, when the reliability falls below a predetermined value, the control content determination means 113 determines to notify a user of a warning message indicating that the reliability of the position and orientation measurement has decreased, but this is not limiting. Any other method may be used as long as a means for notifying the user of the decrease in reliability is provided, and for example, it may be determined to sound an alarm or output the information to an external device.

[0052] Furthermore, as long as the determined control content can be maintained, the control content determination means 113 may determine the content of notification to notify that the reliability has recovered when the reliability exceeds a predetermined value from the next time onward. For example, when the reliability exceeds a predetermined value, the control content determination means 113 may determine to notify a message indicating that the reliability of the position and orientation measurement has improved. Similarly, when the reliability exceeds a predetermined value, the control content determination means 113 may determine to stop the warning sound.

[0053] In this embodiment, the first sensor is a camera and the first sensor data is image data, but this is not limiting. For example, the first sensor may be a LiDAR, and the first sensor data may be two-dimensional or three-dimensional point cloud data measured by the LiDAR.

[0054] Similarly, in this embodiment, the second sensor is an IMU, and the second sensor data is inertial measurement data, but this is not limited thereto. That is, if the mobile body system 100 is equipped with wheels, for example, the second sensor may be a rotary encoder that measures the rotation of the wheels, and the second sensor data may be the rotation angle of the wheels measured by the rotary encoder. Also, in this embodiment, inertial measurement data that combines acceleration and angular velocity is used as the second sensor data, but this is not limited thereto. Only acceleration or only angular velocity may be used as the second sensor data.

[0055] Methods for measuring position and orientation using two types of sensor data include a loose coupling method and a tight coupling method. The loose coupling method integrates the results of each position and orientation measurement, while the tight coupling method integrates data from two types of sensors to measure position and orientation. In this embodiment, the position and orientation measurement unit 103 uses the tight coupling method for position and orientation measurement, but this is not limiting and a loose coupling method may also be used. For example, the position and orientation measurement unit 103 performs position and orientation measurement using image data and position and orientation measurement using inertial measurement data, and corrects the result of the position and orientation measurement using image data with the result of the position and orientation measurement using inertial measurement data.

[0056] [Variation 1-1] A modification of the first embodiment will be described. In this modification, an example will be described in which the method of the present invention is applied to the movement control of a moving object. Specifically, when the reliability of position and orientation measurement decreases, the moving object slows down or stops.

[0057] In this modification, when the control content determination means 113 determines the control content, a list of a plurality of pairs of reliability and control content is used. The list describes a control content to slow down the vehicle for a first reliability value, and a control content to stop the vehicle for a second reliability value.

[0058] In this modification, the following describes changes from FIG. 3 , a flowchart showing the processing procedure of the information processing device 110 described in the first embodiment. In step S304, the control content determination means 113 refers to the list. If the reliability calculated by the reliability calculation means 112 becomes lower than a first value, the control content determination means 113 determines to slow down the traveling of the mobile body system 100. On the other hand, if the reliability becomes lower than a second value, the control content determination means 113 determines to stop the traveling of the mobile body system 100. Furthermore, if the reliability that was lower than the second value becomes higher than the second value, the control content determination means 113 determines to restart the traveling of the stopped mobile body system 100. Similarly, if the reliability that was lower than the first value becomes higher than the first value, the control content determination means 113 determines to restore the speed of the decelerated mobile body system 100.

[0059] The moving means 105 controls the travel of the mobile body system 100 in accordance with the control content determined by the control content determining means 113 .

[0060] In this modification, the control content determination means 113 determines to slow down or stop the traveling of the mobile body system 100 based on the reliability, but this is not limited to this. If the mobile body system 100 can secure a sufficiently wide traveling route, for example, the attitude of the mobile body system 100 may be changed. Specifically, when the reliability falls below a predetermined value, the control content determination means 113 determines to rotate the attitude of the mobile body system 100 by a predetermined angle. The moving means 105 changes the attitude of the mobile body system 100 accordingly. The above process is repeated until the reliability becomes higher than the predetermined value.

[0061] [Second embodiment] In this embodiment, an example will be described in which the method of the present invention is applied to the movement control of a moving object that transports luggage. In Modification 1-1, it was shown that the moving object can operate stably by deciding to decelerate or stop the moving object when the reliability of position and orientation measurement decreases.

[0062] In this embodiment, a reliability distribution is maintained, which is a pair of the position and orientation of a moving object and its reliability. When the reliability of the position and orientation measurement decreases, the reliability of the vicinity of the current position of the moving object is searched for, and the orientation of the moving object is changed in a direction that increases the reliability. This method for improving the reliability of the position and orientation measurement will be described.

[0063] Fig. 4 is a diagram showing an example of the configuration of a mobile system including an information processing device 110 in the second embodiment. Explanation of modules that are the same as those in Fig. 1, which shows an example of the configuration of a mobile system including the information processing device described in the first embodiment, will be omitted, and only modules that differ from the first embodiment will be explained.

[0064] In this embodiment, the information processing device 110 includes a dependency obtaining unit 111 , a reliability calculating unit 112 , a control content determining unit 113 , a position and orientation obtaining unit 411 , and a reliability distribution holding unit 412 .

[0065] The position and orientation acquisition unit 411 acquires the results of position and orientation measurement from the position and orientation measurement unit 103 .

[0066] The reliability distribution holding means 412 holds a list consisting of pairs of the position and orientation acquired by the position and orientation measurement means 103 and the reliability calculated by the reliability calculation means 112 based on the dependency corresponding to the position and orientation. Hereinafter, the list consisting of pairs associating the position and orientation of a moving object with reliability will be referred to as a reliability distribution list.

[0067] The control content determination means 113 determines the control content for the mobile body system 100 based on the reliability distribution list held by the reliability distribution holding means 412. In this embodiment, the control content determination means 113 determines the movement content for the mobile body system 100.

[0068] Fig. 5 is a flowchart showing the processing procedure of the information processing device 110 in the second embodiment. Explanation of the same steps as in Fig. 3, which is a flowchart showing the processing procedure of the information processing device described in the first embodiment, will be omitted, and only the steps different from the first embodiment will be explained. Note that Fig. 5 also performs repeated processing, just like Fig. 3.

[0069] In step S301, the confidence distribution list is initialized. After initialization, the confidence distribution list has zero elements.

[0070] In step S 501 , the position and orientation acquisition unit 411 acquires the position and orientation from the position and orientation measurement unit 103 .

[0071] In step S502, the reliability distribution holding unit 412 adds the position and orientation acquired in step S501 and the reliability calculated in step S303 based on the degree of dependency corresponding to the position and orientation to a reliability distribution list as a set.

[0072] In step S503, the control content determination means 113 determines the movement content of the mobile body system 100 based on the pair of position / posture and reliability newly added in step S502 and the reliability distribution list held up to that point. The method for determining the movement content will be described in detail later.

[0073] In step 503, if the newly added reliability falls below a predetermined value, the control content determination means 113 determines to change the posture of the mobile body system 100 in a direction that improves the accuracy of position and orientation measurement using image data. Here, the direction that improves the accuracy of position and orientation measurement using image data refers to a direction determined based on a position and orientation paired with a reliability higher than a predetermined value in the reliability distribution list. Specifically, the control content determination means 113 first obtains the pair of position and orientation and reliability newly added by the reliability distribution storage means 412 from the reliability distribution list. If the newly added reliability is lower than a predetermined value, the control content determination means 113 searches the reliability distribution list to determine whether a pair of position and orientation and reliability with a reliability higher than the predetermined value exists in the vicinity of the position in the newly added position and orientation. The vicinity of the newly added position is determined by whether the difference between the newly added position and other positions referenced in the reliability distribution list is within a predetermined distance.

[0074] If a pair of a position / posture and reliability with a reliability higher than a predetermined value exists in the vicinity of the newly added position, the newly added position is rotated by a predetermined angle to calculate the camera's viewing frustum, and the posture of the mobile body system 100 is changed so that it overlaps most with the viewing frustum of the camera at the position / posture with a reliability higher than the predetermined value.

[0075] If there are multiple pairs of position and orientation and reliability with reliability higher than a predetermined value in the vicinity of the newly added position, the pair with the smallest difference from the newly added position and orientation is selected.

[0076] If there is no pair of position, posture and reliability in the vicinity of the newly added position, the pair of position, posture and reliability having a reliability higher than a predetermined value, the control content determination means 113 determines to slow down the traveling speed of the mobile system 100, as in variant example 1-1.

[0077] According to the method of this embodiment, even if the reliability of the position and orientation measurement is reduced, the reliability of the position and orientation measurement can be improved by changing the orientation of the moving body, and the moving body can operate stably.

[0078] In this embodiment, the control content determination means 113 changes the orientation of the mobile system 100 so as to maximize overlap with the camera's viewing frustum at a position and orientation where the reliability is higher than a predetermined value, but this is not limiting. Other methods may be used as long as they can calculate an orientation where image features detected from image data at that position and orientation can be captured. For example, the orientation may be changed so that the image feature point group of the image data at that position and orientation is included in the viewing frustum at the newly added position. Alternatively, the orientation may be changed so that a plane obtained by cutting the viewing frustum at that position and orientation at a predetermined depth is included in the viewing frustum at the newly added position.

[0079] In this embodiment, when the reliability becomes lower than a predetermined value, the control content determination means 113 determines to change the attitude of the mobile system 100 in a direction that improves the accuracy of position and orientation measurement using image data, but this is not limited to this.

[0080] For example, if the newly added reliability falls below a predetermined value, the control content determination means 113 may determine to change the position of the mobile body system 100 to a position where the accuracy of position and orientation measurement using image data is improved. Here, the position where the accuracy of position and orientation measurement using image data is improved refers to a position determined based on a position and orientation paired with a reliability higher than a predetermined value in the reliability distribution list. As with the orientation change, the reliability distribution list is searched, and if a pair of a position and orientation and a reliability with a reliability higher than a predetermined value is found in the vicinity of the newly added position, the position of the mobile body system 100 is changed so that it moves to the corresponding position. Furthermore, in addition to the position, the orientation of the mobile body system 100 may be changed so that it approximately matches the orientation of the search result.

[0081] Furthermore, if the control content determination means 113 can retain the determined control content, it may decide to return the attitude of the mobile system 100 to the attitude before the change if the reliability becomes higher than a predetermined value from the next time onwards.

[0082] In step S301 of this embodiment, the reliability distribution list is in an initial state with 0 elements, but this is not limited to this. For example, other methods may be used as long as the reliability distribution holding means 412 holds the reliability distribution list even after the end of the processing shown in Fig. 5 and the mobile system 100 travels on the same two-dimensional plane the next time. For example, the reliability distribution list held in the previous processing may be read in at the time of initialization and reused.

[0083] [Third embodiment] In this embodiment, an example will be described in which the method of the present invention is applied to the movement control of a mobile object that carries luggage. In the second embodiment, a method has been shown in which a reliability distribution, which is a pair of position and orientation and reliability, is maintained, and when the reliability of position and orientation measurement decreases, a nearby reliability distribution is searched for and the orientation of the mobile object is changed in a direction that increases the reliability, thereby improving the reliability of position and orientation measurement.

[0084] In this embodiment, a feature distribution that combines position and orientation and image features is maintained, and when the reliability of position and orientation measurement decreases, a nearby image feature distribution is searched for, and the travel route of the moving object is changed to a route with higher reliability. This method for improving the reliability of position and orientation measurement will be described.

[0085] Fig. 6 is a diagram showing an example of the configuration of a mobile system including an information processing device 110 in the third embodiment. Explanation of modules that are the same as those in Fig. 4, which shows an example of the configuration of a mobile system including the information processing device described in the second embodiment, will be omitted, and only modules that differ from the second embodiment will be explained.

[0086] The information processing device 110 includes a dependency obtaining means 111, a reliability calculating means 112, a control content determining means 113, a position and orientation obtaining means 411, and a reliability distribution holding means 412. In this embodiment, the information processing device 110 also includes a feature distribution obtaining means 611.

[0087] The feature distribution storage means 601 acquires a set of three-dimensional positions of image feature points detected during position and orientation measurement using image data, which is first sensor data, from the position and orientation measurement means 103, as well as the position and orientation at the time of capturing the image data.The feature distribution storage means 601 then stores the set of positions and orientations and the three-dimensional positions of image feature points as a list.Hereinafter, this list will be referred to as a feature distribution list.

[0088] The feature distribution acquisition means 611 acquires the feature distribution list from the feature distribution storage means 601 .

[0089] The control content determination means 113 determines the movement content for the mobile body system 100 based on the pairs of position and orientation and reliability stored in the reliability distribution storage means 412 and the feature distribution list acquired by the feature distribution acquisition means.

[0090] Fig. 7 is a flowchart showing the processing procedure of the information processing device 110 in the third embodiment. Explanation of the same steps as in Fig. 5, which is a flowchart showing the processing procedure of the information processing device described in the second embodiment, will be omitted, and only the steps different from the second embodiment will be explained. Note that Fig. 7 also performs repeated processing, similar to Fig. 5.

[0091] In step S 701 , the feature distribution acquisition means 611 acquires a feature distribution list from the feature distribution storage means 601 .

[0092] In step S702, the control content determination means 113 determines the movement content of the mobile body system 100 based on the pair of position and orientation and reliability held in step S502 and the feature distribution list acquired in step S701. The method for determining the movement content will be described in detail later.

[0093] In step S702, if the reliability becomes lower than a predetermined value, the control content determination means 113 determines to change the route (travel route) of the mobile system 100 to a route that improves the accuracy of position and orientation measurement using image data.

[0094] Specifically, the control content determination means 113 first acquires a pair of position and orientation and reliability stored in the reliability distribution storage means 412. If the stored reliability is lower than a predetermined value, the control content determination means 113 searches the feature distribution list acquired by the feature distribution acquisition means 611 to determine whether there are more than a predetermined number of image feature points in the vicinity area of ​​the position in the stored position and orientation. The vicinity area of ​​the stored position is determined by whether the difference between the stored position and other positions referenced in the feature distribution list is within a predetermined distance.

[0095] If there are more than a predetermined number of sets of image feature points in the area near the stored position, the travel route of the mobile system 100 is changed so that the mobile system 100 travels through the corresponding position. If there are more than a predetermined number of sets of image feature points in the area near the current position, the set of image feature points that differs least from the current position and orientation is selected. If there is no set of image feature points in the vicinity of the stored position whose reliability is greater than a predetermined value, the control content determination means 113 determines to slow down the traveling speed of the mobile system 100, as in variant example 1-1.

[0096] According to the method of this embodiment, even if the reliability of the position and orientation measurement is reduced, the reliability of the position and orientation measurement can be improved by changing the route of the mobile object, and the mobile object can operate stably.

[0097] In this embodiment, the control content determination means 113 may determine to restore the driving route of the mobile system 100 to its original state if the reliability becomes higher than a predetermined value from the next time onwards, as long as the determined control content can be maintained.

[0098] Furthermore, the control content determination means 113 makes a determination based on whether the number of image feature points is greater than a predetermined number, but this is not limiting and any index that can be used as an index for alignment may be used. For example, an edge component or a plane component detected from image data, or a luminance distribution, or an area with a pattern such as a marker may be used as an index.

[0099] In this embodiment, the feature distribution list is in an initial state with 0 elements, but this is not limiting. As long as the feature distribution storage means 601 stores the feature distribution list even after the processing shown in Fig. 7 is completed and the mobile system 100 travels on the same two-dimensional plane the next time, other methods may be used. For example, the feature distribution list stored in the previous processing may be read and reused.

[0100] [Fourth embodiment] In this embodiment, an example will be described in which the method of the present invention is applied to display control in a moving body. Specifically, the distribution of the calculated reliability of position and orientation measurement is displayed on the screen.

[0101] The diagram showing an example of the configuration of a mobile system including an information processing device in this embodiment is the same as FIG. 4 showing the example of the configuration of a mobile system including an information processing device described in the second embodiment.

[0102] The flowchart showing the processing procedure of the information processing device 110 in this embodiment is the same as the flowchart showing the processing procedure of the information processing device 110 described in the second embodiment in Fig. 5. Only steps that differ from the second embodiment will be described below.

[0103] In step S503, the control content determination means 113 determines to display the distribution of the reliability determined from the reliability distribution list held in step S502. Details of the display content will be described later.

[0104] In this embodiment, the control content determining means 113 determines to distinguishably display areas according to the level of reliability for the position and orientation and reliability held by the reliability distribution holding means 412 .

[0105] 8 is a diagram showing a display screen in the fourth embodiment. Specifically, it shows an example of a screen displayed on the display unit 16 by the display means 106 based on the display contents determined by the control content determination means 113.

[0106] A map 800 in FIG. 8 is a map that represents the space in which the mobile body system 100 travels on a two-dimensional plane. A position 801 is the current position of the mobile body system 100. A route 802 is the route that the mobile body system 100 is scheduled to travel. A legend 803 indicates the reliability of the position and orientation measurement by color intensity. In the legend 803, the reliability of the spatial region in which the mobile body system 100 travels is indicated by color intensity. A white region indicates a region where the reliability is 0.8 to 1.0 and the reliability of the position and orientation measurement is high. A light gray region indicates a region where the reliability is 0.5 to 0.8 and the reliability of the position and orientation measurement is relatively high. A dark gray region indicates a region where the reliability is less than 0.5 and the reliability of the position and orientation measurement is low. In FIG. 8, a region 804 is a region where the reliability is less than 0.5 and the reliability of the position and orientation measurement is low. The reliability of the region 805 is between 0.5 and 0.8, and the reliability of the position and orientation measurement is relatively high.

[0107] The method of this embodiment visualizes the reliability of position and orientation measurement for each region and displays it to the user, thereby prompting the user to change the driving control of the mobile body system 100, thereby enabling the mobile body to operate stably.

[0108] The control content determination means 113 may display the movement content determined based on the position, orientation, and reliability. For example, point 806 in Fig. 8 is a point that reflects the movement content determined based on the position, orientation, and reliability. Explanation 807 in Fig. 8 explains the movement content determined based on the position, orientation, and reliability.

[0109] Furthermore, in order to be able to determine whether or not the attitude of the mobile body system 100 should be changed, the control content determining means 113 may determine to display the reliability of each direction at a predetermined position based on the position and attitude and the reliability.

[0110] The method of the fourth embodiment visualizes the reliability of the position and orientation measurement for each region and displays it to the user, thereby prompting the user to change the driving control of the mobile body system 100, thereby enabling the mobile body to operate stably.

[0111] [Other embodiments] Although the present invention has been described in detail based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various modifications within the scope of the present invention are also included in the present invention. In addition, parts of the above-described embodiments may be combined as appropriate.

[0112] In the above-described embodiment, the moving body moves on a two-dimensional plane, but this is not limiting, and the moving body may move in a three-dimensional space, such as a drone. In this case, the position and orientation are six parameters: three parameters (X, Y, Z) that indicate the position in the three-dimensional space, and three parameters (roll, pitch, yaw) that indicate the orientation in the three-dimensional space.

[0113] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0114] 100...Mobile Systems 101...first sensor data acquisition means 102...second sensor data acquisition means 103...Position and orientation measurement means 104…Notification means 105...Transportation 106...Display means 110...Information processing device 111…Dependency acquisition means 112...Reliability calculation means 113...Control content determination means

Claims

1. 1. An information processing apparatus that determines control details of a moving object and peripheral devices of the moving object based on results of position and orientation measurement based on a first position and orientation measurement using first sensor data and a second position and orientation measurement using second sensor data, a dependency acquiring means for acquiring a dependency indicating a degree of dependency of the position and orientation measurement on the second position and orientation measurement; a reliability calculation means for calculating a reliability of the result of the position and orientation measurement based on the magnitude of the accumulated value related to the degree of dependency acquired by the degree of dependency acquisition means; a control content determination means for determining the control content based on the reliability calculated by the reliability calculation means; Equipped with When the accuracy of the first position and orientation measurement is equal to or greater than a predetermined threshold, the second position and orientation measurement is not used, and when the accuracy of the first position and orientation measurement is less than the predetermined threshold, the second position and orientation measurement is also used; the second position and orientation measurement has a larger error accumulated over time or distance than the first position and orientation measurement; the accumulated value related to the dependency is a value accumulated since the second position and orientation measurement started to be used.

1. An information processing device comprising:

2. the reliability calculation means calculates the reliability such that the reliability becomes lower as the accumulated value related to the degree of dependency becomes larger.

2. The information processing apparatus according to claim 1, wherein:

3. the reliability calculation means calculates the reliability such that the reliability becomes lower as the duration of the state in which the degree of dependency exceeds a predetermined value increases.

2. The information processing apparatus according to claim 1, wherein:

4. the reliability calculation means calculates the reliability such that the reliability becomes lower as the distance traveled in a state in which the dependency exceeds a predetermined value increases.

2. The information processing apparatus according to claim 1, wherein:

5. the control content determination means determines notification content indicating the magnitude of the reliability based on the reliability.

5. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

6. the control content determination means determines, based on the reliability, one of the progress of the moving object, the speed of the moving object, the attitude of the moving object, or the path of the moving object.

5. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

7. the control content determination means determines, based on the reliability, a display content that allows the magnitude of the reliability to be identified.

5. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

8. a position and orientation acquisition means for acquiring a result of the position and orientation measurement; a reliability distribution holding means for holding the reliability calculated by the reliability calculation means in association with the position acquired by the position and orientation acquisition means; Furthermore, the control content determination means determines the control content based on the reliability and the position stored by the reliability distribution storage means so as to improve accuracy of the first position and orientation measurement.

8. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

9. a position and orientation acquisition means for acquiring a result of the position and orientation measurement; a reliability distribution holding means for holding the reliability calculated by the reliability calculation means in association with the position and orientation acquired by the position and orientation acquisition means; Furthermore, the accuracy of the first position and orientation measurement depends on the orientation of the moving object at the time of measuring the first sensor data; the control content determination means determines the control content based on the reliability and the position and orientation stored by the reliability distribution storage means so as to change the orientation of the moving object in a direction that improves accuracy of the first position and orientation measurement.

8. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

10. the control content determination means determines the control content so as to change the route of the moving object to a route that improves accuracy of the first position and orientation measurement, based on the distribution of the reliability stored by the reliability distribution storage means.

10. The information processing device according to claim 8, wherein the information processing device is a computer.

11. the control content determining means determines the control content so as to display the distribution of the reliability stored by the reliability distribution storing means.

10. The information processing device according to claim 8, wherein the information processing device is a computer.

12. the first sensor data is image data; The second sensor data is inertial measurement data.

12. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

13. 1. A control method for an information processing device that determines control details for a moving object and peripheral devices of the moving object based on results of position and orientation measurement that are based on a first position and orientation measurement using first sensor data and a second position and orientation measurement using second sensor data, comprising: a dependency obtaining step of obtaining a dependency indicating a degree of dependency of the position and orientation measurement on the second position and orientation measurement; a reliability calculation step of calculating a reliability of the result of the position and orientation measurement based on the magnitude of the accumulated value related to the degree of dependency acquired in the degree of dependency acquisition step; a control content determination step of determining the control content based on the reliability calculated in the reliability calculation step; and When the accuracy of the first position and orientation measurement is equal to or greater than a predetermined threshold, the second position and orientation measurement is not used, and when the accuracy of the first position and orientation measurement is less than the predetermined threshold, the second position and orientation measurement is also used; the second position and orientation measurement has a larger error accumulated over time or distance than the first position and orientation measurement; the accumulated value related to the dependency is a value accumulated since the second position and orientation measurement started to be used.

2. A method for controlling an information processing apparatus comprising:

14. 13. A program causing a computer to function as each of the means of the information processing apparatus according to claim 1.

Images