Information processing device, information processing method, and program

Abstract

To stably operate a moving body while avoiding degradation of precision in estimating a position and posture during movement of the moving body.SOLUTION: Pieces of information on a position and posture of a moving body and the reliability in estimating the position and posture are acquired based on a measured value of a sensor incorporated in the moving body. A reliability map exhibiting the reliability for each posture of the sensor is created based on the position and posture and the reliability retained in retention means. The posture of the moving body incorporating the sensor is determined based on the reliability map so that the reliability becomes equal to or larger than a predetermined value when the moving body is moved or controlled.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technique for controlling the attitude of a moving object. [Background technology]

[0002] A technology for estimating the position and orientation of a moving object has been proposed in order to autonomously move the moving object, such as a transport vehicle (e.g., an AGV (Automated Guided Vehicle)), in an environment such as a factory or a logistics warehouse. When moving the moving object autonomously, it is desirable to move the moving object along a route that allows the position and orientation of the moving object to be estimated with high accuracy. For example, in Patent Document 1, reliability map data is created that associates each position in space with the reliability of the position and orientation estimation, and the reliability map data is referred to to determine a high reliability area where the reliability is equal to or higher than a threshold, and a route that passes through the high reliability area is generated. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-166702 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the method of Patent Document 1, areas where the position and orientation estimation is determined to be low reliable are excluded from passable areas to make them impassable, and the vehicle passes only through areas where the position and orientation estimation is highly reliable. The present invention has been made in consideration of the above problems, and has an object to allow a moving body to operate stably while avoiding a decrease in the accuracy of position and orientation estimation while moving. [Means for solving the problem]

[0005] An information processing device according to the present invention has the following configuration: Namely, the device includes a position and orientation information acquisition means for acquiring information on the position and orientation of a moving body and the reliability of the position and orientation estimation based on measurement values ​​of a sensor mounted on the moving body, a creation means for creating a reliability map having the reliability for each orientation of the sensor based on the position and orientation and the reliability held by a holding means, and a determination means for determining the orientation of the moving body mounting the sensor based on the reliability map created by the creation means such that the reliability is equal to or greater than a predetermined value when controlling the movement of the moving body. Effect of the Invention

[0006] According to the present invention, it is possible to ensure stable operation of a moving body while avoiding a decrease in the accuracy of position and orientation estimation while the moving body is moving. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram showing an example of a functional configuration of an information processing device according to a first embodiment. [Diagram 2] FIG. 1 is a diagram showing an example of a hardware configuration of an information processing device according to a first embodiment. [Diagram 3] 1 is a flowchart showing a process executed by an information processing device according to the first embodiment. [Figure 4] 1 is a flowchart showing a process executed by a creation unit according to the first embodiment. [Diagram 5] FIG. 1 is a diagram for explaining a method for creating a reliability map in the first embodiment. [Figure 6] FIG. 1 is a diagram for explaining a GUI of an information processing device according to a first embodiment. [Figure 7] 11 is a flowchart showing a process executed by an information processing device according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] [Embodiment 1] In this embodiment, an example will be described in which a mobile object that transports luggage in a logistics warehouse is controlled.

[0009] Here, the logistics warehouse assumed in this embodiment is a warehouse where multiple moving objects or people work, and the moving objects repeatedly go around the route that each moving object is responsible for in the warehouse. In a logistics warehouse, the number and distribution of feature points detected from an image captured by a camera mounted on a moving object may change due to changes in the lighting environment such as turning lights on and off, or changes in the shooting environment such as changing the arrangement of items placed on shelves, and the accuracy of the estimation of the position and orientation may change. Note that when the reliability of the position and orientation estimation is high, the accuracy and stability of the position and orientation estimation are generally high. Here, even at the same location, the captured image changes depending on the orientation at the time of shooting, so the reliability also differs for each camera orientation. The orientation of the camera is linked to the orientation of the moving object.

[0010] Therefore, in this embodiment, a method is described in which a reliability map is created for each posture at a position in space, and when this reliability map is used to control the movement of a moving body, the posture of the moving body is determined so that the reliability of position and posture estimation is equal to or greater than a predetermined value, thereby enabling stable operation.

[0011] The position and orientation of a moving body in this embodiment refers to three parameters: two parameters (X, Y) that represent an arbitrary position on a two-dimensional plane, and one parameter (θ) that represents the orientation of the moving body.

[0012] FIG. 1 is a diagram showing an example of the configuration of a mobile system including an information processing device according to the present embodiment.

[0013] The mobile system 100 includes a sensor 101, a position and orientation estimation unit 102, a control unit 103, and an information processing device 110. The information processing device 110 includes a position and orientation information acquisition unit 111, a storage unit 112, a creation unit 113, and a determination unit 114. In this embodiment, the sensor 101 is a camera that captures images.

[0014] The position and orientation estimation unit 102 estimates the position and orientation of the moving object based on the measurement values ​​obtained from the sensor 101 and the reliability of the estimated position and orientation values, and outputs the results to a position and orientation information acquisition unit 111 .

[0015] The position and orientation information acquisition unit 111 acquires the estimation result input by the position and orientation estimation unit 102, and outputs it to the holding unit 112 as position and orientation information.

[0016] The holding unit 112 holds the position and orientation information input by the position and orientation information acquisition unit 111 .

[0017] The creation unit 113 creates a reliability map based on the position and orientation information stored in the storage unit 112 , and outputs the reliability map to the determination unit 114 .

[0018] The determination unit 114 determines the posture of the moving object based on the reliability map input by the creation unit 113 , and outputs the posture to the control unit 103 .

[0019] FIG. 2 is a diagram showing a hardware configuration of the information processing device 110. H11 is a CPU, which controls various devices connected to the system bus H20. H12 is a ROM, which stores a BIOS program and a boot program. H13 is a RAM, which is used as a main storage device of the CPU H11. H14 is an external memory, which stores a program to be processed by the information processing device 110. The input unit H15 is a keyboard, a mouse, or a robot controller, which performs processing related to input of information, etc. The display unit H16 outputs the calculation result of the information processing device 110 to a display device according to an instruction from H11. The display device may be a liquid crystal display device, a projector, an LED indicator, or any other type. H17 is a communication interface, which performs information communication via a network, and the communication interface may be Ethernet, or any type such as USB, serial communication, or wireless communication. H17 is an I / O, which inputs image data from a sensor H18. The sensor H18 is the sensor 101 described above. H19 is the control unit 103 described above.

[0020] Fig. 3 is a flowchart showing a processing procedure in this embodiment. Hereinafter, it is assumed that the flowchart is realized by a CPU executing a control program. The processing described in Fig. 3 starts when an instruction is given to start the running of a moving object.

[0021] In step S101, the information processing device 110 is initialized. That is, a program is read from the external memory H14, and the information processing device 110 is made operable. In addition, a reliability threshold value used in the determination unit 114 and route information of the moving object are read into the RAM H13. Here, the route information refers to information on a route along which the moving object is scheduled to travel, and includes target position coordinates and attitudes of the moving object at one or more waypoints set on the route.

[0022] In step S102, the position and orientation information acquisition unit 111 acquires the position and orientation and reliability estimated by the position and orientation estimation unit 102 based on the measurement value acquired from the sensor 101, and outputs them to the holding unit 112 as position and orientation information. In this embodiment, the sensor 101 is fixed to the moving body, and the sensor moves and rotates in conjunction with the movement and rotation of the moving body, so that the position and orientation of the moving body corresponds to the position and orientation of the sensor. In addition, as a position and orientation estimation method using a sensor, SLAM (Simultaneous Localization and Mapping) technology, a method by Engel et al. (J. Engel, T. Schps, and D. Cremers. LSD-SLAM: Large-Scale Direct Monocular SLAM. In European Conference on Computer Vision (ECCV), 2014) is used. This method simultaneously executes self-position and orientation estimation and map creation in a wide-area environment. The position and orientation and reliability are obtained as the estimation results of SLAM. In this embodiment, the reliability is calculated based on the number of feature points extracted from the captured image of the sensor 101 and the uniformity of their distribution. The more uniformly distributed the feature points are, the higher the reliability. If the feature points are unevenly distributed and the number of feature points included in a specific area is small, the reliability will be low. The storage unit 112 stores the positions and orientations and reliability previously estimated by the position and orientation information acquisition unit 102, and the position and orientation information is array data in which a pair of position and orientation and reliability is one element.

[0023] In step S103, the creation unit 113 creates a reliability map for each posture of the moving object with respect to the position in space, based on the position and posture information previously output by the position and posture information acquisition unit 111, which is held in the holding unit 112. The method of creating this reliability map will be described later.

[0024] In step S104, the determination unit 114 determines an attitude so that the reliability is equal to or greater than a threshold stored in the RAM of H13 when controlling the movement of the position of the moving object, based on the route information read into the RAM of H13 and the reliability map created by the creation unit 113. The method of determining the attitude will be described in detail later.

[0025] In step S105, the attitude of the moving object is controlled so that it assumes the attitude determined by the determination unit 114.

[0026] In step S106, it is determined whether or not to end the entire process of the information processing device 110. Specifically, the process ends when an end instruction from the user is received via the input unit H15. If not, the process returns to step S102 and continues.

[0027] 4 is a flowchart showing the procedure of the reliability map creation method in step S 103. In this embodiment, it is assumed that the moving object moves on a two-dimensional plane, and the posture that can be specified in the movement control is rotation around the vertical axis.

[0028] In step S201, the posture of the moving object is classified by angle at any position (x, y) on a two-dimensional plane of the moving object. Specifically, the posture is classified into eight regions divided by 45 degrees between the X axis and the Y axis as shown in Fig. 5(a). The dotted frame in Fig. 5(b) indicates the angle range of the posture classified in the positive direction of the Y axis.

[0029] In step S202, a representative value of the reliability is calculated for each classified posture. First, from among the combinations of position and posture and reliability stored in the storage unit 112, postures and reliability that are within a predetermined range from the position (x, y) are acquired. Furthermore, an average value of the target reliability is calculated for each posture among them. For example, in the case of the positive Y-axis direction, the reliability in which the posture falls within the angle range within the dotted frame in FIG. 5(b) is targeted, and if the target reliability is 0.6, 0.7, and 0.8 as shown in FIG. 5(c), the representative value of the reliability is the average value 0.7. When the posture is expressed by the angle θ formed with the positive X-axis direction, the reliability is the reliability for the position and posture in which the posture of the moving body falls within the range of θ=67.5 degrees to 112.5 degrees, for example.

[0030] In step S203, it is determined whether the calculation of the reliability representative value in step S202 has been completed for all postures. "All postures" refers to all classified postures, for example, all eight of the number of arrows in FIG. 5(b). If the calculation has been completed for all postures, proceed to step S204, and if not, return to step S202. When the calculation of the reliability representative value for all postures of position (x, y) has been completed, a reliability map for each posture of the moving object as shown in FIG. 5(d) is obtained. In FIG. 5(d), the reliability for each classified posture is plotted with the origin at 0 and the maximum value at 1, and adjacent reliability values ​​are connected by solid lines.

[0031] In step S204, it is determined whether or not the creation of the pose-specific reliability map is completed for all positions of the positions and orientations stored in the storage unit 112. If the creation is completed, the process ends, and if not, the process returns to step S201 and continues.

[0032] In this manner, a reliability map for each pose of the moving object with respect to the position in space is created.

[0033] Next, a method for determining the attitude in step S104 will be described.

[0034] In step S103, the attitude of the moving object is determined based on the route information read into the RAM of H13 and the reliability map created by the creation unit 113.

[0035] For each position on the path that the moving object is scheduled to pass, the reliability of the default posture is compared with a threshold stored in the RAM of H13 based on the reliability map for each posture. If the reliability is greater than the threshold, the posture is not changed, and if the reliability is less than the threshold, the posture is changed to one that is greater than or equal to the threshold based on the reliability map.

[0036] A specific example will be described with reference to Fig. 5. For example, at position (x, y), the reliability increases when the camera is directed in the positive direction of the X axis according to the reliability map for each posture of the moving body in Fig. 5(d), so the posture of the moving body at position (x, y) is determined to be in the positive direction of the X axis. The determined posture is output to the control unit 103, and the determined posture is reflected in the movement control of the moving body.

[0037] In this embodiment, the user may use a GUI to confirm and edit the posture determined by the determination unit 114. Fig. 6 shows an example of a GUI 100 that receives parameter designation from the user and is presented by the presentation unit 104.

[0038] G110 indicates the horizontal plane when the coordinate space is viewed from above regarding the movement path of the mobile object 100. The dotted arrow indicates the travel path, and G111 indicates the position on the path currently selected by the user.

[0039] G120 is a slider for setting a reliability threshold used when determining the posture of the moving object by the determination unit 114. The determination unit 114 determines the posture of the moving object so that the reliability is greater than the threshold set here.

[0040] G130 is a slider for setting the granularity for discretizing the posture when creating a reliability map by the creation unit 113. Here, a value for classifying in increments of 45 degrees, similar to the example in FIG.

[0041] G140 visualizes the reliability map created by the creation unit 113 and the posture determined by the determination unit 114. The displayed reliability map is a reliability map for each posture of the moving body at the point G111 currently specified by the user, and on the reliability map, the currently set posture of the moving body is illustrated by a solid arrow, and the posture of the moving body determined by the determination unit 114 is illustrated by a dotted arrow. In this example, it is recommended that the moving body travels at the point G111 while tilting at an angle of 45 degrees to the right.

[0042] The GUI shown in FIG. 6 allows the user to visually recognize the reliability map for each posture of the moving object and the posture of the moving object determined based on the reliability map.

[0043] In the first embodiment, an information processing device that estimates its own position and orientation using a camera mounted on a moving object can operate the moving object stably even in an environment where the accuracy of position and orientation measurement decreases.

[0044] <Variation 1-1> In the first embodiment, the position and orientation information acquisition unit 111 acquires the position and orientation estimated by the SLAM technique using the sensor 101, and the reliability is calculated based on the number of feature points extracted from the captured image of the sensor 101 and the uniformity of the distribution. However, the present invention is not limited to this, and a sensor capable of estimating the position and orientation and reliability in combination with the SLAM technique can be used. The position and orientation and reliability may be estimated by combining an active distance sensor such as Time of Flight or pattern light projection with the SLAM technique. The reliability may be calculated by any calculation method as long as it can calculate a value representing the accuracy and stability of the position and orientation. For example, the reliability may be calculated based on the luminance change between the captured image and the previous image in the time series.

[0045] The estimation method is not limited to SLAM technology, and may be any method capable of estimating the position, orientation, and reliability. For example, it may be a method based on radio wave intensity measurement in wireless communication such as Wifi. It may also be a method using satellite positioning such as GPS, a magnetic tracking sensor, or a time difference between detections by multiple optical sensors. It may also be a method of model fitting based on CAD data using images acquired by a camera installed at a specific location such as a surveillance camera.

[0046] Furthermore, the acquisition of position and orientation information is not limited to estimation using a sensor, and may be estimated without using a sensor if the position and orientation and the reliability are obtained. For example, the position and orientation may be virtually set, and the reliability may be calculated based on the number and distribution of feature points that can be observed at that time.

[0047] <Variation 1-2> In the first embodiment, the creation unit 113 uses an average as a method for calculating a representative value of reliability for each posture of the moving body. However, the method for calculating the representative value of reliability is not limited to this, and any method may be used as long as it can obtain a representative value of reliability. For example, the median value of reliability may be used as the representative value. Also, a method may be used in which the reliability of the position and posture that is the angle closest to the reference angle for each posture is used as the representative value by using nearest neighbor interpolation. A method may be used in which the magnitude of the angle between the reference angle of the posture and the angle of the posture of the moving body is used as the weight, and a weighted average value is weighted to the reliability and used as the representative value.

[0048] Furthermore, although the posture of the moving body is discretized to obtain the reliability for each posture, a method that does not require discretization may be used as long as the reliability for each posture can be obtained. For example, a method may be used in which a value that is closest to the current position and posture of the moving body is used from all the positions and postures and reliabilities stored in the storage unit 112. Also, a method may be used in which the reliability for each posture is obtained by function approximation such as spline approximation.

[0049] <Modification 1-3> In the first embodiment, the determination unit 114 determines the posture based on the reliability map so that the reliability is equal to or greater than a predetermined value when controlling the movement of the moving body along the travel path. However, the method of determining the posture is not limited to this, and any method may be used as long as the posture has a reliability equal to or greater than a predetermined value. For example, the posture may be determined not only based on the reliability of the position and posture at a single point in time, but also based on the total amount of reliability at multiple points in time. A specific method is to turn the moving body 180 degrees by turning right if the reliability map for each posture of the moving body shows that turning right is better than turning left while maintaining a higher reliability.

[0050] <Variation 1-4> In the first embodiment, the control unit 103 controls the movement of the moving body so that the moving body assumes the attitude determined by the determination unit 114, but this is not limited thereto, and the controlled object may be anything as long as the attitude of the sensor becomes the desired attitude. For example, the attitude of the moving body may be left unchanged, and only the sensor mounted on the moving body may be rotated using an electric pan head or the like. Alternatively, the rotation of the moving body and the rotation of the sensor may be used in combination.

[0051] <Variation 1-5> In the first embodiment, the mobile system 100 is applied to a wheeled robot that moves on a two-dimensional plane. However, the present invention is not limited to this, and may be applied to a flying robot that moves in a three-dimensional space. In this case, the three position and posture parameters of the first embodiment are expanded to a total of six parameters, including three position parameters (X, Y, Z) and three posture parameters (Roll, Pitch, Yaw), and the creation unit 113 creates a reliability map by discretizing the three posture parameters for each position (x, y, z).

[0052] <Variation 1-6> In the first embodiment, the posture determined by the determination unit 114 is presented using a GUI, but the presenting method is not limited to this, and any presentation method may be used as long as the determined posture can be presented. For example, if the moving object is equipped with directional indicator lights at every 45 degrees, the determined posture can be visualized by turning on the lights corresponding to the determined posture by the determination unit 114. Any device may be used, such as an LED lamp or a liquid crystal display, as long as it can present information about the determined posture to the user. The presentation device may be a speaker, and may be configured to play a specific alarm sound or a specific melody according to the determined posture. The information to be presented may be information other than the determined posture, and may be, for example, a reliability map created by the creation unit 113.

[0053] [Embodiment 2] In the first embodiment, it has been shown that a reliability map for each posture of a single moving body is created, and the posture that the moving body should take next is determined based on the reliability map, thereby enabling the moving body to travel stably. In this embodiment, a case where a plurality of moving bodies are traveling simultaneously and the mounting positions of cameras mounted on each model of moving body are different is taken as an example. A method of creating a reliability map for each camera mounting position, selecting a reliability map where the camera mounting position is within a predetermined distance, and determining the posture of the moving body based on the reliability map is described. In this embodiment, it is assumed that only the vertical component of the camera mounting position differs for each moving body.

[0054] A configuration diagram of this embodiment is omitted because it is the same as FIG. 1 which illustrates the configuration of the information processing device 110 described in the first embodiment. This embodiment differs from the first embodiment in that an extraction unit is present and in the processing of the creation unit and the determination unit. Here, only the parts that differ from the first embodiment will be described.

[0055] The extraction unit classifies and extracts vertical components of positions into predetermined ranges from the positions and orientations and reliability levels stored in the storage unit 112. The extraction unit outputs the extracted position and orientation information to the creation unit 113.

[0056] The creation unit 113 creates a reliability map for each camera mounting position based on the position and orientation information extracted by the extraction unit, and outputs the reliability map to the determination unit 114.

[0057] The determination unit 114 selects the reliability map closest to the camera mounting position of the moving object from the reliability maps for the camera mounting positions created by the creation unit 113, and determines the posture of the moving object based on the reliability map.

[0058] Fig. 7 is a diagram showing a flowchart of the processing procedure in this embodiment. Explanations of steps that are the same as those in Fig. 3 that explain the processing procedure of the information processing device 110 described in the first embodiment will be omitted, and only the processing procedure that differs from the first embodiment will be explained.

[0059] In step S201, the extraction unit extracts combinations of position and orientation and reliability stored in the storage unit 112 by classifying and extracting the position and orientation so that the vertical component of the position falls within a predetermined range. The storage unit 112 stores all past position and orientation information of multiple moving objects with various different camera mounting positions. If the camera mounting positions differ greatly, the captured images also differ greatly, and the number and distribution of feature points that can be extracted from the captured images change, which may result in completely different reliability. Therefore, by extracting only position and orientation information with close camera mounting positions by this extraction, the position and orientation information to be input to the reliability map is limited to only position and orientation information including appropriate reliability.

[0060] In step S103, a reliability map for each posture of the moving body is created for each camera mounting position based on the position, posture, and reliability extracted by the extraction unit.

[0061] In step S104, the determination unit 114 selects the reliability map closest to the camera mounting position of the moving body whose posture is to be determined from the reliability maps for each camera mounting position created by the creation unit 113, and determines the posture of the moving body based on that reliability map.

[0062] In the second embodiment, when multiple moving bodies are operating simultaneously and the mounting positions of the cameras mounted on each moving body model are different, the moving bodies can be operated stably even in an environment in which the accuracy of position and orientation measurement decreases.

[0063] <Variation 2-1> In the second embodiment, the extraction unit extracts the position and orientation information by comparing the vertical component of the position held by the holding unit 112 with the mounting position of the camera. However, this is not limited to this, and any extraction method may be used as long as it can classify the mounting position of the camera and extract the position and orientation information. For example, a method of extracting the information by comparing with the mounting angle of the camera may be used.

[0064] <Other embodiments> The present invention can also be realized by executing the following process. That is, software (programs) that realize the functions of the above-described embodiments are supplied to a system or device via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or device reads and executes the programs. The programs may also be provided by recording them on computer-readable storage media. [Explanation of symbols]

[0065] 100 Mobile Systems 101 Sensors 102 Position and orientation estimation unit 103 Control section 104 Presentation section 110 Information Processing Systems 111 Position and orientation information acquisition unit 112 Holding part 113 Creation Department 114 Decision Section

Claims

1. An information processing device, a position and orientation information acquisition means for acquiring information regarding a position and orientation of a moving object and a reliability of the position and orientation estimation based on a measurement value of a sensor mounted on the moving object; a generating means for generating a reliability map having the reliability for each orientation of the sensor based on the position and orientation and the reliability held by a holding means; a determination means for determining an attitude of a moving body on which the sensor is mounted so that a reliability becomes equal to or greater than a predetermined value when controlling the movement of the moving body, based on the reliability map created by the creation means; An information processing device comprising:

2. The reliability map is a map having reliability obtained by discretizing the posture of a moving object with respect to a position in space.

2. The information processing apparatus according to claim 1,

3. further comprising an extraction means for extracting only information that is within a predetermined range when a vertical component of a position is compared with a mounting position of a sensor used for measuring a position by the acquisition means, from the information on the position, orientation and reliability held by the holding means; the creation means creates the reliability map based on the position and orientation and the reliability extracted by the extraction means; The determining means determines the posture of the moving body so as to have a high reliability based on the reliability map in which the vertical direction component is limited, which is created by the creating means.

3. The information processing device according to claim 1, wherein:

4. The display means further includes a display unit, the display unit outputting the attitude of the moving body determined by the determination unit to a display device.

4. The information processing device according to claim 1, wherein:

5. The system further includes a display unit, and the display unit outputs the reliability map created by the creation unit to a display device.

4. The information processing device according to claim 1, wherein:

6. The vehicle further includes a control means for controlling the movement of the moving body, The control means controls the movement of the moving body so that the moving body assumes the attitude determined by the determination means.

6. The information processing device according to claim 1,

7. 1. An information processing method, comprising: a position and orientation information acquisition step of acquiring information regarding a position and orientation of the moving body and a reliability of the position and orientation estimation based on a measurement value of a sensor mounted on the moving body; a generating step of generating a reliability map having the reliability for each orientation of the sensor based on the position and orientation and the reliability held by a holding means; a determination step of determining an attitude of a moving body on which the sensor is mounted so as to increase the reliability based on the reliability map created in the creation step; An information processing method comprising:

Images