Information processing device, information processing method, and program

Abstract

To detect deviation of a position and posture of a sensor with respect to a moving body from a fixed position and posture.SOLUTION: When a moving body 1 moves in a predetermined manner with a position and a posture of a sensor 12 relative to the moving body being maintained, based on an expected positional information of the moving body obtained based on input information input from the sensor and based on positional information of the moving body estimated based on the input information input from the sensor after the moving body has moved in the predetermined manner, there is output information indicating how the position and posture of the sensor relative to the moving body after the moving body has moved in the predetermined manner are changed from the held position and posture.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technique for estimating the position of a moving object from input information of a sensor. [Background technology]

[0002] There is a technology for estimating the position of a moving object equipped with a sensor that measures the distance between the sensor and an object in the environment, based on measurement information obtained from the sensor. In this case, the positional relationship between the moving object and the sensor is measured (calibrated) in advance, so that the position of the moving object can be estimated with higher accuracy. In Patent Document 1, the sensor measures a reference target when the robot is moved a predetermined amount so that the robot and the reference target have a predetermined relative positional relationship, and this is repeated until the measurement result falls within the error range. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2009-279677 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the method of Patent Document 1 does not take into consideration the possibility that the relative positional relationship between the measurement device (sensor) and the robot (moving body) may change to an inappropriate positional relationship due to some unpredictable cause, making it difficult to detect the occurrence of such an event. Furthermore, if such an event occurs, it will ultimately be impossible to accurately estimate the position of the moving body. The present invention has been made in consideration of the above problems, and aims to make it possible to detect the deviation of the position and orientation of the sensor relative to the moving body from the calibrated position and orientation. [Means for solving the problem]

[0005] The information processing device according to the present invention for solving the above-mentioned problems comprises the following configuration: That is, an estimation means for estimating first position information indicating a position of a moving body equipped with a sensor based on input information input from the sensor when the moving body makes a predetermined movement, an acquisition means for acquiring second position information indicating a position of the moving body that is expected to be obtained based on the input information input from the sensor when the moving body makes the predetermined movement while maintaining the position and orientation of the sensor with respect to the moving body held by a holding means, and an acquisition means for acquiring second position information indicating a position of the moving body that has made the predetermined movement based on the first position information and the second position information. case and an output unit that outputs information relating to changes in the position and orientation of the sensor relative to the moving body and the position and orientation of the sensor relative to the moving body held by the holding unit. Effect of the Invention

[0006] According to the present invention, it is possible to detect that the position and orientation of the sensor relative to the moving object has shifted from the held position and orientation. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram showing an example of a system configuration of an information processing system. [Diagram 2] FIG. 1 is a block diagram showing an example of a functional configuration of an information processing device; [Diagram 3] FIG. 1 is a diagram showing an example of a hardware configuration of an information processing device. [Figure 4] A flowchart for explaining a process executed by an information processing device. [Diagram 5] A flowchart for explaining a process executed by a change amount acquisition unit. [Figure 6] A diagram explaining a method for acquiring the amount of change in the attitude parameter of a moving object. [Figure 7] A flowchart for explaining a process executed by a change amount acquisition unit. [Figure 8] A diagram explaining a method for acquiring the amount of change in the position parameter of a moving object. [Figure 9]A flowchart for explaining a process executed by an information processing device. [Figure 10] A diagram showing an example of a GUI [Figure 11] FIG. 1 is a block diagram showing an example of a functional configuration of an information processing device; [Figure 12] A flowchart for explaining a process executed by an information processing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] <Embodiment 1> In this embodiment, the present invention is applied to an information processing system including a mobile body equipped with a sensor for measuring the distance between an object in the environment and the sensor. In this embodiment, the mobile body is specifically an automatic guided vehicle. When the mobile body plans a route using a position and orientation estimated based on the measurement value of the sensor and travels along the plan, it is necessary to convert the measurement results from the sensor into a format used in the movement control of the mobile body. This conversion requires a relative positional relationship between the mobile body and the sensor. Therefore, the positional relationship between the mobile body and the sensor is measured (calibrated) in advance before the mobile body starts traveling. The relative positional relationship between the calibrated sensor and the mobile body is defined as an appropriate positional relationship. However, while the mobile body travels, the sensor may move from the position or orientation of the calibrated state. In other words, the relative positional relationship between the sensor and the mobile body may become inappropriate due to the movement of the mobile body. If there is a change in the actual physical relative positional relationship, the mobile body travels based on the position estimation result of the mobile body that includes the change in the position and orientation of the sensor as noise, so that the mobile body may deviate from the predetermined route. Therefore, in this embodiment, a change in the positional relationship between the sensor and the mobile body when the mobile body is moved straight ahead a predetermined distance is detected. This allows the system to detect a change in the relative positional relationship between the sensor and the moving object, and if there is a change, notify the user that the relative positional relationship between the sensor and the moving object is inappropriate. In addition, a method for allowing the user to correct the positional relationship between the sensor and the moving object will be described. In this embodiment, the sensor refers to a stereo camera (image sensor).

[0009] 1 shows an example of the configuration of an information processing system 1 in this embodiment. The information processing system 1 is composed of a mobile object 10 that travels along a predetermined route, a sensor 12 that measures the shape of an object present in the environment, an information processing device 11 that estimates the position and orientation of the sensor, a control device 13 that controls the mobile object, and an output device that outputs information regarding deviations in positional relationship. Note that in this embodiment, the mobile object moves on a horizontal plane, and is, for example, an unmanned guided vehicle that transports luggage.

[0010] In this embodiment, the movement control refers to controlling a motor, which is an actuator serving as the control device 13 of the moving body, and a steering wheel that changes the direction of the wheels. Specifically, the control device 13 causes the moving body to move straight forward, backward, left and right, and to turn at a predetermined angle. By inputting control information into the control device 13, the moving body is moved along a predetermined path. The movement control information is information necessary for controlling the control device 13, and refers to the rotation torque and traveling direction of the wheels. By setting the number of rotations of the wheels and the traveling direction in advance as the movement control information for a predetermined operation, the moving body can be made to move along a certain trajectory regardless of the position and orientation of the moving body or the sensor.

[0011] In this embodiment, the position of the sensor 12 is defined by three parameters (X, Y, Z) that represent the position of the sensor 12 in an arbitrary world coordinate system defined in real space. Moreover, the orientation of the sensor 12 is defined by three parameters (Roll, Pitch, Yaw) that represent the orientation of the sensor 12. These six parameters combined indicate the position and orientation of the sensor 12. Moreover, a three-dimensional coordinate system defined on the sensor with the optical axis of the sensor 12 as the Z axis, the horizontal direction of the image as the X axis, and the vertical direction as the Y axis is called the sensor coordinate system.

[0012] Meanwhile, the position of a moving body in this embodiment is defined by two position parameters (X and Y positions on a plane horizontal to the floor of the real space) of the moving body on an arbitrary world coordinate system defined in the real space. The attitude of the moving body is defined by and one attitude parameter (direction of rotation on a plane horizontal to the floor of the real space). A total of three parameters, combining these, define the position and attitude of the moving body. A two-dimensional coordinate system defined on the moving body, with the Y axis representing the moving direction of the moving body and the X axis representing a direction perpendicular to the moving direction, is called a moving body coordinate system. It is assumed that the moving body in this embodiment moves on a horizontal plane.

[0013] Furthermore, the relative positional relationship between the sensor 12 and the moving body in this embodiment refers to the relative position and orientation between the origin of the sensor coordinate system and the origin of the moving body coordinate system, and is expressed by a total of six parameters, including three parameters representing the position and three parameters representing the orientation. By using this relative position and orientation, the position and orientation are mutually converted between the sensor coordinate system and the moving body coordinate system. It is assumed that the positional relationship between the sensor and the moving body has been calibrated in advance using a known technique. In the following process, if the orientation (position) of the sensor has changed from the initial state due to vibration of the moving body or some other external factor, it is detected that the positional relationship has changed from the positional relationship known in advance by calibration.

[0014] 2 is a diagram showing an example of a functional configuration of the information processing device 11 in this embodiment. The information processing device 11 is composed of a movement control unit 111, a position and orientation estimation unit 112, a position and orientation conversion unit 113, a change amount acquisition unit 114, a calibration necessity determination unit 115, a relative position and orientation calibration unit 116, and an output unit 117. The position and orientation estimation unit 112 is connected to the sensor 12. The movement control unit 111 is connected to the control device 13. The information processing device 11 is also connected to the control device 13 that controls the movement of the moving body, and an output device 14 that outputs information on the relative positional relationship between the sensor and the moving body.

[0015] The movement control unit 111 generates movement control information for controlling the movement of the moving object, and outputs the information to the control device 13 and the change amount acquisition unit 114 .

[0016] The position and orientation estimation unit 112 estimates the position and orientation of the sensor based on the measurement value obtained from the sensor 12, and outputs the position and orientation of the sensor to the position and orientation conversion unit 113. The sensor 12 is, for example, a stereo camera, and acquires image information as input information. Hereinafter, the input information from the sensor 12 will be described as measurement information or measurement value for convenience. The position and orientation conversion unit 113 converts the position and orientation of the sensor input by the position and orientation estimation unit 112 from the sensor coordinate system to the moving body coordinate system, and outputs it to the change amount acquisition unit 114. The change amount acquisition unit 114 outputs the change amount of the relative positional relationship between the sensor 12 and the moving body based on the conversion result input by the position and orientation conversion unit 113 and the displacement of the position and orientation of the moving body when a predetermined operation that is known in advance is performed. The acquired change amount is output to the calibration necessity determination unit 115 or the output unit 117. The calibration necessity determination unit 115 determines whether calibration is necessary using the change amount of the relative position and orientation as an input. The determination result is output to the relative position and orientation calibration unit 116. The relative position and orientation calibration unit 116 calibrates the relative position and orientation of the sensor 12 and the moving body based on the result of the determination of whether or not calibration is required. The output unit 117 outputs output information indicating whether the positional relationship between the sensor 12 and the moving body after a predetermined operation is maintained based on the amount of change in the relative positional relationship between the sensor 12 and the moving body. That is, the output unit 117 outputs output information according to the difference between the position of the moving body obtained from the position of the moving body estimated by the position and orientation conversion unit 113 (first position information) and the predicted position of the moving body after a predetermined operation associated with the predetermined operation (second position information) to the output device 14. The second position information is the position of the moving body that is expected to be obtained when the moving body performs a predetermined operation while the relative positional relationship between the sensor and the moving body is maintained. This makes it possible to detect that the relative positional relationship between the sensor and the moving body measured in advance has shifted before the start of running or during running, and to notify the user of this.

[0017] FIG. 3 is a diagram showing a hardware configuration of the information processing device 11. H11 is a CPU, which controls various devices connected to the system bus H20. H12 is a ROM, which stores a BIOS program and a boot program. H13 is a RAM, which is used as a main storage device of the CPU H11. H14 is an external memory, which stores a program to be processed by the information processing device 11. The input unit H15 is a keyboard, a mouse, or a robot controller, which performs processing related to input of information, etc. The display unit H16 outputs the calculation result of the information processing device 11 to a display device according to an instruction from H11. The display device may be a liquid crystal display device, a projector, an LED indicator, or any other type. H17 is a communication interface, which performs information communication via a network, and the communication interface may be Ethernet (registered trademark), or any type such as USB, serial communication, or wireless communication. H17 is an I / O, which inputs an image from a sensor H18. The sensor H18 is the sensor 12 described above. H19 is the control device 13 described above.

[0018] Fig. 4 is a flowchart showing the processing procedure in this embodiment. The processing includes initialization S101, movement control S102, position and orientation estimation S103, position and orientation conversion S104, change amount acquisition S105, calibration necessity determination S106 and S107, relative position and orientation calibration S108, and system termination determination S109. The processing shown in the flowchart in Fig. 4 is executed by the CPU H11 in Fig. 3, which is a computer, in accordance with a computer program stored in the external memory H14. In the following description, each process (step) is represented by adding an S to the beginning, and the description of the process (step) is omitted.

[0019] In S101, the system is initialized. That is, a program is read from the external memory H14, and the information processing device 11 is put into an operable state. In addition, parameters of each device connected to the information processing device 11 and the relative position and orientation of the sensor 12 and the moving object are read into the RAM H13. In addition, each control device of the moving object is started up, and put into an operable and controllable state.

[0020] In S102, the movement control unit 111 causes the moving object to perform a predetermined action based on control information for the moving object to perform the predetermined action. Here, the predetermined action is an action of moving straight a predetermined distance. Control information such as how many meters to move forward may be acquired or generated. Note that in this embodiment, control is performed to fix the moving direction to forward (or backward) so that the moving object moves straight. This makes it possible to detect changes in the attitude (direction) of the sensor without human intervention.

[0021] In S103, the position and orientation estimation unit 112 estimates the position and orientation of the sensor based on the measurement information measured by the sensor 12 at a second time point after the predetermined operation. When the sensor is an image sensor, the measurement information is image information. Here, the trajectory of the position and orientation of the sensor is estimated based on the measurement information measured by the sensor 12 from a first time point when the predetermined operation is started to a second time point after the predetermined operation. The estimated position and orientation of the sensor is called first position information. In this embodiment, a SLAM (Simultaneous Localization and Mapping) technique is used. A specific example of SLAM processing will be described. SLAM is a method of using feature points detected on an image to estimate the position and orientation of a sensor or the like. The information processing device holds, for example, three-dimensional point cloud information indicating structural features of walls and stationary objects in a scene in which a moving body carrying a sensor runs, as map information of the scene. Specifically, the information of the three-dimensional point cloud in the scene is the three-dimensional coordinates of the feature points in the world coordinate system. In this embodiment, map information of a scene is generated in advance based on images of the shapes of objects existing in the environment captured while a mobile body is traveling, and is stored in a storage device. Based on the input image and the map information, the information processing device minimizes the difference (reprojection error) between the positions of feature points (edge ​​features indicating the surrounding shape) on the input image and the positions when the feature points on the map are projected onto the input image using the position and orientation of the sensor when the input image was captured. This estimates the position and orientation of the sensor. Any other method may be used as long as it is a method of estimating the position and orientation of the sensor from measurement information.

[0022] Here, we use the method of Engel et al. (J. Engel, T. Schps, and D. Cremers. LSD-SLAM: Large-Scale Direct Monocular SLAM. ECCV, 2014). This method simultaneously performs self-location estimation and map creation in a wide-area environment such as outdoors. When a moving object is in a stationary state before starting to move, the position and orientation estimation unit 112 outputs the origin of the sensor coordinate system as the position and orientation of the sensor. Also, while the moving object is performing a predetermined operation, the position and orientation of the sensor is continuously measured at regular time intervals or distance intervals, and all the measurement information is output to the position and orientation conversion unit 113. Note that the measurement information is array data of six position and orientation parameters.

[0023] In S104, the position and orientation conversion unit 113 converts the trajectory of the position and orientation of the sensor 12 input by the position and orientation estimation unit 112 from the sensor coordinate system to the moving body coordinate system. That is, the estimated change (trajectory) of the position and orientation of the sensor is converted to the position and orientation of the moving body based on the known positional relationship. Specifically, first, the position and orientation of the sensor 12 is rotated or translated based on the relative position and orientation of the sensor 12 and the moving body stored in a storage unit such as a RAM, thereby matching each axis (X, Y, Z) of the coordinate system for three position parameters and three orientation parameters with the moving body coordinate system. Next, since the moving body coordinate system has three degrees of freedom, the six position and orientation parameters of the sensor 12 are reduced to three parameters. Specifically, data that cannot exist in the moving body coordinate system is deleted from the array data of the position information. For the position parameters, data of the Z axis, which is the vertical direction, is deleted. On the other hand, for the orientation parameters, data of Roll (X axis) and Pitch (Y axis) other than Yaw (Z axis), which is the rotation direction on the horizontal plane, is deleted. The multiple pieces of position information of the sensor 12 converted into the moving body coordinate system are output to the change amount acquisition unit 114. When the moving body is moved in a straight line, the position of the moving body at a first time point when a predetermined motion is started and the position of the moving body at a second time point after the predetermined motion may be acquired based on the positional relationship between the sensor and the moving body. This has the effect of reducing the processing load since it can be estimated by a simpler calculation than estimating the trajectory of the moving body. When the trajectory of the moving body is used, it has the effect of being able to estimate the amount of change in the attitude of the sensor with higher accuracy.

[0024] In S105, the change amount acquisition unit 114 acquires the change amount between the relative position and orientation held in the RAM and the physical relative positional relationship between the actual moving body and the sensor, using multiple pieces of position information of the sensor 12 input by the position and orientation conversion unit 113. That is, the position and orientation conversion unit 113 acquires the result of converting the position (first position information) of the moving body or the sensor after a predetermined operation from the measurement information measured by the sensor when the moving body executes a predetermined operation. Then, the predicted position (second position information) of the moving body or the sensor when the predetermined operation is performed in a preset positional relationship is compared with the position of the moving body or the sensor after the predetermined operation. If the difference between the position of the moving body or the sensor from the first position information and the second position information is within a predetermined threshold, it can be said that the relative positional relationship between the sensor and the moving body has not changed from a preset value. When the moving body moves straight as in this embodiment, the change amount of the orientation parameter in the relative position and orientation is obtained. The details of the process will be described below.

[0025] FIG. 5 is a flowchart for explaining the process executed by the change amount acquisition unit 114. In S201, the change amount acquisition unit 114 judges whether the trajectory represented by the position information input by the position and orientation conversion unit 113 is sufficient in number for estimating the change amount of the sensor's orientation. For example, the Euclidean distance between the position parameters (X, Y) of the latest position and orientation and the origin (0, 0) is obtained from the position information, and if the distance is equal to or greater than a predetermined threshold, it is determined that the number of data of the position and orientation is sufficient, and the process proceeds to S202. If not, the change amount acquisition process is terminated without executing the change amount acquisition. The more position information there is, the more accurately the change amount of the sensor's orientation can be estimated. Note that this step may be skipped when the change amount of the sensor's orientation is obtained only from the positions of the moving body at the first time point when a predetermined operation is started and the second time point when the predetermined operation is ended.

[0026] Here, the subsequent processing will be explained with reference to FIG. 6. FIG. 6(a) is a diagram in which the position information of the sensor 12 in the moving body coordinate system is mapped onto a two-dimensional plane. The moving body 10 equipped with the sensor 12 moves linearly for a predetermined distance in the illustrated direction. D101 is the origin of the moving body coordinate system and is also the initial position of the sensor 12. The black circle in D102 is a plot of each position parameter included in the sensor position information. The trajectory is shown by plotting the positions that the moving body has passed through between a first time point when the moving body starts a predetermined operation and a second time point after the moving body has performed the predetermined operation in the moving body coordinate system.

[0027] In S202, the change amount acquisition unit 114 estimates first position information indicating the position of the moving body after a predetermined movement from measurement information measured when the moving body performs a predetermined movement based on the positional relationship between the sensor and the moving body. The first position information may be the trajectory of the sensor, and an approximate straight line is acquired by inputting all position parameters included in the position information of the sensor 12. In FIG. 6, D103 is the approximate straight line. When the trajectory of the moving body is used, the change in the attitude of the sensor shown in D104 can be detected more accurately. Note that, in order to maintain the detection accuracy of the change in the attitude of the sensor, points whose positions could not be accurately estimated or measured may not be included in the approximate straight line of D1003. For example, when the reliability of the acquired position information of the sensor or the moving body is lower than a predetermined value, it is suppressed so as not to be used in estimating the approximate straight line. Note that, when the change in attitude is detected from the position of the sensor at the first time point when the predetermined movement is started and the position of the sensor after the predetermined movement, this step may be skipped. In this case, there is an effect of reducing the processing load because the amount of information to be handled is reduced.

[0028] In S203, the change amount acquisition unit 114 outputs the difference in the position of the sensor from the first position information and the second position information indicating the predicted position of the moving body (sensor) associated with the predetermined operation. That is, the change amount of the attitude of the sensor is acquired based on the position (trajectory) of the moving body (sensor) before and after the predetermined operation. Specifically, the angle D104 formed by the approximation line D103 and the Y axis in FIG. 6 is acquired. In the moving body coordinate system, since the Y axis direction is the moving body's moving direction, when the moving body moves straight, the X coordinate of the position parameter should always be 0 (predicted position). Therefore, when the approximation line of the position information deviates from the Y axis, it is found that the position and attitude conversion unit 113 has performed an incorrect conversion in the coordinate system conversion of the position information (sensor coordinate system → moving body coordinate system). This incorrect conversion is caused by a change from the relative positional relationship between the sensor and the moving body held in the RAM, that is, a change in the attitude parameter of the sensor. The change amount of the attitude parameter is the angle formed by the approximation line of the position information and the Y axis. The angle between these two lines is calculated using trigonometric functions. Then, the process of S105 ends.

[0029] The amount of change in the attitude of the sensor acquired as described above is output to the calibration necessity determination unit 115. In S106, the calibration necessity determination unit 115 acquires the amount of change from the change amount acquisition unit 114. If the acquired amount of change is equal to or greater than a predetermined threshold, it is determined that calibration of the relative position and attitude is necessary, and if not, calibration is not required. In S107, if the calibration necessity determination unit 115 determines that calibration is necessary, the process proceeds to S108, and if not, the process skips to S109. The determination result here is input to the output unit 117, and output information is presented to the user through a predetermined output device 14. For example, if it is determined that calibration is necessary, output information is output to inform the user that the mounting position and attitude of the sensor have shifted. The output device 14 and the output information will be described later.

[0030] In S108, the relative position and orientation calibration unit 116 corrects the relative position and orientation based on the amount of change in the relative position and orientation acquired by the change amount acquisition unit 114 so that it matches the physical relative positional relationship between the actual moving body and the sensor. That is, the amount of change in the orientation of the sensor is estimated from the first position information, and the relative positional relationship between the sensor and the moving body is updated according to the amount of change. Specifically, the angle of the amount of change is subtracted from the parameter that is Yaw (Z axis) in the moving body coordinates among the orientation parameters of the relative position and orientation. The relative position and orientation after calibration is overwritten and held in H13, which is a RAM.

[0031] In S109, it is determined whether or not to terminate the system. Specifically, if the amount of change acquired by the change amount acquisition unit 114 is equal to or less than the threshold value, or if an end instruction from the user is received by the input unit H15, the calibration of the relative position and orientation is considered to be completed and the system is terminated. If not, the system returns to S101 and continues the process.

[0032] In the first embodiment, when a moving object moves straight, the position and orientation are measured based on an image acquired from a sensor, and the amount of change in the orientation parameters of the relative position and orientation of the sensor and the moving object is acquired based on position information consisting of the positions and orientations at multiple points in time. Furthermore, the orientation parameters are corrected from the acquired amount of change. This makes it possible to correct the relative position and orientation of the sensor and the moving object without using a highly accurate amount of movement of the moving object, and makes it possible to easily correct changes in the relative positional relationship between the moving object and the position and orientation estimation device.

[0033] (Variation 1-1) In the first embodiment, the position and orientation estimation unit 112 measures the position and orientation by the SLAM technique using a camera. However, the sensor combined with the SLAM technique is not limited to a camera, and any sensor may be used. The position and orientation may be measured by combining an active distance sensor such as LiDAR, Time of Flight, or pattern light projection with the SLAM technique.

[0034] Furthermore, the position and orientation estimation unit 112 is not limited to SLAM technology, and may be any technology capable of measuring the position and orientation. For example, it may be a method based on measuring radio wave intensity in wireless communication such as Wifi. It may be a method using a magnetic tracking sensor or a time difference between detections by multiple optical sensors. It may also be a method of performing model fitting based on CAD data using an image acquired by a camera installed at a specific location such as a surveillance camera.

[0035] (Variation 1-2) In the first embodiment, the position and orientation conversion unit 113 converts the position and orientation of the sensor 12 from the sensor coordinate system to the moving body coordinate system and outputs the converted data to the change amount acquisition unit 114. However, any coordinate system may be used as long as the coordinate system can be unified to a single coordinate system when the change amount acquisition unit 114 acquires the amount of change in the relative position and orientation. For example, when unifying to the sensor coordinate system, the following processing may be performed. The position and orientation conversion unit 113 converts the position and orientation of the moving body estimated based on the movement control information acquired from the movement control unit 111 from the moving body coordinate system to the sensor coordinate system and outputs the converted data to the change amount acquisition unit 114. On the other hand, the position and orientation of the sensor obtained from the position and orientation estimation unit 112 may be output to the change amount acquisition unit 114 in the sensor coordinate system, and the change amount acquisition unit 114 may acquire the amount of change based on both the position and orientation of the sensor and the moving body in the sensor coordinate system.

[0036] (Variation 1-3) In the first embodiment, the information processing system 10 is not limited to an unmanned guided vehicle (mobile body). For example, the information processing system 10 may be a self-driving car or an autonomous mobile robot, and the movement control described in the present embodiment may be applied to them. It may also be applied to an unmanned aerial vehicle 10' as shown in FIG. 8(b). Specifically, it is a drone. In the case of an aerial vehicle, for example, the inclination of the sensor 12 can be detected by moving up and down a predetermined distance in the direction of gravity (Z axis). Alternatively, it may be applied to a robot arm equipped with a sensor on the arm.

[0037] (Variation 1-4) The output unit 117 outputs output information indicating whether the positional relationship after the predetermined operation is the maintained positional relationship based on the trajectory (first position information) of the moving body estimated after the predetermined operation and the predicted trajectory (second position information) of the moving body after the predetermined operation. For example, when the change amount of the sensor or the position and orientation of the moving body acquired by the change amount acquisition unit 114 is greater than a predetermined threshold, output information indicating whether the positional relationship between the sensor and the moving body is the maintained positional relationship is output to the output device. For example, if three-color lights of red, yellow, and green are used for the display unit, red is displayed when the change amount of the relative position and orientation is equal to or greater than a predetermined threshold, and green is displayed when the change amount is less than the threshold. This makes it possible to visualize the change amount acquisition result to the user. The threshold may be set in stages, and the notification method may be changed depending on which threshold range the change amount falls within. For example, a first threshold at a stage where calibration is definitely required and a second threshold for notifying that the position estimation has not yet been affected but that the initial position and orientation has changed are set in advance. For example, when the acquired change in the position and orientation of the sensor is equal to or greater than a first threshold, the light turns red, when the change is less than the first threshold and equal to or greater than a second threshold, the light turns yellow, and when the change is equal to or less than the second threshold, the light turns green. That is, when the difference in the position of the moving object indicated by the first measurement information and the second measurement information is greater than the first threshold, the output information is output to the first output device. When the difference in the position of the moving object indicated by the first measurement information and the second measurement information is greater than a second threshold that is smaller than the first threshold, the output information is output to the second output device. Also, as long as it can present information on the change amount acquisition result to the user, an output device such as an LED lamp or a liquid crystal display may be used. The output device may be an audio output device such as a speaker, and may be configured to play specific audio information (such as an alarm sound or a melody) according to the map update status. Also, the output device may be a projection device. For example, when it is detected that the relative positional relationship between the sensor and the moving object is inappropriate, the correct mounting position and orientation of the sensor may be projected by a projection device or a display device. Specifically, by projecting the appropriate position and orientation of the sensor using a projector, the attachment position and orientation of the sensor can be presented to the user in an easy-to-understand manner.

[0038] (Variation 1-5) The mobile object may be set to automatically perform a predetermined operation at a predetermined timing. For example, when the information processing system 1 is started or ended, the mobile object may be made to move straight ahead a predetermined distance and immediately return as a predetermined operation. Since a change in the positional relationship between the mobile object and the sensor can be detected before the mobile object starts autonomous traveling using the sensor, the mobile object can be controlled more safely. If the attitude of the sensor has changed, the sensor will pass through the trajectory of D103' twice as shown in FIG. 6(b). If the attitude of the sensor has not changed, the sensor will move forward a predetermined distance on the Y axis and return. Since the sensor moves along the same trajectory twice, the amount of measurement information increases, and the amount of change in the sensor's attitude can be obtained more accurately. In addition, the mobile object may be set to go to a predetermined location (for example, a charging spot or a waiting space) when the mobile object is started or ended. In such a case, the amount of change in the sensor's attitude may be obtained when the mobile object approaches the predetermined location (when the target point and the position of the mobile object are within a predetermined range). By checking the relative positional relationship between the sensor and the mobile object at a predetermined timing in this way, the position of the mobile object can be measured more accurately and more stably. In addition, by setting a trigger at a timing that allows the user to immediately check the detection results, the above processing can be executed at any timing, making it easy to detect whether the relative positional relationship between the sensor and the moving object is inappropriate.

[0039] <Embodiment 2> In the first embodiment, when the movement trajectory of the moving object becomes a straight line, the amount of change in the relative attitude parameter between the sensor and the moving object is acquired based on the position information of the sensor, and the relative position and attitude between the moving object and the sensor is corrected based on the acquired result. According to this embodiment, the relative position parameter between the sensor and the moving object can also be corrected. Therefore, in the second embodiment, the amount of change in the relative position parameter between the sensor and the moving object is acquired based on the position information of the sensor when the moving object turns, and the relative position and attitude is corrected using the acquired result.

[0040] The configuration diagram of this embodiment is the same as FIG. 2 which illustrates the configuration of the information processing device 11 described in the first embodiment, and therefore the description will be omitted. The hardware configuration may also be the same as that of the first embodiment, as shown in FIG. 3. The diagram of the processing procedure of this embodiment is the same as FIG. 4 which illustrates the processing procedure of the information processing device 11 described in the first embodiment, and therefore the description will be omitted. Only the processing procedures which differ from the first embodiment will be described, and the other procedures will be omitted, as they are the same processes as those in FIG. 4.

[0041] In S102, the movement control unit 111 controls the movement of the moving body by outputting movement control information for rotating the moving body by a predetermined rotation amount to the control device 13. Note that in this embodiment, the movement of the moving body is controlled so that the position parameters do not move from the origin, which is the initial position, in the moving body coordinate system, and only the attitude parameters are changed to perform a rotational movement.

[0042] In S105, the change amount acquisition unit 114 acquires the amount of change between the relative position and orientation held in the RAM and the actual physical relative positional relationship between the moving body and the sensor, using the position information of the sensor 12 input by the position and orientation conversion unit 113. In the case where the moving body has undergone a rotational motion in which only the orientation changes as in this embodiment, the amount of change in the position parameter of the relative position and orientation is obtained.

[0043] 7 is a flowchart showing a detailed flow of the change amount acquisition process. In S301, the change amount acquisition unit 114 judges whether the number of positions and orientations in the position information input by the position and orientation conversion unit 113 is sufficient for acquiring the amount of change. Since the position and orientation estimation unit 112 measures the position and orientation at a fixed time interval, the number of pieces of position and orientation data are proportional to the movement time. If the movement time is equal to or longer than the fixed time, it is determined that the number of pieces of data is sufficient, and the process proceeds to S302. If not, the change amount acquisition process is terminated without executing the change amount acquisition.

[0044] Here, the following process will be explained with reference to Fig. 8(a). Fig. 8(a) is a diagram in which the position information of the sensor 12 in the moving body coordinate system is mapped onto a two-dimensional plane. D201 is the origin of the moving body coordinate system, and is also the initial position of the sensor 12. The black circles in D202 are plots of each position parameter included in the position information. The moving body 10 performs an operation of turning in the traveling direction shown in the figure.

[0045] In S302, the change amount acquisition unit 114 obtains an approximation circle using all position parameters included in the position information of the sensor 12 as input. In this embodiment, since a rotational motion in which only the orientation parameters change is performed, the original position parameters do not change from the origin. However, if there is a change in the position parameters of the relative position and orientation, the position information of the sensor 12 will have a trajectory close to a circle, so this approximation circle is obtained. In Fig. 6, D203 is the approximation circle.

[0046] In S303, the change amount acquisition unit 114 acquires the difference between the center coordinate of the approximation circle and the initial position coordinate. Since the initial position coordinate is the origin, the difference is the center coordinate of the approximation circle of the position information, which becomes the change amount. In FIG. 8, the center coordinate D204 of the approximation circle of the position information changes from the origin in the X-axis direction, so it can be seen that there is a change in the X-coordinate of the position parameter. The change amount acquired in this manner is output to the calibration necessity determination unit 115.

[0047] In S108, the relative position and orientation calibration unit 16 corrects the relative position and orientation based on the amount of change in the relative position and orientation acquired by the change amount acquisition unit 114 as in the first embodiment, so that the relative position and orientation coincides with the physical relative positional relationship between the actual moving object and the sensor. Specifically, among the position parameters of the relative position and orientation, the amount of change is subtracted from the parameters in the X-axis direction and the Y-axis direction in the moving object coordinates. In the example of Fig. 8, since there is a change in the X-axis direction, the amount of change is subtracted from the X coordinate. The calibrated relative position and orientation is overwritten and stored in H13, which is a RAM.

[0048] In the second embodiment, when a moving body undergoes a rotational motion that changes only its orientation, the amount of change in the relative position parameters between the moving body and the sensor is obtained from the position information of the sensor. Furthermore, the relative position and orientation are corrected from the obtained amount of change. This makes it possible to correct the relative position and orientation between the sensor and the moving body without using a highly accurate amount of movement of the moving body, and makes it possible to easily correct the change in the relative positional relationship between the moving body and the position and orientation estimation device.

[0049] (Variation 2-1) In the first and second embodiments, the movement control unit 111 controls the movement of the moving body so that the moving body moves straight (first embodiment) or turns (second embodiment), but this is not limited thereto. The movement of the moving body may be controlled in any manner as long as the movement trajectory is composed of straight movement, turning, or both. For example, the movement trajectory may be a U-turn, an L-shaped movement, or a rectangular movement trajectory. In the case of movement control in which straight movement and turning are mixed like this, straight line parts and non-straight line parts, for example, corners or turning points in the route, are separated from the trajectory of the position information of the sensor 12. The change amount acquisition process described in the first and second embodiments is applied to each separated trajectory to acquire the change amount of each parameter of the position and orientation of the relative position and orientation.

[0050] (Variation 2-2) In the first and second embodiments, the movement control unit 111 moves the moving object itself by outputting movement control information to the control device 13. However, the moving object itself does not need to move as long as it can move along a predetermined movement trajectory. For example, the moving object may be placed on a device such as a moving stage or a rotating plate and moved or rotated. In this case, since the measurement information can be obtained by the sensor without controlling the moving object itself, the change in the positional relationship between the sensor and the moving object can be detected with higher accuracy.

[0051] <Embodiment 3> In the first and second embodiments, when the moving object is controlled to move in a prescribed manner such as going straight or turning, the amount of change in the relative position and orientation between the sensor and the moving object is acquired based on the position and orientation of the sensor, and the relative position and orientation are corrected based on the amount of change. In the third embodiment, a method for automatically calibrating the relative position and orientation between the sensor and the moving object while the moving object is automatically traveling along a predetermined route will be described. In particular, in this embodiment, a case will be described in which a section that forms a prescribed movement is set in advance together with route information. Note that, in this embodiment, the prescribed movement section is a section that forms a prescribed movement such as going straight or turning.

[0052] An example of the system configuration in this embodiment is similar to that of embodiment 1, and is composed of a moving body 10 having an information processing device and a sensor 12 as shown in FIG. 1. The hardware configuration in this embodiment is similar to that of embodiment 1, and is shown in FIG. 3. The functions of each piece of hardware are similar to those of embodiment 1, so their explanation will be omitted. An example of the functional configuration in this embodiment is shown in FIG. 11. It differs from FIG. 2 in embodiment 1 in that a movement path storage unit 216 and a prescribed movement determination unit 217 are present. Here, only the parts that differ from embodiment 1 will be explained, and explanations of the similar parts will be omitted.

[0053] The moving path storage unit 216 stores path information and specified moving section information required for a moving body to travel a predetermined path. This information includes the end point and intermediate points of the moving body's traveling path, and the start and end points of the specified moving section, and is array data of three parameters, the position and orientation of the moving body.

[0054] The movement control unit 211 generates movement control information using the route information stored in the movement route storage unit as an input, and outputs the information to the control device 13. Alternatively, the movement control unit 211 acquires movement control information generated by an external device or the like, or movement control information set in advance, and controls the moving object based on the acquired movement control information.

[0055] The prescribed movement determination unit 217 receives the position and orientation of the sensor 12 input by the position and orientation conversion unit 113 and the prescribed movement section information stored in the movement path storage unit, and determines whether the current position of the moving object is inside or outside the prescribed movement section. The determination result is output to the change amount acquisition unit 114.

[0056] Fig. 9 is a diagram showing a flowchart of the processing procedure in this embodiment. The description of the same processes as those in Fig. 4 which explain the processing procedure of the information processing device 11 described in the first embodiment will be omitted, and only S302, S305, S306, and S309 which are different from the first embodiment will be described.

[0057] In S302, the movement control unit 111 acquires movement control information so as to travel along a specified route based on the route information stored in the movement route storage unit 216, and outputs the information to the control device 13, thereby controlling the movement of the moving object.

[0058] In S305, the prescribed movement determination unit 217 determines whether the moving object is within the prescribed movement section. Based on the position and orientation of the sensor 12 input by the position and orientation conversion unit 113 and the prescribed movement section information held by the movement path holding unit, a determination is made as to whether the current position of the moving object is within or outside the prescribed movement section. When the moving object is within the prescribed movement section, the type information of the prescribed movement is also output as a determination result. The type information of the prescribed movement is, for example, a movement pattern including a straight line as described in the first embodiment and a turning as described in the second embodiment. Furthermore, even if the position of the moving object is within the prescribed movement section, if the running time within the prescribed movement section is short, the number of position and orientation data in the position information of the sensor 12 may not be sufficient to obtain the amount of change. Therefore, if the moving object has not run within the prescribed movement section for a predetermined time or more, a false result is output even if the moving object is within the prescribed movement section.

[0059] In S306, if the determination result by the prescribed movement determination unit 217 is true, the process proceeds to S307. If not, the process skips to S310. In S307, the change amount acquisition unit 114 acquires the amount of change between the relative position and orientation stored in the RAM and the physical relative positional relationship between the actual moving body and the sensor, using the position information of the sensor 12 input by the position and orientation conversion unit 113. At this time, the change amount acquisition method is determined based on the type information of the prescribed movement input by the prescribed movement determination unit. For example, if the type of the prescribed movement is straight, the method described in the first embodiment is used, and if it is turning, the method described in the second embodiment is used.

[0060] In S309, the relative position and orientation calibration unit 216 corrects the relative position and orientation so as to match the actual physical relative positional relationship between the moving body and the sensor, based on the amount of change in the relative position and orientation acquired by the change amount acquisition unit 114. In this embodiment, the correction is made in accordance with the parameters acquired by the change amount acquisition unit 114. Specifically, if the type of prescribed movement is straight ahead, the orientation parameters are corrected, and if it is turning, the position parameters are corrected.

[0061] In the third embodiment, when a section of a specified movement is set in advance together with route information in the autonomous driving of a moving body, it is determined whether the moving body is within the specified movement section. Based on the result of the determination, acquisition of the amount of change in the relative position and orientation of the sensor and the moving body and implementation of correction are dynamically switched. This enables correction of the relative position and orientation while the moving body is autonomously driving, and makes it possible to easily correct changes in the relative positional relationship between the moving body and the position and orientation estimation device.

[0062] (Variation 3-1) In the third embodiment, the moving path storage unit stores route information and prescribed moving section information required for the moving body to travel along a predetermined path. Furthermore, the prescribed moving judgment unit performs a determination as to whether the position of the moving body is inside or outside the prescribed moving section based on the prescribed moving section information. However, this is not limited to these, and any method may be used as long as the prescribed moving judgment unit can determine whether the position of the moving body is inside or outside the prescribed moving section based on the information stored in the moving path storage unit. For example, the moving path storage unit may store a history of the moving trajectory of the moving body, and the prescribed moving judgment unit may extract a section that is a prescribed movement such as going straight or turning by using least squares fitting to the history of the moving trajectory.

[0063] Furthermore, the prescribed movement determination unit may make a determination using information other than the information stored in the movement path storage unit. For example, a sensor capable of observing a moving object from a third-party viewpoint, such as a surveillance camera, may be used to obtain a movement trajectory of the moving object, and the inside / outside of the prescribed movement section or the type of prescribed movement may be determined based on the movement trajectory.

[0064] The output unit 117 may output information related to the calibration of the relative position and orientation, such as a specified movement section, a change amount of the relative position and orientation, and calibration parameters. Fig. 10 shows an example of a GUI 100 that uses a display as the output device 14 and presents route information and specified movement section information of a moving object, an acquisition result of the change amount of the relative position and orientation, and calibration parameters.

[0065] G110 is a GUI that presents information about the moving path and the prescribed moving section of the moving object. G111 is a moving object that is moving automatically, G112 is the start point of the moving path, and G113 is the end point. The arrow line connecting the start point G112 and the end point G113 is the moving path, and in particular, the solid arrow indicates the prescribed moving section.

[0066] G120 is a GUI for selecting the type of movement control of a moving object. Specifically, in the movement control for turning a moving object, the type is selected as either one in which only the attitude parameters change as shown in the second embodiment, or one in which both the position parameters and the attitude parameters change as in an automobile.

[0067] G130 is a GUI that presents the results of the change in the relative position and orientation. G131 displays the change in the acquired relative position and orientation numerically, dividing it into position and orientation. G132 displays each axis (solid arrow) of the coordinate system to which the acquired change has been added to the current moving body coordinate system (dotted arrow). This makes it possible to visually understand the change in the relative position and orientation. The user can decide whether or not to correct the sensor mounting position and orientation. Note that Fig. 10 shows an example in which only the orientation (angle) has changed.

[0068] G140 is a GUI that allows the user to adjust the calibration of the relative position and orientation. G141 allows the user to enable automatic calibration while driving by clicking a checkbox. G142 allows the user to move the threshold adjustment bar left and right to determine the threshold to be used when determining whether calibration is necessary.

[0069] As described above, the GUI that presents the output of the change amount acquisition results and the input of parameters during calibration can encourage the user to understand intuitively, thereby further improving convenience when correcting the relative position and orientation.

[0070] <Embodiment 4> In the first and second embodiments, when the moving body is controlled to move in a specified direction such as going straight or turning, the amount of change in the relative position and attitude between the sensor and the moving body is acquired based on the position and attitude of the sensor, and the relative position and attitude is corrected based on the amount of change. In both the first and second embodiments, the moving body is described as an automatic guided vehicle, and since it is assumed that the moving body moves on the ground, i.e., on a two-dimensional plane, the position and attitude of the moving body has three degrees of freedom (two position parameters and one attitude parameter).

[0071] In this embodiment, a case where the present invention is applied to a moving body such as a drone that can move and rotate in the height direction (perpendicular to the floor in the real space) will be described. Therefore, in the fourth embodiment, a method for correcting the relative position and orientation in a drone in which the movement trajectory is a straight line will be described. In the case of such a moving body, the position and orientation of the moving body have six degrees of freedom (three position parameters and three orientation parameters).

[0072] The configuration example of the system in this embodiment is the same as that of the first embodiment, and is composed of a moving body 10 having an information processing device and a sensor 12 as shown in FIG. 1. However, the difference is that the moving direction of the moving body 10 can move not only in the horizontal direction but also in the vertical direction. In addition, the posture parameter of the moving body is one parameter (Yaw) in the first embodiment, whereas in this embodiment, it is three parameters (Roll, Pitch, Yaw), and it is possible to rotate around three axes of XYZ. Here, the Z axis is a coordinate axis perpendicular to the floor of the real space, and Roll and Pitch are defined as posture parameters that indicate rotation around the X axis and Y axis, respectively. The hardware configuration in this embodiment is the same as that of the first embodiment, and is shown in FIG. 3. The functions of each hardware are the same as those of the first embodiment, so the description will be omitted. A diagram of the processing procedure in this embodiment is shown in FIG. 12. Only the processing procedures different from the first embodiment will be described, and the description of the other procedures will be omitted as they are the same as those in FIG. 4.

[0073] Since the drone can move three-dimensionally, in this embodiment, the drone moves on the coordinate axes in the order of the X axis, the Y axis, and the Z axis. The order of the coordinate axes to be moved here may be any order. The flowchart in FIG. 12 is a flowchart showing the processing procedure for each axis.

[0074] In S401, the movement control unit 111 determines the route and stopping points along which the drone will move, and generates control information. When a drone moves forward, backward, left, or right, it is difficult for it to move in a straight line because it is prone to wobbling. In this embodiment, it is necessary to make the movement trajectory of the moving object a straight line. A stopping point is provided in the middle of the movement path where the drone hovers or lands to stabilize, and the movement trajectory is pseudo-linearized by using the position and orientation of the sensor at that point. Linearization refers to making it easier to obtain an approximate straight line of the movement trajectory by using a method of eliminating position measurement data that deviates from the straight line, or a method of averaging a group of position measurement data that includes variation.

[0075] In S102, the movement control unit 111 inputs the control information generated in S401 and moves the drone, while temporarily stopping it at the stopping point.

[0076] In S103, the position and orientation estimation unit 112 estimates the position and orientation of the sensor near the stopping point, and outputs an average value to the position and orientation conversion unit 113 as an estimation result.

[0077] In S104, the position and orientation conversion unit 113 converts the trajectory of the position and orientation of the sensor 12 input by the position and orientation estimation unit 112 from the sensor coordinate system to the moving body coordinate system. In the first embodiment, the moving body coordinate system had three degrees of freedom, so six parameters of the position and orientation were reduced to three parameters, but in this embodiment, the moving body has six degrees of freedom, so there is no need to reduce the parameters.

[0078] In S105, the change amount acquisition unit 114 acquires the amount of change between the relative position and orientation held in the RAM and the physical relative positional relationship between the actual moving body and the sensor, using the position information at the start point and the stop point of the sensor 12 input by the position and orientation conversion unit 113. Since the moving body is moved in a straight line, the amount of change in the orientation parameters of the relative position and orientation can be obtained, but the rotational components of the obtained change amount differ depending on the moving direction. Specifically, the rotational components of Pitch, Yaw, and Roll can be obtained from the position information when moving on the X axis, Y, and Z, respectively. Note that the same method as in the first embodiment can be used to calculate the amount of change regardless of the moving direction. In the case of moving on the Z axis, the moving direction in the first embodiment is replaced by the Z axis, and the coordinate axis perpendicular to it is replaced by the Y axis, and the comparison axis when obtaining the amount of change is replaced by the Z axis, so that the amount of change in the orientation parameter Roll is the angle between the approximate straight line of the sensor's position information and the Z axis. On the other hand, in the case of movement along the X axis, the traveling direction in the first embodiment is replaced with the X axis, the coordinate axis perpendicular to it is replaced with the Z axis, and the comparison axis when calculating the amount of change is replaced with the X axis. Therefore, the amount of change in the attitude parameter Pitch is the angle between the approximation line of the sensor position information and the X axis.

[0079] In S108, the relative position and orientation calibration unit 116 corrects the relative position and orientation so that it matches the physical relative positional relationship between the actual moving object and the sensor, based on the amount of change in the relative position and orientation acquired by the change amount acquisition unit 114. The relative position and orientation are corrected by subtracting the amount of change in the three orientation parameters calculated in S105 from each component of the relative position and orientation.

[0080] In the fourth embodiment, for a moving body that can move in any direction in three-dimensional space, it is possible to correct the relative position and orientation between the sensor and the moving body without using a highly accurate movement amount of the moving body, and it is possible to easily correct changes in the relative positional relationship between the moving body and the position and orientation estimation device.

[0081] (Variation 4-1) In the fourth embodiment, the movement control unit 111 moves the drone along the coordinate axes of the X-axis, Y-axis, and Z-axis. However, the movement direction is not limited to these, and may be any direction not along the coordinate axes as long as the change amount acquisition unit 114 can compare the predicted position of the sensor with the position of the moving body. The change amount acquisition unit determines a two-dimensional plane to be used as a reference and then compares the predicted position of the sensor with the position of the moving body. The two-dimensional plane to be used as a reference is determined by the rotation component of the change amount to be acquired. Specifically, the rotation component to be acquired is the XY plane when the rotation component to be acquired is the YZ plane when the rotation component to be acquired is the Yaw, and the ZX plane when the rotation component to be acquired is the Pitch. For example, when it is desired to acquire the change amount of the rotation component of Roll, the movement control unit 111 moves the moving body straight on the YZ plane and compares the predicted position of the sensor with the position of the moving body. Furthermore, even if the moving body is moved in any direction in the XYZ space that is not on the YZ plane, the predicted position of the sensor and the position of the moving body may be projected onto the YZ plane, and the change amount of the Pitch may be acquired by comparing the position information after the plane projection.

[0082] (Variation 4-2) In the fourth embodiment, the position and orientation estimation unit 112 estimates the position and orientation of the sensor near the stopping point, and outputs the average value as the estimation result. However, this is not limited to this, and any position and orientation group of the estimation result may be linearized. For example, since a drone is likely to wobble immediately after starting to move, the position and orientation may be estimated at a certain time interval after a certain time has elapsed since the start of movement. Alternatively, after moving a certain distance from the start of movement, the position and orientation may be estimated at a certain distance interval and output to the relative position and orientation change amount acquisition unit 114. Furthermore, regardless of the distance or time from the start of movement, a position and orientation that deviates from a straight line may be excluded.

[0083] (Variation 4-3) In the fourth embodiment, the movement control unit 111 corrects the three attitude parameters of the relative position and attitude by moving in a straight line in each of the X-axis, Y-axis, and Z-axis directions. However, the three position parameters of the relative position and attitude may be corrected by rotating the drone in turn around each coordinate axis instead of moving in a straight line. Here, the rotation may be a method of rotating the drone in the air by controlling the movement of the drone, or a method of fixing the drone to a jig so that the drone rotates around a designed rotation axis and rotating the drone. As in the case of moving in a straight line, the position parameters that can be corrected differ depending on the rotation axis. Specifically, the Y·Z coordinates can be corrected by X-axis rotation (Roll), the Z·X coordinates can be corrected by Y-axis rotation (Pitch), and the X·Y coordinates can be corrected by Z-axis rotation (Yaw). The same method as in the second embodiment can be used for the method of acquiring the change amount during rotation and the calibration method. For example, in the case of X-axis rotation, the two-dimensional plane on which the position information in the second embodiment is mapped is replaced with a YZ plane, and the change amount of the position parameter Y·Z coordinate is the difference between the center coordinate of the approximation circle of the position information of the sensor and the initial position coordinate (origin).

[0084] (Variation 4-4) In the fourth embodiment, since it is difficult for a drone to move stably in a straight line, the position and orientation of the sensor near the stopping point are used to obtain the amount of change, but this is not limited to drones. For example, it can be applied to moving objects that move on a two-dimensional plane, such as automatic guided vehicles, self-driving cars, and autonomous mobile robots.

[0085] (Variation 4-5) In the modified example 4-2, the position and orientation estimation unit 112 removes positions and orientations that deviate from a straight line in order to linearize the estimated sensor position and orientation group, but this is not limited to drones. For example, it can also be applied to moving objects that move on a two-dimensional plane, such as automated guided vehicles, self-driving cars, and autonomous mobile robots.

[0086] Of the above-mentioned processing units, the position and orientation measurement unit 112 and the movement control unit 111 may use a trained model that has been machine-learned instead. In that case, for example, a plurality of combinations of input data and output data to the processing unit are prepared as training data, knowledge is acquired from them by machine learning, and a trained model that outputs output data for the input data as a result based on the acquired knowledge is generated. The trained model can be configured, for example, as a neural network model. The trained model performs the processing of the processing unit by operating in cooperation with a CPU or GPU as a program for performing the same processing as the processing unit. The trained model may be updated after a certain amount of processing as necessary.

[0087] The present invention can also be realized by executing the following process. That is, software (programs) that realize the functions of the above-described embodiments are supplied to a system or device via a data communication network or various storage media. Then, a computer (or a CPU, MPU, etc.) of the system or device reads and executes the program. The program may also be provided by recording it on a computer-readable recording medium. [Explanation of symbols]

[0088] 10. Information Processing Systems 11 Information processing equipment 12 Sensors 13 Control device

Claims

1. an estimation means for estimating first position information indicating a position of a moving object mounted with a sensor based on input information input from the sensor when the moving object moves in a predetermined manner; an acquisition means for acquiring second position information indicating a position of the moving object that is expected to be obtained based on input information input from the sensor when the moving object held by the holding means makes the predetermined movement while maintaining the position and orientation of the sensor with respect to the moving object; an output means for outputting information regarding a change in a position and orientation of the sensor relative to the moving body when the moving body makes the predetermined movement and a change in a position and orientation of the sensor relative to the moving body held by the holding means, based on the first position information and the second position information; 13. An information processing device comprising:

2. 2 . The information processing apparatus according to claim 1 , wherein the output means outputs output information indicating a difference in position of the moving object based on the first position information and the second position information.

3. The information processing device according to claim 2, characterized in that the output means outputs the output information indicating that the position and orientation of the sensor with respect to the moving body is the held position and orientation when a difference in position of the moving body based on the first position information and the second position information is less than a predetermined threshold value.

4. The information processing device according to claim 2, characterized in that the output means outputs the output information indicating that the position and orientation of the sensor relative to the moving body is not the retained position and orientation when a difference in position of the moving body based on the first position information and the second position information is equal to or greater than a predetermined threshold value.

5. The information processing device according to claim 2, characterized in that the output means outputs to a display device the output information indicating that the position and orientation of the sensor relative to the moving body is not the retained position and orientation when a difference in position of the moving body based on the first position information and the second position information is equal to or greater than a predetermined threshold value.

6. The information processing device according to claim 2, characterized in that the output means outputs the output information for displaying the position of the moving body indicated by the first or second position information on a display device when a difference in the position of the moving body based on the first position information and the second position information is equal to or greater than a predetermined threshold value.

7. The information processing device according to claim 2, characterized in that the output means outputs the output information indicating that the position and orientation of the sensor relative to the moving body is not the retained position and orientation to an audio output device when a difference in position of the moving body based on the first position information and the second position information is equal to or greater than a predetermined threshold.

8. 3. The information processing device according to claim 2, wherein the output means outputs the output information indicating the position and orientation of the sensor that satisfies the held position and orientation to a projection device or a display device when a difference in position of the moving body based on the first position information and the second position information is greater than a predetermined value.

9. 3. The information processing device according to claim 2, wherein the output means outputs different output information depending on the magnitude of the difference between the positions of the moving body based on the first position information and the second position information based on a plurality of threshold values.

10. The information processing device according to claim 8, characterized in that the output means outputs the output information to a first output device when a difference in the position of the moving body based on the first position information and the second position information is greater than a first threshold, and outputs the output information to a second output device when a difference in the position of the moving body based on the first position information and the second position information is greater than a second threshold that is smaller than the first threshold.

11. The predetermined movement is a movement of the moving body moving in a straight line for a predetermined distance, 9. The information processing device according to claim 8, characterized in that the output means outputs the output information indicating a change in attitude of the sensor based on a difference in position of the moving body based on the first position information and the second position information.

12. The predetermined movement is a movement in which the moving body turns by a predetermined angle, 9. The information processing device according to claim 8, characterized in that the output means outputs the output information indicating a change in the position of the sensor based on a difference between the position of the moving body based on the first position information and the second position information.

13. 9. The information processing device according to claim 8, further comprising an update means for updating the position and orientation of the sensor relative to the moving body based on the output information when a difference in the position of the moving body based on the first position information and the second position information is greater than a predetermined threshold value.

14. a measuring means for measuring a position of the sensor from the input information based on map information obtained by measuring a moving path of the moving object in advance; 14. The information processing device according to claim 1, wherein the estimation means estimates the position of the moving body by converting the position or orientation of the sensor measured by the measurement means based on the held position and orientation of the sensor with respect to the moving body.

15. 15. The information processing apparatus according to claim 1, wherein the sensor is an imaging device that captures an image of an object existing in the environment.

16. 15. The information processing apparatus according to claim 1, wherein the sensor is a distance sensor that measures a distance between an object present in an environment and the sensor.

17. A program for causing a computer to function as each of the means included in the information processing device according to any one of claims 1 to 15.

18. an estimation step of estimating first position information indicating a position of a moving object equipped with a sensor based on input information input from the moving object when the moving object moves in a predetermined manner; an acquisition step of acquiring second position information indicating a position of the moving object that is expected to be obtained based on input information input from the sensor when the moving object moves in the predetermined manner while maintaining the position and orientation of the sensor relative to the moving object held by a holding means; and an output step of outputting information regarding changes in the position and orientation of the sensor relative to the moving body when the moving body performs the predetermined movement, and the position and orientation of the sensor relative to the moving body held by the holding means, based on the first position information and the second position information.

Images