Information processing device, mobile object control system, control method, and program

Abstract

The purpose of the present invention is to provide an information processing device (10) which can prevent an accident such as a collision in advance. The information processing device (10) according to the present disclosure comprises: an extraction unit (11) which extracts, from among a plurality of first captured images captured by using a first imaging device for creating a three-dimensional image of a prescribed area, at least one first captured image similar to a second captured image captured by using a second imaging device mounted on a moving device that is moving in the area; an estimation unit (12) which estimates a first imaging orientation of the first imaging device that has captured the extracted first captured image and a second imaging orientation of the second imaging device when the second captured image is captured; and a control unit (13) which controls an operation of the moving device on the basis of the first imaging orientation and the second imaging orientation.

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, a mobile object control system, a control method, a program, and the like. [Background technology]

[0002] In recent years, labor shortages in the logistics industry have become increasingly serious. Therefore, there is a demand for labor-saving and unmanned logistics operations through the use of autonomous mobile robots such as forklifts and AGVs (Automatic Guided Vehicles) to transport goods. In unmanned operations using mobile robots, the mobile robots must be able to estimate their own position.

[0003] Patent Document 1 discloses the configuration of an autonomous mobile device that edits a map created using the SLAM (Simultaneous Localization and Mapping) method when it is determined that the difference between the device's own position obtained from odometry and the device's own position estimated using the SLAM method exceeds a predetermined error range. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-107425 Summary of the Invention [Problem to be solved by the invention]

[0005] However, when the autonomous mobile device disclosed in Patent Document 1 determines that the deviation of its estimated position exceeds a predetermined error range, the autonomous mobile device is traveling significantly off the predetermined route. In such a case, the autonomous mobile device may collide with another object. In other words, the autonomous mobile device disclosed in Patent Document 1 has a problem in that it cannot prevent accidents such as collisions in advance.

[0006] One object of the present disclosure is to provide an information processing device, a mobile object control system, a control method, a program, etc. that can prevent accidents such as collisions in advance. [Means for solving the problem]

[0007] An information processing device according to a first aspect of the present disclosure includes an extraction unit that extracts, from a plurality of first photographed images taken using a first photographing device to create a three-dimensional image of a specified area, at least one first photographed image that is similar to a second photographed image taken using a second photographing device mounted on a mobile device moving through the area; an estimation unit that estimates a first photographing attitude of the first photographing device that captured the extracted first photographed image and a second photographing attitude of the second photographing device when the second photographed image was captured; and a control unit that controls the operation of the mobile device based on the first photographing attitude and the second photographing attitude.

[0008] A mobile object control system according to a second aspect of the present disclosure includes a mobile device that moves autonomously within a predetermined area; an extraction unit that extracts at least one first photographed image from a plurality of first photographed images taken using a first photographing device to create a three-dimensional image of the area, the first photographed image being similar to a second photographed image taken using a second photographing device mounted on the mobile device while the mobile device is moving within the area; an estimation unit that estimates a first photographing attitude of the first photographing device that captured the extracted first photographed image and a second photographing attitude of the second photographing device when the second photographed image was captured; and a control unit that determines the operation of the mobile device based on the first photographing attitude and the second photographing attitude, and the mobile device operates based on operation instructions obtained from the remote control device via a network.

[0009] A control method according to a third aspect of the present disclosure is executed in an information processing device, and extracts at least one first photographed image from a plurality of first photographed images taken using a first photographing device to create a three-dimensional image of a specified area, the first photographed image being similar to a second photographed image taken using a second photographing device mounted on a mobile device moving through the area, estimates a first photographing attitude of the first photographing device that captured the extracted first photographed image and a second photographing attitude of the second photographing device when the second photographed image was captured, and controls the operation of the mobile device based on the first photographing attitude and the second photographing attitude.

[0010] A program according to a fourth aspect of the present disclosure causes a computer to perform the following steps: extract, from a plurality of first photographed images taken using a first photographing device to create a three-dimensional image of a specified area, at least one first photographed image that is similar to a second photographed image taken using a second photographing device mounted on a mobile device moving through the area; estimate a first photographing attitude of the first photographing device that captured the extracted first photographed image and a second photographing attitude of the second photographing device when the second photographed image was captured; and control the operation of the mobile device based on the first photographing attitude and the second photographing attitude. [Effects of the Invention]

[0011] The present disclosure makes it possible to provide an information processing device, a mobile object control system, a control method, a program, and the like that can prevent accidents such as collisions in advance. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a configuration diagram of an information processing device according to a first embodiment. [Figure 2] FIG. 10 is a diagram showing experimental results regarding self-location estimation according to the first embodiment. [Figure 3] FIG. 10 is a diagram showing experimental results regarding self-location estimation according to the first embodiment. [Figure 4]FIG. 10 is a configuration diagram of a mobile robot according to a second embodiment. [Figure 5] FIG. 10 is a diagram illustrating matching of feature points according to the second embodiment. [Figure 6] 10A and 10B are diagrams illustrating a difference in posture of an image capturing apparatus according to a second embodiment. [Figure 7] 10A and 10B are diagrams illustrating a difference in posture of an image capturing apparatus according to a second embodiment. [Figure 8] FIG. 10 is a diagram showing a flow of controlling the movement of a mobile robot according to the second embodiment. [Figure 9] FIG. 10 is a configuration diagram of a mobile object control system according to another embodiment. [Figure 10] FIG. 10 is a configuration diagram of an information processing device according to another embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] (Embodiment 1) Hereinafter, embodiments of the present invention will be described with reference to the drawings. An example of the configuration of an information processing device 10 according to the first embodiment will be described with reference to FIG. 1. The information processing device 10 may be a computer device that operates when a processor executes a program stored in a memory. The information processing device 10 has an extraction unit 11, an estimation unit 12, and a control unit 13. The components of the information processing device 10, such as the extraction unit 11, the estimation unit 12, and the control unit 13, may be software or modules that perform processing when a processor executes a program stored in a memory. Alternatively, the components of the information processing device 10 may be hardware, such as a chip or a circuit.

[0014] The extraction unit 11 extracts at least one image from a plurality of first photographed images taken using a first photographing device in order to create a three-dimensional image of a predetermined area. Specifically, the extraction unit 11 extracts at least one first photographed image similar to a second photographed image taken using a second photographing device mounted on a mobile device moving in the predetermined area.

[0015] The predetermined area may be, for example, a predetermined space, or a space including a location where the mobile device moves, such as inside a building or a factory. The first image capturing device may be, for example, a camera that generates RGB (Red Green Blue) images, and may be, for example, a camera mounted on a mobile terminal such as a smartphone. The second image capturing device may be a camera mounted on the mobile device and generate RGB images. The mobile device may be a vehicle, such as an autonomously moving forklift or AGV (Automatic Guided Vehicle). The autonomously moving mobile device may be referred to as a mobile robot or simply as a robot.

[0016] The first imaging device may be used as the second imaging device. That is, the first imaging device may be used as the second imaging device, or an imaging device different from the first imaging device may be used.

[0017] A three-dimensional image is an image that identifies the three-dimensional position of an object existing within a predetermined area. The three-dimensional image may also be referred to as a three-dimensional map, an environmental map, or the like. The three-dimensional image may be generated, for example, by executing Structure from Motion (SfM) using multiple images captured by a first image capture device. SfM calculates all feature points in a series of previously acquired two-dimensional images (or frames) and estimates matching feature points from multiple images that occur before and after in time. Furthermore, SfM accurately estimates the three-dimensional position or orientation of the camera that captured each frame based on the difference in the position on a two-dimensional plane in the frame in which each feature point appears. The three-dimensional image may be generated by the information processing device 10 or by a computer device different from the information processing device 10. The information processing device 10 may receive a three-dimensional image of a predetermined area generated by another computer device via a network.

[0018] The extraction unit 11 compares a second captured image captured using a second image capturing device mounted on a mobile device moving within a predetermined area with each of the first captured images used to generate the three-dimensional image, and determines the degree of similarity between them. The extraction unit 11 may extract the first captured image having the highest degree of similarity with the second captured image. Alternatively, the extraction unit 11 may extract two or more first captured images in descending order of the degree of similarity between the first captured image and the second captured image. The multiple first captured images used to generate the three-dimensional image may be stored in a memory or the like mounted on the information processing device 10.

[0019] The estimation unit 12 estimates the shooting attitude of the first camera device that captured the first captured image extracted by the extraction unit 11. Furthermore, the estimation unit 12 estimates the shooting attitude of the second camera device that captured a second captured image similar to the extracted first captured image. The estimation unit 12 estimates the attitude of the first camera device, for example, based on the difference in the positions of feature points that appear in multiple captured images taken before and after the first captured image in time. Similarly, the estimation unit 12 estimates the attitude of the second camera device based on the difference in the positions of feature points that appear in multiple captured images taken before and after the second captured image in time. Furthermore, the estimation unit 12 may use information about the attitude of the first camera device that was estimated in advance when generating the 3D image as estimated information about the attitude of the first camera device. In other words, the estimation unit 12 may estimate the orientation of the first imaging device when estimating the orientation of the second imaging device, or may use the orientation of the first imaging device that has been estimated in advance as estimation information.

[0020] The image capturing posture may include, for example, the image capturing position and image capturing direction of each image capturing device. The image capturing direction may indicate, for example, the direction in which the image capturing device is facing, with the image capturing device as the origin.

[0021] The control unit 13 controls the operation of the mobile device based on the shooting attitude of the first image capturing device and the shooting attitude of the second image capturing device. Controlling the operation of the mobile device may be, for example, controlling the speed of the mobile device or controlling warning information output by the mobile device. Furthermore, controlling the operation of the mobile device may be changing the moving path of the mobile device.

[0022] Here, the contents of an experiment conducted by the inventors when examining the contents of the present disclosure will be described with reference to Figures 2 and 3. The inventors conducted an experiment to investigate the relationship between a self-location estimated as an offline process and a self-location estimated in real time.

[0023] In this experiment, an environmental map showing a predetermined area is generated in advance using multiple captured images. Furthermore, a self-location is estimated in real time using captured images captured while moving within the predetermined area. Furthermore, a self-location is estimated using captured images captured while moving within the predetermined area using a method substantially similar to the method used to generate the environmental map in advance. Estimating the self-location includes estimating the attitude of the image capture device. Specifically, estimation of the position of the image capture device and estimation of the image capture direction of the image capture device are also performed. Furthermore, estimating the self-location may include estimation of a three-dimensional position or generation of an environmental map.

[0024] The generation of an environmental map performed in advance and the estimation of a self-location using a method substantially similar to the method used to generate the environmental map are referred to as offline processing. Furthermore, the estimation of a self-location performed in real time is referred to as real-time processing. In order to reduce the processing load, real-time processing performs calculations with a smaller amount of calculations than those performed in offline processing. Furthermore, the number of captured images used for self-location estimation in real-time processing is smaller than the number of captured images used for generating an environmental map or estimating a self-location in offline processing. As a result, the accuracy of generating an environmental map and estimating a self-location in offline processing is higher than the accuracy of estimating a self-location in real-time processing. Hereinafter, the estimation result of a self-location in offline processing using captured images captured while moving within a specified area, i.e., captured images captured in real time, will be described as a true value. The true value indicates that the estimation result is substantially identical to the actual value.

[0025] The horizontal axis of Figure 2 shows the difference between the attitude value of the camera that captured the images used to generate the environmental map and the estimated value of the attitude of the camera that captured the images in real-time processing. Furthermore, the vertical axis of Figure 2 shows the difference between the true attitude value in offline processing using the images captured in real time and the estimated value of the attitude estimated in real-time processing. Here, attitude primarily refers to the direction in which the camera is facing. Figure 2 shows the relationship between the difference between the attitude of the camera that was used to generate the environmental map and the attitude of the camera that was estimated in real-time processing, and the difference between the true attitude value in offline processing and the estimated value of the attitude estimated in real-time processing.

[0026] Figure 2 shows that as the difference on the horizontal axis increases, the difference between the true value of the pose in offline processing and the estimated value of the pose estimated in real time processing also increases. In other words, Figure 2 shows that the larger the difference on the horizontal axis, the more likely it is that the accuracy of the estimated value of the pose estimated in real time processing will be poor.

[0027] The horizontal axis of Figure 3 shows the difference between the attitude value of the camera that captured the image used to generate the environmental map and the estimated value of the attitude of the camera that captured the image in real-time processing. Furthermore, the vertical axis of Figure 3 shows the difference between the true value of the position of the camera captured in real time and the estimated value of the position estimated in real-time processing. Here, the attitude mainly refers to the direction in which the camera is facing. Figure 3 shows the relationship between the difference between the position value of the camera that was used to generate the environmental map and the estimated value of the position of the camera estimated in real-time processing, and the difference between the true value of the position in offline processing and the estimated value of the position estimated in real-time processing.

[0028] 3 shows that as the difference on the horizontal axis increases, the difference between the true position value obtained in offline processing and the estimated position value obtained in real-time processing also increases. In other words, FIG. 3 shows that the larger the difference on the horizontal axis, the more likely it is that the accuracy of the estimated position value obtained in real-time processing will decrease.

[0029] Based on the results of the above experiments, the inventors who considered the contents of the present disclosure discovered the need to control a mobile device according to the difference between the attitude of the camera that captured the images used to generate the environmental map and the attitude of the camera during real-time processing.

[0030] Specifically, when controlling the operation of a mobile device, it is necessary to take into consideration that the accuracy of self-location estimation in real-time processing deteriorates as the difference in camera attitude between offline processing and real-time processing increases. Therefore, the inventors of the present disclosure have discovered a way to execute operation control that takes into consideration the safety of the mobile device, depending on the difference in attitude.

[0031] The information processing device 10 in Fig. 1 estimates the difference between the attitude value of the camera that captured the images used to create a 3D image of a predetermined area and the estimated attitude value of the camera that captured the images while moving through the predetermined area. Furthermore, the information processing device 10 controls the operation of the mobile device based on the estimation result. As a result, the information processing device 10 can execute control that takes into consideration the safety of the mobile device when there is a high possibility that the accuracy of self-location estimation of the mobile device will deteriorate.

[0032] Furthermore, as a control that takes safety into consideration, the information processing device 10 may cause the mobile device to output a warning. As a result, other mobile devices that recognize the mobile device that output the warning can take action to avoid the mobile device that output the warning. Furthermore, as a control that takes safety into consideration, the information processing device 10 may control the operation of the mobile device and notify or report the same to an administrator or the like via a network.

[0033] (Embodiment 2) Next, a configuration example of the mobile robot 30 according to the second embodiment will be described with reference to Fig. 4. The mobile robot 30 may be an autonomously moving forklift or an AGV. The mobile robot 30 includes an information processing device 20, a photographing device 31, and a driving unit 32.

[0034] The image capturing device 31 may be, for example, a camera that captures RGB images or RGB-D (RGB and Depth) images. The image capturing device 31 captures an image of the environment around the mobile robot 30 while the mobile robot 30 is moving. The image capturing device 31 outputs the captured image to the information processing device 20.

[0035] The driving unit 32 may include tires of the vehicle and peripheral devices that operate the tires. The information processing device 20 may be a computer device like the information processing device 10, and controls the operation of the driving unit 32. For example, the information processing device 20 may control the driving unit 32 to change the direction in which the mobile robot 30 moves or to change the moving speed of the mobile robot 30. Changing the moving speed of the mobile robot 30 also includes stopping the mobile robot 30.

[0036] Next, a description will be given of an example configuration of the information processing device 20. The information processing device 20 has a configuration in which an attitude difference calculation unit 21, an environmental map storage unit 22, and a path plan storage unit 23 are added to the information processing device 10 in Fig. 1. In the following description of the information processing device 20, detailed description of functions or operations similar to those of the information processing device 10 will be omitted.

[0037] The attitude difference calculation unit 21 may be software or a module whose processing is performed by a processor executing a program stored in a memory. Alternatively, the attitude difference calculation unit 21 may be hardware such as a circuit or a chip. The environmental map storage unit 22 and the path plan storage unit 23 may be memories mounted on the information processing device 20. Alternatively, the environmental map storage unit 22 and the path plan storage unit 23 may be removable external memories attached to the information processing device 20.

[0038] The environmental map storage unit 22 stores an environmental map that has been generated in advance. The environmental map storage unit 22 also stores captured images, which are two-dimensional images used to generate the environmental map, which is a three-dimensional image. Furthermore, the environmental map storage unit 22 stores information about the orientation of the image capture device when the two-dimensional image was captured, in association with the image capture device.

[0039] "Generated in advance" may mean, for example, that the environmental map is generated before the mobile robot 30 starts moving. The environmental map may be generated using images captured by a camera mounted on a mobile terminal such as a smartphone. The environmental map may be generated in the information processing device 20 or in a computer device different from the information processing device 20. The information regarding the attitude of the camera includes, for example, information indicating the position and direction of the camera when the camera captured the image. The attitude of the camera is estimated when generating the environmental map by, for example, executing SfM. Furthermore, when generating the environmental map, a person holding a smartphone or the like may capture images for generating the environmental map while moving along a predetermined path for the mobile robot 30. In this way, the captured images used to generate the environmental map and the captured images captured by the camera device 31 while the mobile robot 30 is moving are generated from similar viewpoints.

[0040] The extraction unit 11 extracts, from among the plurality of captured images stored in the environmental map storage unit 22, a captured image that is similar to the captured image output from the photographing device 31. For example, the extraction unit 11 may extract feature points from each of the plurality of captured images stored in the environmental map storage unit 22 and the captured image output from the photographing device 31, and determine the similarity of the images using the feature points. For example, the feature points of each image may be extracted using an algorithm such as SIFT, SURF, ORB, or AKAZE.

[0041] For example, as shown in FIG. 5 , the extraction unit 11 may match feature points between the captured image stored in the environmental map storage unit 22 and the captured image output from the camera device 31, and determine the similarity depending on the number of matching feature points. For example, the extraction unit 11 may determine that the greater the number of matching feature points, the higher the similarity between the captured images. For example, the extraction unit 11 may extract from the environmental map storage unit 22 the captured image that is most similar to the captured image output from the camera device 31, or may extract any number of captured images from the environmental map storage unit 22 in descending order of similarity. The extraction unit 11 outputs the captured image extracted from the environmental map storage unit 22, information about the attitude of the camera device associated with the extracted captured image, and the captured image output from the camera device 31 to the estimation unit 12.

[0042] The estimation unit 12 estimates the attitude of the camera device 31 that captured the received captured image. For example, the estimation unit 12 executes Visual SLAM (VSALM) using a plurality of captured images that are taken before and after the captured image for which the attitude of the camera device 31 is to be estimated. In this way, the estimation unit 12 estimates the attitude of the camera device 31 that captured the captured image for which the attitude is to be estimated. The estimation unit 12 outputs, to the attitude difference calculation unit 21, information on the attitude of the camera device associated with the captured image extracted from the environmental map storage unit 22 and information on the attitude of the camera device 31 that captured the captured image estimated by the estimation unit 12.

[0043] The attitude difference calculation unit 21 calculates the difference between the attitude of the camera that captured the captured image extracted from the environmental map storage unit 22 and the attitude of the camera that captured the captured image output from the camera device 31. The attitude difference of the camera that is calculated by the attitude difference calculation unit 21 will be described with reference to FIG.

[0044] In FIG. 6, it is assumed that the captured images used to generate the environmental map were captured using a smartphone. In FIG. 6, the capturing point 41 is the capturing point when the smartphone captured the image, and the capturing direction 42 indicates the capturing direction of the smartphone. The angle of view 43 indicates the angle of view of the camera mounted on the smartphone. The capturing point 51 is the capturing point when the image was captured by the image capturing device 31, and the capturing direction 52 indicates the capturing direction when the image capturing device 31 captured the captured image. The angle of view 53 indicates the angle of view of the image capturing device 31. The points in the captured image indicate feature points of the object.

[0045] 6 indicates the difference in attitude between the image capturing device 31 and the smartphone. Specifically, the attitude difference calculation unit 21 moves the image capturing point 41 to the image capturing point 51 while maintaining the image capturing direction. The attitude difference calculation unit 21 sets the matching image capturing point 41 and image capturing point 51 as the starting point, and determines the angle between the image capturing direction 42 and image capturing direction 52 as r1.

[0046] An example of the attitude difference of the image capturing device calculated by the attitude difference calculation unit 21, which is different from that shown in FIG. 6, will be described with reference to FIG. 7. FIG. 7 shows a reference line 61. The reference line 61 is a line indicating a predetermined direction in a predetermined space. FIG. 7 shows that the angle formed between the reference line 61 and the image capturing direction 42 is r2, and the angle formed between the reference line 61 and the image capturing direction 52 is r3. The attitude difference calculation unit 21 may calculate, for example, r2±r3 as the attitude difference.

[0047] The control unit 13 controls the driving unit 32 based on the attitude difference calculated by the attitude difference calculation unit 21 to change the operation of the mobile robot 30. Specifically, the control unit 13 may change a predetermined path plan for the mobile robot 30. For example, the path plan storage unit 23 stores a path plan in which a path for the mobile robot 30 is predetermined. The control unit 13 extracts information about the path plan from the path plan storage unit 23 and controls the driving unit 32 so that the mobile robot 30 moves along the path plan. Here, if the attitude difference exceeds a predetermined threshold, the control unit 13 may change the path plan stored in the path plan storage unit 23 and control the driving unit 32 so that the mobile robot 30 moves along the changed path plan. If the attitude difference exceeds a predetermined threshold, there is a possibility that the distance between the estimated position and the actual position of the mobile robot 30 is greater than a predetermined distance. Therefore, for example, when the posture difference exceeds a predetermined threshold, the control unit 13 may change the path plan to narrow the movement range of the mobile robot 30 in consideration of the safety of the mobile robot 30.

[0048] Furthermore, the control unit 13 may change the movement speed based on the posture difference calculated by the posture difference calculation unit 21. For example, when the posture difference exceeds a predetermined threshold, the control unit 13 may control the drive unit 32 to slow down the movement speed of the mobile robot 30 or to stop the mobile robot 30. Furthermore, the control unit 13 may use a plurality of thresholds with different values to control the drive unit 32 to, for example, slow down the movement speed as the posture difference increases. Alternatively, when the posture difference does not exceed the predetermined threshold, the control unit 13 may control the drive unit 32 to increase the movement speed of the mobile robot 30.

[0049] For example, the control unit 13 may use a step function, a sigmoid function, or another function to convert the attitude difference calculated by the attitude difference calculation unit 21 into the moving speed of the mobile robot 30. Furthermore, when the extraction unit 11 extracts a plurality of photographed images, for example, when the number of photographed images in which the attitude difference from the image capturing device 31 exceeds a threshold value exceeds a predetermined number, the control unit 13 may perform control such as slowing down the speed of the mobile robot 30.

[0050] Next, a flow of controlling the movement of the mobile robot 30 according to the second embodiment will be described with reference to Fig. 8. First, the extraction unit 11 acquires real-time images captured by the image capturing device 31 (S11). The real-time images refer to images captured by the image capturing device 31 while the mobile robot 30 is moving.

[0051] Next, the extraction unit 11 extracts or acquires, from the environmental map storage unit 22, photographed images similar to the real-time image and attitude information of the photographing device that captured the photographed images (S12). The extraction unit 11 acquires at least one photographed image similar to the real-time image and acquires attitude information associated with the acquired photographed image. The photographed image is the photographed image used to generate the environmental map.

[0052] Next, the estimation unit 12 estimates the attitude related to the real-time image (S13). Specifically, the estimation unit 12 estimates the position of the image capturing device 31 that captured the real-time image and the direction in which the image capturing device 31 is facing.

[0053] Next, the attitude difference calculation unit 21 calculates the difference between the attitude of the image capturing device 31 that captured the real-time image and the attitude associated with the captured image acquired in step S12 (S14).

[0054] Next, the control unit 13 controls the drive unit 32 based on the attitude difference calculated in step S14 (S15). For example, the control unit 13 may control the drive unit 32 to decelerate as the attitude difference increases, or may stop the drive unit 32 when the attitude difference exceeds a predetermined threshold. Alternatively, the control unit 13 may control the drive unit 32 to accelerate from its current speed when the attitude difference is less than the predetermined threshold.

[0055] As described above, the mobile robot 30 according to the second embodiment can control the driving unit 32 according to the difference between the posture of the image capturing device 31 while moving and the posture of the image capturing device that captured the images used to generate the environmental map.

[0056] 2 and 3, as the difference between the attitude related to the real-time image and the attitude related to the captured image used to generate the environmental map increases, the difference between the current position of the mobile robot 30 and the estimated position may also increase. Therefore, by controlling the operation of the mobile robot 30 in accordance with the attitude difference, the mobile robot 30 can achieve safe movement before the estimated value of its own position deviates significantly.

[0057] Furthermore, by slowing down the speed of the mobile robot 30 as the posture difference increases, the following effect is achieved. The mobile robot 30 estimates its own position using multiple real-time images captured by the image capture device 31 while moving. At this time, if the moving speed of the mobile robot 30 is slow, the overlapping area between the multiple real-time images captured by the image capture device 31 becomes larger. In other words, if the moving speed of the mobile robot 30 is slow, the number of common feature points between the real-time images becomes greater than when the moving speed is fast. This is because, when the moving speed of the mobile robot 30 is slow, the intervals between positions at which real-time images are captured become shorter than when the moving speed is fast, and it is possible to increase the number of real-time images containing the same object.

[0058] As a result, the mobile robot 30 can improve the accuracy of its self-location estimation by using similar real-time images, thereby enabling the mobile robot 30 to move more safely.

[0059] (Other embodiments) In other embodiments, the overall configuration of a mobile object control system including an information processing device 20 and hardware resources for realizing the information processing device 20 will be described. First, Fig. 9 shows a block diagram of a mobile object control system according to other embodiments. The mobile object control system shown in Fig. 9 is a schematic block diagram of the system. Fig. 9 shows an example of the configuration of a system in which a device equivalent to the information processing device 20 in Fig. 4 remotely controls a mobile robot 30.

[0060] The mobile object control system shown in FIG. 9 includes a remote control device 101, a network 102, an imaging device 103, and a mobile object 104. The remote control device 101 corresponds to the information processing device 20. The imaging device 103 corresponds to the photographing device 31. The mobile object 104 corresponds to the mobile robot 30 through which the information processing device 20 is viewed. The remote control device 101 includes a mobile object control unit 201 and a communication unit 202. The mobile object control unit 201 corresponds to the control unit 13 described above. The communication unit 202 is an interface that enables the mobile object control unit 201 to communicate with the mobile object 104 and the imaging device 103.

[0061] The network 102 is configured to enable mutual communication between the remote control device 101, the image capturing device 103, and the mobile object 104. The image capturing device 103 is, for example, a camera.

[0062] The moving body 104 is, for example, a mobile robot 30 to be controlled. The moving body 104 has a communication unit 301, a drive control unit 302, and a drive unit 303. The communication unit 301 is an interface that enables the moving body 104 to communicate with the remote control device 101. The drive control unit 302 gives an operation instruction to the drive unit 303 based on a first control input value given from the remote control device 101. The drive unit 303 is configured with a motor that drives drive wheels, etc.

[0063] Next, the hardware resources constituting the information processing device 20 will be described with reference to Fig. 10. The hardware resources shown in Fig. 10 are an example, and other components such as an input interface may also be included. In other words, the hardware resources 400 shown in Fig. 10 are not intended to limit the configuration shown in Fig. 10.

[0064] 10, the hardware resource 400 includes a processing unit 401, a memory 402, a network interface 403, and the like, which are interconnected by an internal bus 404. The processing unit 401 may be referred to as a processor.

[0065] The hardware resources 400 may include hardware (for example, an input / output interface) that is not shown. Alternatively, the number of units such as the arithmetic unit 401 included in the device is not limited to the example shown in Fig. 10, and for example, the device may include a plurality of arithmetic units 401. The arithmetic unit 401 may be, for example, a central processing unit (CPU), a microprocessor unit (MPU), or the like.

[0066] The memory 402 may be, for example, a random access memory (RAM), a read only memory (ROM), a hard disk drive (HDD), or a solid state drive (SSD).

[0067] The network interface 403 may be, for example, a LAN (Local Area Network) card, a network adapter, a network interface card, or the like.

[0068] 10, the memory 402 is used to store a group of software modules. The calculation unit 401 reads and executes the group of software modules from the memory 402, thereby performing the processing of the information processing device 20 and the like described in the above-described embodiment.

[0069] As explained using FIG. 10, each of the calculation units 401 possessed by the information processing device 20 etc. in the above-described embodiment executes one or more programs including a group of instructions for causing a computer to perform the algorithm explained using the drawings.

[0070] In the above example, the program can be stored and supplied to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). The program may also be supplied to a computer by various types of transitory computer-readable media. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer-readable media can supply the program to a computer via a wired communication path such as an electric wire or optical fiber, or via a wireless communication path.

[0071] The present disclosure is not limited to the above-described embodiment, and can be modified as appropriate within the scope of the present disclosure.

[0072] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes. (Appendix 1) an extracting unit that extracts, from a plurality of first photographed images taken by a first photographing device in order to create a three-dimensional image of a predetermined area, at least one first photographed image that is similar to a second photographed image taken by a second photographing device mounted on a mobile device moving within the area; an estimation unit that estimates a first photographing posture of the first photographing device that captured the extracted first photographed image and a second photographing posture of the second photographing device when the second photographed image was captured; a control unit that controls an operation of the moving device based on the first shooting attitude and the second shooting attitude. (Appendix 2) The control unit 2. The information processing device according to claim 1, wherein the speed of the moving device is changed based on the first shooting attitude and the second shooting attitude. (Appendix 3) the first photographing attitude is a first angle of a photographing direction of the first photographing device with respect to a specific direction, and the second photographing attitude is a second angle of a photographing direction of the second photographing device with respect to the specific direction; The control unit 3. The information processing device according to claim 1, wherein the speed of the mobile device is reduced when a difference between the first angle and the second angle exceeds a predetermined threshold. (Appendix 4) The control unit 4. The information processing device according to claim 3, wherein the speed of the mobile device is increased when a difference between the first angle and the second angle is below a predetermined threshold. (Appendix 5) the first photographing attitude is a first photographing direction of the first photographing device, and the second photographing attitude is a second photographing direction of the second photographing device; The control unit 3. The information processing device according to claim 1, wherein the speed of the moving device is reduced when an angle formed between the first imaging direction and the second imaging direction exceeds a predetermined threshold. (Appendix 6) The control unit 6. The information processing device according to claim 5, wherein the speed of the moving device is increased when an angle formed between the first imaging direction and the second imaging direction is below a predetermined threshold. (Appendix 7) 7. The information processing device according to claim 1, wherein the three-dimensional image is generated using SfM (Structure from Motion). (Appendix 8) a mobile device that autonomously moves within a predetermined area; a remote control device including: an extracting unit that extracts at least one first photographed image similar to a second photographed image taken by a second photographing device mounted on the mobile device moving through the area from among a plurality of first photographed images taken by a first photographing device to create a three-dimensional image of the area; an estimating unit that estimates a first photographing attitude of the first photographing device that captured the extracted first photographed image and a second photographing attitude of the second photographing device when the second photographed image was captured; and a control unit that determines an operation of the mobile device based on the first photographing attitude and the second photographing attitude; The moving device is A mobile object control system that operates based on operational instructions obtained from the remote control device via a network. (Appendix 9) The control unit 9. The mobile object control system according to claim 8, wherein the speed of the mobile device is changed based on the first shooting attitude and the second shooting attitude. (Appendix 10) extracting at least one first photographed image from a plurality of first photographed images taken by a first photographing device in order to create a three-dimensional image of a predetermined area, the first photographed image being similar to a second photographed image taken by a second photographing device mounted on a mobile device moving within the area; estimating a first photographing posture of the first photographing device that photographed the extracted first photographed image and a second photographing posture of the second photographing device when the second photographed image was photographed; A control method executed in an information processing device, which controls the operation of the mobile device based on the first shooting attitude and the second shooting attitude. (Appendix 11) extracting at least one first photographed image from a plurality of first photographed images taken by a first photographing device in order to create a three-dimensional image of a predetermined area, the first photographed image being similar to a second photographed image taken by a second photographing device mounted on a mobile device moving within the area; estimating a first photographing posture of the first photographing device that photographed the extracted first photographed image and a second photographing posture of the second photographing device when the second photographed image was photographed; a program for causing a computer to execute the following: controlling the operation of the moving device based on the first shooting attitude and the second shooting attitude; [Explanation of symbols]

[0073] 10. Information processing equipment 11 Extraction part 12 Estimation part 13 Control Unit 20 Information processing equipment 21 Posture difference calculation section 22 Environmental map storage section 23 Route plan storage unit 30 Mobile Robot 31 Imaging equipment 32 Drive unit 41 Shooting Location 42 Shooting direction 43 Angle of View 51 Shooting location 52 Shooting direction 53 Angle of View 61 Reference Line 101 Remote control device 102 Network 103 Imaging device 104 Mobile 201 Mobile control unit 202 Communications Department 301 Communications Department 302 Drive control unit 303 Drive unit 400 Hardware Resources 401 Arithmetic section 402 memory 403 Network Interface 404 Internal Bus

Claims

1. an extracting unit that extracts, from a plurality of first photographed images taken by a first photographing device in order to create a three-dimensional image of a predetermined area, at least one first photographed image that is similar to a second photographed image taken by a second photographing device mounted on a mobile device moving within the area; an estimation unit that estimates a first photographing posture of the first photographing device that captured the extracted first photographed image and a second photographing posture of the second photographing device when the second photographed image was captured; and a control unit that, when the distance between the estimated position and the actual position of the mobile device is greater than a predetermined distance and the attitude difference between the first shooting attitude and the second shooting attitude exceeds a predetermined threshold, modifies a predetermined route plan for the mobile device and controls the operation of the mobile device to move along the modified route plan.

2. The control unit The information processing apparatus according to claim 1 , wherein the speed of the moving device is changed based on the first shooting attitude and the second shooting attitude.

3. the first imaging attitude is a first angle of an imaging direction of the first imaging device with respect to a specific direction, and the second imaging attitude is a second angle of an imaging direction of the second imaging device with respect to the specific direction; The control unit The information processing device according to claim 1 , wherein the speed of the mobile device is reduced when the difference between the first angle and the second angle exceeds a predetermined threshold value.

4. The control unit The information processing device according to claim 3 , wherein the speed of the mobile device is increased when the difference between the first angle and the second angle is below a predetermined threshold.

5. the first imaging posture is a first imaging direction of the first imaging device, and the second imaging posture is a second imaging direction of the second imaging device; The control unit The information processing device according to claim 1 , wherein the speed of the mobile device is reduced when the angle between the first imaging direction and the second imaging direction exceeds a predetermined threshold value.

6. The control unit The information processing device according to claim 5 , wherein the speed of the moving device is increased when the angle formed between the first imaging direction and the second imaging direction is below a predetermined threshold.

7. The information processing device according to claim 1 , wherein the three-dimensional image is generated using Structure from Motion (SfM).

8. a mobile device that autonomously moves within a predetermined area; a remote control device including: an extraction unit that extracts at least one first photographed image similar to a second photographed image taken by a second photographing device mounted on the mobile device moving through the area from among a plurality of first photographed images taken by a first photographing device to create a three-dimensional image of the area; an estimation unit that estimates a first photographing attitude of the first photographing device that captured the extracted first photographed image and a second photographing attitude of the second photographing device when the second photographed image was captured; and a control unit that, when a distance between the estimated position and the actual position of the mobile device is a predetermined distance or more and therefore an attitude difference between the first photographing attitude and the second photographing attitude exceeds a predetermined threshold, modifies a predetermined route plan for the mobile device and determines an operation of the mobile device to move along the modified route plan; The moving device is A mobile object control system that operates based on operational instructions obtained from the remote control device via a network.

9. extracting at least one first photographed image from a plurality of first photographed images taken by a first photographing device in order to create a three-dimensional image of a predetermined area, the first photographed image being similar to a second photographed image taken by a second photographing device mounted on a mobile device moving within the area; estimating a first photographing posture of the first photographing device that photographed the extracted first photographed image and a second photographing posture of the second photographing device when the second photographed image was photographed; A control method executed on an information processing device, which, when the distance between the estimated position and the actual position of the mobile device is a predetermined distance or more and therefore the attitude difference between the first shooting attitude and the second shooting attitude exceeds a predetermined threshold, modifies a predetermined route plan for the mobile device and controls the operation of the mobile device to move along the modified route plan.

10. extracting at least one first photographed image from a plurality of first photographed images taken by a first photographing device in order to create a three-dimensional image of a predetermined area, the first photographed image being similar to a second photographed image taken by a second photographing device mounted on a mobile device moving within the area; estimating a first photographing posture of the first photographing device that photographed the extracted first photographed image and a second photographing posture of the second photographing device when the second photographed image was photographed; A program that causes a computer to execute the following steps: when the distance between the estimated position and the actual position of the mobile device is greater than a predetermined distance and the attitude difference between the first shooting attitude and the second shooting attitude exceeds a predetermined threshold, modifying a predetermined route plan for the mobile device and controlling the operation of the mobile device so that the mobile device moves along the modified route plan.

Images