Information processing device, information processing method, and computer program

Abstract

To provide an information processing device capable of notifying of information on an object serving as a change factor of accuracy in self-location estimation.SOLUTION: An information processing device comprises: self-location estimation means for estimating a self-location on the basis of an image; accuracy determination means for determining accuracy in self-location estimation; image analysis means for detecting an object from the image; and notification means for notifying of information on the object detected by the image analysis means when accuracy is out of a prescribed range.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, a computer program, and the like that perform self-position estimation. [Background technology]

[0002] For example, when a self-propelled robot or the like is made to travel within an environment such as a factory, in order to stably control the movement of the moving body, a method is known in which the amount of movement is controlled based on a self-position created from image information, as in Patent Document 1. Also, Patent Document 2 is known as a technology for providing information that assists in the calculation accuracy of the self-position based on features in an image captured by an imaging device. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-125345 [Patent Document 2] Patent No. 6723798 Summary of the Invention [Problem to be solved by the invention]

[0004] However, these conventional techniques have the problem that it is difficult to grasp the cause of a change in the accuracy of self-location estimation.

[0005] The present invention has been made in view of the above-mentioned problems, and one of its objects is to provide an information processing device that can notify information about an object that causes a change in the accuracy of self-location estimation. [Means for solving the problem]

[0006] The information processing device according to the present invention comprises: a self-position estimation means for estimating a self-position based on an image; an accuracy determination means for determining accuracy in estimating the self-position; image analysis means for detecting an object from the image; If the accuracy is outside a predetermined range, a notification means for notifying the user of the information; death, The notification means notifies the image, the feature points detected from the image, the accuracy, and information indicating the object. It is characterized by: [Effects of the Invention]

[0007] According to the present invention, it is possible to realize an information processing device that can notify information about an object that causes a change in the accuracy of self-location estimation. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram showing a system configuration according to an embodiment of the present invention; [Figure 2] 1 is a functional block diagram illustrating an example of the configuration of an information processing device according to an embodiment of the present invention. [Figure 3] 10 is a flowchart illustrating an example of a processing flow of an information processing apparatus according to an embodiment of the present invention. [Figure 4] FIG. 10 is a diagram showing an example of displaying the results of image analysis according to the embodiment of the present invention. [Figure 5] FIG. 10 is a diagram showing an example of displaying the results of image analysis on a map according to an embodiment of the present invention. [Figure 6] 1 is a block diagram illustrating an example of a hardware configuration of an information processing apparatus according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiments. In each drawing, the same members or elements are designated by the same reference numerals, and duplicated descriptions will be omitted or simplified.

[0010] In this embodiment, a method for controlling the movement of a mobile body such as an automated guided vehicle (AGV) or an autonomous mobile robot (AMR), particularly a route setting method for automatic driving, will be described. Note that, although the following description will be given using an AGV as an example of a mobile body, the mobile body may also be an AMR or a service mobile robot (SMR).

[0011] 1 is a diagram showing a system configuration according to an embodiment of the present invention. An information management system 100 in this embodiment is composed of multiple mobile objects 101 (101-1, 101-2, ...), a process management system 103, and a mobile object management system 102. The information management system 100 is a logistics system, a production system, or the like.

[0012] A plurality of moving bodies 101 (101-1, 101-2, ...) are guided vehicles (AGVs) that transport objects according to a process schedule determined by a process control system 103. A plurality of moving bodies are moving (traveling) within the environment.

[0013] The process control system 103 manages the processes executed by the information control system 100. For example, it is an MES (Manufacturing Execution System) that manages processes in a factory or a logistics warehouse. It communicates with the mobile object control system 102.

[0014] The mobile object management system 102 is a system for managing mobile objects, and communicates with a process management system 103. It also communicates with the mobile object 101 (for example, via Wi-Fi communication) to send and receive operation information in both directions.

[0015] Fig. 2 is a functional block diagram showing an example of the configuration of an information processing device according to an embodiment of the present invention. Note that some of the functional blocks shown in Fig. 2 are realized by causing a CPU or the like serving as a computer included in the information processing device to execute a computer program stored in a memory serving as a storage medium.

[0016] However, some or all of these functions may be implemented by hardware. Examples of hardware that can be used include dedicated circuits (ASICs) and processors (reconfigurable processors, DSPs). Furthermore, the functional blocks shown in Figure 2 do not have to be built into the same housing, and may be configured as separate devices connected to each other via signal paths.

[0017] The information processing device 200 has a self-position estimation unit 201, an image analysis unit 202, and a notification unit 205. The information processing device 200 is also connected to an image acquisition unit 203 that acquires images from an imaging device such as a camera, and a sensor information acquisition unit 204 that acquires information from a distance sensor, an acceleration sensor, an infrared sensor, etc. The information processing device 200 is further connected to an input device 206 that accepts user operations such as a mouse, keyboard, or touch panel, and an output device 209 that outputs notified content.

[0018] The travel control unit 208 acquires position information 207 from the self-position estimation unit 201 and controls the mobile object. The information processing device 200 can be mounted on the mobile object 101. The information processing device 200 can also be a device that communicates with the mobile object 101 via Wi-Fi or the like using a network I / F 603 described later. When the information processing device 200 receives data from the outside, it is physically connected to the outside via a bus or the like, or the network I / F 603 is used.

[0019] The self-position estimation unit 201 has a configuration of SLAM (Simultaneous Localization and Mapping) that estimates the self-position while creating a map based on the image acquired by the image acquisition unit 203. Here, the self-position estimation unit 201 estimates the self-position based on the image.

[0020] Many SLAM methods have been proposed and can be used. For example, a method can be used in which feature values ​​obtained from images taken from multiple different viewpoints using algorithms such as SIFT or ORB are saved as keyframes and then matched to estimate the self-localization.

[0021] SIFT stands for Scale-Invariant Feature Transform, and ORB stands for Oriented FAST and Rotated BRIEF. Alternatively, a method may be used in which the brightness values ​​of images taken from multiple different viewpoints are compared to estimate the self-position so as to minimize the error.

[0022] In addition, there are well-known algorithms such as the RGB-D SLAM algorithm, which generates a map while tracking the depth of feature points detected from an RGB image as depth values ​​of a depth sensor, and these may also be used.Furthermore, a position estimation method using a deep learning-based technique may also be used.

[0023] The image analysis unit 202 collects features (Haar-Like features, HOG (Histograms of Oriented Gradients) features, etc.) of training images and performs training using a classifier such as a discriminator. The results of the training are saved as a detector dictionary and used during detection. The likelihood of detection is calculated as a detection score. For example, a weighted sum of the outputs can be output as the detection score.

[0024] The higher the detection score, the more likely the output obtained by each detector is to be the detected object that it is targeting. For example, one method for detecting humans is to use a classifier trained by a statistical learning method to extract local features using sample images of humans. Note that the method for detecting and counting humans is not limited to the above algorithm, and any algorithm may be used.

[0025] If the AGV is equipped with sensors such as inertial sensors such as a gyroscope or an IMU (Inertial Measurement Unit), or an encoder for acquiring the amount of tire rotation, the sensor information acquisition unit 204 outputs the sensor values. The self-position estimation unit 201 can also calculate the self-position of the image acquisition unit 203 by using both the image and the above-mentioned sensor values. In this way, by using the image information of the image acquisition unit 203 and the sensor information together, the self-position can be calculated with high accuracy and robustness.

[0026] The accuracy of these SLAMs may be a value that can be calculated using features, scores, etc. that can be obtained by an algorithm, and is not limited to a specific calculation method.

[0027] Next, Fig. 3 is a flowchart showing an example of a processing flow of an information processing device according to an embodiment of the present invention. Note that the operation of each step in the flowchart of Fig. 3 is performed by a CPU or the like serving as a computer in the information processing device executing a computer program stored in memory.

[0028] In step S301 (self-location estimation step), the self-location estimation unit 201 acquires an image from the image acquisition unit 203 and estimates the self-location based on the image. Next, in step S302, it is determined that the number of feature points in the estimation of the self-location is insufficient. That is, if the number of feature points in the estimation of the self-location is equal to or less than a predetermined number, it is determined that the accuracy of the estimation of the self-location is lower than a predetermined range.

[0029] For example, if the number of feature points contained in the image is below the first threshold (predetermined number) of 200, that is, if the answer to step S302 is Yes, the process proceeds to step S303. In step S303, image analysis is performed to detect objects.

[0030] In step S302, the image may be divided into a plurality of regions, and the determination criterion may be whether or not a feature point is contained in each region. The image may be divided into a grid or concentric circles, and the shape of the division is not limited. By dividing the image and making a determination, if there are a lack of feature points on the image even in cases where feature points are present locally, it is determined that objects with few features, such as walls or luggage, occupy the majority of the image.

[0031] Therefore, in step S303, image analysis is performed to detect an object, and in step S304, information about the object detected in step S303 is displayed, and the user is notified that the lack of feature points is the cause of the decrease in accuracy. The display device that displays (notifies) the information can be provided within the information processing device 200, or an external display device (output device 209) can be used. The notification unit 205 notifies the content of the notification (display). The input device 206 can also function as an input / output device such as a tablet.

[0032] Next, in step S305, it is determined whether the likelihood of the feature point is lower than a predetermined second threshold. For example, an example will be described in which the accuracy is normalized to 100 levels. When the arbitrarily determined second threshold is set to 40, if the likelihood falls below the second threshold of 40, the determination in step S305 is Yes, and the process proceeds to step S306.

[0033] Here, steps S302 and S305 function as an accuracy determination step (accuracy determination means) for determining the accuracy of the self-position estimation, and steps S303 and S306 function as an image analysis step (image analysis means) for detecting an object from an image.

[0034] Furthermore, steps S304 and S307 function as a notification step (notification means) for notifying information about an object detected by the image analysis step when the accuracy of self-position estimation is outside a predetermined range.

[0035] In step S306, image analysis is performed to detect an object. Furthermore, in step S307, information about the object detected in step S306 is displayed, and a display (notification) is given to the user that low likelihood of feature points is the cause of the decrease in accuracy. As described above, the display device that performs the display (notification) can be provided within the information processing device 200, or an external display device (output device 209) can be used. The notification unit 205 notifies the content of the notification (display). The input device 206 can also function as an input / output device such as a tablet.

[0036] As described above, in this embodiment, in step S305, it is determined whether the likelihood of the feature points is lower than a predetermined threshold. That is, if the likelihood of the feature points in the estimation of the self-position is equal to or lower than the predetermined threshold, it is determined that the accuracy is lower than a predetermined range. If the result in step S305 is Yes, image analysis is performed to detect an object, and information about the object can be displayed as a factor in the decrease in accuracy.

[0037] To reduce the number of analyses, the analysis may be performed at the following timing. For example, the analysis may be performed at a position close to the position where the key frame recording the feature amount of the 3D point cloud exists during self-location estimation. Alternatively, the analysis may be performed at an arbitrary interval, such as every second.

[0038] By doing this, it is possible to suppress the frequent implementation of image analysis and reduce the resources used by the information processing device. Because equipment such as AGVs operates in an environment with limitations such as battery power, by avoiding unnecessary calculations, it is possible to operate for a longer period of time.

[0039] In this way, when the accuracy of self-location estimation (lack of feature points or likelihood of feature points) is low, information on objects detected by image analysis is displayed as a factor that reduces the accuracy of self-location estimation. Therefore, the user can take measures to address the factors that reduce the accuracy of self-location estimation based on the displayed information.

[0040] Conversely to the above description, an instruction to perform image analysis may be issued when the number of feature points exceeds a third threshold that is greater than the first threshold, or when the likelihood of the feature points exceeds a fourth threshold that is greater than the second threshold. In other words, when the accuracy of self-location estimation is higher than a predetermined range, this may be notified.

[0041] When image analysis detects markers such as posters or tags, the system presents them as factors that have improved the user's position, allowing the user to confirm that the posters, tags, etc. have improved accuracy.

[0042] Next, Fig. 4 is a diagram showing an example of displaying the results of image analysis according to an embodiment of the present invention. Using Fig. 4, a method of displaying the image acquired by the image acquisition unit 203, the self-position calculated by the self-position estimation unit 201, and the detection results of the object detected by the image analysis unit 202 will be described. The contents of Fig. 4 are displayed on the output device 209. As mentioned above, the input device 206 may function as an input / output device such as a tablet at this time.

[0043] Reference numeral 401 denotes feature points detected by the self-location estimation unit 201. Reference numeral 402 displays the current accuracy value of the self-location estimation by the self-location estimation unit 201. A human body 403 and a cardboard box 404 indicate the results detected by the image analysis unit 202.

[0044] The correspondence list 405 lists countermeasures (instructions) corresponding to objects detected by image analysis. For example, if the detected object is a person as a result of image analysis, instructions such as "Self-location estimation in progress" or "Please move (since self-location estimation is in progress)" may be given to the person by voice or image display. The voice and image are output by the output device 209. As described above, the input device 206 can also function as an input / output device such as a tablet in this case.

[0045] Furthermore, for example, when a static object that is easily movable, such as a cardboard box, is detected by image analysis, it is necessary to determine whether to move it, so instructions such as "Please move (if necessary)" may be given by voice or image display. The voice and image are output by the output device 209. As described above, in this case, the input device 206 can also function as an input / output device such as a tablet.

[0046] Furthermore, if a location with few features, such as a wall, is detected by image analysis, instructions such as "Please increase the number of features by attaching (posters, tags, etc.)" may be given by voice or image display. In other words, if the accuracy is lower than a predetermined range, a countermeasure to improve accuracy is notified based on the object detected by the image analysis means. In this way, by providing a storage means for storing notification content linked to the detected object, storing a correspondence list 405 in the storage means, and presenting the correspondence to the user, the accuracy of self-location estimation can be improved.

[0047] Next, Fig. 5 is a diagram showing an example of displaying the results of image analysis on a map according to an embodiment of the present invention. Note that, although this embodiment will be described using a window for presenting two-dimensional map information, three-dimensional map information may also be presented. The contents of Fig. 5 are displayed on the output device 209. As mentioned above, the input device 206 may also function as an input / output device such as a tablet.

[0048] 5 shows an example of a map 500 held by the self-position estimation unit 201, but the self-position of the AGV may also be synthesized onto the map based on the self-position of the self-position estimation unit 201. Reference numeral 501 indicates a lack of feature points at feature point A, and reference numeral 505 is a cardboard box icon that is displayed when a cardboard box is detected as an object detection result.

[0049] Similarly, 504 indicates that there are insufficient feature points at feature point D, and 507 is a cardboard box icon that is displayed when a cardboard box is detected as the object detection result. 506 indicates that the feature points at feature point B have changed by more than a predetermined number, and 506 is a person icon that is displayed when a person is detected as the object detection result. 503 indicates that there are sufficient feature points at feature point C.

[0050] That is, when the accuracy of self-location estimation is lower than a predetermined range as in 501, 502, and 504, the notification means notifies the cause (insufficient number of feature points, change in feature points, etc.) based on the object detected by the image analysis means. The notification means may also notify the cause of low accuracy of self-location estimation, such as low likelihood of feature points. Note that a message such as "Please move" may be displayed in 501, 502, and 504 as a countermeasure.

[0051] Reference numeral 508 denotes an accuracy value list, which displays the accuracy values ​​of the self-position at feature points A to D. In this way, by presenting the map, the AGV's position, the object detection results, and the route on the map created by SLAM, the user can easily understand the status of the self-position accuracy, its causes, countermeasures, etc. The notification means mentioned above may also notify the value related to the accuracy of the self-position estimation as shown in the accuracy value list 508.

[0052] In this embodiment, the moving body is not limited to an AGV (Automated Guided Vehicle). For example, the moving body may be a self-driving car or an autonomous mobile robot, and the movement control described in this embodiment may be applied to them.

[0053] 6 is a block diagram showing an example of the hardware configuration of an information processing device according to an embodiment of the present invention, and as shown in Fig. 6, the information processing device 200 includes a CPU 600, a main memory device 601, an auxiliary memory device 602, and a network I / F 603. Also, an image acquisition unit 203, a sensor information acquisition unit 204, an input device 206, and a driving control unit 208 are connected to, for example, a bus of the information processing device.

[0054] The CPU 600 of the information processing device 200 executes processing using computer programs and data stored in the main memory device 601. As a result, the CPU 600 controls the overall operation of the information processing device 200 and also executes or controls each of the above-mentioned processes executed by the information processing device.

[0055] For example, the CPU 600 executes processing using computer programs and data stored in the main storage device 601, thereby realizing the operations of the flowchart shown in FIG.

[0056] The main memory device 601 is a memory device such as a RAM (Random Access Memory). The main memory device 601 stores computer programs and data loaded from the auxiliary memory device 602. The main memory device 601 also has an area for storing captured images acquired by the image acquisition unit 203 and various data received from the driving control unit 208 via the network I / F 603.

[0057] Furthermore, the main memory device 601 has a work area that is used when the CPU 600 executes various processes. In this way, the main memory device 601 can provide various areas as needed.

[0058] The auxiliary storage device 602 is a large-capacity information storage device such as a hard disk drive (HDD), a read-only memory (ROM), or a solid-state drive (SSD).

[0059] The auxiliary storage device 602 stores an OS (operating system) and computer programs and data for causing the CPU 600 to execute or control the processes described above as those performed by the information processing device. The auxiliary storage device 602 also stores data (e.g., the above-mentioned imaging parameters) received from the driving control unit 208 and the like via the network I / F 603.

[0060] The computer programs and data stored in the auxiliary storage device 602 are loaded into the main storage device 601 as appropriate under the control of the CPU 600, and are processed by the CPU 600. The network I / F 603 is an interface used by the information processing device 200 to communicate data with the driving control unit 208 via a network.

[0061] Although the present invention has been described in detail above based on the preferred embodiments, the present invention is not limited to the above embodiments, and various modifications are possible based on the spirit of the present invention, and these modifications are not excluded from the scope of the present invention. The above embodiments include the following combinations.

[0062] (Configuration 1) An information processing device characterized by having a self-position estimation means for estimating a self-position based on an image, an accuracy determination means for determining the accuracy of the estimation of the self-position, an image analysis means for detecting an object from the image, and a notification means for notifying information about the object detected by the image analysis means when the accuracy is outside a predetermined range.

[0063] (Configuration 2) The information processing device according to configuration 1, wherein the notification means notifies the cause based on an object detected by the image analysis means when the accuracy is lower than the predetermined range.

[0064] (Configuration 3) The information processing device according to configuration 1 or 2, characterized in that the notification means notifies of countermeasures to improve the accuracy based on the object detected by the image analysis means when the accuracy is lower than the predetermined range.

[0065] (Configuration 4) The information processing device according to any one of configurations 1 to 3, wherein the notification means issues a notification when the accuracy is higher than the predetermined range.

[0066] (Configuration 5) The information processing device according to any one of configurations 1 to 3, wherein the notification means notifies the value relating to the accuracy.

[0067] (Configuration 6) The information processing device according to any one of configurations 1 to 4, wherein the notification means has storage means for storing notification content linked to the object.

[0068] (Configuration 7) The information processing device according to any one of configurations 1 to 6, wherein the self-position estimation means creates a map based on the image.

[0069] (Configuration 8) The information processing device described in any one of configurations 1 to 7, characterized in that the notification means determines that the accuracy is lower than the predetermined range when the number of feature points in the estimation of the self-position is less than a predetermined number.

[0070] (Configuration 9) The information processing device described in any one of configurations 1 to 8, characterized in that the notification means determines that the accuracy is lower than the predetermined range when the likelihood of a feature point in the estimation of the self-position is below a predetermined threshold.

[0071] (Method) An information processing method comprising a self-position estimation process for estimating a self-position based on an image, an accuracy determination process for determining the accuracy of the self-position estimation, an image analysis process for detecting an object from the image, and a notification process for notifying information regarding the object detected by the image analysis process if the accuracy is outside a predetermined range.

[0072] (Program) A computer program for controlling each means in the information processing device according to any one of configurations 1 to 9 by a computer.

[0073] In order to realize part or all of the control in the above-described embodiments, a computer program that realizes the functions of the above-described embodiments may be supplied to an information processing device or the like via a network or various storage media. Then, a computer (or a CPU, MPU, or the like) in the information processing device or the like may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0074] 100: Information Management System 101: Mobile 102: Mobile management system 103: Process control system 200: Information processing device 201: Self-position estimation part 202: Image analysis unit 203: Image acquisition unit 204: Sensor information acquisition unit 206: Input device 208: Driving control unit

Claims

1. a self-position estimation means for estimating a self-position based on an image; an accuracy determination means for determining accuracy in estimating the self-position; image analysis means for detecting an object from the image; a notification means for notifying information about the object detected by the image analysis means when the accuracy is outside a predetermined range, The information processing device is characterized in that the notification means notifies by displaying the image, the feature points detected from the image, the accuracy, and information indicating the object.

2. 2. The information processing apparatus according to claim 1, wherein the notification means notifies a cause based on an object detected by the image analysis means when the accuracy is lower than the predetermined range.

3. 2 . The information processing apparatus according to claim 1 , wherein the notification means notifies the user of countermeasures for improving the accuracy based on the object detected by the image analysis means when the accuracy is lower than the predetermined range.

4. 2. The information processing apparatus according to claim 1, wherein the notification means issues a notification when the accuracy is higher than the predetermined range.

5. 2. The information processing apparatus according to claim 1, wherein the notification means notifies a value relating to the accuracy.

6. 2. The information processing apparatus according to claim 1, wherein the notification means includes storage means for storing notification content associated with the object.

7. 2. The information processing apparatus according to claim 1, wherein the self-position estimation means creates a map based on the image.

8. 2. The information processing apparatus according to claim 1, wherein the notification means determines that the accuracy is lower than the predetermined range when the number of feature points in the estimation of the self-location is equal to or less than a predetermined number.

9. The information processing apparatus according to claim 1 , wherein the notification means determines that the accuracy is lower than the predetermined range when the likelihood of the feature points in the estimation of the self-location is equal to or less than a predetermined threshold value.

10. a self-location estimation step of estimating a self-location based on an image; an accuracy determination step of determining accuracy in the estimation of the self-location; an image analysis step of detecting an object from the image; a notification step of notifying information about the object detected by the image analysis step when the accuracy is outside a predetermined range, The information processing method is characterized in that the notification step notifies by displaying the image, the feature points detected from the image, the accuracy, and information indicating the object.

11. A computer program for controlling each means in the information processing device according to any one of claims 1 to 9 by a computer.

Images