Information processing device, method, and program

Abstract

To make it possible for an autonomous moving body to perform robust traveling even in an environment where reflection is present.SOLUTION: An information processing device comprises: image input means for acquiring image information photographed by an imaging device; detection means for detecting feature point information from the acquired image; determination means for determining whether or not the detected feature point information is feature point information due to reflection; control means for setting feature point information in which an influence of the feature point information due to the reflection is reduced from the detected feature point information as control feature point information when the determination means determines that the detected feature point information is the feature point information due to the reflection; position attitude estimation means for estimating a position attitude of the imaging device by using the control feature point information; and movement control means for controlling a movement of a moving body on which the imaging device is disposed, based on the estimated position attitude of the imaging device.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing device, a method, and a program. [Background technology]

[0002] Conventionally, there is known a mobile body that acquires its own position and orientation using a sensor such as a camera, and further moves autonomously using map data of the surrounding environment. Such an autonomous mobile body moves autonomously by matching the detected feature points of the surrounding environment with the map data and estimating the position and orientation of the autonomous mobile body.

[0003] However, depending on the location where the vehicle autonomously moves, there is a possibility that reflections may occur on the floor surface. Patent Document 1 discloses a method for detecting areas of road surface reflections caused by headlights from an input image, excluding those areas before processing, and thus accurately detecting the vehicle even when the road surface reflections caused by headlights are united and spread out. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2007-265016 A Summary of the Invention [Problem to be solved by the invention]

[0005] Here, reflection will be described. When reflection is present, a moving object may erroneously detect a feature point in the reflection. For example, a case where reflection is present on the floor surface will be described with reference to Figs. 6(A), 6(B), 6(C) and 6(D).

[0006] Fig. 6(A) is a diagram showing a moving object 501 traveling. The moving object 501 has an imaging device 502. The moving object 501 travels through a passageway whose floor surface has no pattern while being imaged by the imaging device 502. Lighting 503 is disposed on the ceiling. Reflection 504 is the light from the lighting 503 reflected on the floor surface and reflected in the image captured by the imaging device 502. Note that walls are not shown in Figs. 6(A), 6(B), 6(C), and 6(D).

[0007] Fig. 6(B) shows an example of feature points detected from an image captured by the imaging device 502 in the situation of Fig. 6(A). Feature point 505 is a feature point on the wall on the left side of the moving object 501. Feature point 506 is a feature point on the wall on the right side of the moving object 501. Feature point 507 is a feature point due to reflection 504 on the floor surface in Fig. 6(A).

[0008] Fig. 6(C) is a diagram showing a state when the moving object 501 advances to the position of the reflection 504 on the floor surface in Fig. 6(A). Reflection 508 is the reflection of the lighting 503 on the floor surface and reflected in the image captured by the imaging device 502. Reflection 508 is the reflection on the floor surface when the moving object 501 advances to the position of the reflection 504 on the floor surface in Fig. 6(A). Note that the same members or elements as those in Fig. 6(A) are given the same reference numerals and their description will be omitted.

[0009] Fig. 6(D) shows an example of feature points detected from an image captured by the imaging device 502 in the situation of Fig. 6(C). Feature point 509 is a feature point on the wall on the left side of the moving object 501. Feature point 506 is a feature point on the wall on the right side of the moving object 501. Because the moving object 501 has moved forward, feature point 506 is located closer to the moving object feature point than in Fig. 6(B). Feature point 510 is a feature point caused by reflection 508 of the floor surface in Fig. 6(C).

[0010] Since there is no pattern on the floor surface, feature points should not be detected. However, as can be seen by referring to FIG. 6(B) and FIG. 6(D), feature points due to reflections are detected. This detection of feature points due to reflections is a false detection. Furthermore, feature points due to reflections continue to exist in the forward direction of the moving body even when the moving body moves forward. In such a case, the false detection may affect self-location estimation. For example, there is a possibility that a problem will occur in which the false detection part is detected as an obstacle and the desired operation is not performed during autonomous driving.

[0011] In self-location estimation, if features detected from a floor with reflections are deemed to be unreliable and are not used at all, features such as the floor pattern will also not be used, which is likely to reduce the accuracy of self-location estimation in places with few features in the surrounding area.

[0012] Therefore, it is desirable to determine whether the detected feature points are due to reflection or not, but the method disclosed in Patent Document 1 cannot be applied to reflections due to ceiling lighting, for example. For this reason, conventionally, an autonomous moving body may not be able to travel robustly in an environment with reflections.

[0013] Therefore, an object of the present invention is to enable an autonomous moving body to travel robustly even in an environment where glare is present. [Means for solving the problem]

[0014] An information processing device according to one embodiment of the present invention has an image input means for acquiring image information captured by an imaging device, a detection means for detecting feature point information from the acquired image, a determination means for determining whether the detected feature point information is feature point information due to reflection, a control means for, when the determination means determines that the detected feature point information is feature point information due to reflection, setting feature point information obtained by reducing the influence of the feature point information due to reflection from the detected feature point information as controlled feature point information, a position and orientation estimation means for estimating a position and orientation of the imaging device using the control feature point information, and a movement control means for controlling movement of a moving body in which the imaging device is placed based on the estimated position and orientation of the imaging device. Effect of the Invention

[0015] According to the present invention, it is possible to enable an autonomous moving body to travel robustly even in an environment where glare is present. [Brief description of the drawings]

[0016] [Figure 1] 1 is a diagram showing a configuration of an autonomous moving body according to an embodiment of the present invention; [Diagram 2] 1 is a block diagram showing a functional configuration of an information processing device according to an embodiment of the present invention; [Diagram 3] 1 is a diagram illustrating a hardware configuration of an information processing device according to an embodiment of the present invention. [Figure 4] 4 is a flowchart illustrating a process executed by an information processing apparatus according to an embodiment of the present invention. [Diagram 5] 4 is a flowchart illustrating a process executed by an information processing apparatus according to an embodiment of the present invention. [Figure 6] 11A and 11B are diagrams illustrating erroneous detection of feature points due to reflections; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In this embodiment, an autonomous moving body that moves autonomously with an imaging device and an information processing device arranged therein will be described. FIG. 1 is a diagram showing the configuration of an autonomous moving body according to an embodiment of the present invention. A moving body 10 according to this embodiment has an imaging device 11, an information processing device 12, and a moving body control device 13. The moving body 10 is an example of an autonomous moving body.

[0018] The mobile object 10 is, for example, an autonomous mobile robot or an automated guided vehicle (AGV) that performs predetermined operations such as cleaning and transporting luggage in the interior space of a building such as an office building. AGV is an abbreviation for Automatic Guided Vehicle.

[0019] The moving body 10 is provided with an imaging device 11 as a sensor. The moving body 10 acquires an image captured by the imaging device 11 as sensor data. The moving body 10 autonomously moves through space by grasping the space around itself through self-position and orientation measurement such as SLAM using the sensor data from the imaging device 11. SLAM is an abbreviation for Simultaneous Localization And Mapping. The information processing device 12 executes various processes such as SLAM processing using the captured image of the imaging device 11.

[0020] 2 is a block diagram showing the functional configuration of the information processing device 12. The information processing device 12 includes an image input unit 121, a detection unit 122, a determination unit 123, a control unit 124, a map information storage unit 125, a position and orientation estimation unit 126, a route information storage unit 127, and a movement control unit 128.

[0021] The imaging device 11 is, for example, a stereo camera. The imaging device 11 captures an image of the surroundings of the moving object 10 to obtain a stereo image. The imaging device 11 outputs the obtained stereo image to the information processing device 12.

[0022] The image input unit 121 acquires images captured by the imaging device 11 in time series (for example, 60 frames per second) and outputs the acquired images to the detection unit 122.

[0023] The detection unit 122 detects feature points on the image from the image input unit 121, and further performs stereo measurement on the detected feature points to estimate the three-dimensional positions of the feature points. Note that the detection unit 122 detects locations in the image where there is a large change in the brightness gradient, such as edges and corners, as feature points.

[0024] Furthermore, the detection unit 122 outputs feature point information to the determination unit 123. The feature point information here refers to the position (two-dimensional coordinates) of the feature point on the image and the three-dimensional position of the feature point. The three-dimensional position of the feature point refers to the three-dimensional coordinates (X, Y, Z) in the world coordinate system. This feature point information will be referred to as "frame feature point information" below. The detection unit 122 also outputs the frame feature point information to a storage unit (not shown).

[0025] Determination unit 123 determines whether or not each of the feature points included in the feature point information from detection unit 122 is a feature point caused by reflection. This determination method will be described later with reference to Figs. 4 and 5. Determination unit 123 also outputs the determination result of determination unit 123 and the feature point information from detection unit 122 to control unit 124. The determination result here is feature point information caused by reflection, and is hereinafter referred to as "reflection feature point information".

[0026] Control unit 124 acquires frame feature point information and reflection feature point information from determination unit 123, and creates feature point information by masking the frame feature point information with the reflection feature point information. Here, the masked feature point information created by control unit 124 is called "control feature point information." Masking the frame feature point information with the reflection feature point information means excluding feature points included in the reflection feature point information from feature points included in the frame feature point information. In other words, the control feature point information plus the reflection feature point information is the frame feature point information.

[0027] Note that the control unit 124 may create control feature point information by lowering the reliability of feature points included in the reflection feature point information, rather than masking the frame feature point information with the reflection feature point information. In this case, the feature point information includes the reliability of each feature point included in the feature point information. In this case, for feature points with low reliability, the contribution of the feature points with low reliability to determining the position and orientation of the imaging device 11 by matching the control feature point information with the feature points of the three-dimensional map is reduced. In other words, the control unit 124 may use feature point information from the frame feature point information in which the influence of the reflection feature point information has been reduced as the control feature point information.

[0028] The map information storage unit 125 stores three-dimensional map information. The map information storage unit 125 can also be provided on an external server. The three-dimensional map information here refers to feature point information detected from an image and position and orientation information of the imaging device 11 that captured the image in which the feature points were detected. The map information storage unit 125 outputs the stored three-dimensional map information to the position and orientation estimation unit 126.

[0029] The position and orientation estimation unit 126 acquires three-dimensional map information from the map information storage unit 125, and further acquires control feature point information from the control unit 124. The position and orientation estimation unit 126 also matches the acquired control feature point information with feature point information included in the three-dimensional map information, and estimates and calculates the position and orientation of the imaging device 11 based on the matching result. In addition, the position and orientation estimation unit 126 outputs the calculated position and orientation to the movement control unit 128.

[0030] The route information storage unit 127 stores moving route information. The route information storage unit 127 can also be provided on an external server. The moving route information here is a list of coordinates (waypoints) and destination coordinates on the way to the destination of the moving object 10. The route information storage unit 127 outputs the stored moving route information to the movement control unit 128.

[0031] The movement control unit 128 acquires the position and orientation calculated by the position and orientation estimation unit 126 and the movement route information held by the route information holding unit 127. The movement control unit 128 calculates a control value for controlling the mobile object control device 13 from the acquired position and orientation so that the positions and orientations of the waypoints and destination included in the movement route information match the position and orientation calculated by the position and orientation estimation unit 126. The control value refers to the number of rotations of each tire installed on the mobile object 10. Furthermore, the movement control unit 128 outputs the calculated control value to the mobile object control device 13.

[0032] The mobile body control device 13 is a control device for controlling the mobile body 10 based on a control value calculated by the movement control unit 128. The mobile body control device 13 is, for example, an actuator. In this embodiment, the mobile body control device 13 is a tire motor that moves the mobile body 10. The mobile body control device 13 rotates the tires based on the control value from the movement control unit 128, and realizes the straight-line and turning movements of the mobile body 10.

[0033] Next, the hardware configuration of the information processing device 12 will be described with reference to Fig. 3. Fig. 3 is a diagram showing the hardware configuration of the information processing device 12. The information processing device 12 has a CPU 201, a ROM 202, a RAM 203, an external memory 204, an input unit 205, a display unit 206, an I / O 207, and a system bus 210. CPU is an abbreviation for Central Processing Unit. ROM is an abbreviation for Read Only Memory. RAM is an abbreviation for Random Access Memory. I / O is an abbreviation for Input / Output. The imaging device 208 is the imaging device 11 described above. The mobile object control device 209 is the mobile object control device 13 described above.

[0034] The CPU 201 controls various devices connected to a system bus 210. The ROM 202 records a BIOS program and a boot program. BIOS is an abbreviation for Basic Input Output System. The RAM 203 is used as a main storage device for the CPU 201. The external memory 204 stores programs executed by the information processing device 12.

[0035] The input unit 205 is a keyboard, mouse, or robot controller, and performs processing related to input of information, etc. The display unit 206 outputs the calculation results of the information processing device 12 to a display device according to instructions from the CPU 201. The display device is, for example, a liquid crystal display device. The I / O 207 is a communication interface, and performs information communication via a network. The communication interface may be Ethernet (registered trademark), or any type of interface such as USB or serial communication. USB is an abbreviation for Universal Serial Bus. The I / O 207 also inputs image information from an imaging device 208, and outputs control values ​​to a mobile object control device 209.

[0036] Next, an example of a process for determining whether a detected feature point is due to a reflection and masking the feature point due to a reflection will be described with reference to Fig. 4 and Fig. 5. Fig. 4 is a flowchart showing an example of a process executed by information processing device 12, for determining whether a detected feature point is due to a reflection and masking the feature point due to a reflection. Fig. 5 is a flowchart showing an example of a process executed by information processing device 12, for determining whether a detected feature point is due to a reflection and masking the feature point due to a reflection, and is a diagram continuing from Fig. 4. The circled numbers shown in Fig. 4 are connected to the circled numbers shown in Fig. 5 with the same numbers.

[0037] In step S401, the information processing device 12 initializes the system. That is, the CPU 201 reads the program, map information, and route information from the external memory 204 into the RAM 203. In addition, in step S401, the information processing device 12 acquires internal parameters of the imaging device 11 that are necessary for the position and orientation estimation unit 126 to estimate the position of the moving body 10. Examples of the internal parameters of the imaging device 11 include a horizontal focal length, a vertical focal length, a center position of an image in the horizontal direction, a center position of an image in the vertical direction, and lens distortion parameters. In addition, in step S401, the information processing device 12 reads parameters of the moving body control device 13 from the external memory 204 into the RAM 203. Examples of the parameters of the moving body control device 13 include the size of the moving body 10, the diameter of the tires, and the torque value of the motor. In addition, in step S401, the information processing device 12 starts up the imaging device 11, and makes the system including the information processing device 12 operable. Also, in step S401, the information processing device 12 starts autonomous traveling in which the mobile object 10 moves toward the destination based on the loaded map information and route information.

[0038] In step S402, the image input unit 121 acquires an image captured by the imaging device 11. The image input unit 121 converts the acquired image into a corrected image in which the influence of lens distortion is removed using internal parameters, and outputs the corrected image to the detection unit 122. Hereinafter, this corrected image will be simply referred to as an "image."

[0039] In step S403, the detection unit 122 acquires an image from the image input unit 121, and detects feature points from the acquired image. The detection of feature points may be performed using any known method.

[0040] In step S404, the detection unit 122 further calculates the three-dimensional positions of the feature points detected in step S403, and outputs feature point information (frame feature point information) to the determination unit 123.

[0041] In step S405, the determination unit 123 acquires feature point information up to the previous frame stored in a storage unit (not shown), and acquires the frame feature point information output in step S404 by the detection unit 122. Furthermore, in step S405, the determination unit 123 matches the feature points up to the previous frame with the frame feature points.

[0042] In step S406 of FIG. 5, the determination unit 123 determines whether the feature point matched in step S405 continues to be reflected in the angle of view. When the feature point matched in step S405 exists in, for example, all of the most recent 300 frames including the current frame, the determination unit 123 determines that the matched feature continues to be reflected in the angle of view. When the determination unit 123 determines that the feature point matched in step S405 continues to be reflected in the angle of view, the process of step S407 is executed. When the determination unit 123 determines that the feature point matched in step S405 does not continue to be reflected in the angle of view, the process of step S410 is executed. Note that, here, the most recent 300 frames including the current frame are targeted for the determination of whether or not the feature point continues to be reflected, but this target range may be arbitrarily changed by the user.

[0043] In step S407, the determination unit 123 calculates the horizontal movement amount of the matched feature point. Hereinafter, the horizontal movement amount is simply called the movement amount. Hereinafter, the movement amount refers to the movement amount between the matched feature point in the current frame and 300 frames ago.

[0044] In step S408, the determination unit 123 extracts feature points whose movement amount calculated in step S407 falls within a threshold value. Also, in step S408, the determination unit 123 outputs feature point information (reflection feature point information) of the extracted feature points and frame feature point information acquired from the detection unit 122 in step S405 to the control unit 124. The threshold value here is, for example, 15% of the average of the calculated movement amounts. Note that, although the threshold value is set to 15% of the average of the calculated movement amounts here, this threshold value may be changed arbitrarily by the user. In other words, the determination unit 123 matches the feature point information with past feature point information, and determines that, among feature points that continue to be reflected in the angle of view, feature points whose movement amount falls within the threshold value are feature point information due to reflection.

[0045] In step S409, control unit 124 creates feature point information (control feature point information) by masking the frame feature point information output by determination unit 123 in step S408 with the reflection feature point information, and outputs the control feature point information to position and orientation estimation unit 126.

[0046] In step S410, the position and orientation estimation unit 126 acquires control feature point information from the control unit 124, and acquires three-dimensional map information from the map information storage unit 125. Furthermore, in step S410, the position and orientation estimation unit 126 matches the acquired control feature point information with feature point information included in the three-dimensional map information, and estimates and calculates the position and orientation of the imaging device 11 based on the matching result. Furthermore, in step S410, the position and orientation estimation unit 126 outputs the calculated position and orientation of the imaging device 11 to the movement control unit 128.

[0047] In step S411, the movement control unit 128 acquires the position and orientation calculated by the position and orientation estimation unit 126 and the movement route information stored in the route information storage unit 127. In step S411, the movement control unit 128 calculates a control value for controlling the mobile body control device 13 from the acquired position and orientation so that the positions and orientations of the waypoints and destination included in the movement route information match the positions and orientations calculated by the position and orientation estimation unit 126. Furthermore, in step S411, the movement control unit 128 outputs the calculated control value to the mobile body control device 13.

[0048] In step S412, the information processing device 12 determines whether or not to shut down the system. If a command to shut down the system is input by the user via the mouse, keyboard, or the like, the information processing device 12 determines to shut down the system. If the information processing device 12 determines to shut down the system, it shuts down the system. If the information processing device 12 determines not to shut down the system, it executes the process of step S402 in FIG. 4.

[0049] The information processing device 12 according to this embodiment has been described above. The information processing device 12 estimates the position and orientation without using feature points determined by the determination unit 123 to be feature points due to reflection. Therefore, the information processing device 12 can perform stable self-location estimation. Furthermore, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0050] (Other embodiments) The present invention can also be realized by a process in which a program for realizing one or more functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. Also, the present invention can be realized by a circuit (e.g., ASIC) for realizing one or more functions.

[0051] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0052] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) an image input means for acquiring image information captured by an imaging device; A detection means for detecting feature point information from the acquired image; a determination means for determining whether the detected feature information is feature information due to reflection; a control means for, when the determination means determines that the detected feature point information is feature point information caused by reflection, setting feature point information obtained by reducing an influence of the feature point information caused by reflection from the detected feature point information as control feature point information; a position and orientation estimation means for estimating a position and orientation of the imaging device using the control feature point information; a movement control means for controlling the movement of a moving object on which the imaging device is disposed based on the estimated position and orientation of the imaging device; 13. An information processing device comprising: (Configuration 2) The feature point information includes a position of the feature point on the image and a three-dimensional position of the feature point. 2. The information processing device according to configuration 1. (Configuration 3) the determining means matches the feature point information with past feature point information, and determines that feature points whose movement amount falls within a threshold value among feature points that continue to be reflected in the angle of view are feature point information caused by reflection; 3. The information processing device according to configuration 1 or 2. (Configuration 4) the control means creates feature point information by masking feature points whose movement amounts fall within a threshold value from among the feature point information. 4. The information processing device according to configuration 3. (Configuration 5) The feature point information further includes a reliability. 4. The information processing device according to configuration 3. (Configuration 6) the control means creates feature point information by lowering the reliability of features whose movement amount falls within a threshold value, from among the feature point information. 6. The information processing device according to configuration 5. (Method 1) an image input step of acquiring image information captured by an imaging device; a detection step of detecting feature point information from the acquired image; a determination step of determining whether the detected feature point information is feature point information due to reflection; a control step of, when the determination step determines that the detected feature point information is feature point information caused by reflection, determining, as control feature point information, feature point information obtained by reducing an influence of the feature point information caused by reflection from the detected feature point information; a position and orientation estimation step of estimating a position and orientation of the imaging device using the control feature point information; a movement control step of controlling a movement of a moving object on which the imaging device is disposed based on the estimated position and orientation of the imaging device; The method according to claim 1, further comprising: (Program 1) Computer, an image input means for acquiring image information captured by an imaging device; A detection means for detecting feature point information from the acquired image; a determination means for determining whether the detected feature point information is feature point information due to reflection; a control means for setting, when the determination means determines that the detected feature point information is feature point information caused by reflection, feature point information obtained by reducing an influence of the feature point information caused by reflection from the detected feature point information as control feature point information; a position and orientation estimation means for estimating a position and orientation of the imaging device using the control feature point information; and a movement control means for controlling the movement of a moving object on which the imaging device is disposed based on the estimated position and orientation of the imaging device; A program characterized by causing the program to function as a [Explanation of symbols]

[0053] 10: Information processing systems 11: Imaging device 12: Information processing device 13: Mobile control device

Claims

1. an image input means for acquiring image information captured by an imaging device; A detection means for detecting feature point information from the acquired image; a determination means for determining whether the detected feature information is feature information due to reflection; a control means for, when the determination means determines that the detected feature point information is feature point information caused by reflection, setting feature point information obtained by reducing an influence of the feature point information caused by reflection from the detected feature point information as control feature point information; a position and orientation estimation means for estimating a position and orientation of the imaging device using the control feature point information; a movement control means for controlling the movement of a moving object on which the imaging device is disposed based on the estimated position and orientation of the imaging device; 13. An information processing device comprising:

2. The feature point information includes a position of the feature point on the image and a three-dimensional position of the feature point.

2. The information processing apparatus according to claim 1,

3. the determining means matches the feature point information with past feature point information, and determines that feature points whose movement amount falls within a threshold value among feature points that continue to be reflected in the angle of view are feature point information caused by reflection; 2. The information processing apparatus according to claim 1,

4. the control means creates feature point information by masking feature points whose movement amounts fall within a threshold value from among the feature point information.

4. The information processing apparatus according to claim 3.

5. The feature point information further includes a reliability. The information processing device according to claim 3 .

6. the control means creates feature point information by lowering the reliability of features whose movement amount falls within a threshold value, from among the feature point information.

6. The information processing apparatus according to claim 5,

7. an image input step of acquiring image information captured by an imaging device; a detection step of detecting feature point information from the acquired image; a determination step of determining whether the detected feature point information is feature point information due to reflection; a control step of, when the determination step determines that the detected feature point information is feature point information caused by reflection, determining, as control feature point information, feature point information obtained by reducing an influence of the feature point information caused by reflection from the detected feature point information; a position and orientation estimation step of estimating a position and orientation of the imaging device using the control feature point information; a movement control step of controlling a movement of a moving object on which the imaging device is disposed based on the estimated position and orientation of the imaging device; The method according to claim 1, further comprising:

8. Computer, an image input means for acquiring image information captured by an imaging device; A detection means for detecting feature point information from the acquired image; a determination means for determining whether the detected feature point information is feature point information due to reflection; a control means for setting, when the determination means determines that the detected feature point information is feature point information caused by reflection, feature point information obtained by reducing an influence of the feature point information caused by reflection from the detected feature point information as control feature point information; a position and orientation estimation means for estimating a position and orientation of the imaging device using the control feature point information; and a movement control means for controlling the movement of a moving object on which the imaging device is disposed based on the estimated position and orientation of the imaging device; A program characterized by causing the program to function as a

Images