Information processing device and information processing method

Abstract

Provided is an information processing device that can reduce accumulated errors in a three-dimensional map even when it is difficult to arrange multiple markers of normal size. [Solution] The information processing device has a detection means that detects a first marker and feature points from an image captured by an imaging device and detects a second marker based on the feature points, and an estimation means that estimates the position and orientation of the imaging device based on information about the first marker and information about the second marker.

Description

[Technical Field]

[0001] The present invention relates to an information processing device and an information processing method. [Background technology]

[0002] In a technology for estimating the three-dimensional shape of a subject using camera images, a three-dimensional map (including three-dimensional point cloud data representing the three-dimensional shape of the subject) is successively updated and expanded based on the measured position and orientation of the camera. For example, SLAM (Simultaneous Localization and Mapping) is a well-known technology for estimating three-dimensional shape.

[0003] In SLAM technology that uses images, errors occur in the estimation of the camera position and orientation due to sampling errors of feature points detected from the images and noise in the images. Therefore, the 3D map generated based on the camera position and orientation contains errors. Furthermore, the errors are propagated to the camera position and orientation estimated based on the 3D map containing errors. The propagated errors are accumulated in the 3D map.

[0004] Non-Patent Document 1 discloses a method for reducing accumulated errors by using image similarity to recognize when the camera has returned to approximately the same position (loop closure) and correcting a 3D map based on the results of this recognition. Patent Document 1 also discloses a method for reducing accumulated errors by placing artificial square markers in the scene and comparing the positions of the markers in a world coordinate system measured in advance with the positions of the markers in a camera coordinate system detected from the image. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2019-020778 [Non-patent literature]

[0006] [Non-Patent Document 1] Jacob Engel et al., “Large-Scale Direct SLAM with Stereo Cameras”, [online], 2015, [Retrieved August 22, 2024], Internet<URL:https: / / jakobengel.github.io / pdf / engel2015_stereo_lsdslam.pdf> Summary of the Invention [Problem to be solved by the invention]

[0007] The method disclosed in Non-Patent Document 1 may not reduce accumulated errors when the camera's movement range is limited. Furthermore, the method disclosed in Patent Document 1 may have difficulty placing multiple markers of normal size in a scene with a large number of facilities, such as a factory. Furthermore, when multiple markers are used, there is a problem in that preparation time is required to determine the position and orientation of each marker in space in advance.

[0008] Therefore, an object of the present invention is to provide an information processing device that can reduce accumulated errors in a three-dimensional map even when it is difficult to arrange multiple markers of normal size. [Means for solving the problem]

[0009] The information processing device according to the present invention includes a detection unit that detects a first marker and a feature point from an image captured by an imaging device and detects a second marker based on the feature point; and an estimation means for estimating the position and orientation of the imaging device based on information about the first marker and information about the second marker. [Effects of the Invention]

[0010] According to the present invention, even when it is difficult to arrange multiple markers of normal size, it is possible to reduce accumulated errors in a three-dimensional map. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a diagram illustrating an example of the configuration of an information processing system according to a first embodiment. [Figure 2] FIG. 10 is a diagram illustrating a detailed configuration example of an index information management unit. [Figure 3] FIG. 2 is a diagram illustrating a detailed configuration example of a correction unit. [Figure 4] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device. [Figure 5] FIG. 10 is a diagram showing an example of a scene in which indexes A and B are placed. [Figure 6] 10 is a diagram illustrating an example of the shape of an index B. FIG. [Figure 7] 5A to 5C are diagrams illustrating processing by a correction unit according to the first embodiment. [Figure 8] 10 is a flowchart illustrating an example of an image display process. [Figure 9] 10 is a flowchart illustrating an example of an index information management process. [Figure 10] 10 is a diagram for explaining the detection process of the index B. FIG. [Figure 11] 10 is a diagram for explaining the posture estimation process of the indicator B. FIG. [Figure 12] 10 is a flowchart illustrating an index information correction process. [Figure 13] FIG. 10 is a diagram illustrating an example of the configuration of an information processing system according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0013] First Embodiment The configuration of an information processing device 100 according to the present invention will be described with reference to FIGS. 1 to 4. FIG. 1 is a block diagram showing an example of the configuration of an information processing system 10 according to the first embodiment. The information processing system 10 to which the first embodiment is applied includes an information processing device 100, an imaging unit 103, a display unit 110, and an operation unit 120. The information processing system 10 may be, for example, a system including an electronic device such as a head-mounted display (HMD) including the imaging unit 103, the display unit 110, and the operation unit 120, and an information processing device 100 capable of communicating with the electronic device. The information processing system 10 may be an electronic device such as an HMD including the imaging unit 103, the display unit 110, and the operation unit 120, and having the information processing device 100 built therein.

[0014] The imaging unit 103 is a stereo camera and has a first camera (for example, a left camera) and a second camera (for example, a right camera). Each of the first camera and the second camera captures a moving image of a scene and outputs the captured moving image of the scene (captured image of each frame) to the information processing device 100. Note that the imaging unit 103 is not limited to a stereo camera and may be a monocular camera.

[0015] The operation unit 120 is an operation member such as a button, a mouse, and a keyboard, and inputs input from the user to the information processing device 100. The display unit 110 is a display for presenting to the user a composite image, which is a composite of a real-life image and a virtual image output from the information processing device 100.

[0016] The index A (first marker) and the index B (second marker) will be described with reference to Fig. 5 and Fig. 6(A) to 6(C). Fig. 5 is a diagram showing an example of a scene in which the index A and the index B are arranged and are photographed by the imaging unit 103. In the scene of Fig. 5, a rectangular parallelepiped workbench 500 is placed on the floor.

[0017] Marker 510 placed on the floor surface is an example of index A. Markers 520a to 520c (collectively referred to as markers 520) placed on the surface of workbench 500 are examples of index B. Index A is, for example, a square with sides of 20 cm, and is placed in an area larger than a 20 cm square rectangle. The information processing device 100 can estimate the position and orientation of the camera (imaging unit 103) with high accuracy by estimating the position and orientation of the camera based on the shape of index A, which is larger than index B. Note that the following description will be given of a case where index A is a square, but index A is not limited to a rectangular shape such as a square and may be a polygon. Feature points 530 indicated by crosses are used in SLAM.

[0018] The number of indicators A placed in the space is not limited to one, and multiple indicators A may be placed. In the scene shown in Fig. 5, workbench 500 has drawers, handles, etc. on the side, and has slits 540 on the top surface that divide the work space. Therefore, workbench 500 does not have a flat surface large enough to place indicator A. In the scene of Fig. 5, a small indicator B is placed on the surface of workbench 500 to place a CG model in the correct position relative to workbench 500.

[0019] 6(A) to 6(C) are diagrams illustrating the shape of index B. Index B is an index with a checkered pattern, as shown in Fig. 6(A), in which the area within a circle is divided into four parts by two straight lines that intersect at right angles at the center point of the circle. The four divided areas are colored so that the difference in brightness and hue between adjacent areas is large.

[0020] The shape of the index B is not limited to that shown in Fig. 6(A). As shown in Fig. 6(B), the index B may be an index in a checkered pattern formed by dividing the area inside a rectangle into four parts by two straight lines that pass through opposing vertices of the rectangle and intersect at the center point. Alternatively, as shown in Fig. 6(C), the index B may be an index in which the inside of the outer frame of the rectangle is checkered.

[0021] The information processing device 100 shown in Fig. 1 will be described. The information processing device 100 includes an image acquisition unit 105, an operation acquisition unit 125, a storage unit 150, an index information management unit 155, a virtual image generation unit 160, a synthesis unit 170, a correction unit 180, and a position and orientation estimation unit 190.

[0022] The image acquisition unit 105 receives images captured by the stereo camera of the imaging unit 103 and stores them in the storage unit 150. The operation acquisition unit 125 receives information (input signals) of user operations input via the operation unit 120 and stores them in the storage unit 150.

[0023] The storage unit 150 stores data used for executing various processes by the information processing device 100. The storage unit 150 holds, for example, the following information: (a) Camera image (b) CG model information (c) Camera parameters (d) Position and orientation information (e) Keyframe information (3D map) (f) Information on Indicator A (g) Information on Indicator B (h) Minutiae information (i) Virtual Image (j) Composite image

[0024] The data stored in the storage unit 150 is not limited to the above information or data. As long as the information on the index B and the feature point information can be corrected, the items of data stored in the storage unit 150 may be added, modified, or deleted. Each functional unit of the information processing device 100 is 4, and can be used. Furthermore, each functional unit of the information processing device 100 can write processing results to the storage unit 150 via the RAM 407. The data stored in the storage unit 150 will now be specifically described.

[0025] (a) The camera image is an image captured by the imaging unit 103, and is associated with a frame ID and stored in the storage unit 150. The storage unit 150 stores a plurality of camera images. If the camera images are stereo camera images, the storage unit 150 may manage the left and right images separately.

[0026] (b) CG model information includes the vertex positions of 3D polygons in a model coordinate system unique to the CG model, edge information, texture images, and spatial position and orientation. The model coordinate system is set in advance as a reference coordinate system that represents the coordinates of the vertices of 3D polygons on the surface of the CG model.

[0027] (c) The camera parameters include the camera's inherent principal point position and focal length. If the camera is a stereo camera, the camera parameters include parameters that represent the relative positions and orientations of the left and right cameras. (d) The position and orientation information is information about the position and orientation of the camera (imaging unit 103) obtained by inputting the camera image and camera parameters into SLAM.

[0028] (e) The key frame information includes the camera image and frame ID of the camera image at the time of creating the key frame, the key frame ID, and the position and orientation of the camera at the time of creating the key frame. The key frame information also includes the index ID and detected coordinate position of index A detected in the camera image, and the index ID and detected coordinate position of index B detected in the camera image. The key frame information also includes the feature point ID, detected coordinate position, and distance value of the feature point detected in the camera image. The key frame information also includes key frame correspondence information exemplified in JP 2019-020778 A. The three-dimensional map is also referred to as key frame information. Hereinafter, the three-dimensional map will be referred to as key frame information.

[0029] (f) The information on index A includes the position and orientation of index A in space, the three-dimensional positions of the four vertices of index A, an index ID, and a keyframe ID associated with the index ID. The position and orientation of index A in space can be calibrated in advance by, for example, the method disclosed in Japanese Patent Laid-Open No. 2005-326274 (U.S. Patent Application Publication No. 2006 / 004280).

[0030] (g) The information on index B includes the three-dimensional position of index B in space, the normal vector of index B, an index ID, a keyframe ID associated with the index ID, and a group feature point ID. The group feature point ID is an ID that identifies the group to which the feature point grouped with index B belongs. The feature point grouped with index B is a candidate for position correction to match the plane on which index B is placed. The feature point grouped with index B is, for example, a feature point surrounding index B, and is a feature point that exists within a predetermined distance from the position of index B.

[0031] (h) The feature point information includes the three-dimensional position of the feature point in space, the feature point ID, and the group feature point ID. The feature point information is recorded in the storage unit in association with the key frame corresponding to the camera image in which the feature point was detected.

[0032] (i) A virtual image is an image that is created by rendering CG model information based on the current camera position and orientation and camera parameters obtained by SLAM. The virtual image is synthesized onto the camera image (real image). If the camera is a stereo camera, a virtual image is generated for each camera. (j) A synthesized image is an image that has a virtual image synthesized onto a camera image.

[0033] The index information management unit 155 detects feature points, index A, and index B from the camera image. The index information management unit 155 updates the information on index A and the information on index B based on the key frame information stored in the storage unit 150. The updated information on index A and the information on index B are stored in the storage unit 150.

[0034] The virtual image generation unit 160 generates a virtual image by rendering it based on the CG model information and position and orientation information stored in the storage unit 150. The virtual image generation unit 160 stores the generated virtual image in the storage unit 150. The composition unit 170 combines the camera image stored in the storage unit 150 with the corresponding virtual image, and displays the combined image on the display unit 110.

[0035] The correction unit 180 corrects the index B information, feature point information, and key frame information based on the index A information, index B information, feature point information, and key frame information stored in the storage unit 150. The correction unit 180 stores the corrected index B information, feature point information, and key frame information in the storage unit 150.

[0036] The position and orientation estimation unit 190 estimates the position and orientation of the camera from the camera image, camera parameters, key frame information, index A information, index B information, and feature point information stored in the storage unit 150. The position and orientation estimation unit 190 can estimate the position and orientation of the camera using, for example, SLAM. The position and orientation estimation unit 190 stores the estimated position and orientation of the camera in the storage unit 150. If the SLAM processing satisfies predetermined conditions for adding a key frame, the correction unit 180 generates key frame information and updates the key frame information in the storage unit 150.

[0037] The index information management unit 155 will be described in detail with reference to Fig. 2. Fig. 2 is a diagram showing a detailed configuration example of the index information management unit 155. The index information management unit 155 includes a binarization unit 210, an edge detection unit 220, an edge tracking unit 230, an index A detection unit 240, an index B detection unit 250, a feature point grouping unit 260, and an index B orientation estimation unit 280.

[0038] The index A detection unit 240 detects a rectangular marker (index A) from the camera image when the key frame was created, by referring to the key frame information in the storage unit 150. The index A detection unit 240 can detect index A by using a detection process for a rectangular marker having an identification ID such as an ArUco marker. The index A detection unit 240 detects a rectangular area from the camera image and acquires the index ID and the image coordinates of the four vertices of the rectangle by performing homography transformation on the bit pattern arranged in the rectangular area to identify it. If the index ID of the detected index A matches the index ID of the information on index A in the storage unit 150, the index A detection unit 240 records the key frame ID of the key frame corresponding to the camera image in which index A was detected, in association with the index ID of the information on index A in the storage unit 150.

[0039] The binarization unit 210 references the key frame information in the storage unit 150, and performs binarization processing on the camera image at the time of key frame creation using a predetermined binarization threshold to generate a binarized image. The binarization unit 210 outputs the generated binarized image to the edge detection unit 220.

[0040] The edge detection unit 220 detects edges from the binarized image output by the binarization unit 210. The edge detection unit 220 generates an edge-detected image by extracting the line segments of the index B captured in the camera image. The edge detection unit 220 outputs the generated edge-detected image to the edge tracking unit 230.

[0041] The edge tracking unit 230 acquires edge information by determining the linearity and length of the edge line segments included in the edge-detected image output by the edge detection unit 220. The edge tracking unit 230 outputs the acquired edge information to the index B detection unit 250.

[0042] The index B detection unit 250 detects index B based on the feature points. Specifically, the index B detection unit 250 acquires the detected coordinate values, index ID, and three-dimensional position in space of index B captured in the camera image based on the feature point information in the storage unit 150, the camera image at the time of key frame creation, and edge information. Details of the index B detection process will be described in step S930 of FIG. 9. The index B detection unit 250 may use, as the index ID, a hash value obtained by inputting XYZ values, which are the three-dimensional position in space, into a hash function as a key, for example.

[0043] The feature point grouping unit 260 groups index B with the feature points existing around index B. The feature point grouping unit 260 associates the information of index B with the feature point information of the feature points existing around index B. The feature points grouped with index B become candidates for correction by the correction unit 180.

[0044] 5, the feature point 530 grouped with the marker 520 (index B) can be, for example, a feature point 530 that exists within a predetermined distance from the position (coordinates) of the index B that appears in the image when the key frame is created. Also, the feature point 530 grouped with the marker 520 (index B) may be a feature point 530 that is detected together with the marker 520 in the camera image of more than a predetermined number of key frames (for example, three or more). When multiple indices B exist in the space, the feature point grouping unit 260 performs a process of grouping the feature points 530 for each index B.

[0045] The index B orientation estimation unit 280 estimates the normal of the index B from the edge information in order to determine the orientation of each detected index B. A method for calculating the orientation of the index B will be described later with reference to FIGS.

[0046] The correction unit 180 will be described in detail with reference to Fig. 3. Fig. 3 is a diagram showing a detailed configuration example of the correction unit 180. The correction unit 180 includes an index B orientation correction unit 310, a feature point correction unit 320, and a key frame information correction unit.

[0047] The index B attitude correcting unit 310 corrects the detection error of the normal direction (normal vector) estimated by the index B attitude estimating unit 280. Specifically, the index B attitude correcting unit 310 determines whether the tilt of index A and the tilt of index B approximately match, and if the tilt of index A and the tilt of index B approximately match, corrects the tilt of index B based on the tilt of index A.

[0048] The index B attitude correcting unit 310 can correct the tilt of index B in space by correcting the normal direction of index B using the normal direction of index A. For example, if the difference between the tilt of index A and the tilt of index B is smaller than a predetermined tilt threshold, the index B attitude correcting unit 310 determines that the tilt of index A and the tilt of index B approximately match. If the tilt of index A and the tilt of index B approximately match, the index B attitude correcting unit 310 corrects the value of the normal vector of index B to the value of the normal vector of index A.

[0049] The processing of the correction unit 180 will be described using Figures 7(A) to 7(C). The correction unit 180 corrects the posture, feature points, and key frame information of index B. Figures 7(A) to 7(C) show the scene shown in Figure 5 as viewed from the right side. Index A 510 is placed on the floor. Indexes B 520a and 520b are placed on the workbench 500. Normal vectors 710a and 710b indicate the normal directions of indexes B 520a and 520b, respectively.

[0050] FIG. 7A shows that the inclinations of the normal vectors 710a and 710b estimated from the orientations of the indices B 520a and 520b are different from those of the work table 500 due to an error in detecting the orientations of the indices B 520a and 520b. For example, the inclination of plane 720 on which index B 520a is detected is different from the inclination of the work table 500's top surface.

[0051] 7(B) shows a state in which the attitudes of the indices B 520a and 520b have been corrected. The index B attitude correcting unit 310 can correct the attitudes of the indices B 520a and 520b in accordance with the tilt of the index A 510 by setting the values ​​of the normal vectors of the indices B 520a and 520b to the value of the normal vector of the index A 510.

[0052] The feature point correction unit 320 corrects the three-dimensional positions of feature points grouped with index B, which is stored in the storage unit 150 as information about index B, based on the corrected information about index B. A method for correcting the positions of feature points 530a to 530d will be described using FIGS. 7(B) and 7(C). Plane 730 in FIG. 7(B) is a plane on which index B 520a after correction is placed. Plane 730 on which index B 520a is placed is a plane that includes the three-dimensional position of index B 520a and has normal vector 710a of index B as its normal line.

[0053] Of the three feature points 530a, 530b, and 530c grouped with index B, the feature point correction unit 320 corrects the positions of feature points 530b and 530c whose distance from the plane on which index B is placed is shorter than a predetermined distance threshold.

[0054] 7(B), the feature point correction unit 320 calculates the distance to each feature point, and moves the positions of feature points 530b and 530c whose distance from the plane 730 is shorter than a predetermined distance threshold to positions on the plane 730. Fig. 7(C) shows the result of the feature point correction unit 320 correcting the positions of feature points 530b and 530c.

[0055] The key frame information correction unit 330 corrects the key frame information based on the camera parameters, index A information, index B information, and feature point information stored in the storage unit 150. The key frame information correction unit 330 can correct the key frame information in a manner similar to the processing of the second derivation unit and correction unit of Japanese Patent Application Laid-Open No. 2019-020778, for example. That is, the key frame information correction unit 330 uses the relative position and orientation between key frames derived using index A and index B placed in the scene to correct the key frame information.

[0056] Even when the camera movement range is limited, the information processing device 100 can generate highly accurate key frame information by correcting the key frame information using the key frame information correcting unit 330. Furthermore, the index B attitude correcting unit 310 corrects the attitude (tilt) of index B, and the feature point correcting unit 320 corrects the positions of feature points around index B, so that the information processing device 100 can generate more accurate key frame information than before.

[0057] Fig. 4 is a diagram illustrating an example of the hardware configuration of the information processing device 100. In the example of Fig. 4, the information processing device 100 is configured as a device separate from an electronic device such as an HMD that includes an imaging unit 103, a display unit 110, and an operation unit 120. The information processing device 100 is, for example, a personal computer.

[0058] The information processing device 100 is connected to the imaging unit 103 and the display unit 110 via an interface (I / F) 403. The information processing device 100 includes a CPU (Central Processing Unit) 401, a ROM (Read Only Memory) 402, a storage medium drive 405, an external storage device 406, and a RAM 407. The information processing device 100 may include input devices such as a mouse 408 and a keyboard 409, and may also include a monitor 411 as a display device. The devices constituting the information processing device 100 are connected to each other via a bus 410.

[0059] The CPU 401 controls the entire information processing device 100 by reading a control program stored in the ROM 402 into the RAM 407 and executing it. The external storage device 406 stores an operating system (OS) and various programs for executing the processing of each functional unit of the information processing device 100. The CPU 401 implements the processing of each functional unit of the information processing device 100 by loading a program read from the ROM 402 or the external storage device 406 into the RAM 407 and executing it. Note that all or part of the software processing by the CPU 401 may be implemented by a hardware circuit corresponding to the processing of each functional unit of the information processing device 100.

[0060] The monitor 411 is, for example, a CRT (Cathode Ray Tube) or a liquid crystal panel, etc. The monitor 411 can display the same image as the display unit 110 of the information processing system 10.

[0061] The storage medium drive 405 is a drive device for reading programs and data recorded on a storage medium such as an optical disk, and for writing programs and data to the storage medium. The external storage device 406 is a storage device such as a hard disk drive (HDD) or a solid state drive (SSD). The external storage device 406 may function as the storage unit 150 shown in FIG. 1.

[0062] The mouse 408 and keyboard 409 are examples of input devices, and when operated by a user, can input various instructions to the CPU 401. The mouse 408 and keyboard 409 may function as the operation unit 120 shown in FIG.

[0063] 8 is a flowchart illustrating an example of image display processing of the information processing device 100. In step S810, the CPU 401 reads pre-calibrated camera parameters and information on index A from the storage unit 150 as initialization processing.

[0064] In step S820, the image acquisition unit 105 acquires a camera image (captured image). The image acquisition unit 105 receives a signal from the imaging unit 103, generates a camera image, and stores the generated image in the storage unit 150.

[0065] In step S830, the position and orientation estimation unit 190 estimates the position and orientation of the image capture unit 103 based on the information read from the storage unit 150 in step S810.

[0066] In step S840, the position and orientation estimation unit 190 determines whether to add the camera image acquired in step S820 as a key frame. The position and orientation estimation unit 190 sets, as the nearest key frame, the key frame corresponding to the position and orientation that most closely matches the position and orientation of the camera estimated from the camera image acquired in step S820, among the key frames stored in the storage unit 150. If at least one of the difference between the position of the nearest key frame and the estimated position and the difference between the orientation of the nearest key frame and the estimated orientation exceeds a threshold, the position and orientation estimation unit 190 determines to add the key frame of the camera image.

[0067] If it is determined that a key frame should be added, the position and orientation estimation unit 190 may add the key frame in the same manner as the processing in step S1040 in FIG. 4 of JP 2019-020778 A. That is, the position and orientation estimation unit 190 detects pixels on the camera image whose brightness gradient is equal to or greater than a threshold as feature points, and performs stereo matching of the feature points between the stereo cameras. The position and orientation estimation unit 190 calculates the distance value of the feature point from the three-dimensional position of the added key frame by stereo matching. The calculated image coordinates and distance value of the feature point are added to the key frame information in the storage unit 150. If it is determined that a key frame should be added, the processing The process proceeds to step S850. If it is determined that a key frame is not to be added, the process proceeds to step S860.

[0068] In step S850, the index information management unit 155 executes index information management processing to use the key frame information to detect index A and index B. Details of the index information management processing in step S850 will be described later with reference to FIG.

[0069] In step S860, composition unit 170 generates a composite image of the camera image and the image of the CG model (virtual image), and displays it on display unit 110. Specifically, virtual image generation unit 160 renders the CG model based on the position and orientation information of the camera stored in storage unit 150. Composition unit 170 composites the image of the CG model generated by virtual image generation unit 160 with the camera image. Display unit 110 displays the composite image generated by composition unit 170.

[0070] In step S870, CPU 401 determines whether or not to end the image display process. When CPU 401 receives a signal instructing the end from operation acquisition unit 125, it determines to end the image display process. When it determines to end the image display process, the process shown in Fig. 8 ends. When it determines not to end the image display process, the process returns to step S820.

[0071] FIG. 9 is a flowchart showing details of the index information management process in step S850 in FIG. 8. In step S910, the index information management unit 155 detects index A from the camera image of the key frame information added in step S840. The index information management unit 155 binarizes the camera image to generate a binarized image, and detects a quadrangular area from the generated binarized image. The index information management unit 155 calculates a projective transformation matrix from a square to image coordinates based on the image coordinates of each vertex of the detected quadrangular area. The index information management unit 155 converts the quadrangular area into a square using the calculated projective transformation matrix. The index information management unit 155 identifies the index ID of index A by reading out a brightness value corresponding to a predetermined position of the square from the camera image.

[0072] The index information management unit 155 compares the index ID of the identified index A with the index ID of the information on index A stored in the storage unit 150. If the index IDs match, the index information management unit 155 stores the correspondence between the image coordinates of each vertex of index A and the three-dimensional position of each vertex of index A in space. The three-dimensional positions of the four vertices of index A in space are calculated based on the three-dimensional positions of the four vertices in the coordinate system defined for index A and the position and orientation of index A.

[0073] In step S920, the index information management unit 155 determines whether there is information for detecting index B from the camera image of the key frame information added in step S840. The index information management unit 155 detects feature points from the camera image, associates information about the detected feature points with key frame information corresponding to the camera image, and records the information in the storage unit. The index information management unit 155 generates a list of feature points that were also detected in camera images of more than a predetermined number of past key frames, out of the feature points detected in the camera image of the key frame information.

[0074] The index information management unit 155 determines that there is information for detecting index B when there are feature points detected in camera images of more than a predetermined number of key frames. The index information management unit 155 determines that there is no information for detecting index B when there are no feature points detected in camera images of more than a predetermined number of key frames. When there is information for detecting index B, the process proceeds to step S930. When there is no information for detecting index B, the index information management process shown in FIG. 9 ends.

[0075] In step S930, the index B detection unit 250 of the index information management unit 155 detects index B based on the feature points included in the list generated in step S920. The index B detection unit 250 detects index B based on information about pixels surrounding the feature points included in the list of feature points generated in step S920. The index B detection unit 250 refers to the feature point coordinates on the camera image of the key frame to which the feature point ID of the feature point is linked, and determines whether the area including the feature point is index B based on information about pixels surrounding the feature point coordinates.

[0076] The detection process of index B will be described using Figures 10(A) and 10(B). The shape of index B in Figure 10(A) is rotationally symmetric (two-fold symmetry, which means that it overlaps with the original shape when rotated 180 degrees). A feature point 1005 located at the center of index B 520 in Figure 10(A) is a feature point included in the list generated in step S920. The index B detection unit 250 acquires information on the luminance and hue of pixels surrounding feature point 1005.

[0077] For example, in Fig. 10(A), three reference points (pixels) are arranged radially in each of four directions with feature point 1005 at the center. The three reference points in each of the four directions are arranged rotationally symmetrically with feature point 1005 at the center. For example, in Fig. 10(A), reference points 1001a to 1003a are arranged rotationally symmetrically with reference points 1001b to 1003b, respectively.

[0078] The index B detection unit 250 determines whether the difference in luminance and the difference in hue between the pixels on the image at the reference point 1001a and the reference point 1001b are smaller than their respective predetermined thresholds. For example, if at least one of the difference in luminance and the difference in hue between the pixels at the reference point 1001a and the reference point 1001b is smaller than its respective predetermined threshold, the index B detection unit 250 determines that index B is present at the position of the feature point 1005.

[0079] The index B detection unit 250 may determine that index B exists at the position of the feature point 1005 when both the difference in luminance and the difference in hue between pixels are smaller than their respective predetermined thresholds. The index B detection unit 250 may also determine that index B exists at the position of the feature point 1005 when the difference in luminance and the difference in hue between two or more pairs of reference points out of the three rotationally symmetric reference point pairs are smaller than their respective predetermined thresholds.

[0080] The index B detection unit 250 may detect index B based on whether information on pixels located around the feature point 1005 corresponds to the shape of index B, in addition to the brightness and hue conditions between rotationally symmetric pixels. For example, the index B detection unit 250 may determine a condition regarding the line segments output by the edge tracking unit 230. As shown in FIG. 10B , the index B detection unit 250 determines whether the intersection of four line segments 1010, 1020, 1030, and 1040 from the line segments output by the edge tracking unit 230 is close to the feature point 1005. If the distance between the intersection of the four line segments and the feature point 1005 is shorter than a predetermined distance threshold, the index B detection unit 250 determines that index B is present at the position of the feature point 1005.

[0081] 11(A) and 11(B), the orientation estimation process for index B will be described. The index B orientation estimation unit 280 acquires the three-dimensional position of the feature point in space based on the feature point ID and feature point information of the feature point 1005 present at the position of index B.

[0082] FIG. 11(A) is a perspective view of index B. FIG. 11(B) is a view of index B when viewed from directly in front. Four points 1110 to 1140 on the circumference of index B form a square with point 1150 as the center. The index B orientation estimation unit 280 can calculate a homography matrix based on, for example, the points 1110 to 1140 and corresponding points on four line segments 1010 to 1040 detected in the camera image of the key frame. Using the homography matrix, the index B orientation estimation unit 280 estimates the orientation of index B in the camera image when viewed from directly in front (in the direction opposite to the visual axis of the camera coordinate system). By finding the orientation of index B on the image, the orientation of index B in the camera coordinate system defined by the camera image can be found.

[0083] The orientation of index B is recorded in the storage unit 150 as the normal vector of index B used in the index information correction process described in Fig. 12. When the detection of index B is completed, the process proceeds to step S940 in Fig. 9.

[0084] In step S940, the feature point grouping unit 260 groups the feature points corresponding to each index B detected in step S930 by linking them to the respective index B. For example, the feature point grouping unit 260 can group feature points that exist within a predetermined distance from the position of index B captured in the camera image by linking them to the index B. In the example of FIG. 7(A), the feature point grouping unit 260 groups feature points 530a, 530b, and 530c that exist within a predetermined distance from the position of index B 520a with index B 520a. Feature point 530d is grouped with index B 520b.

[0085] In step S950, the correction unit 180 executes index information correction processing to correct the slope of index B and the feature points grouped with index B, based on the information of index A. Details of the index information correction processing in step S950 will be described with reference to FIG. 12.

[0086] 12 is a flowchart showing the details of the process of step S950 in FIG. 9. In step S1210, the correction unit 180 corrects the tilt of the index B. The correction unit 180 can correct the tilt of the index B by correcting the normal direction of the index B. 7(A) and 7(B) are used to explain the correction of the inclination of the index B. A side view of the scene from the right is shown.

[0087] The correction unit 180 acquires the normal vector of the index A 510 based on the orientation information of the index A 520 stored in the storage unit 150. The correction unit 180 acquires the normal vector of the index B 520a based on the orientation information of the index A 520 stored in the storage unit 150. The degree of coincidence between the normal vector of the index A510 and the normal vector of the index A510 is determined. If the normal vector 710a of index B 520a is greater than the threshold value for the normal vector of index A 510, the normal vector 710a of index B 520a is replaced with the information of the normal vector of index A 510. Similarly, the correction unit 180 replaces the normal vector 710b of index B 520b with the information of the normal vector of index A 510. In this way, the correction unit 180 calculates the normal directions of indexes B 520a and 520b using the normal direction of index A 510. By correcting the inclination of the indicators B520a and 520b, the inclination of the indicators B520a and 520b is corrected. Furthermore, the normal direction of the corrected indices B 520a and 520b coincides with the normal direction of the indices A.

[0088] Index B is a smaller marker than index A, and its actual size is smaller than index A, so it appears smaller than index A in the camera image. For this reason, the orientation of index B estimated in step S930 may not be obtained correctly due to the influence of sampling error. Correction unit 180 corrects the tilt of index B to reduce the influence of sampling error.

[0089] Index A is a larger marker than index B, and has a larger actual size than index B. Therefore, the influence of sampling errors when estimating the attitude of index A is smaller than when estimating the attitude of index B. If it is determined that the attitude of index B substantially matches the attitude of index A, the correction unit 180 can correct the tilt of index B by replacing the normal direction of index B with the normal direction of index A.

[0090] For example, in the scene shown in Fig. 5, the top of the workbench 500 is approximately parallel to the floor surface on which the index A is placed, so the correction unit 180 can apply correction using the normal direction. Unlike the scene in Fig. 5, when the index A is placed on a wall surface, the correction unit 180 applies correction using the normal direction of the index B (for example, the marker 520c in Fig. 5) placed on a surface perpendicular to the floor surface of the workbench 500. The direction can be corrected.

[0091] Note that the correction of the tilt of index B is not limited to the case where it is performed using the normal direction, and it may be corrected using other information that expresses the tilt of index B in space. For example, if the tilt of an index in space is defined as a 3 × 3 orientation matrix, the correction unit 180 can correct the tilt of index B by replacing the orientation matrix of index B with the orientation matrix of index A.

[0092] Furthermore, the correction unit 180 may switch the correction method depending on the reliability at the time of detecting the index B. The reliability at the time of detecting the index B is set based on the following parameters, for example. (1) The lengths of the four line segments 1010 to 1040 in Figure 10(B) (2) The smaller angle formed by the four line segments 1010 to 1040 in Figure 10(B)

[0093] (1) When the length of line segments 1010 to 1040 exceeds a predetermined threshold length, index B is displayed larger, allowing position and orientation estimation unit 190 to stably estimate the position and orientation of the camera. Also, (2) when the smaller angle formed by line segments 1010 to 1040 exceeds a predetermined threshold angle, the area in which index B is displayed increases, allowing position and orientation estimation unit 190 to stably estimate the position and orientation of the camera.

[0094] The reliability of index B may be set according to the value of each of the above parameters, or may be set based on whether or not the parameter exceeds a threshold value. If each parameter exceeds its threshold value, the reliability of index B may be set higher, and the correction unit 180 may use the normal direction of index B as is without correcting the tilt of index B.

[0095] In step S1220, the correction unit 180 corrects the three-dimensional positions of the feature points grouped with index B. The feature points grouped with index B are candidates for correction. The correction of feature points will be described with reference to FIGS. 7(B) and 7(C). The correction unit 180 acquires information on the positions of three feature points 530a, 530b, and 530c grouped with index B 520a. The correction unit 180 acquires plane 730, which has the same normal vector as normal vector 720a of index B 520a and passes through the three-dimensional position of index B 520a.

[0096] The correction unit 180 calculates the distances between the feature points 530a, 530b, and 530c and the plane 730. The correction unit 180 determines that the feature points 530b and 530c, whose distances from the plane 730 are shorter than a predetermined distance threshold, exist on the same plane 730 as the index B 520a. The correction unit 180 updates the positions of the feature points 530b and 530c to the three-dimensional positions obtained by projecting the feature points 530b and 530c onto the plane 730.

[0097] In step S1230, the correction unit 180 determines whether the tilt of the index B and the position of the feature point have been corrected for each index B detected in the camera image. If the correction process for each index B has been completed, the index information correction process shown in FIG. 12 ends. If the correction process for each index B has not been completed, the process returns to step S1210. In the example of FIG. 7(B), when the correction process for the index B 520a is completed and the process returns to step S1210, the correction unit 180 also performs the same correction process for the index B 520b as for the index B 520a. By the index information correction process of FIG. 12, the correction unit 180 can correct the positions of the feature points 530b to 530d to match the height of the workbench 500, as shown in FIG. 7(C).

[0098] In the first embodiment described above, the information processing device 100 estimates the position and orientation of the camera based on information on index A and index B. By using information on both index A and index B, the information processing device 100 can more accurately estimate the three-dimensional shape by detecting and correcting index B even in a scene where there is no area in which multiple indices A are placed. By using the key frame information storing the shape, the information processing device 100 can obtain the position and orientation of the camera with higher accuracy. Furthermore, by correcting the tilt of the index B, the information processing device 100 can reduce accumulated errors in the three-dimensional map (key frame information).

[0099] (Variation 1) 12, the correction unit 180 selects the feature points grouped with index B as the correction target. In Modification 1, the correction unit 180 does not select the feature points grouped with index B as the correction target, but detects feature points that satisfy predetermined conditions and selects them as the correction target.

[0100] For example, the correction unit 180 may extract, as correction targets (candidates), feature points that have the same normal vector as the normal vector of index B and that are within a predetermined distance from the three-dimensional position of index B on a plane that passes through the three-dimensional position of index B, from the feature point information in the storage unit 150. The predetermined distance from the position of index B may be set to, for example, 1 m when correcting feature points on the top plate of the workbench 500 illustrated in FIG. 5. The predetermined distance from index B when selecting feature points to be corrected may be set individually for each index B based on information such as the scene in which index B is located and the distance from adjacent index B.

[0101] In variant example 1, even if the feature points are not pre-grouped with index B, the information processing device 100 can reduce the estimation error of the camera position and orientation by correcting the positions of the feature points that satisfy predetermined conditions.

[0102] (Variation 2) 9, the index B detection unit 250 detects index B based on the luminance and hue of a reference point set in a region surrounding a feature point located at the center of index B. In Modification 2, the index B detection unit 250 detects index B by using, for example, pattern matching, rather than using pixel information such as the luminance and hue of the reference point.

[0103] The index B detection unit 250 can detect index B by comparing an area including a feature point detected in a camera image with a pattern indicating the shape of index B. In the pattern matching process, various patterns indicating the shape of index B are prepared in advance. The storage unit 150 stores image patterns in which the center point of index B is viewed from various directions and angles as patterns indicating the shape of index B. The index B detection unit 250 detects index B by comparing an area including a feature point with the pattern indicating the shape of index B. The index B detection unit 250 can determine the degree of match between the pattern indicating the shape of index B and the pattern of pixels surrounding the feature point. If the degree of match is higher than a threshold for the degree of match, the index B detection unit 250 can detect the area including the feature point as index B.

[0104] In the second modification, the information processing device 100 can detect the index B with high accuracy by pattern matching processing.

[0105] Second Embodiment In the first embodiment, the index B attitude correcting unit 310 of the correcting unit 180 corrects the tilt of index B using the normal direction of index A. Specifically, if the normal directions of index A and index B substantially match in step S1210 of Fig. 12, the index B attitude correcting unit 310 replaces the normal vector of index B with the normal vector of index A. In contrast, in the second embodiment, the index B attitude correcting unit 310 corrects the tilt of index B based on the gravity axis direction measured by a sensor.

[0106] 13 is a block diagram showing an example of the configuration of an information processing system 10 according to the second embodiment. The information processing system 10 to which the second embodiment is applied includes an information processing device 100, an imaging unit 103, a display unit 110, an operation unit 120, and a sensor 104. In the second embodiment, the same functional components as those in the first embodiment are denoted by the same reference numerals, and detailed description thereof will be omitted. The hardware configuration of the information processing device 100 according to the second embodiment is the same as the hardware configuration of the information processing device 100 according to the first embodiment described in FIG. 4.

[0107] The sensor 104 is an inertial sensor fixed to the image capturing unit 103. The sensor 104 has an angular velocity sensor and an acceleration sensor mounted therein. The sensor 104 measures the direction of the gravity axis in a coordinate system based on the image capturing unit 103, using the angular velocity sensor and the acceleration sensor. The sensor 104 records the measured direction of the gravity axis in the storage unit 150 in association with the captured camera image. The index B attitude correction unit 310 corrects the normal direction of index B using the direction of the gravity axis associated with the camera image.

[0108] 12, the index B attitude correction unit 310 inverts the gravity axis direction associated with the camera image by 180 degrees and converts it into a vector U representing the vertically upward direction opposite to the gravity axis direction. The index B attitude correction unit 310 compares the normal vector of the index B to be corrected with the vector U and determines whether the angular difference is within a predetermined angle range. If the angular difference is within the predetermined angle range, the index B attitude correction unit 310 replaces the normal vector of index B with the vector U.

[0109] In the second embodiment described above, the information processing device 100 corrects the normal direction of index B using the vector U in the vertically upward direction of the gravity axis obtained by the sensor 104, rather than the normal direction of index A. Therefore, when index B is placed on a horizontal surface such as the top plate of the workbench 500, the information processing device 100 can accurately correct the normal direction of index B even if an error occurs when detecting the normal direction of index A. Furthermore, when index A is placed on a horizontal surface, the information processing device 100 can also correct the normal direction of index A using the gravity axis direction. In the second embodiment as well, the information processing device 100 can more accurately determine the position and orientation of the camera. Furthermore, by correcting the tilts of indexes A and B, the information processing device 100 can reduce accumulated errors in the three-dimensional map (keyframe information).

[0110] The various controls described above may or may not be performed by a single piece of hardware (e.g., a processor or circuit). The entire device may be controlled by multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) sharing the processing.

[0111] The above processor is a processor in a broad sense, and includes general-purpose processors and dedicated processors. General-purpose processors include, for example, CPUs (Central Processing Units), MPUs (Micro Processing Units), and DSPs (Digital Signal Processors). Dedicated processors include, for example, GPUs (Graphics Processing Units), ASICs (Application Specific Integrated Circuits), and PLDs (Programmable Logic Devices). Programmable logic devices include, for example, FPGAs (Field Programmable Gate Arrays) and CPLDs (Complex Programmable Logic Devices).

[0112] Furthermore, although the embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. The above-described embodiments merely represent one embodiment of the present invention, and the embodiments can be combined as appropriate.

[0113] <Other embodiments> The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program, or by a circuit that realizes one or more of the functions.

[0114] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) a detection means for detecting a first marker and a feature point from an image captured by an imaging device, and detecting a second marker based on the feature point; an estimation means for estimating the position and orientation of the imaging device based on information about the first marker and information about the second marker; An information processing device comprising: (Configuration 2) The estimation means estimates the position and orientation of the imaging device based on the shape of the first marker detected by the detection means. 2. The information processing device according to configuration 1, (Configuration 3) The detection means detects the second marker based on information about pixels located around the feature point. 3. The information processing device according to configuration 1 or 2. (Configuration 4) the second marker has a rotationally symmetric shape; The detection means detects the second marker by determining whether or not at least one of a difference in luminance and a difference in hue between pixels arranged rotationally symmetrically around the feature point is smaller than a respective predetermined threshold value. 4. The information processing device according to configuration 3. (Configuration 5) The detection means detects the second marker based on whether information on pixels located around the feature point corresponds to the shape of the second marker. 5. The information processing device according to configuration 3 or 4. (Configuration 6) The detection means detects the second marker by comparing a region including the feature point with a pattern indicating the shape of the second marker that has been prepared in advance. 3. The information processing device according to configuration 1 or 2. (Configuration 7) The apparatus further includes a first correction means for correcting the tilt of the second marker based on the tilt of the first marker when the difference between the tilt of the first marker and the tilt of the second marker is smaller than a predetermined tilt threshold. 7. The information processing device according to any one of configurations 1 to 6. (Configuration 8) The first correction means corrects the tilt of the second marker by correcting the normal direction of the second marker using the normal direction of the first marker. 8. The information processing device according to configuration 7. (Configuration 9) The apparatus further includes a first correction means for correcting the tilt of the second marker based on the gravity axis direction measured by the sensor. 7. The information processing device according to any one of configurations 1 to 6. (Configuration 10) The first correction means corrects the tilt of the second marker when the difference in angle between the direction opposite to the gravity axis direction and the normal direction of the second marker is within a predetermined angle range. 10. The information processing device according to configuration 9. (Configuration 11) The image capturing device further includes a second correction means for correcting the positions of the feature points around the second marker based on the tilt and position of the second marker. 11. The information processing device according to any one of configurations 1 to 10. (Configuration 12) The second correction means corrects the feature points that are present within a predetermined distance range from the position of the second marker and whose distance from the plane on which the second marker is arranged is shorter than a predetermined distance threshold. 12. The information processing device according to configuration 11. (Configuration 13) The second correction means corrects the feature points that are detected in the captured images of more than a predetermined number of past key frames among the feature points detected by the detection means. 13. The information processing device according to configuration 11 or 12. (Configuration 14) The detection means detects the second marker based on the feature points detected in the captured images of more than a predetermined number of past key frames. 14. The information processing device according to any one of configurations 1 to 13. (Configuration 15) The second marker is smaller than the first marker. 15. The information processing device according to any one of configurations 1 to 14. (method) detecting a first marker and a feature point from an image captured by an imaging device, and detecting a second marker based on the feature point; estimating the position and orientation of the image capturing device based on information about the first marker and information about the second marker; An information processing method comprising: (program) 16. A program for causing a computer to function as each means of the information processing device according to any one of configurations 1 to 15. [Explanation of symbols]

[0115] 100: Information processing device, 103: Image capture unit, 105: Image acquisition unit, 155: Index information management unit, 190: Position and orientation estimation unit

Claims

1. a detection means for detecting a first marker and a feature point from an image captured by an imaging device, and detecting a second marker based on the feature point; an estimation means for estimating the position and orientation of the imaging device based on information about the first marker and information about the second marker; An information processing device comprising:

2. The estimation means estimates the position and orientation of the imaging device based on the shape of the first marker detected by the detection means.

2. The information processing apparatus according to claim 1, wherein:

3. The detection means detects the second marker based on information about pixels located around the feature point.

2. The information processing apparatus according to claim 1, wherein:

4. the second marker has a rotationally symmetric shape; The detection means detects the second marker by determining whether or not at least one of a difference in luminance and a difference in hue between pixels arranged rotationally symmetrically around the feature point is smaller than a respective predetermined threshold value.

4. The information processing apparatus according to claim 3,

5. The detection means detects the second marker based on whether information on pixels located around the feature point corresponds to the shape of the second marker.

4. The information processing apparatus according to claim 3,

6. The detection means detects the second marker by comparing a region including the feature point with a pattern indicating the shape of the second marker that has been prepared in advance.

2. The information processing apparatus according to claim 1, wherein:

7. The apparatus further includes a first correction means for correcting the tilt of the second marker based on the tilt of the first marker when a difference between the tilt of the first marker and the tilt of the second marker is smaller than a predetermined tilt threshold.

2. The information processing apparatus according to claim 1, wherein:

8. The first correction means corrects the tilt of the second marker by correcting the normal direction of the second marker using the normal direction of the first marker.

8. The information processing apparatus according to claim 7,

9. The apparatus further includes a first correction means for correcting the tilt of the second marker based on the gravity axis direction measured by the sensor.

2. The information processing apparatus according to claim 1, wherein:

10. The first correction means corrects the tilt of the second marker when the difference in angle between the direction opposite to the gravity axis direction and the normal direction of the second marker is within a predetermined angle range.

10. The information processing apparatus according to claim 9,

11. The image capturing device further includes a second correction means for correcting the positions of the feature points around the second marker based on the tilt and position of the second marker.

2. The information processing apparatus according to claim 1, wherein:

12. The second correction means corrects the feature points that are present within a predetermined distance range from the position of the second marker and whose distance from the plane on which the second marker is arranged is shorter than a predetermined distance threshold.

12. The information processing apparatus according to claim 11,

13. The second correction means corrects, among the feature points detected by the detection means, feature points that are also detected in the captured images of more than a predetermined number of past key frames.

12. The information processing apparatus according to claim 11,

14. The detection means detects the second marker based on the feature points detected in the captured images of more than a predetermined number of past key frames.

2. The information processing apparatus according to claim 1, wherein:

15. The second marker is smaller than the first marker.

2. The information processing apparatus according to claim 1, wherein:

16. detecting a first marker and a feature point from an image captured by an imaging device, and detecting a second marker based on the feature point; estimating the position and orientation of the image capturing device based on information about the first marker and information about the second marker; An information processing method comprising:

17. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 15.

Images