Information processing device, information processing method, and program

Abstract

In order to properly estimate at least one of the position and orientation of a target object, an information processing device (1) is provided with: a depth information acquisition unit (11) that acquires depth information; a captured image acquisition unit (12) that acquires a captured image; a generation unit (13) that generates a candidate solution for at least one of the position and orientation of a target object in a three-dimensional space with reference to first two-dimensional data obtained with reference to the depth information and third two-dimensional data obtained from a three-dimensional model relating to the target object; and a calculation unit (14) that calculates at least one of the position and orientation of the target object in the three-dimensional space using the candidate solution with reference to second two-dimensional data obtained with reference to the captured image and fourth two-dimensional data obtained from the three-dimensional model.

Description

[Technical Field]

[0001] The present invention relates to an information processing device and an information processing method for calculating at least one of the position and the orientation of an object. law, and program Regarding. [Background technology]

[0002] 2. Description of the Related Art Conventionally, there is known a technique for estimating the position and orientation of an object in real space by analyzing a captured image that includes the object in its angle of view.

[0003] For example, Non-Patent Document 1 discloses a technology for estimating the position and orientation of an object by comparing two-dimensional data obtained by projecting three-dimensional point cloud data of the object generated in advance into two dimensions with a captured image that includes the object in its angle of view. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Martin A. Fischler and Robert C. Bolles, "Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography," June 1, 1981. Summary of the Invention [Problem to be solved by the invention]

[0005] The technology in Non-Patent Document 1 requires searching a space of six axes, namely, position (x, y, z) and attitude (roll, pitch, yaw), which results in an enormous search space, resulting in increased computational costs and time.

[0006] One aspect of the present invention has been made in consideration of the above-mentioned problems, and one example of its purpose is to provide a technology that can suitably estimate at least one of the position and orientation of an object while reducing calculation costs and calculation time. [Means for solving the problem]

[0007] An information processing device according to one aspect of the present invention includes a depth information acquisition means for acquiring depth information obtained by a depth sensor having an object in its sensing range, an image acquisition means for acquiring an image obtained by an imaging sensor having the object in its angle of view, a generation means for generating one or more candidate solutions for at least one of the position and orientation of the object in three-dimensional space by referring to first two-dimensional data obtained by a first feature point extraction process that references the depth information and a three-dimensional model of the object, and a calculation means for calculating at least one of the position and orientation of the object in three-dimensional space by referring to second two-dimensional data obtained by a second feature point extraction process that references the image and the three-dimensional model, using the one or more candidate solutions.

[0008] An information processing device according to one aspect of the present invention includes a depth information acquisition means for acquiring depth information obtained by a depth sensor having an object in its sensing range, an image acquisition means for acquiring an image obtained by an image sensor having the object in its angle of view, a first matching means for performing a first matching process by referring to first two-dimensional data obtained by a first feature point extraction process with reference to the depth information and a three-dimensional model related to the object, a second matching means for performing a second matching process by referring to second two-dimensional data obtained by a second feature point extraction process with reference to the image and the three-dimensional model, and a calculation means for calculating at least one of the position and orientation of the object in three-dimensional space by referring to the results of the first matching process and the results of the second matching process.

[0009] An information processing method according to one aspect of the present invention includes acquiring depth information obtained by a depth sensor that includes an object in its sensing range, acquiring an image obtained by an imaging sensor that includes the object in its angle of view, generating one or more candidate solutions regarding at least one of the position and orientation of the object in three-dimensional space by referring to first two-dimensional data obtained by a first feature point extraction process that references the depth information and a three-dimensional model of the object, and calculating at least one of the position and orientation of the object in three-dimensional space by referring to second two-dimensional data obtained by a second feature point extraction process that references the captured image and the three-dimensional model, using the one or more candidate solutions.

[0010] An information processing method according to one aspect of the present invention includes acquiring depth information obtained by a depth sensor that includes an object in its sensing range, acquiring an image obtained by an imaging sensor that includes the object in its angle of view, performing a first matching process that references first two-dimensional data obtained by a first feature point extraction process that references the depth information and a three-dimensional model related to the object, performing a second matching process that references second two-dimensional data obtained by a second feature point extraction process that references the captured image and the three-dimensional model, and calculating at least one of the position and orientation of the object in three-dimensional space by referring to the results of the first matching process and the results of the second matching process.

[0011] According to one aspect of the present invention program teeth, A program that causes a computer to function as an information processing device,A program that causes a computer to function as a depth information acquisition means that acquires depth information obtained by a depth sensor that includes an object in its sensing range, an image acquisition means that acquires an image obtained by an image sensor that includes the object in its angle of view, a generation means that generates one or more candidate solutions regarding at least one of the position and orientation of the object in three-dimensional space by referring to first two-dimensional data obtained by a first feature point extraction process that references the depth information and a three-dimensional model of the object, and a calculation means that calculates at least one of the position and orientation of the object in three-dimensional space by referring to second two-dimensional data obtained by a second feature point extraction process that references the image and the three-dimensional model, using the one or more candidate solutions.

[0012] According to one aspect of the present invention program teeth, A program that causes a computer to function as an information processing device, A program that causes a computer to function as a depth information acquisition means that acquires depth information obtained by a depth sensor that includes an object in its sensing range, an image acquisition means that acquires an image obtained by an image sensor that includes the object in its angle of view, a first matching means that performs a first matching process by referring to first two-dimensional data obtained by a first feature point extraction process that references the depth information and a three-dimensional model related to the object, a second matching means that performs a second matching process by referring to second two-dimensional data obtained by a second feature point extraction process that references the image and the three-dimensional model, and a calculation means that calculates at least one of the position and orientation of the object in three-dimensional space by referring to the results of the first matching process and the results of the second matching process. [Effects of the Invention]

[0013] According to one aspect of the present invention, it is possible to suitably estimate at least one of the position and orientation of an object while suppressing calculation costs and calculation time. [Brief explanation of the drawings]

[0014] [Figure 1]1 is a block diagram showing a configuration of an information processing device according to a first exemplary embodiment of the present invention. [Figure 2] 1 is a flowchart showing the flow of an information processing method according to a first exemplary embodiment of the present invention. [Figure 3] 1 is a block diagram showing a configuration of an information processing system according to a first exemplary embodiment of the present invention. [Figure 4] FIG. 10 is a block diagram showing the configuration of an information processing device according to a second exemplary embodiment of the present invention. [Figure 5] FIG. 10 is a flowchart showing the flow of an information processing method according to a second exemplary embodiment of the present invention. [Figure 6] FIG. 10 is a block diagram showing the configuration of an information processing system according to a second exemplary embodiment of the present invention. [Figure 7] FIG. 10 is a block diagram showing the configuration of an information processing system according to a third exemplary embodiment of the present invention. [Figure 8] FIG. 11 is a diagram showing cameras that capture an image of a vessel of a truck, which is an object, and the positions of the cameras in the third exemplary embodiment of the present invention. [Figure 9] 10A to 10C are diagrams illustrating a method by which an RGB image position estimating unit according to the third exemplary embodiment of the present invention calculates the position and orientation of an object in three-dimensional space. [Figure 10] 10 is a flowchart showing the flow of processing executed by an information processing device according to a third exemplary embodiment of the present invention. [Figure 11] 10A to 10C are diagrams showing examples of images referenced and generated in each process executed by an information processing device according to a third exemplary embodiment of the present invention. [Figure 12] FIG. 10 is a block diagram showing the configuration of an information processing system according to a fourth exemplary embodiment of the present invention. [Figure 13] FIG. 10 is a block diagram showing the configuration of an information processing system according to a fifth exemplary embodiment of the present invention. [Figure 14] 10 is a flowchart showing the flow of processing executed by an information processing device according to a fifth exemplary embodiment of the present invention. [Figure 15] FIG. 10 is a block diagram showing the configuration of an information processing system according to a sixth exemplary embodiment of the present invention. [Figure 16] FIG. 2 is a block diagram illustrating an example of a hardware configuration of an information processing device and an information processing system according to each exemplary embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0015] Exemplary Embodiment 1 A first exemplary embodiment of the present invention will be described in detail with reference to the drawings. This exemplary embodiment is a basic form of the exemplary embodiments described below.

[0016] (Configuration of information processing device 1) The configuration of an information processing device 1 according to this exemplary embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of an information processing device 1 according to this exemplary embodiment.

[0017] The information processing device 1 is a device that calculates at least one of the position and posture of an object by referring to depth information obtained by a depth sensor that includes the object in its sensing range and an image obtained by an imaging sensor that includes the object in its angle of view.

[0018] Examples of the object include, but are not limited to, a vessel (bed) of a dump truck, and a box surrounded by edges that can store items inside.

[0019] The information processing device 1 can be widely applied to one or more AGVs (Automatic Guided Vehicles), construction machinery, self-driving vehicles, monitoring systems, etc. For example, the information processing device 1 can be used in a system that calculates at least one of the position and attitude of the dump truck vessel as an object at a work site where earth and sand excavated by a backhoe is loaded into the vessel of a dump truck, and loads earth and sand into the vessel by referring to at least one of the calculated position and attitude.

[0020] Examples of depth sensors include, but are not limited to, a stereo camera that has multiple cameras and determines the distance (depth) to an object based on the parallax between the cameras, or a LiDAR (Light Detection and Ranging) that uses a laser to measure the distance (depth) to an object. Examples of depth information include a depth image representing the depth acquired by the stereo camera, or coordinate data indicating the coordinates of each point acquired by the LiDAR, but these do not limit the present exemplary embodiment. Depth can also be expressed in the form of an image by converting the coordinate data acquired by the LiDAR.

[0021] In this exemplary embodiment, the position of an object is the position of the object in three-dimensional space, and is a concept that includes the translational position of the object. Furthermore, the orientation of an object is the orientation of the object in three-dimensional space, and is a concept that includes the orientation of the object. However, the specific parameters by which the position and orientation of the object are expressed are not intended to limit this exemplary embodiment.

[0022] As an example, the position and orientation of an object can be expressed by the center of gravity (x, y, z) of the object and the orientation (roll, pitch, yaw) of the object, respectively. In this case, the position and orientation of the object are expressed by six parameters (x, y, z, roll, pitch, yaw).

[0023] 1, the information processing device 1 includes a depth information acquisition unit 11, a captured image acquisition unit 12, a generation unit 13, and a calculation unit 14. In this exemplary embodiment, the depth information acquisition unit 11, the captured image acquisition unit 12, the generation unit 13, and the calculation unit 14 respectively function as a depth information acquisition means, a captured image acquisition means, a generation means, and a calculation means.

[0024] The depth information acquisition unit 11 acquires depth information obtained by a depth sensor that includes an object in its sensing range. The depth information acquisition unit 11 supplies the acquired depth information to the generation unit 13.

[0025] The captured image acquisition unit 12 acquires a captured image obtained by an imaging sensor that includes the target object in its angle of view. The captured image acquisition unit 12 supplies the acquired captured image to the calculation unit 14.

[0026] The generation unit 13 generates one or more candidate solutions related to at least one of the position and orientation of the object in three-dimensional space by referring to the first two-dimensional data obtained by the first feature point extraction process with reference to the depth information supplied from the depth information acquisition unit 11 and a three-dimensional model of the object. As an example, the generation unit 13 maps the first two-dimensional data obtained by the first feature point extraction process with reference to the depth information and the three-dimensional model of the object into two-dimensional space, and generates one or more candidate solutions related to at least one of the position and orientation of the object in three-dimensional space by referring to the first two-dimensional data obtained by the first feature point extraction process. The generation unit 13 supplies the generated one or more candidate solutions to the calculation unit 14.

[0027] Here, the first feature point extraction process refers to a process of referring to depth information and extracting one or more feature points included in the depth information. An example of the first feature point extraction process is an edge extraction process of an object using an edge extraction filter. With this configuration, the edge extraction process can be performed on the depth information, allowing the information processing device 1 to suitably extract feature points of the object.

[0028] Furthermore, a three-dimensional model of an object is a model that includes data that represents the size and shape of the object in three-dimensional space, and an example of this is three-dimensional data that is a collection of point data that represents each point contained in the object.

[0029] The calculation unit 14 refers to the second two-dimensional data obtained by the second feature point extraction process with reference to the captured image supplied from the captured image acquisition unit 12 and the three-dimensional model of the object, and calculates at least one of the position and orientation of the object in three-dimensional space using one or more candidate solutions generated by the generation unit 13. As an example, the calculation unit 14 refers to the second two-dimensional data obtained by the second feature point extraction process with reference to the captured image and fourth two-dimensional data obtained by mapping the three-dimensional model of the object into two-dimensional space, and calculates at least one of the position and orientation of the object in three-dimensional space using one or more candidate solutions generated by the generation unit 13.

[0030] Here, the second feature point extraction process refers to a process of extracting one or more feature points contained in a captured image by referring to the captured image. An example of the second feature point extraction process is an edge extraction process of an object using an edge extraction filter. With this configuration, the edge extraction process can be performed on the captured image, and the information processing device 1 can preferably extract feature points of the object.

[0031] Furthermore, the edge extraction filter used in the second feature point extraction processing may be the same as the edge extraction filter used in the first feature point extraction processing, or may be an edge extraction filter different from the edge extraction filter used in the first feature point extraction processing. For example, the edge extraction filter used in the second feature point extraction processing may be a filter having filter coefficients different from those of the edge extraction filter used in the first feature point extraction processing.

[0032] As described above, the information processing device 1 according to this exemplary embodiment is configured to include a depth information acquisition unit 11 that acquires depth information obtained by a depth sensor that includes the object in its sensing range; an image acquisition unit 12 that acquires an image obtained by an imaging sensor that includes the object in its angle of view; a generation unit 13 that generates one or more candidate solutions regarding at least one of the position and orientation of the object in three-dimensional space by referring to first two-dimensional data obtained by a first feature point extraction process that references the depth information and a three-dimensional model of the object; and a calculation unit 14 that calculates at least one of the position and orientation of the object in three-dimensional space by referring to second two-dimensional data obtained by a second feature point extraction process that references the image and a three-dimensional model of the object using one or more candidate solutions.

[0033] More specifically, the information processing device 1 according to this exemplary embodiment is configured to include a depth information acquisition unit 11 that acquires depth information obtained by a depth sensor that includes the object in its sensing range; an image acquisition unit 12 that acquires an image obtained by an imaging sensor that includes the object in its angle of view; a generation unit 13 that generates one or more candidate solutions regarding at least one of the position and orientation of the object in three-dimensional space by referring to first two-dimensional data obtained by a first feature point extraction process that references the depth information and a three-dimensional model of the object onto two-dimensional space and third two-dimensional data obtained by the first feature point extraction process; and a calculation unit 14 that calculates at least one of the position and orientation of the object in three-dimensional space by referring to second two-dimensional data obtained by a second feature point extraction process that references the image and a three-dimensional model of the object onto two-dimensional space and fourth two-dimensional data obtained by the second feature point extraction process, using one or more candidate solutions.

[0034] Therefore, according to the information processing device 1 of this exemplary embodiment, one or more candidate solutions regarding at least one of the position and orientation of the object in three-dimensional space are generated by referring to first two-dimensional data obtained by referring to depth information that has a smaller amount of information than the captured image, so that one or more candidate solutions regarding at least one of the position and orientation of the object in three-dimensional space can be derived with reduced computational cost and time compared to when referring to second two-dimensional data obtained by referring to the captured image.

[0035] Furthermore, the information processing device 1 according to this exemplary embodiment calculates at least one of the position and orientation of the object in three-dimensional space by using one or more candidate solutions with reference to second two-dimensional data obtained by reference to a captured image that contains a larger amount of information than depth information. Therefore, the information processing device 1 according to this exemplary embodiment can calculate at least one of the position and orientation of the object in three-dimensional space with higher accuracy than the first two-dimensional data obtained by reference to depth information. Furthermore, the information processing device 1 according to this exemplary embodiment can reduce calculation costs and calculation time by using one or more candidate solutions compared to when no candidate solutions are used.

[0036] Therefore, the information processing device 1 according to this exemplary embodiment can suitably estimate at least one of the position and orientation of an object while suppressing calculation costs and calculation time.

[0037] (Flow of information processing method S1) The flow of the information processing method S1 according to this exemplary embodiment will be described with reference to Fig. 2. Fig. 2 is a flow diagram showing the flow of the information processing method S1 according to this exemplary embodiment.

[0038] (Step S11) In step S11, the depth information acquisition unit 11 acquires depth information obtained by a depth sensor that includes the object in its sensing range. The depth information acquisition unit 11 supplies the acquired depth information to the generation unit 13.

[0039] (Step S12) In step S12, the captured image acquisition unit 12 acquires a captured image obtained by an imaging sensor that includes the object in its angle of view. The captured image acquisition unit 12 supplies the acquired captured image to the calculation unit .

[0040] (Step S13) In step S13, the generation unit 13 generates one or more candidate solutions related to at least one of the position and orientation of the object in three-dimensional space by referring to the first two-dimensional data obtained by the first feature point extraction process with reference to the depth information supplied from the depth information acquisition unit 11 in step S11 and a three-dimensional model of the object. As an example, the generation unit 13 maps the first two-dimensional data obtained by the first feature point extraction process with reference to the depth information and the three-dimensional model of the object into two-dimensional space, and generates one or more candidate solutions related to at least one of the position and orientation of the object in three-dimensional space by referring to the third two-dimensional data obtained by the first feature point extraction process. The generation unit 13 supplies the generated one or more candidate solutions to the calculation unit 14.

[0041] (Step S14) In step S14, the calculation unit 14 calculates second two-dimensional data obtained by a second feature point extraction process with reference to the captured image supplied from the captured image acquisition unit 12 in step S12. Furthermore, the calculation unit 14 refers to the second two-dimensional data and the three-dimensional model, and calculates at least one of the position and orientation of the object in three-dimensional space using one or more candidate solutions supplied from the generation unit 13 in step S13. As an example, the calculation unit 14 refers to the second two-dimensional data obtained by the second feature point extraction process with reference to the captured image and fourth two-dimensional data obtained by mapping the three-dimensional model of the object into two-dimensional space, and calculates at least one of the position and orientation of the object in three-dimensional space using one or more candidate solutions supplied from the generation unit 13.

[0042] As described above, in the information processing method S1 according to this exemplary embodiment, in step S11, the depth information acquisition unit 11 acquires depth information obtained by a depth sensor whose sensing range includes the object. In step S12, the captured image acquisition unit 12 acquires a captured image obtained by an image sensor whose angle of view includes the object. Also, in the information processing method S1 according to this exemplary embodiment, in step S13, the generation unit 13 references first two-dimensional data obtained by a first feature point extraction process that references the depth information and a three-dimensional model of the object to generate one or more candidate solutions regarding at least one of the position and orientation of the object in three-dimensional space. Also, in the information processing method S1 according to this exemplary embodiment, in step S14, the calculation unit 14 references second two-dimensional data obtained by a second feature point extraction process that references the captured image and a three-dimensional model of the object to calculate at least one of the position and orientation of the object in three-dimensional space using one or more candidate solutions.

[0043] More specifically, in the information processing method S1 according to this exemplary embodiment, in step S11, the depth information acquisition unit 11 acquires depth information obtained by a depth sensor whose sensing range includes the object, and in step S12, the captured image acquisition unit 12 acquires a captured image obtained by an image sensor whose angle of view includes the object. Also, in the information processing method S1 according to this exemplary embodiment, in step S13, the generation unit 13 maps first two-dimensional data obtained by a first feature point extraction process with reference to the depth information and a three-dimensional model of the object onto two-dimensional space, and generates one or more candidate solutions regarding at least one of the position and orientation of the object in three-dimensional space with reference to third two-dimensional data obtained by the first feature point extraction process. Furthermore, in the information processing method S1 according to this exemplary embodiment, in step S14, the calculation unit 14 maps the second two-dimensional data obtained by the second feature point extraction process with reference to the captured image and a three-dimensional model of the object into two-dimensional space, and calculates at least one of the position and orientation of the object in three-dimensional space using one or more candidate solutions with reference to the fourth two-dimensional data obtained by the second feature point extraction process.

[0044] Therefore, according to the information processing method S1 according to this exemplary embodiment, the same effects as those of the information processing device 1 are achieved.

[0045] (Configuration of information processing system 10) The configuration of the information processing system 10 according to this exemplary embodiment will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the configuration of the information processing system 10 according to this exemplary embodiment.

[0046] 3, the information processing system 10 includes a depth information acquisition unit 11, a captured image acquisition unit 12, a generation unit 13, and a calculation unit 14. Also, as shown in FIG. 3, in the information processing system 10, the depth information acquisition unit 11, the captured image acquisition unit 12, the generation unit 13, and the calculation unit 14 are connected to each other via a network N so as to be able to communicate with each other.

[0047] The specific configuration of the network N does not limit this embodiment, but as an example, a wireless LAN (Local Area Network), a wired LAN, a WAN (Wide Area Network), a public line network, a mobile data communication network, or a combination of these networks can be used.

[0048] The depth information acquisition unit 11 acquires depth information obtained by a depth sensor that includes an object in its sensing range. The depth information acquisition unit 11 outputs the acquired depth information to the generation unit 13 via the network N.

[0049] The captured image acquisition unit 12 acquires a captured image obtained by an imaging sensor that includes the object in its angle of view. The captured image acquisition unit 12 outputs the acquired captured image to the calculation unit 14 via the network N.

[0050] The generation unit 13 generates one or more candidate solutions related to at least one of the position and orientation of the object in three-dimensional space by referring to first two-dimensional data obtained by a first feature point extraction process with reference to the depth information output from the depth information acquisition unit 11 and a three-dimensional model of the object. As an example, the generation unit 13 maps the first two-dimensional data obtained by the first feature point extraction process with reference to the depth information and the three-dimensional model of the object into two-dimensional space, and generates one or more candidate solutions related to at least one of the position and orientation of the object in three-dimensional space by referring to third two-dimensional data obtained by the first feature point extraction process. The generation unit 13 outputs the generated one or more candidate solutions to the calculation unit 14 via the network N.

[0051] The calculation unit 14 refers to the second two-dimensional data obtained by the second feature point extraction process with reference to the captured image output from the captured image acquisition unit 12 and the three-dimensional model of the object, and calculates at least one of the position and orientation of the object in three-dimensional space using one or more candidate solutions output from the generation unit 13. As an example, the calculation unit 14 refers to the second two-dimensional data obtained by the second feature point extraction process with reference to the captured image and fourth two-dimensional data obtained by mapping the three-dimensional model of the object into two-dimensional space, and calculates at least one of the position and orientation of the object in three-dimensional space using one or more candidate solutions generated by the generation unit 13.

[0052] As described above, the information processing system 10 according to this exemplary embodiment is configured to include a depth information acquisition unit 11 that acquires depth information obtained by a depth sensor that includes the object in its sensing range; an image acquisition unit 12 that acquires an image obtained by an imaging sensor that includes the object in its angle of view; a generation unit 13 that generates one or more candidate solutions regarding at least one of the position and orientation of the object in three-dimensional space by referring to first two-dimensional data obtained by a first feature point extraction process that references the depth information and a three-dimensional model of the object; and a calculation unit 14 that calculates at least one of the position and orientation of the object in three-dimensional space by referring to second two-dimensional data obtained by a second feature point extraction process that references the image and a three-dimensional model of the object using one or more candidate solutions.

[0053] More specifically, the information processing system 10 according to this exemplary embodiment is configured to include a depth information acquisition unit 11 that acquires depth information obtained by a depth sensor that includes the object in its sensing range; an image acquisition unit 12 that acquires an image obtained by an imaging sensor that includes the object in its angle of view; a generation unit 13 that generates one or more candidate solutions regarding at least one of the position and orientation of the object in three-dimensional space by referring to first two-dimensional data obtained by a first feature point extraction process that references the depth information and a three-dimensional model of the object onto two-dimensional space and third two-dimensional data obtained by the first feature point extraction process; and a calculation unit 14 that calculates at least one of the position and orientation of the object in three-dimensional space by referring to second two-dimensional data obtained by a second feature point extraction process that references the image and fourth two-dimensional data obtained by the second feature point extraction process that maps a three-dimensional model of the object onto two-dimensional space and uses one or more candidate solutions.

[0054] Therefore, the information processing system 10 according to this exemplary embodiment provides the same effects as the information processing device 1.

[0055] Exemplary Embodiment 2 A second exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first exemplary embodiment are given the same reference numerals, and their description will be omitted as appropriate.

[0056] (Configuration of information processing device 2) The configuration of the information processing device 2 according to this exemplary embodiment will be described with reference to Fig. 4. Fig. 4 is a block diagram showing the configuration of the information processing device 2 according to this exemplary embodiment.

[0057] The information processing device 2 is a device that calculates at least one of the position and orientation of an object by referring to depth information obtained by a depth sensor that includes the object in its sensing range and an image captured by an imaging sensor that includes the object in its angle of view. The object, depth information, and position and orientation of the object are as described in the above-mentioned embodiment.

[0058] 4, the information processing device 2 includes a depth information acquisition unit 11, a captured image acquisition unit 12, a first matching unit 23, a second matching unit 24, and a calculation unit 25. In this exemplary embodiment, the depth information acquisition unit 11, the captured image acquisition unit 12, the first matching unit 23, the second matching unit 24, and the calculation unit 25 respectively implement a depth information acquisition means, a captured image acquisition means, a first matching means, a second matching means, and a calculation means.

[0059] The depth information acquisition unit 11 acquires depth information obtained by a depth sensor that includes an object in its sensing range. The depth information acquisition unit 11 supplies the acquired depth information to the first matching unit 23.

[0060] The captured image acquisition unit 12 acquires a captured image obtained by an imaging sensor that includes the target object in its angle of view. The captured image acquisition unit 12 supplies the acquired captured image to the second matching unit 24.

[0061] The first matching unit 23 performs a first matching process by referring to the first two-dimensional data obtained by the first feature point extraction process by referring to the depth information supplied from the depth information acquisition unit 11 and a three-dimensional model of the object. As an example, the first matching unit 23 performs a first matching process by referring to the first two-dimensional data obtained by the first feature point extraction process by referring to the depth information and the third two-dimensional data obtained by mapping the three-dimensional model of the object into two-dimensional space and performing the first matching process. The first feature point extraction process is as described in the above-mentioned embodiment.

[0062] The first matching process is a process of referring to the first two-dimensional data and a three-dimensional model of the object, and determining whether the position of the object included in the first two-dimensional data matches the position of the object indicated by the three-dimensional model. The first matching unit 23 supplies the result of the first matching process to the calculation unit 25.

[0063] The second matching unit 24 executes a second matching process by referring to the second two-dimensional data obtained by the second feature point extraction process by referring to the captured image supplied from the captured image acquisition unit 12 and the three-dimensional model. As an example, the second matching unit 24 executes a second matching process by referring to the second two-dimensional data obtained by the second feature point extraction process by referring to the captured image and the fourth two-dimensional data obtained by mapping a three-dimensional model of the object into two-dimensional space and executing the second feature point extraction process. The second feature point extraction process is as described in the above-mentioned embodiment.

[0064] The second matching process is a process of referring to the second two-dimensional data and a three-dimensional model of the object, and determining whether the position of the object included in the second two-dimensional data matches the position of the object indicated by the three-dimensional model. The second matching unit 24 supplies the result of the second matching process to the calculation unit 25.

[0065] The calculation unit 25 calculates at least one of the position and orientation of the object in three-dimensional space by referring to the result of the first matching process supplied from the first matching unit 23 and the result of the second matching process supplied from the second matching unit 24.

[0066] As described above, the information processing device 2 according to this exemplary embodiment employs a configuration including a depth information acquisition unit 11 that acquires depth information obtained by a depth sensor that includes the object in its sensing range, an image acquisition unit 12 that acquires an image obtained by an imaging sensor that includes the object in its angle of view, a first matching unit 23 that performs a first matching process that references first two-dimensional data obtained by a first feature point extraction process that references the depth information and a three-dimensional model related to the object, a second matching unit 24 that performs a second matching process that references second two-dimensional data obtained by a second feature point extraction process that references the image and a three-dimensional model related to the object, and a calculation unit 25 that calculates at least one of the position and orientation of the object in three-dimensional space by referring to the results of the first matching process and the results of the second matching process.

[0067] More specifically, the information processing device 2 according to this exemplary embodiment is configured to include a depth information acquisition unit 11 that acquires depth information obtained by a depth sensor that includes the object in its sensing range; an image acquisition unit 12 that acquires an image obtained by an imaging sensor that includes the object in its angle of view; a first matching unit 23 that performs a first matching process by referencing first two-dimensional data obtained by a first feature point extraction process that references the depth information and a three-dimensional model of the object onto two-dimensional space and third two-dimensional data obtained by the first feature point extraction process; a second matching unit 24 that performs a second matching process by referencing second two-dimensional data obtained by a second feature point extraction process that references the image and a three-dimensional model of the object onto two-dimensional space and fourth two-dimensional data obtained by the second feature point extraction process; and a calculation unit 25 that calculates at least one of the position and orientation of the object in three-dimensional space by referencing the results of the first matching process and the second matching process.

[0068] Therefore, according to the information processing device 2 of this exemplary embodiment, at least one of the position and orientation of the object in three-dimensional space is calculated by referring to the results of a first matching process that references first two-dimensional data obtained by referring to depth information that has less information volume than the captured image, and the results of a second matching process that references second two-dimensional data obtained by referring to the captured image that has more information volume than the depth information.

[0069] Therefore, according to the information processing device 2 of this exemplary embodiment, it is possible to derive at least one of the position and orientation of the object in three-dimensional space by referring to the results of the first matching process, which refers to depth information that has less information volume than the captured image, while reducing computational costs and computational time.

[0070] On the other hand, the information processing device 2 according to this exemplary embodiment can calculate with higher accuracy at least one of the position and orientation of the object in three-dimensional space by referring to the result of the second matching process that refers to the captured image, which has a larger amount of information than the depth information. That is, the information processing device 2 according to this exemplary embodiment can preferably estimate at least one of the position and orientation of the object while suppressing the calculation cost and calculation time.

[0071] (Information Processing Method S 2 Flow) The flow of the information processing method S2 according to this exemplary embodiment will be described with reference to Fig. 5. Fig. 5 is a flow diagram showing the flow of the information processing method S2 according to this exemplary embodiment.

[0072] (Step S11) In step S11, the depth information acquisition unit 11 acquires depth information obtained by a depth sensor that includes an object in its sensing range. The depth information acquisition unit 11 converts the acquired depth information into First matching unit 23 Supply to.

[0073] (Step S12) In step S12, the captured image acquisition unit 12 acquires a captured image obtained by an imaging sensor that includes the object in its angle of view. Second matching unit 24 Supply to.

[0074] (Step S23) In step S23, the first matching unit 23 performs a first matching process by referring to the first two-dimensional data obtained by the first feature point extraction process with reference to the depth information supplied from the depth information acquisition unit 11 in step S11 and a three-dimensional model of the object. As an example, the first matching unit 23 performs a first matching process by referring to the first two-dimensional data obtained by the first feature point extraction process with reference to the depth information and third two-dimensional data obtained by mapping the three-dimensional model of the object into two-dimensional space and performing the first matching process. The first matching unit 23 supplies the result of the first matching process to the calculation unit 25.

[0075] (Step S24) In step S24, the second matching unit 24 performs a second matching process by referring to the second two-dimensional data obtained by the second feature point extraction process by referring to the captured image supplied from the captured image acquisition unit 12 in step S12 and the three-dimensional model. As an example, the second matching unit 24 performs the second matching process by referring to the second two-dimensional data obtained by the second feature point extraction process by referring to the captured image and fourth two-dimensional data obtained by mapping a three-dimensional model of the object into two-dimensional space and performing the second feature point extraction process. The second matching unit 24 supplies the result of the second matching process to the calculation unit 25.

[0076] (Step S25) In step S25, the calculation unit 25 calculates at least one of the position and orientation of the object in three-dimensional space by referring to the result of the first matching process supplied from the first matching unit 23 in step S23 and the result of the second matching process supplied from the second matching unit 24 in step S24.

[0077] As described above, in the information processing method S2 according to this exemplary embodiment, in step S11, the depth information acquisition unit 11 acquires depth information obtained by a depth sensor whose sensing range includes the object, and in step S12, the captured image acquisition unit 12 acquires a captured image obtained by an image sensor whose angle of view includes the object. Also, in the information processing method S2 according to this exemplary embodiment, in step S23, the first matching unit 23 performs a first matching process by referring to first two-dimensional data obtained by a first feature point extraction process with reference to the depth information and a three-dimensional model of the object, and in step S24, the second matching unit 24 performs a second matching process by referring to second two-dimensional data obtained by a second feature point extraction process with reference to the captured image and the three-dimensional model of the object. Furthermore, in the information processing method S2 according to this exemplary embodiment, in step S25, the calculation unit 25 calculates at least one of the position and orientation of the object in three-dimensional space by referring to the results of the first matching process and the results of the second matching process.

[0078] More specifically, in the information processing method S2 according to this exemplary embodiment, in step S11, the depth information acquisition unit 11 acquires depth information obtained by a depth sensor whose sensing range includes the object, and in step S12, the captured image acquisition unit 12 acquires a captured image obtained by an image sensor whose angle of view includes the object. Also, in the information processing method S2 according to this exemplary embodiment, in step S23, the first matching unit 23 maps first two-dimensional data obtained by a first feature point extraction process using the depth information and a three-dimensional model of the object onto two-dimensional space, and performs a first matching process using third two-dimensional data obtained by the first feature point extraction process. In step S24, the second matching unit 24 maps second two-dimensional data obtained by a second feature point extraction process using the captured image and a three-dimensional model of the object onto two-dimensional space, and performs a second matching process using fourth two-dimensional data obtained by the second feature point extraction process. Furthermore, in the information processing method S2 according to this exemplary embodiment, in step S25, the calculation unit 25 calculates at least one of the position and orientation of the object in three-dimensional space by referring to the results of the first matching process and the results of the second matching process.

[0079] Therefore, the information processing method S2 according to this exemplary embodiment provides the same effects as the information processing device 2.

[0080] (Configuration of information processing system 20) The configuration of the information processing system 20 according to this exemplary embodiment will be described with reference to Fig. 6. Fig. 6 is a block diagram showing the configuration of the information processing system 20 according to this exemplary embodiment.

[0081] 6, the information processing system 20 includes a depth information acquisition unit 11, a captured image acquisition unit 12, a first matching unit 23, a second matching unit 24, and a calculation unit 25. Also, as shown in FIG. 6, in the information processing system 20, the depth information acquisition unit 11, the captured image acquisition unit 12, the first matching unit 23, the second matching unit 24, and the calculation unit 25 are connected to each other so as to be able to communicate with each other via a network N. The network N is as described in the above-mentioned embodiment.

[0082] The depth information acquisition unit 11 acquires depth information obtained by a depth sensor that includes an object in its sensing range. The depth information acquisition unit 11 outputs the acquired depth information to the first matching unit 23 via the network N.

[0083] The captured image acquisition unit 12 acquires a captured image obtained by an imaging sensor that includes the target object in its angle of view. The captured image acquisition unit 12 outputs the acquired captured image to the second matching unit 24 via the network N.

[0084] The first matching unit 23 performs a first matching process by referring to the first two-dimensional data obtained by the first feature point extraction process with reference to the depth information output from the depth information acquisition unit 11 and a three-dimensional model of the object. As an example, the first matching unit 23 performs a first matching process by referring to the first two-dimensional data obtained by the first feature point extraction process with reference to the depth information and third two-dimensional data obtained by mapping the three-dimensional model of the object into two-dimensional space and performing the first matching process. The first matching unit 23 outputs the result of the first matching process to the calculation unit 25 via the network N.

[0085] The second matching unit 24 performs a second matching process by referring to the second two-dimensional data obtained by the second feature point extraction process by referring to the captured image output from the captured image acquisition unit 12 and the three-dimensional model. As an example, the second matching unit 24 performs the second matching process by referring to the second two-dimensional data obtained by the second feature point extraction process by referring to the captured image and fourth two-dimensional data obtained by mapping a three-dimensional model of the object into two-dimensional space and performing the second feature point extraction process. The second matching unit 24 outputs the result of the second matching process to the calculation unit 25 via the network N.

[0086] The calculation unit 25 calculates at least one of the position and orientation of the object in three-dimensional space by referring to the result of the first matching process output from the first matching unit 23 and the result of the second matching process output from the second matching unit 24.

[0087] As described above, the information processing system 20 according to this exemplary embodiment includes a depth information acquisition unit 11 that acquires depth information obtained by a depth sensor that includes the object in its sensing range, an image acquisition unit 12 that acquires an image obtained by an imaging sensor that includes the object in its angle of view, a first matching unit 23 that performs a first matching process by referring to first two-dimensional data obtained by a first feature point extraction process that references the depth information and a three-dimensional model related to the object, a second matching unit 24 that performs a second matching process by referring to second two-dimensional data obtained by a second feature point extraction process that references the image and a three-dimensional model related to the object, and a calculation unit 25 that calculates at least one of the position and orientation of the object in three-dimensional space by referring to the results of the first matching process and the results of the second matching process.

[0088] More specifically, the information processing system 20 according to this exemplary embodiment includes a depth information acquisition unit 11 that acquires depth information obtained by a depth sensor that includes the object in its sensing range; an image acquisition unit 12 that acquires an image obtained by an imaging sensor that includes the object in its angle of view; a first matching unit 23 that performs a first matching process by referring to first two-dimensional data obtained by a first feature point extraction process that references the depth information and a three-dimensional model of the object onto two-dimensional space and third two-dimensional data obtained by the first feature point extraction process; a second matching unit 24 that performs a second matching process by referring to second two-dimensional data obtained by a second feature point extraction process that references the image and a three-dimensional model of the object onto two-dimensional space and fourth two-dimensional data obtained by the second feature point extraction process; and a calculation unit 25 that calculates at least one of the position and orientation of the object in three-dimensional space by referring to the results of the first matching process and the second matching process.

[0089] Therefore, the information processing system 20 according to this exemplary embodiment provides the same effects as the information processing device 2.

[0090] Exemplary Embodiment 3 A third exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the above exemplary embodiment are denoted by the same reference numerals, and their description will not be repeated.

[0091] (Configuration of information processing system 100) The configuration of the information processing system 100 according to this exemplary embodiment will be described with reference to Fig. 7. Fig. 7 is a block diagram showing the configuration of the information processing system 100 according to this exemplary embodiment.

[0092] As shown in FIG. 7, the information processing system 100 includes an information processing device 3, a depth sensor 4, and an RGB (Red, Green, Blue) camera 5. In the information processing system 100, the information processing device 3 acquires depth information including an object captured by the depth sensor 4 within its sensing range, and acquires imaging information including the object captured by the RGB camera 5 within its angle of view. The information processing device 3 then refers to the acquired depth information and imaging information to calculate at least one of the position and orientation of the object. The object, depth information, and position and orientation of the object are as described in the above-described embodiment.

[0093] The depth sensor 4 is a sensor that outputs depth information indicating the distance to an object included in the sensing range. As described in the above embodiment, examples of the depth sensor 4 include, but are not limited to, a stereo camera equipped with multiple cameras and a LiDAR. As described in the above embodiment, examples of the depth information include, but are not limited to, a depth image indicating depth and coordinate data indicating the coordinates of each point.

[0094] The RGB camera 5 is a camera equipped with an imaging sensor that captures an object included in its angle of view and outputs imaging data that includes the object in its angle of view. The information processing system 100 is not limited to the RGB camera 5, and may be configured to include a camera that outputs a multi-value image. For example, instead of the RGB camera 5, the information processing system 100 may be configured to include a monochrome camera that outputs a black-and-white image that expresses the captured object in gradations of black and white.

[0095] (Configuration of information processing device 3) As shown in FIG. 7, the information processing device 3 includes a control unit 31, an output unit 32, and a storage unit 33.

[0096] The output unit 32 is a device that outputs data supplied from the control unit 31, which will be described later. One example of the output unit 32 outputting data is a configuration in which the output unit 32 is connected to a network (not shown) and outputs data to another device that can communicate via the network. Another example of the output unit 32 outputting data is a configuration in which the output unit 32 is connected to a display (not shown, for example, a display panel) and outputs data indicating an image to be displayed on the display. These examples do not limit the present exemplary embodiment.

[0097] The storage unit 33 stores various data referenced by the control unit 31, which will be described later. As an example, the storage unit 33 stores a 3D model 331, which is a three-dimensional model of an object. The 3D model 331 may be defined by a mesh or surface used in 3D modeling, or may be a model that explicitly includes data on the edges (contours) of the object, or may define a texture that indicates features in an image of the object. Since the 3D model 331 is configured to explicitly include data on the edges (contours) of the object, edge extraction processing can be performed on the 3D model 331, allowing the information processing device 3 to suitably extract feature points of the object. Furthermore, the 3D model 331 may include data on the vertices of the object. The three-dimensional model of the object is as described in the above-mentioned embodiment.

[0098] (Control unit 31) The control unit 31 controls each component of the information processing device 3. For example, the control unit 31 acquires data from the storage unit 33 and outputs data to the output unit 32.

[0099] 7, the control unit 31 also functions as a depth information acquisition unit 311, a depth image feature point extraction unit 312, a depth image position estimation unit 313, an RGB image acquisition unit 314, an RGB image feature point extraction unit 315, and an RGB image position estimation unit 316. In this exemplary embodiment, the depth information acquisition unit 311, the depth image position estimation unit 313, the RGB image acquisition unit 314, and the RGB image position estimation unit 316 respectively function as a depth information acquisition unit, a generation unit, a captured image acquisition unit, and a calculation unit.

[0100] The depth information acquisition unit 311 acquires depth information obtained by the depth sensor 4 that includes the target object in its sensing range. Even if the target object does not exist in the sensing range, the depth information acquisition unit 311 acquires depth information related to the sensing range that is obtained by the depth sensor 4. The depth information acquisition unit 311 supplies the acquired depth information to the depth image feature point extraction unit 312.

[0101] The depth image feature point extraction unit 312 executes a first feature point extraction process with reference to the depth information supplied from the depth information acquisition unit 311, and generates first two-dimensional data. The depth image feature point extraction unit 312 supplies the generated first two-dimensional data to the depth image position estimation unit 313. The first feature point extraction process is as described in the above-mentioned embodiment. An example of the process executed by the depth image feature point extraction unit 312 will be described later with reference to different drawings.

[0102] The depth image position estimation unit 313 generates one or more candidate solutions regarding at least one of the position and orientation of the object in three-dimensional space by referring to the first two-dimensional data supplied from the depth image feature point extraction unit 312 and the 3D model 331 stored in the storage unit 33. The depth image position estimation unit 313 supplies the generated one or more candidate solutions to the RGB image position estimation unit 316. An example of processing executed by the depth image position estimation unit 313 will be described later with reference to different drawings.

[0103] The RGB image acquisition unit 314 acquires an RGB image (captured image) that includes the target object in its angle of view, obtained by the RGB camera 5. The RGB image acquisition unit 314 supplies the acquired RGB image to the RGB image feature point extraction unit 315.

[0104] The RGB image feature point extraction unit 315 executes second feature point extraction processing with reference to the RGB image supplied by the RGB image acquisition unit 314, and generates second two-dimensional data. The RGB image feature point extraction unit 315 supplies the generated second two-dimensional data to the RGB image position estimation unit 316. The second feature point extraction processing is as described in the above-mentioned embodiment. An example of the processing executed by the RGB image feature point extraction unit 315 will be described later with reference to different drawings.

[0105] The RGB image position estimation unit 316 refers to the second two-dimensional data supplied from the RGB image feature point extraction unit 315 and the 3D model 331 stored in the storage unit 33, and calculates at least one of the position and orientation of the object in three-dimensional space using one or more candidate solutions supplied from the depth image position estimation unit 313. The RGB image position estimation unit 316 supplies at least one of the calculated position and orientation of the object in three-dimensional space to the output unit 32. An example of the processing executed by the RGB image position estimation unit 316 will be described later.

[0106] (Example of a method for calculating the position and orientation of an object in three-dimensional space) An example of a method by which the RGB image position estimating unit 316 calculates the position and orientation of an object in three-dimensional space will be described with reference to Fig. 8 and Fig. 9. Fig. 8 is a diagram showing the positions of cameras CA1 and CA2 that capture an image of a truck vessel RT, which is an object, in this exemplary embodiment. Fig. 9 is a diagram showing a method by which the RGB image position estimating unit 316 according to this exemplary embodiment calculates the position and orientation of an object in three-dimensional space.

[0107] For example, when an object RT is captured using the camera CA1 shown in Fig. 8, the image output by the camera CA1 is image P1 shown in Fig. 9. The RGB image position estimation unit 316 calculates the coordinates of the object RT in the global coordinate system (the position and orientation of the object RT in three-dimensional space) by moving and rotating the 3D model based on the position parameters to determine the position of the object RT included in the image P1.

[0108] Here, the position parameters represent the positions and orientations that the object RT can take. Examples of the position parameters will be described later with reference to different drawings.

[0109] As another example, when an image of an object RT is captured using the camera CA2 shown in Fig. 8, the image output by the camera CA2 is image P2 shown in Fig. 9. The RGB image position estimation unit 316 calculates the coordinates of the object RT in the global coordinate system (the position and orientation of the object RT in three-dimensional space) by moving and rotating the 3D model based on the position parameters to determine the position of the object RT included in the image P2.

[0110] (Flow of processing executed by information processing device 3) The flow of processing executed by the information processing device 3 will be described with reference to FIGS. 10 and 11. FIG. 10 is a flowchart showing the flow of processing executed by the information processing device 3 according to this exemplary embodiment. FIG. 11 is a diagram showing examples of images referenced and generated in each process executed by the information processing device 3 according to this exemplary embodiment. In the example shown in FIG. 11, a dump truck vessel will be described as an example of the target object. A 3D model image P11 of the vessel in FIG. 11 is an image showing a 3D model of the vessel, which is the target object. As shown in FIG. 11, the 3D model of the vessel includes data related to the edge of the vessel.

[0111] (Step S31) In step S31, the information processing device 3 acquires the 3D model 331. The information processing device 3 stores the acquired 3D model 331 in the storage unit 33.

[0112] (Step S32) In step S32, the depth image position estimation unit 313 acquires a set of position parameters of the object to be evaluated.

[0113] As described above, position parameters represent the positions and orientations that an object can take. In the example shown in Fig. 11, a set of positions and orientations that the vessel can take (a set of position parameters) is applied to a 3D model image P11 of the vessel, and the resulting two-dimensional image is image P12. Image P12 is also referred to as a "model edge."

[0114] (Step S33) In step S33, the depth image position estimation unit 313 selects one unevaluated position parameter from the set of position parameters indicating the position and orientation of the vessel. In the example shown in Fig. 11, the depth image position estimation unit 313 selects the position parameter applied to the unevaluated vessel from among the multiple two-dimensional vessels included in the image P12.

[0115] (Step S34) In step S34, the depth image position estimation unit 313 moves and rotates the 3D model 331 stored in the storage unit 33 based on the selected position parameters.

[0116] (Step S35) In step S35, the depth image position estimation unit 313 maps the moved and rotated 3D model 331 onto a two-dimensional space to generate a mapped image. The mapped image generated by the depth image position estimation unit 313 is characterized by being an image that indicates depth information of the 3D model 331.

[0117] (Step S36) In step S36, the depth image position estimation unit 313 extracts the contour (edge) of the object in the mapped image. As an example, the depth image position estimation unit 313 extracts the contour, which is a feature point of the object, by applying a first feature point extraction process to the mapped image, and generates third two-dimensional data indicating the contour. The third two-dimensional data generated by the depth image position estimation unit 313 is also referred to as "template data."

[0118] (Step S37) In step S37, the depth information acquisition unit 311 acquires depth information obtained by the depth sensor 4 that includes the target object in its sensing range. Then, the depth information acquisition unit 311 supplies the acquired depth information to the depth image feature point extraction unit 312.

[0119] The depth image feature point extraction unit 312 Depth information acquisition unit 311 For example, the depth image feature point extraction unit 312 acquires depth information in which the object is included in the sensing range and depth information in which the object is not included in the sensing range, and generates a depth image in which the object is included and a depth image in which the object is not present.

[0120] In the example shown in Figure 11, the depth image feature point extraction unit 312 generates a depth image P14 of the recognition target, which is a depth image that includes the target RT in the sensing range, and a background depth image P13, which is a depth image when the target RT is not present in the sensing range.

[0121] (Step S38) In step S38, the depth image feature point extraction unit 312 refers to the depth image and extracts the contour of the object. The data obtained by the depth image feature point extraction unit 312 extracting the contour of the object is first two-dimensional data, which is also referred to as a "depth edge" or "search data."

[0122] In the example shown in FIG. 11, the depth image feature point extraction unit 312 first calculates the difference between a depth image P14 of the recognition target and a background depth image P13, and generates a difference image P15 that is difference information.

[0123] Next, the depth image feature point extraction unit 312 executes a first feature point extraction process by referring to the generated difference information, and extracts one or more feature points included in the difference image. With this configuration, the information processing device 3 performs the extraction process of the object and the feature points of the object included in the depth information by referring to the depth information with a small amount of information, thereby reducing the calculation cost and calculation time.

[0124] 11, the depth image feature point extraction unit 312 uses an edge extraction filter on the difference image P15 to extract an edge OL2 from the difference image to generate an image P16. The image P16 is the first two-dimensional data (depth edge or search data). The depth image feature point extraction unit 312 supplies the first two-dimensional data to the depth image position estimation unit 313.

[0125] Here, the depth image feature point extraction unit 312 may execute the first feature point extraction process by referring to the binarized difference information obtained by applying a binarization process to the difference information. With this configuration, the information processing device 3 refers to the binarized difference information obtained by applying a binarization process with a small amount of information, thereby reducing the calculation cost and calculation time.

[0126] The processes of steps S37 and S38 are an example of processes executed by the depth image feature point extraction unit 312.

[0127] Note that steps S37 and S38 may be executed in parallel with steps S31 to S36, or may be executed before steps S31 to S36, or may be executed after steps S31 to S36.

[0128] (Step S39) In step S39, the depth image position estimation unit 313 matches the template data (third two-dimensional data) extracted in step S36 with the search data (first two-dimensional data) supplied from the depth image feature point extraction unit 312 in step S38, and calculates a matching error. As an example, the depth image position estimation unit 313 calculates the matching error by template matching processing with reference to the third two-dimensional data and the first two-dimensional data.

[0129] Here, an example of template matching processing is chamfer matching, but this does not limit the present embodiment. As another example, the depth image position estimation unit 313 may use PnP (Perspective n Point), ICP (Inverse Coordinate System), etc. to calculate the matching error. Interactive Examples of methods include, but are not limited to, methods using Closest Point (CPM) and Directional Chamfer Matching (DCM).

[0130] 11, an image P17 is an image in which image P16, which is search data, and contour OL1, which is template data to be applied to image P16, are superimposed. The depth image position estimation unit 313 calculates the error between edge OL2 included in image P16 and contour OL1 as a matching error. The error calculated by the depth image position estimation unit 313 is also referred to as a "matching error (depth)" to indicate that it is a matching error using depth information.

[0131] (Step S40) In step S40, the depth image position estimation unit 313 determines whether or not there are any unevaluated position parameters.

[0132] In step S40, if it is determined that there are unevaluated position parameters (step S40: YES), the depth image position estimation unit 313 returns to the processing of step S33.

[0133] (Step S41) If it is determined in step S40 that there are no unevaluated position parameters (step S40: NO), in step S41, the depth image position estimation unit 313 selects up to N position parameters with small matching errors (depths) equal to or less than a predetermined threshold, and sets these as N candidate solutions. Here, the depth image position estimation unit 313 may select N position parameters with relatively small errors, and set these as N candidate solutions. With this configuration, the information processing device 3 generates one or more candidate solutions by template matching processing with reference to first two-dimensional data obtained with reference to depth information with less information volume than an RGB image, thereby reducing calculation cost and calculation time. The depth image position estimation unit 313 supplies the N candidate solutions to the RGB image position estimation unit 316.

[0134] The above steps S32 to S36 and steps S39 to S41 are examples of the processing executed by the depth image position estimation unit 313.

[0135] (Step S42) In step S42, when the RGB image position estimation unit 316 acquires candidate solutions, which are N position parameters, from the depth image position estimation unit 313, the RGB image position estimation unit 316 uses the candidate solutions as position parameters to be evaluated.

[0136] (Step S43) In step S43, the RGB image position estimation unit 316 selects one unevaluated position parameter from among the N position parameters.

[0137] (Step S44) In step S44, the RGB image position estimation unit 316 moves and rotates the 3D model 331 stored in the storage unit 33 based on the selected position parameters.

[0138] (Step S45) In step S45, the RGB image position estimation unit 316 maps the moved and rotated 3D model 331 onto a two-dimensional space to generate a mapped image. The mapped image generated by the RGB image position estimation unit 316 is characterized by being an image that includes texture information of the 3D model 331.

[0139] (Step S46) In step S46, the RGB image position estimation unit 316 extracts the contour of the object in the mapped image. As an example, the RGB image position estimation unit 316 extracts the contour (edge) of the object by applying a second feature point extraction process to the mapped image, and generates fourth two-dimensional data indicating the contour. The contour extracted by the RGB image position estimation unit 316 may be a rectangular contour. The fourth two-dimensional data generated by the RGB image position estimation unit 316 is also referred to as "template data."

[0140] (Step S47) In step S47, the RGB image acquisition unit 314 acquires an RGB image including the object captured by the RGB camera 5 in the angle of view. The RGB image acquisition unit 314 supplies the acquired RGB image to the RGB image feature point extraction unit 315.

[0141] (Step S48) In step S48, the RGB image feature point extraction unit 315 refers to the RGB image supplied from the RGB image acquisition unit 314, executes second feature point extraction processing, and generates second two-dimensional data.

[0142] In the example shown in FIG. 11, the RGB image feature point extraction unit 315 extracts the rectangular contour of an object included in the RGB image P18 as a feature point. A known method may be used as an example of a method for extracting the rectangular shape. The RGB image feature point extraction unit 315 generates an image P19 including the extracted contour OL4 as second two-dimensional data. The image P19 generated by the RGB image feature point extraction unit 315 is also referred to as an "RGB edge" or "search data." The RGB image feature point extraction unit 315 supplies the generated second two-dimensional data to the RGB image position estimation unit 316.

[0143] The process of step S48 is an example of a process executed by the RGB image feature point extraction unit 315.

[0144] Note that steps S47 and S48 may be executed in parallel with steps S42 to S46, or may be executed before steps S42 to S46, or may be executed after steps S42 to S46.

[0145] (Step S49) In step S49, the RGB image position estimation unit 316 matches the template data (fourth two-dimensional data) extracted in step S46 with the search data (second two-dimensional data) supplied from the RGB image feature point extraction unit 315 in step S48, and calculates a matching error. As an example, the RGB image position estimation unit 316 calculates the matching error by template matching processing with reference to the fourth two-dimensional data and the second two-dimensional data. Here, an example of the template matching processing is chamfer matching, but this is not intended to limit the present embodiment. As other examples, the RGB image position estimation unit 316 may use methods such as PnP, ICP, and DCM to calculate the matching error, but is not limited to these.

[0146] 11, an image P20 is an image in which image P19, which is search data, and contour OL3, which is template data to be applied to image P19, are superimposed. RGB image position estimation unit 316 calculates the error between contour OL4 included in image P19 and contour OL3 as a matching error. The error calculated by RGB image position estimation unit 316 is also referred to as a "matching error (image)" to indicate that it is a matching error using an RGB image (image).

[0147] (Step S50) In step S50, the RGB image position estimation unit 316 determines whether or not there are any unevaluated position parameters.

[0148] In step S50, if it is determined that there are unevaluated position parameters (step S50: YES), the RGB image position estimation unit 316 returns to the processing in step S43.

[0149] (Step S51) In step S51, the RGB image position estimation unit 316 calculates a total error from the matching error (depth) and matching error (image) calculated for each position parameter, and selects the position parameter with the smallest total error. In other words, the RGB image position estimation unit 316 calculates at least one of the position and orientation of the object in three-dimensional space. With this configuration, the information processing device 3 calculates at least one of the position and orientation of the object in three-dimensional space by template matching processing with reference to second two-dimensional data obtained with reference to an RGB image that contains more information than depth information, so that at least one of the position and orientation of the object can be suitably estimated. The RGB image position estimation unit 316 supplies the selected parameters to the output unit 32.

[0150] As an example, the RGB image position estimator 316 can calculate the total error e using the following equation (1), but this is not intended to limit the present exemplary embodiment. e=wd*ed+wi*ei···(1) The variables in equation (1) represent the following: wd: weighting parameter wi: weighting parameter ed: Matching error (depth) ei: Matching error (image) That is, the RGB image position estimation unit 316 uses, as the total error e, the sum of the product of the matching error (depth) ed calculated by the depth image position estimation unit 313 in step S39 and the weighting parameter wd, and the product of the matching error (image) ei calculated by the RGB image position estimation unit 316 in step S49 and the weighting parameter wi.

[0151] As another example, the RGB image position estimation unit 316 can also calculate the total error e using the following equation (2). e=βd*exp(αd*ed)+ βi*exp(αi*ei)···(2) The variables in equation (2) represent the following: βd: weighting parameter βi: weighting parameter αd: parameter αi: parameter ed: Matching error (depth) ei: Matching error (image) That is, the RGB image position estimation unit 316 first calculates the exponential of the product of the matching error (depth) ed calculated by the depth image position estimation unit 313 in step S39 and the parameter αd. Then, the RGB image position estimation unit 316 calculates the product (value d) of the calculated value and the weighting parameter βd.

[0152] Next, the RGB image position estimation unit 316 calculates the exponential of the product of the matching error (image) e i calculated by the RGB image position estimation unit 316 in step S49 and the parameter α i. Subsequently, the RGB image position estimation unit 316 calculates the product (value i) of the calculated value and the weighting parameter β i.

[0153] Then, the RGB image position estimation unit 316 uses the sum of the value d and the value i as the total error e.

[0154] Here, the RGB image position estimation unit 316 may apply a data deletion process to the RGB image or the second two-dimensional data, which deletes data that is a predetermined distance or more away from the positions indicated by the N candidate solutions (in other words, data that indicates that the data is away by a predetermined distance or more). In this case, the RGB image position estimation unit 316 may refer to the captured image or the second two-dimensional data after the data deletion process, and calculate at least one of the position and orientation of the object in three-dimensional space. With this configuration, the information processing device 3 calculates at least one of the position and orientation of the object in three-dimensional space without processing data other than the object, thereby reducing calculation costs and calculation time.

[0155] The above steps S49 to S51 are an example of the process executed by the RGB image position estimation unit 316.

[0156] Furthermore, in the flowchart shown in FIG. 10, the information processing device 3 may be configured to reverse the order of executing steps S37 to S39 and steps S47 to S49, and further, have the RGB image position estimation unit 316 execute the processes of steps S32 to S36 and steps S39 to S41 instead of the depth image position estimation unit 313, and have the depth image position estimation unit 313 execute the processes of steps S42 to S46 and steps S49 to S51 instead of the RGB image position estimation unit 316.

[0157] In other words, in step S47, the RGB image acquisition unit 314 acquires a captured image taken by the RGB camera 5 that includes the target object in its angle of view, and in step S39, the RGB image position estimation unit 316 refers to the second two-dimensional data obtained by the second feature point extraction process that references the captured image and the three-dimensional model to generate one or more candidate solutions regarding at least one of the position and orientation of the target object in three-dimensional space.

[0158] Next, in step S37, the depth information acquisition unit 311 acquires depth information obtained by the depth sensor 4 that includes the object in its sensing range, and in step S49, the depth image position estimation unit 313 refers to the first two-dimensional data obtained by the first feature point extraction process that references the depth information and the three-dimensional model, and calculates at least one of the position and orientation of the object in three-dimensional space using one or more candidate solutions.

[0159] Even with this configuration, the information processing device 3 achieves substantially the same effects as the information processing device 1 described above.

[0160] As described above, in the information processing system 100 according to this exemplary embodiment, the information processing device 3 includes a depth information acquisition unit 311 that acquires depth information obtained by a depth sensor 4 that includes the object in its sensing range, an RGB image acquisition unit 314 that acquires an RGB image obtained by an RGB camera 5 that includes the object in its angle of view, a depth image position estimation unit 313 that generates one or more candidate solutions regarding at least one of the position and orientation of the object in three-dimensional space by referring to first two-dimensional data obtained by a first feature point extraction process that references the depth information and a 3D model 331 related to the object, and an RGB image position estimation unit 316 that calculates at least one of the position and orientation of the object in three-dimensional space by referring to second two-dimensional data obtained by a second feature point extraction process that references the RGB image and the 3D model 331, using one or more candidate solutions.

[0161] Therefore, according to the information processing system 100 according to this exemplary embodiment, the information processing device 3 achieves the same effects as the information processing device 1 described above.

[0162] Exemplary Embodiment 4 A fourth exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the above exemplary embodiment are denoted by the same reference numerals, and their description will not be repeated.

[0163] (Configuration of information processing system 100A) The configuration of the information processing system 100A according to this exemplary embodiment will be described with reference to Fig. 12. Fig. 12 is a block diagram showing the configuration of the information processing system 100A according to this exemplary embodiment.

[0164] 12, the information processing system 100A includes an information processing device 3A, a depth sensor 4, an RGB camera 5, and a terminal device 6. The depth sensor 4 and the RGB camera 5 are as described in the above-described embodiment.

[0165] In the information processing system 100A, the terminal device 6 acquires depth information obtained by the depth sensor 4, the depth information including the object in its sensing range, and acquires imaging information obtained by the RGB camera 5, the imaging information including the object in its angle of view. Then, in the information processing system 100A, the information processing device 3A calculates at least one of the position and orientation of the object in three-dimensional space by referring to the depth information and imaging information acquired by the terminal device 6. The object, the depth information, and the position and orientation of the object are as described in the above-mentioned embodiment.

[0166] (Configuration of terminal device 6) As shown in FIG. 12, the terminal device 6 includes a depth information acquisition unit 311 and an RGB image acquisition unit 314.

[0167] The depth information acquisition unit 311 acquires depth information obtained by the depth sensor 4 that includes an object in its sensing range. Even if the object is not present in the sensing range, the depth information acquisition unit 311 acquires depth information related to the sensing range that is obtained by the depth sensor 4. The depth information acquisition unit 311 outputs the acquired depth information to the information processing device 3A.

[0168] The RGB image acquisition unit 314 acquires an RGB image (captured image) that includes the target object in its angle of view, obtained by the RGB camera 5. The RGB image acquisition unit 314 outputs the acquired RGB image to the information processing device 3A.

[0169] (Configuration of information processing device 3A) 12, the information processing device 3A includes a control unit 31A, an output unit 32, and a storage unit 33. The output unit 32 and the storage unit 33 are as described in the above-described embodiment.

[0170] The control unit 31A controls the respective components of the information processing device 3A. As shown in Fig. 12, the control unit 31A also functions as a depth image feature point extraction unit 312, a depth image position estimation unit 313, an RGB image feature point extraction unit 315, and an RGB image position estimation unit 316. The depth image position estimation unit 313 and the RGB image position estimation unit 316 are as described in the above-mentioned embodiment.

[0171] The depth image feature point extraction unit 312 executes a first feature point extraction process with reference to the depth information output from the terminal device 6, and generates first two-dimensional data. The depth image feature point extraction unit 312 supplies the generated first two-dimensional data to the depth image position estimation unit 313. An example of the process executed by the depth image feature point extraction unit 312 is as described in the above-mentioned embodiment.

[0172] The RGB image feature point extraction unit 315 executes second feature point extraction processing with reference to the RGB image output from the terminal device 6, and generates second two-dimensional data. The RGB image feature point extraction unit 315 supplies the generated second two-dimensional data to the RGB image position estimation unit 316. An example of the processing executed by the RGB image feature point extraction unit 315 is as described in the above-mentioned embodiment.

[0173] As described above, in the information processing system 100A according to this exemplary embodiment, the terminal device 6 acquires depth information and an RGB image and outputs the acquired depth information and RGB image to the information processing device 3A. The information processing device 3A refers to the depth information and the RGB image output from the terminal device 6 and calculates at least one of the position and orientation of the object in three-dimensional space. Therefore, in the information processing system 100A according to this exemplary embodiment, the information processing device 3A does not need to acquire the depth information and the RGB image directly from the depth sensor 4 and the RGB camera 5, and can therefore be realized by a server or the like that is located physically distant from the depth sensor 4 and the RGB camera 5.

[0174] Exemplary Embodiment 5 A fifth exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the above exemplary embodiment are denoted by the same reference numerals, and their description will not be repeated.

[0175] (Configuration of information processing system 100B) The configuration of an information processing system 100B according to this exemplary embodiment will be described with reference to Fig. 13. Fig. 13 is a block diagram showing the configuration of an information processing system 100B according to this exemplary embodiment.

[0176] 13, the information processing system 100B includes an information processing device 3B, a depth sensor 4, and an RGB camera 5. The depth sensor 4 and the RGB camera 5 are as described in the above-described embodiments.

[0177] In the information processing system 100B, the information processing device 3B, like the information processing device 3, acquires depth information including an object in its sensing range obtained by the depth sensor 4, and acquires imaging information including the object in its angle of view obtained by the RGB camera 5. The information processing device 3B then refers to the acquired depth information and imaging information to calculate at least one of the position and orientation of the object. The object, depth information, and position and orientation of the object are as described in the above-mentioned embodiment.

[0178] (Configuration of information processing device 3B) 13, the information processing device 3B includes a control unit 31B, an output unit 32, and a storage unit 33. The output unit 32 and the storage unit 33 are as described in the above-described embodiment.

[0179] 13 , the control unit 31B also functions as a depth information acquisition unit 311, a depth image feature point extraction unit 312, a depth image position estimation unit 313, an RGB image acquisition unit 314, an RGB image feature point extraction unit 315, an RGB image position estimation unit 316, and an integrated determination unit 317. The depth information acquisition unit 311, the depth image feature point extraction unit 312, the RGB image acquisition unit 314, and the RGB image feature point extraction unit 315 are as described in the above-mentioned embodiment.

[0180] The depth information acquisition unit 311, the depth image position estimation unit 313, the RGB image acquisition unit 314, the RGB image position estimation unit 316, and the integrated judgment unit 317 are configured to respectively realize a depth information acquisition means, a first matching means, a captured image acquisition means, a second matching means, and a calculation means in this exemplary embodiment.

[0181] The depth image position estimation unit 313 executes a first matching process by referring to the first two-dimensional data supplied from the depth image feature point extraction unit 312 and the 3D model 331 stored in the storage unit 33. The first two-dimensional data and the first matching process are as described in the above-mentioned embodiment. The depth image position estimation unit 313 supplies the result of the first matching process to the integrated determination unit 317.

[0182] Furthermore, the depth image position estimation unit 313 supplies an image obtained by moving and rotating the 3D model 331 stored in the storage unit 33 to the RGB image position estimation unit 316.

[0183] The RGB image position estimation unit 316 executes a second matching process by referencing the second two-dimensional data supplied from the RGB image feature point extraction unit 315 and the 3D model 331 stored in the storage unit 33. The second two-dimensional data and the second matching process are as described in the above-mentioned embodiment. The RGB image position estimation unit 316 supplies the result of the second matching process to the integration determination unit 317.

[0184] The integrated determination unit 317 calculates at least one of the position and orientation of the object in three-dimensional space by referring to the result of the first matching process supplied from the depth image position estimation unit 313 and the result of the second matching process supplied from the RGB image position estimation unit 316. An example of a method by which the integrated determination unit 317 calculates at least one of the position and orientation of the object in three-dimensional space is the same as the example of the method by which the RGB image position estimation unit 316 described above calculates at least one of the position and orientation of the object in three-dimensional space, and therefore a description thereof will be omitted.

[0185] (Flow of processing executed by information processing device 3B) The flow of processing executed by the information processing device 3B will be described with reference to Fig. 14. Fig. 14 is a flowchart showing the flow of processing executed by the information processing device 3B according to this exemplary embodiment.

[0186] (Step S31) In step S31, the information processing device 3B acquires the 3D model 331. The information processing device 3B stores the acquired 3D model 331 in the storage unit 33.

[0187] (Step S32) In step S32, the depth image position estimation unit 313 acquires a set of position parameters of the object to be evaluated. The position parameters are as described above.

[0188] (Step S33) In step S33, the depth image position estimation unit 313 selects one unevaluated position parameter from the set of position parameters indicating the position and orientation of the vessel.

[0189] (Step S60) In step S60 , deep The depth image position estimation unit 313 moves and rotates the 3D model 331 stored in the storage unit 33 based on the selected position parameters. The depth image position estimation unit 313 supplies the moved and rotated 3D model 331 to the RGB image position estimation unit 316.

[0190] (Step S35) In step S35, the depth image position estimation unit 313 maps the moved and rotated 3D model 331 onto a two-dimensional space to generate a mapped image.

[0191] (Step S36) In step S36, the depth image position estimation unit 313 extracts the contour (edge) of the object in the mapped image. As an example, the depth image position estimation unit 313 extracts the contour, which is a feature point of the object, by applying a first feature point extraction process to the mapped image, and generates third two-dimensional data indicating the contour. The third two-dimensional data generated by the depth image position estimation unit 313 is also referred to as "template data."

[0192] (Step S37) In step S37, the depth information acquisition unit 311 acquires depth information obtained by the depth sensor 4 that includes the target object in its sensing range. Then, the depth information acquisition unit 311 supplies the acquired depth information to the depth image feature point extraction unit 312.

[0193] The depth image feature point extraction unit 312 Depth information acquisition unit 311 For example, the depth image feature point extraction unit 312 acquires depth information in which the object is included in the sensing range and depth information in which the object is not included in the sensing range, and generates a depth image in which the object is included and a depth image in which the object is not present.

[0194] (Step S38) In step S38, the depth image feature point extraction unit 312 refers to the depth image and extracts the contour of the object. The data obtained by the depth image feature point extraction unit 312 extracting the contour of the object is first two-dimensional data, which is also referred to as a "depth edge" or "search data."

[0195] As an example, similar to the above-described embodiment, the depth image feature point extraction unit 312 first references depth information in which the object is included in the sensing range and depth information in which the object is not included in the sensing range, and generates a difference image, which is difference information. An example of the difference image is shown as difference image P15 in FIG. 11 above.

[0196] next , deep The depth image feature point extraction unit 312 executes a first feature point extraction process with reference to the generated difference information, and extracts one or more feature points (contours, edges, etc.) included in the difference image. An example of an image from which the depth image feature point extraction unit 312 extracts one or more feature points is shown in image P16 in FIG. 11 described above.

[0197] Here, similarly to the above-described embodiment, the depth image feature point extraction unit 312 may execute the first feature point extraction process by referring to the binarized difference information obtained by applying the binarization process to the difference information. According to this configuration, the information processing device 3B refers to the binarized difference information obtained by applying the binarization process with a small amount of information, thereby reducing the calculation cost and calculation time.

[0198] Note that steps S37 and S38 may be executed in parallel with steps S31 to S33, step S60, step S35, and step S36, or may be executed before steps S31 to S33, step S60, step S35, and step S36, or may be executed after steps S31 to S33, step S60, step S35, and step S36.

[0199] (Step S39) In step S39, the depth image position estimation unit 313 executes a first matching process between the template data (third two-dimensional data) extracted in step S36 and the search data (first two-dimensional data) supplied from the depth image feature point extraction unit 312 in step S38, and calculates a matching error (depth). As an example, the depth image position estimation unit 313 calculates the matching error (depth) by template matching process with reference to the third two-dimensional data and the first two-dimensional data. The depth image position estimation unit 313 supplies the calculated matching error (depth) to the integration determination unit 317.

[0200] As in the above-described embodiment, an example of template matching processing is chamfer matching, and methods using PnP, ICP, and DCM for calculating matching errors are also possible, but are not limited to these.

[0201] In step S39, an example of the first matching process executed by the depth image position estimation unit 313 is as described above using the image P17 in FIG.

[0202] (Step S61) In step S61, the RGB image position estimation unit 316 maps the moved and rotated 3D model 331 supplied from the depth image position estimation unit 313 onto a two-dimensional space to generate a mapped image.

[0203] (Step S62) In step S62, the RGB image position estimation unit 316 extracts the contour of the object in the mapped image. As an example, the RGB image position estimation unit 316 extracts the contour (edge) of the object by applying a second feature point extraction process to the mapped image, and generates fourth two-dimensional data indicating the contour. The fourth two-dimensional data generated by the RGB image position estimation unit 316 is also referred to as "template data."

[0204] (Step S47) In step S47, the RGB image acquisition unit 314 acquires an RGB image including the object captured by the RGB camera 5 in the angle of view. The RGB image acquisition unit 314 supplies the acquired RGB image to the RGB image feature point extraction unit 315.

[0205] (Step S48) In step S48, the RGB image feature point extraction unit 315 references the RGB image supplied from the RGB image acquisition unit 314, executes second feature point extraction processing, and generates second two-dimensional data. The second two-dimensional data generated by the RGB image feature point extraction unit 315 is also referred to as "RGB edge" or "search data." An example of the second two-dimensional data is shown in image P19 in FIG. 11 described above.

[0206] Note that steps S47 and S48 may be executed in parallel with steps S61 and S62, or may be executed before steps S61 and S62, or may be executed after steps S61 and S62.

[0207] (Step S63) In step S63, the RGB image position estimation unit 316 executes a second matching process between the template data (fourth two-dimensional data) extracted in step S62 and the search data (second two-dimensional data) supplied from the RGB image feature point extraction unit 315 in step S48, and calculates a matching error (image). As an example, the RGB image position estimation unit 316 calculates the matching error by template matching process with reference to the fourth two-dimensional data and the second two-dimensional data. The RGB image position estimation unit 316 supplies the calculated matching error (image) to the integration determination unit 317.

[0208] As in the above-described embodiment, an example of template matching processing is chamfer matching, and methods using PnP, ICP, and DCM for calculating matching errors are also possible, but are not limited to these.

[0209] An example of the second matching process executed by the RGB image position estimation unit 316 in step S63 is as described above using the image P20 in FIG.

[0210] Here, step S63 may be executed in parallel with step S39, or may be executed before step S39, or may be executed after step S39.

[0211] (Step S64) In step S64, the integrated judgment unit 317 calculates an integrated error by referring to the matching error (depth) supplied from the depth image position estimation unit 313 in step S39 and the matching error (image) supplied from the RGB image position estimation unit 316 in step S63.

[0212] Similar to the method by which the RGB image position estimation unit 316 calculates the overall error in the above-described embodiment, one example of how the integration determination unit 317 calculates the integrated error is to calculate the integrated error e using the following formula (3), but this does not limit this exemplary embodiment. e=wd*ed+wi*ei···(3) The variables in equation (3) represent the following: wd: weighting parameter wi: weighting parameter ed: Matching error (depth) ei: Matching error (image) That is, the integrated determination unit 317 Integration The error e is the sum of the product of the matching error (depth) ed calculated by the depth image position estimation unit 313 in step S39 and the weighting parameter wd, and the product of the matching error (image) ei calculated by the RGB image position estimation unit 316 in step S63 and the weighting parameter wi.

[0213] As another example, the integrated determination unit 317 can also calculate the integrated error e using the following equation (4). e=βd*exp(αd*ed)+ βi*exp(αi*ei)···(4) The variables in equation (4) represent the following: βd: weighting parameter βi: weighting parameter αd: parameter αi: parameter ed: Matching error (depth) ei: Matching error (image) That is, the integrated determination unit 317 first calculates the exponential of the product of the matching error (depth) ed calculated by the depth image position estimation unit 313 in step S39 and the parameter αd. Then, the integrated determination unit 317 calculates the product (value d) of the calculated value and the weighting parameter βd.

[0214] Next, the integrated determination unit 317 calculates the exponential of the product of the matching error (image) e i calculated by the RGB image position estimation unit 316 in step S63 and the parameter α i. Subsequently, the integrated determination unit 317 calculates the product (value i) of the calculated value and the weighting parameter β i.

[0215] Then, the integrated determination unit 317 Integration The error e is the sum of the value d and the value i.

[0216] (Step S65) In step S65, the integrated determination unit 317 determines whether or not there are any unevaluated position parameters.

[0217] In step S65, if it is determined that there are unevaluated position parameters (step S65: YES), the processing of the information processing device 3B returns to step S33.

[0218] (Step S66) If it is determined in step S65 that there are no unevaluated position parameters (step S65: NO), the integrated determination unit 317 selects the position parameters that minimize the integrated error. In other words, the integrated determination unit 317 calculates at least one of the position and orientation of the object in three-dimensional space. The integrated determination unit 317 outputs the selected position parameters to the output unit 32.

[0219] As described above, in the information processing system 100B according to this exemplary embodiment, the information processing device 3B includes a depth information acquisition unit 311 that acquires depth information obtained by a depth sensor 4 that includes the object in its sensing range, an RGB image acquisition unit 314 that acquires a captured image obtained by an RGB camera 5 that includes the object in its angle of view, a depth image position estimation unit 313 that performs a first matching process by referring to first two-dimensional data obtained by a first feature point extraction process that references the depth information and a 3D model 331 related to the object, an RGB image position estimation unit 316 that performs a second matching process by referring to second two-dimensional data obtained by a second feature point extraction process that references the captured image and the 3D model 331, and an integrated determination unit 317 that calculates at least one of the position and orientation of the object in three-dimensional space by referring to the results of the first matching process and the results of the second matching process.

[0220] Therefore, in the information processing system 100B according to this exemplary embodiment, the information processing device 3B only needs to perform the process of moving and rotating the 3D model 331 once for each position parameter, thereby reducing calculation costs and calculation time.

[0221] Furthermore, in the information processing system 100B according to this exemplary embodiment, if the matching error is large in the first matching process, which is fast because the amount of information is small, the information processing device 3B does not need to execute the second matching process. Therefore, in the information processing system 100B according to this exemplary embodiment, the information processing device 3B can reduce calculation costs and calculation time.

[0222] Exemplary Embodiment 6 A sixth exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the above exemplary embodiments are denoted by the same reference numerals, and their description will not be repeated.

[0223] (Configuration of information processing system 100C) The configuration of an information processing system 100C according to this exemplary embodiment will be described with reference to Fig. 15. Fig. 15 is a block diagram showing the configuration of an information processing system 100C according to this exemplary embodiment.

[0224] 15, the information processing system 100C includes an information processing device 3C, a depth sensor 4, an RGB camera 5, and a terminal device 6C. The depth sensor 4 and the RGB camera 5 are as described in the above-described embodiments.

[0225] In the information processing system 100C, the terminal device 6C acquires depth information including an object in its sensing range obtained by the depth sensor 4, and acquires imaging information including the object in its angle of view obtained by the RGB camera 5. Then, in the information processing system 100C, the information processing device 3C calculates at least one of the position and orientation of the object in three-dimensional space by referring to the depth information and imaging information acquired by the terminal device 6C. The object, depth information, and position and orientation of the object are as described in the above-mentioned embodiment.

[0226] (Configuration of terminal device 6C) As shown in FIG. 15, the terminal device 6C includes a depth information acquisition unit 311 and an RGB image acquisition unit 314.

[0227] The depth information acquisition unit 311 acquires depth information obtained by the depth sensor 4 that includes an object in its sensing range. Even if the object is not present in the sensing range, the depth information acquisition unit 311 acquires depth information related to the sensing range that is obtained by the depth sensor 4. The depth information acquisition unit 311 outputs the acquired depth information to the information processing device 3C.

[0228] The RGB image acquisition unit 314 acquires an RGB image (captured image) that includes the target object in its angle of view, obtained by the RGB camera 5. The RGB image acquisition unit 314 outputs the acquired RGB image to the information processing device 3C.

[0229] (Configuration of information processing device 3C) 15, the information processing device 3C includes a control unit 31C, an output unit 32, and a storage unit 33. The output unit 32 and the storage unit 33 are as described in the above-described embodiment.

[0230] 15 , the control unit 31C also functions as a depth image feature point extraction unit 312, a depth image position estimation unit 313, an RGB image feature point extraction unit 315, an RGB image position estimation unit 316, and an integration determination unit 317. The depth image position estimation unit 313, the RGB image position estimation unit 316, and the integration determination unit 317 are as described in the above-mentioned embodiments.

[0231] The depth image feature point extraction unit 312 executes a first feature point extraction process with reference to the depth information output from the terminal device 6C, and generates first two-dimensional data. The depth image feature point extraction unit 312 supplies the generated first two-dimensional data to the depth image position estimation unit 313. An example of the process executed by the depth image feature point extraction unit 312 is as described in the above-mentioned embodiment.

[0232] The RGB image feature point extraction unit 315 executes second feature point extraction processing with reference to the RGB image output from the terminal device 6C, and generates second two-dimensional data. The RGB image feature point extraction unit 315 supplies the generated second two-dimensional data to the RGB image position estimation unit 316. An example of the processing executed by the RGB image feature point extraction unit 315 is as described in the above-mentioned embodiment.

[0233] As described above, in the information processing system 100C according to this exemplary embodiment, the terminal device 6C acquires depth information and an RGB image and outputs the acquired depth information and RGB image to the information processing device 3C. The information processing device 3C refers to the depth information and the RGB image output from the terminal device 6C and calculates at least one of the position and orientation of the object in three-dimensional space. Therefore, in the information processing system 100C according to this exemplary embodiment, the information processing device 3C does not need to acquire the depth information and the RGB image directly from the depth sensor 4 and the RGB camera 5, and can therefore be realized by a server or the like located physically distant from the depth sensor 4 and the RGB camera 5.

[0234] [Software implementation example] Some or all of the functions of the information processing devices 1, 2, 3, 3A, 3B, and 3C and the information processing systems 10, 20, 100, 100A, 100B, and 100C may be realized by hardware such as an integrated circuit (IC chip), or by software.

[0235] In the latter case, the information processing devices 1, 2, 3, 3A, 3B, and 3C and the information processing systems 10, 20, 100, 100A, 100B, and 100C are realized, for example, by a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as computer C) is shown in FIG. 16. The computer C includes at least one processor C1 and at least one memory C2. The memory C2 stores a program P for causing the computer C to operate as the information processing devices 1, 2, 3, 3A, 3B, and 3C and the information processing systems 10, 20, 100, 100A, 100B, and 100C. In the computer C, the processor C1 reads and executes the program P from the memory C2, thereby realizing each function of the information processing devices 1, 2, 3, 3A, 3B, and 3C and the information processing systems 10, 20, 100, 100A, 100B, and 100C.

[0236] The processor C1 may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a microcontroller, or a combination thereof. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination thereof.

[0237] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may also include a communication interface for transmitting and receiving data to and from other devices. The computer C may also include an input / output interface for connecting input / output devices such as a keyboard, mouse, display, and printer.

[0238] Furthermore, the program P can be recorded on a non-transitory tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can acquire the program P via such a recording medium M. The program P can also be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also acquire the program P via such a transmission medium.

[0239] [Appendix 1] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means disclosed in the above-described embodiments are also included in the technical scope of the present invention.

[0240] [Appendix 2] Some or all of the above-described embodiments can also be described as follows: However, the present invention is not limited to the following described aspects.

[0241] (Appendix 1) an information processing device comprising: a depth information acquisition means for acquiring depth information obtained by a depth sensor having an object in its sensing range; an image acquisition means for acquiring an image obtained by an imaging sensor having an angle of view of the object; a generation means for generating one or more candidate solutions for at least one of a position and an orientation of the object in the three-dimensional space by referring to first two-dimensional data obtained by a first feature point extraction process with reference to the depth information and third two-dimensional data obtained by the first feature point extraction process, which maps a three-dimensional model of the object into two-dimensional space; and a calculation means for calculating at least one of a position and an orientation of the object in the three-dimensional space by referring to second two-dimensional data obtained by a second feature point extraction process with reference to the image and fourth two-dimensional data obtained by the second feature point extraction process, which maps a three-dimensional model of the object into two-dimensional space.

[0242] (Appendix 2) The information processing device according to claim 1, wherein the first feature point extraction process and the second feature point extraction process include edge extraction processes, and the three-dimensional model includes data relating to edges of the object.

[0243] (Appendix 3) The information processing device described in Appendix 1 or 2, wherein the depth information acquisition means acquires depth information regarding the sensing range even when the object is not present in the sensing range, and the first feature point extraction process is a feature point extraction process that refers to differential information between depth information when the object is present in the sensing range and depth information when the object is not present in the sensing range.

[0244] (Appendix 4) 4. The information processing device according to claim 3, wherein the first feature point extraction process is a feature point extraction process that references binarized difference information obtained by applying a binarization process to the difference information.

[0245] (Appendix 5) The information processing device according to any one of appendices 1 to 4, wherein the calculation means applies a data deletion process to the captured image or the second two-dimensional data to delete data that indicates that the captured image or the second two-dimensional data is more than a predetermined distance away from a position indicated by the one or more candidate solutions, and calculates at least one of the position and orientation of the object in three-dimensional space by referring to the captured image or the second two-dimensional data after the data deletion process.

[0246] (Appendix 6) The information processing device according to any one of appendices 1 to 5, wherein the generation means generates the one or more candidate solutions by a template matching process that references the third two-dimensional data and the first two-dimensional data.

[0247] (Appendix 7) The information processing device according to any one of appendices 1 to 6, wherein the calculation means calculates at least one of the position and orientation of the object in three-dimensional space by template matching processing that references the fourth two-dimensional data and the second two-dimensional data.

[0248] (Appendix 8) an information processing device comprising: a depth information acquisition means for acquiring depth information obtained by a depth sensor having an object in its sensing range; an image acquisition means for acquiring an image obtained by an imaging sensor having an angle of view of the object; a first matching means for executing a first matching process by referring to first two-dimensional data obtained by a first feature point extraction process with reference to the depth information and third two-dimensional data obtained by the first feature point extraction process, mapping a three-dimensional model of the object into two-dimensional space; a second matching means for executing a second matching process by referring to second two-dimensional data obtained by a second feature point extraction process with reference to the image, mapping a three-dimensional model of the object into two-dimensional space, and fourth two-dimensional data obtained by the second feature point extraction process; and a calculation means for calculating at least one of the position and orientation of the object in three-dimensional space with reference to a result of the first matching process and a result of the second matching process.

[0249] (Appendix 9) an information processing method including: acquiring depth information obtained by a depth sensor having a sensing range that includes an object; acquiring an image obtained by an imaging sensor having an angle of view that includes the object; generating one or more candidate solutions for at least one of a position and an orientation of the object in the three-dimensional space by referring to first two-dimensional data obtained by a first feature point extraction process that references the depth information and third two-dimensional data obtained by the first feature point extraction process, which maps a three-dimensional model of the object into two-dimensional space; and calculating at least one of a position and an orientation of the object in the three-dimensional space by referring to second two-dimensional data obtained by a second feature point extraction process that references the image, and fourth two-dimensional data obtained by the second feature point extraction process, which maps a three-dimensional model of the object into two-dimensional space.

[0250] (Appendix 10) an information processing method including: acquiring depth information obtained by a depth sensor having a sensing range that includes an object; acquiring an image obtained by an imaging sensor having an angle of view that includes the object; performing a first matching process by referring to first two-dimensional data obtained by a first feature point extraction process that references the depth information and third two-dimensional data obtained by mapping a three-dimensional model of the object onto two-dimensional space; performing a second matching process by referring to second two-dimensional data obtained by a second feature point extraction process that references the captured image and fourth two-dimensional data obtained by mapping a three-dimensional model of the object onto two-dimensional space; and calculating at least one of a position and an orientation of the object in three-dimensional space by referring to a result of the first matching process and a result of the second matching process.

[0251] (Appendix 11) an information processing system comprising: a depth information acquisition means for acquiring depth information obtained by a depth sensor having an object in its sensing range; an image acquisition means for acquiring an image obtained by an imaging sensor having an angle of view of the object; a generation means for generating one or more candidate solutions for at least one of a position and an orientation of the object in the three-dimensional space by referring to first two-dimensional data obtained by a first feature point extraction process with reference to the depth information and third two-dimensional data obtained by the first feature point extraction process, which maps a three-dimensional model of the object into two-dimensional space; and a calculation means for calculating at least one of a position and an orientation of the object in the three-dimensional space by referring to second two-dimensional data obtained by a second feature point extraction process with reference to the image and fourth two-dimensional data obtained by the second feature point extraction process, which maps a three-dimensional model of the object into two-dimensional space.

[0252] (Appendix 12) an information processing system comprising: a depth information acquisition means for acquiring depth information obtained by a depth sensor having a sensing range of an object; an image acquisition means for acquiring an image obtained by an imaging sensor having an angle of view of the object; a first matching means for executing a first matching process by referring to first two-dimensional data obtained by a first feature point extraction process with reference to the depth information and mapping a three-dimensional model of the object onto two-dimensional space and third two-dimensional data obtained by the first feature point extraction process; a second matching means for executing a second matching process by referring to second two-dimensional data obtained by a second feature point extraction process with reference to the image and mapping a three-dimensional model of the object onto two-dimensional space and fourth two-dimensional data obtained by the second feature point extraction process; and a calculation means for calculating at least one of the position and orientation of the object in three-dimensional space with reference to a result of the first matching process and a result of the second matching process.

[0253] (Appendix 13) A program for causing a computer to operate as the information processing device according to any one of appendices 1 to 8, the program causing the computer to function as each of the means.

[0254] (Appendix 14) An information processing device comprising: a depth information acquisition means for acquiring depth information obtained by a depth sensor that includes an object in its sensing range; an image acquisition means for acquiring an image obtained by an imaging sensor that includes the object in its angle of view; a generation means for generating one or more candidate solutions regarding at least one of the position and orientation of the object in three-dimensional space by referring to first two-dimensional data obtained by a first feature point extraction process that references the depth information and a three-dimensional model of the object; and a calculation means for calculating at least one of the position and orientation of the object in three-dimensional space by referring to second two-dimensional data obtained by a second feature point extraction process that references the image and the three-dimensional model, using the one or more candidate solutions.

[0255] (Appendix 15) An information processing device comprising: a depth information acquisition means for acquiring depth information obtained by a depth sensor having an object in its sensing range; an image acquisition means for acquiring an image obtained by an imaging sensor having an angle of view of the object; a first matching means for performing a first matching process by referring to first two-dimensional data obtained by a first feature point extraction process with reference to the depth information and a three-dimensional model of the object; a second matching means for performing a second matching process by referring to second two-dimensional data obtained by a second feature point extraction process with reference to the image and the three-dimensional model; and a calculation means for calculating at least one of the position and orientation of the object in three-dimensional space with reference to the results of the first matching process and the second matching process.

[0256] (Appendix 16) An information processing method including: acquiring depth information obtained by a depth sensor that includes an object in its sensing range; acquiring an image obtained by an imaging sensor that includes the object in its angle of view; generating one or more candidate solutions regarding at least one of a position and an orientation of the object in three-dimensional space by referring to first two-dimensional data obtained by a first feature point extraction process that references the depth information and a three-dimensional model of the object; and calculating at least one of a position and an orientation of the object in three-dimensional space by referring to second two-dimensional data obtained by a second feature point extraction process that references the image and the three-dimensional model, using the one or more candidate solutions.

[0257] (Appendix 17) An information processing method including: acquiring depth information obtained by a depth sensor that includes an object in its sensing range; acquiring an image obtained by an imaging sensor that includes the object in its angle of view; performing a first matching process that references first two-dimensional data obtained by a first feature point extraction process that references the depth information and a three-dimensional model related to the object; performing a second matching process that references second two-dimensional data obtained by a second feature point extraction process that references the captured image and the three-dimensional model; and calculating at least one of a position and orientation of the object in three-dimensional space by referring to a result of the first matching process and a result of the second matching process.

[0258] (Appendix 18) An information processing system comprising: a depth information acquisition means for acquiring depth information obtained by a depth sensor whose sensing range includes an object; an image acquisition means for acquiring an image obtained by an imaging sensor whose angle of view includes the object; a generation means for generating one or more candidate solutions regarding at least one of the position and orientation of the object in three-dimensional space by referring to first two-dimensional data obtained by a first feature point extraction process that references the depth information and a three-dimensional model of the object; and a calculation means for calculating at least one of the position and orientation of the object in three-dimensional space by referring to second two-dimensional data obtained by a second feature point extraction process that references the image and the three-dimensional model, using the one or more candidate solutions.

[0259] (Appendix 19) An information processing system comprising: a depth information acquisition means for acquiring depth information obtained by a depth sensor having an object in its sensing range; an image acquisition means for acquiring an image obtained by an imaging sensor having an angle of view of the object; a first matching means for performing a first matching process by referring to first two-dimensional data obtained by a first feature point extraction process with reference to the depth information and a three-dimensional model of the object; a second matching means for performing a second matching process by referring to second two-dimensional data obtained by a second feature point extraction process with reference to the image and the three-dimensional model; and a calculation means for calculating at least one of the position and orientation of the object in three-dimensional space by referring to the results of the first matching process and the results of the second matching process.

[0260] (Appendix 20) A program for causing a computer to operate as the information processing device according to claim 14 or 15, the program causing the computer to function as each of the means.

[0261] [Appendix 3] Some or all of the above-described embodiments can also be expressed as follows.

[0262] The apparatus includes at least one processor, and the processor performs a depth information acquisition process to acquire depth information obtained by a depth sensor having a sensing range of an object; a captured image acquisition process to acquire an image obtained by an imaging sensor having an angle of view of the object; a generation process to generate one or more candidate solutions regarding at least one of a position and an orientation of the object in the three-dimensional space by referring to first two-dimensional data obtained by a first feature point extraction process that references the depth information and third two-dimensional data obtained by the first feature point extraction process, which maps a three-dimensional model of the object onto a two-dimensional space; and a generation process to generate one or more candidate solutions regarding at least one of a position and an orientation of the object in the three-dimensional space by referring to second two-dimensional data obtained by a second feature point extraction process that references the captured image and fourth two-dimensional data obtained by the second feature point extraction process, which maps a three-dimensional model of the object onto a two-dimensional space. position and an information processing device that executes a calculation process for calculating at least one of the posture.

[0263] The information processing device may further include a memory that stores a program for causing the processor to execute the depth information acquisition process, the captured image acquisition process, the generation process, and the calculation process. The program may be recorded on a computer-readable, non-transitory, tangible recording medium.

[0264] The information processing device also includes at least one processor, which executes a depth information acquisition process that acquires depth information obtained by a depth sensor that includes the object in its sensing range, a captured image acquisition process that acquires an image obtained by an imaging sensor that includes the object in its angle of view, a first matching process that references first two-dimensional data obtained by a first feature point extraction process that references the depth information and a three-dimensional model of the object onto two-dimensional space and performs third two-dimensional data obtained by the first feature point extraction process, a second matching process that references second two-dimensional data obtained by a second feature point extraction process that references the captured image and a three-dimensional model of the object onto two-dimensional space and performs fourth two-dimensional data obtained by the second feature point extraction process, and a calculation process that references a result of the first matching process and a result of the second matching process to calculate at least one of the position and orientation of the object in three-dimensional space.

[0265] The information processing device may further include a memory that stores a program for causing the processor to execute the depth information acquisition process, the captured image acquisition process, the first matching process, the second matching process, and the calculation process. The program may be recorded on a computer-readable, non-transitory, tangible recording medium.

[0266] The image processing device also includes at least one processor, and the processor performs a depth information acquisition process to acquire depth information obtained by a depth sensor that includes the object in its sensing range, a captured image acquisition process to acquire a captured image obtained by an image sensor that includes the object in its angle of view, a generation process to generate one or more candidate solutions regarding at least one of a position and an orientation of the object in three-dimensional space by referring to first two-dimensional data obtained by a first feature point extraction process that references the depth information and a three-dimensional model related to the object, and a calculation process to calculate at least one of a position and an orientation of the object in three-dimensional space by referring to second two-dimensional data obtained by a second feature point extraction process that references the captured image and the three-dimensional model, using the one or more candidate solutions. Execute Information processing device.

[0267] The information processing device may further include a memory that stores a program for causing the processor to execute the depth information acquisition process, the captured image acquisition process, the generation process, and the calculation process. The program may be recorded on a computer-readable, non-transitory, tangible recording medium.

[0268] The apparatus also includes at least one processor, and the processor is configured to acquire depth information obtained by a depth sensor having a sensing range including the object. process and acquiring a captured image obtained by an imaging sensor including the object in its angle of view. process and a first matching process that performs a first matching process by referring to first two-dimensional data obtained by a first feature point extraction process that refers to the depth information and a three-dimensional model related to the object. process and a second matching process is performed by referring to the second two-dimensional data obtained by the second feature point extraction process with reference to the captured image and the three-dimensional model. processand a calculation unit that calculates at least one of the position and the orientation of the object in a three-dimensional space by referring to the result of the first matching process and the result of the second matching process. process and Execute Information processing device.

[0269] The information processing device may further include a memory that stores a program for causing the processor to execute the depth information acquisition process, the captured image acquisition process, the first matching process, the second matching process, and the calculation process. The program may be recorded on a computer-readable, non-transitory, tangible recording medium. [Explanation of symbols]

[0270] 1, 2, 3, 3A, 3B, 3C Information processing equipment 4 Depth Sensor 5 RGB cameras 6, 6C terminal equipment 10, 20, 100, 100A, 100B, 100C Information Processing Systems 11, 311 Depth information acquisition section 12 Image acquisition unit 13 Generation part 14, 25 Calculation section 23 First Matching Section 24 Second Matching Section 31, 31A, 31B, 31C control section 32 Output section 33 Storage section 312 Depth Image Feature Point Extraction Unit 313 Depth image position estimation unit 314 RGB image acquisition unit 315 RGB image feature point extraction unit 316 RGB Image Position Estimation Unit 317 Integrated Judgment Department

Claims

1. a depth information acquisition means for acquiring depth information obtained by a depth sensor having a sensing range including an object; an image acquisition means for acquiring an image captured by an image sensor including the object in its angle of view; a generating means for generating one or more candidate solutions relating to at least one of a position and an orientation of the object in a three-dimensional space by referring to first two-dimensional data obtained by a first feature point extraction process with reference to the depth information and a three-dimensional model relating to the object; a calculation means for calculating at least one of a position and an orientation of the object in a three-dimensional space by referring to second two-dimensional data obtained by a second feature point extraction process with reference to the captured image and the three-dimensional model, and by using the one or more candidate solutions; The calculation means applying a data deletion process to the captured image or the second two-dimensional data to delete data that represents a distance greater than a predetermined distance from a position indicated by the one or more candidate solutions; calculating at least one of the position and the orientation of the object in three-dimensional space by referring to the captured image or the second two-dimensional data after the data deletion process; Information processing device.

2. the first feature point extraction process and the second feature point extraction process include edge extraction processes, The three-dimensional model includes data about the edges of the object. The information processing device according to claim 1 .

3. the depth information acquisition means acquires depth information regarding the sensing range even when the object is not present in the sensing range; The first feature point extraction process includes:

3. The information processing device according to claim 1, wherein the feature point extraction process refers to differential information between depth information when the object is present in the sensing range and depth information when the object is not present in the sensing range.

4. The first feature point extraction process is a feature point extraction process that refers to binarized difference information obtained by applying a binarization process to the difference information. The information processing device according to claim 3 .

5. one or more processors acquiring depth information obtained by a depth sensor having a sensing range including the object; acquiring an image captured by an image sensor that includes the object in its angle of view; generating one or more candidate solutions relating to at least one of a position and an orientation of the object in a three-dimensional space by referring to first two-dimensional data obtained by a first feature point extraction process that refers to the depth information and a three-dimensional model relating to the object; calculating at least one of a position and an orientation of the object in three-dimensional space by using the one or more candidate solutions, with reference to second two-dimensional data obtained by a second feature point extraction process with reference to the captured image and the three-dimensional model; Including, The calculating step includes: applying a data deletion process to the captured image or the second two-dimensional data to delete data that represents a distance greater than a predetermined distance from a position indicated by the one or more candidate solutions; calculating at least one of the position and the orientation of the object in three-dimensional space by referring to the captured image or the second two-dimensional data after the data deletion process; Information processing methods.

6. A program that causes a computer to function as an information processing device, The computer a depth information acquisition means for acquiring depth information obtained by a depth sensor having a sensing range including an object; an image acquisition means for acquiring an image captured by an image sensor including the object in its angle of view; a generating means for generating one or more candidate solutions relating to at least one of a position and an orientation of the object in a three-dimensional space by referring to first two-dimensional data obtained by a first feature point extraction process with reference to the depth information and a three-dimensional model relating to the object; a calculation means for calculating at least one of a position and an orientation of the object in a three-dimensional space by referring to second two-dimensional data obtained by a second feature point extraction process with reference to the captured image and the three-dimensional model, and by using the one or more candidate solutions; It functions as The calculation means applying a data deletion process to the captured image or the second two-dimensional data to delete data that represents a distance greater than a predetermined distance from a position indicated by the one or more candidate solutions; calculating at least one of the position and the orientation of the object in three-dimensional space by referring to the captured image or the second two-dimensional data after the data deletion process; program.

Images