Information processing device, information processing method, and program

Abstract

To realize a technique capable of estimating a position and a posture of an object without preparing a large number of three-dimensional models.SOLUTION: An information processing apparatus (1) comprises: an acquisition unit (11) for acquiring sensing data obtained by a sensor that includes an object in its sensing range; an estimation unit (12) for deriving one or more candidate solutions regarding a position and a posture of the object through first estimation processing using the sensing data acquired by the acquisition unit (11) and a three-dimensional model; and an extension unit (13) for extending the three-dimensional model for each of the one or more candidate solutions. The estimation unit (12) estimates the position and the posture of the object through second estimation processing using each three-dimensional model extended by the extension unit (13).SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program for estimating the position and orientation of an object.

Background Art

[0002] There is known a technique for estimating the position and orientation of an object in real space by analyzing a captured image that includes the object within its viewing angle. For example, Patent Document 1 describes a technique for discriminating the position and type of different parts (workpieces) in a line handling multiple product types. In this technique, a 3D model is prepared for each type, and the 2D image data obtained when the 3D model is photographed is compared with the actual camera video.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, the technique described in Patent Document 1 has problems in that it is necessary to prepare elaborate 3D models for individual types, and the calculation cost and calculation time increase as the number of types increases.

[0005] One aspect of the present invention has been made in view of the above problems, and an example of its object is to provide a technique capable of estimating the position and orientation of an object without preparing a large number of 3D models.

Means for Solving the Problems

[0006] An information processing device according to one aspect of the present invention comprises: acquisition means for acquiring sensing data obtained by a sensor that includes an object within its sensing range; estimation means for deriving one or more candidate solutions regarding the position and orientation of the object by a first estimation process using the sensing data acquired by the acquisition means and a three-dimensional model; and extension means for extending the three-dimensional model with respect to each of the one or more candidate solutions, wherein the estimation means estimates the position and orientation of the object by a second estimation process using each of the three-dimensional models extended by the extension means.

[0007] An information processing method relating to one aspect of the present invention includes acquiring sensing data obtained by a sensor that includes an object within its sensing range, deriving one or more candidate solutions regarding the position and orientation of the object by a first estimation process using the sensing data and a three-dimensional model, extending the three-dimensional model with respect to each of the one or more candidate solutions, and estimating the position and orientation of the object by a second estimation process using each of the extended three-dimensional models.

[0008] A program relating to one aspect of the present invention is a program that causes a computer to function as an acquisition means for acquiring sensing data obtained by a sensor that includes an object within its sensing range; an estimation means for deriving one or more candidate solutions regarding the position and orientation of the object by performing a first estimation process using the sensing data acquired by the acquisition means and a three-dimensional model; and an extension means for extending the three-dimensional model with respect to each of the one or more candidate solutions, wherein the estimation means estimates the position and orientation of the object by performing a second estimation process using each of the three-dimensional models extended by the extension means. [Effects of the Invention]

[0009] According to one aspect of the present invention, the position and orientation of an object can be estimated without preparing a large number of three-dimensional models. [Brief explanation of the drawing]

[0010] [Figure 1] This is a block diagram showing the configuration of an information processing device according to exemplary embodiment 1. [Figure 2] This is a flowchart showing the flow of the information processing method according to Exemplary Embodiment 1. [Figure 3] This is a block diagram showing the configuration of the information processing system according to Exemplary Embodiment 2. [Figure 4] This diagram shows the camera and the position of the camera used to image the vessel of the truck, which is the object in exemplary embodiment 2. [Figure 5] This is a flowchart showing the processing flow executed by the information processing device according to Exemplary Embodiment 2. [Figure 6] This is a flowchart showing the processing flow executed by the information processing device according to Exemplary Embodiment 2. [Figure 7] This figure shows examples of images that are referenced and generated in each process performed by the information processing device according to Exemplary Embodiment 2. [Figure 8] This figure shows a specific example of how to calculate the degree of manifestation. [Figure 9] This figure shows a concrete example of 3D model extension. [Figure 10] This figure shows a specific example of a 3D model used to derive candidate solutions. [Figure 11] This figure shows an example of mapping a 3D model to a 2D space. [Figure 12] This figure shows an example of mapping a 3D model to a 2D space. [Figure 13] This figure shows an example of mapping a 3D model to a 2D space. [Figure 14] This is a diagram illustrating the method for calculating the second level of manifestation. [Figure 15] This diagram illustrates how to select the effective edges to be extended. [Figure 16] This figure shows an example of a reception screen generated by the 3D model extension unit. [Figure 17] This is a diagram to explain edge grouping. [Figure 18]This is a diagram for explaining the grouping of edges. [Figure 19] This is a block diagram showing an example of the hardware configuration of an information processing apparatus and an information processing system in each exemplary embodiment of the present invention.

Mode for Carrying Out the Invention

[0011] 〔Exemplary Embodiment 1〕 The first exemplary embodiment of the present invention will be described in detail with reference to the drawings. This exemplary embodiment is a basic form for the exemplary embodiments described later.

[0012] (Configuration of Information Processing Apparatus 1) The configuration of the information processing apparatus 1 according to this exemplary embodiment will be described with reference to FIG. 1. FIG. 1 is a block diagram showing the configuration of the information processing apparatus 1 according to this exemplary embodiment. The information processing apparatus 1 derives one or more candidate solutions regarding the position and orientation of the object using the sensing data obtained by a sensor that includes the object within the sensing range and a 3D model.

[0013] The object is the target for estimating the position and orientation. Examples of the object include, but are not limited to, the hopper (loading platform) of a dump truck and a box capable of storing objects inside surrounded by edges.

[0014] The information processing apparatus 1 is widely applicable to one or more AGVs (Automatic Guided Vehicles), construction machines, autonomous vehicles, and monitoring systems, etc. For example, in a work site where an excavator loads the earth and sand excavated by a backhoe into the hopper of a dump truck, the information processing apparatus 1 estimates at least one of the position and orientation of the hopper of the dump truck as the object, and refers to at least one of the estimated position and orientation, and can be used in a system for loading earth and sand into the hopper.

[0015] Sensors include, for example, depth sensors or imaging sensors. Examples of depth sensors include, but are not limited to, stereo cameras equipped with multiple cameras that determine the distance (depth) to an object by the parallax between the cameras, or LiDAR (Light Detection and Ranging) that measures the distance (depth) to an object using a laser. Sensing data obtained by sensors includes, for example, depth information or captured images. Examples of depth information include depth images representing depth acquired by stereo cameras, or coordinate data showing the coordinates of each point acquired by LiDAR, but these are not limited to this exemplary embodiment. It is also possible to represent depth in image format by converting coordinate data acquired by LiDAR.

[0016] In this exemplary embodiment, the position of an object refers to its position in three-dimensional space and includes its translational position. Similarly, the orientation of an object refers to its orientation in three-dimensional space and includes its orientation. However, the specific parameters used to represent the position and orientation of an object are not limited to this exemplary embodiment.

[0017] As an example, the position and orientation of an object can be expressed by the position of its center of gravity (x, y, z) and its orientation (roll, pitch, yaw), respectively. In this case, the position and orientation of the object are expressed by six parameters: (x, y, z, roll, pitch, yaw). x, y, and z represent the position (coordinates) of the object in three-dimensional space.

[0018] As shown in Figure 1, the information processing device 1 comprises an acquisition unit 11, an estimation unit 12, and an expansion unit 13. In this exemplary embodiment, the acquisition unit 11, estimation unit 12, and expansion unit 13 are configured to implement an acquisition means, an estimation means, and an expansion means, respectively.

[0019] The acquisition unit 11 acquires sensing data obtained by a sensor that includes the target object within its sensing range. For example, the acquisition unit 11 may acquire sensing data output by a sensor, or it may receive sensing data from other devices via a communication interface.

[0020] The estimation unit 12 derives one or more candidate solutions regarding the position and orientation of the object by performing a first estimation process using the sensing data acquired by the acquisition unit 11 and the 3D model. More specifically, as an example, the estimation unit 12 derives one or more candidate solutions regarding the position and orientation of the object in 3D space by referring to first 2D data obtained by a feature point extraction process that references depth information and third 2D data obtained by mapping the 3D model to 2D space and performing the first feature point extraction process. The estimation unit 12 supplies the derived one or more candidate solutions to the extension unit 13.

[0021] Here, the first feature point extraction process is a process that refers to depth information and extracts one or more feature points contained in said depth information. An example of the first feature point extraction process is the edge extraction process of an object using an edge extraction filter. With this configuration, since the edge extraction process can be performed on the depth information, the information processing device 1 can suitably extract the feature points of the object.

[0022] The extension unit 13 extends the 3D model for each of the one or more candidate solutions. The extension of the 3D model includes, as an example, determining the length of an edge to be added to the 3D model. Regarding the relationship between the candidate solutions and the 3D models, as an example, for each candidate solution regarding the position and orientation of an object derived by the estimation unit 12, one 3D model having that position and orientation is identified. Alternatively, the estimation unit 12 may be described as having a configuration that derives a 3D model having the estimated position and orientation as a candidate solution regarding the position and orientation of an object.

[0023] The extension unit 13, as an example, extends the 3D model to improve its visibility. Visibility can also be expressed as the degree of visibility or identifiability of the 3D model. Visibility may, as an example, be evaluated on a pixel-by-pixel basis.

[0024] Furthermore, the estimation unit 12 estimates the position and orientation of the object by a second estimation process using each of the 3D models after expansion by the expansion unit 13. For example, the estimation unit 12 estimates the position and orientation of the object in 3D space by referring to the second 2D data obtained by a second feature point extraction process that references the captured image supplied from the acquisition unit 11, and the 3D model, and using one or more candidate solutions generated by the estimation unit 12. For example, the estimation unit 12 estimates the position and orientation of the object in 3D space by referring to the second 2D data obtained by a second feature point extraction process that references the captured image, and the fourth 2D data obtained by mapping the 3D model to 2D space and performing a second feature point extraction process, and using one or more candidate solutions generated by the estimation unit 12.

[0025] Here, the second feature point extraction process is a process that refers to the captured image and extracts one or more feature points contained in the captured image. An example of the second feature point extraction process is the edge extraction process of an object using an edge extraction filter. With this configuration, since the edge extraction process can be performed on the captured image, the information processing device 1 can suitably extract the feature points of the object.

[0026] Furthermore, the edge extraction filter used in the second feature point extraction process may be the same as the edge extraction filter used in the first feature point extraction process, or it may be a different edge extraction filter from the one used in the first feature point extraction process. For example, the edge extraction filter used in the second feature point extraction process may have different filter coefficients than the edge extraction filter used in the first feature point extraction process.

[0027] As described above, the information processing device 1 according to this exemplary embodiment includes an acquisition unit 11 that acquires sensing data obtained by a sensor that includes an object within its sensing range, an estimation unit that derives one or more candidate solutions regarding the position and orientation of the object by performing a first estimation process using the sensing data acquired by the acquisition unit 11 and a three-dimensional model, and an extension unit 13 that extends the three-dimensional model for each of the one or more candidate solutions, and the estimation unit 12 is configured to estimate the position and orientation of the object by performing a second estimation process using each of the three-dimensional models extended by the extension unit 13. For this reason, according to the information processing device 1 according to this exemplary embodiment, the position and orientation of an object can be estimated without preparing a large number of three-dimensional models.

[0028] (Information processing method S1 flow) The flow of the information processing method S1 according to this exemplary embodiment will be explained with reference to Figure 2. Figure 2 is a flowchart showing the flow of the information processing method S1 according to this exemplary embodiment.

[0029] In step S11, the acquisition unit 11 acquires sensing data obtained by a sensor that includes the object within its sensing range. In step S12, the estimation unit 12 derives one or more candidate solutions regarding the position and orientation of the object by performing a first estimation process using the sensing data and the 3D model. In step S13, the expansion unit 13 expands the 3D model for each of the one or more candidate solutions. In step S14, the estimation unit 12 estimates the position and orientation of the object by performing a second estimation process using each of the expanded 3D models.

[0030] As described above, the information processing method S1 according to this exemplary embodiment includes acquiring sensing data obtained by a sensor that includes an object within its sensing range, deriving one or more candidate solutions regarding the position and orientation of the object by a first estimation process using the sensing data and a three-dimensional model, extending the three-dimensional model with respect to each of the one or more candidate solutions, and estimating the position and orientation of the object by a second estimation process using each of the extended three-dimensional models. Therefore, the information processing method S1 according to this exemplary embodiment has the same effects as the information processing device 1.

[0031] [Exemplary Embodiment 2] A second exemplary embodiment of the present invention will be described in detail with reference to the drawings. Components having the same function as those described in Exemplary Embodiment 1 will be denoted by the same reference numerals, and their descriptions will not be repeated.

[0032] (Configuration of Information Processing System 100) The configuration of the information processing system 100 according to this exemplary embodiment will be described with reference to Figure 3. Figure 3 is a block diagram showing the configuration of the information processing system 100 according to this exemplary embodiment.

[0033] As shown in Figure 3, the information processing system 100 comprises an information processing device 3, a depth sensor 4, and an RGB (Red, Green, Blue) camera 5. In the information processing system 100, the information processing device 3 acquires depth information that includes the object obtained by the depth sensor 4 within its sensing range, and acquires imaging information that includes the object obtained by the RGB camera 5 within its field of view. The information processing device 3 then refers to the acquired depth information and imaging information to estimate the position and orientation of the object. The object, depth information, and the position and orientation of the object are as described in the exemplary embodiment described above.

[0034] The depth sensor 4 is a sensor that outputs depth information indicating the distance to an object included in the sensing range. Examples of the depth sensor 4 include, but are not limited to, a stereo camera equipped with multiple cameras and a LiDAR, as described in the embodiments described above. Examples of depth information also include, but are not limited to, a depth image representing depth and coordinate data indicating the coordinates of each point, as described in the embodiments described above.

[0035] The RGB camera 5 is equipped with an image sensor that captures objects within the field of view and outputs image data that includes the objects within the field of view. The information processing system 100 is not limited to the RGB camera 5, and may include any camera that outputs a multi-level image. For example, instead of the RGB camera 5, the system may include a monochrome camera that outputs a black and white image representing the captured object in shades of white and black. The depth sensor 4 and the RGB camera 5 may be separate from the information processing device 3, or they may be built into the information processing device 3.

[0036] (Configuration of Information Processing Device 3) As shown in Figure 3, the information processing device 3 comprises a control unit 31, an input / output unit 32, and a storage unit 33. The input / output unit 32 is a device that outputs data supplied from the control unit 31, which will be described later. One example of the input / output unit 32 outputting data is a configuration in which the input / output unit 32 is connected to a network (not shown) and the data is output to another device that can communicate via the network. Another example of the input / output unit 32 outputting data is a configuration in which the input / output unit 32 is connected to a display (e.g., a display panel, etc.) (not shown) and the data indicating the image to be displayed on the display is output. These are not limiting to this exemplary embodiment. The input / output unit 32 also functions as a means for receiving user input regarding the identification of effective edges, which will be described later. To perform this function, the input / output unit 32 may be equipped with a keyboard, mouse, touch panel, etc.

[0037] The memory unit 33 stores various types of data that the control unit 31, described later, will refer to. As an example, the memory unit 33 stores a group of 3D models 332, which includes multiple 3D models 331. The 3D models 331 may be defined by meshes (e.g., triangle meshes) or surfaces used in 3D modeling, or they may be models that explicitly include data about the edges (contours) of an object, or they may have textures that represent the features of an image of an object defined within them. By configuring the 3D models 331 to explicitly include data about the edges (contours) of an object, edge extraction processing can be performed on the 3D models 331, allowing the information processing device 3 to suitably extract feature points of the object. The 3D models 331 may also include data about the vertices of an object.

[0038] The 3D model group 332 includes the initial 3D model 331 and an extended version of the initial 3D model 331. Hereafter, the initial 3D model 331 will also be referred to as "initial 3D model 331_0". Initial 3D model 331_0 models, as an example, at least some of the features common to various types of bessels. Initial 3D model 331_0 is a 3D model representing a part of a bessel, including six vertices and four edges. Here, the length of each edge in initial 3D model 331_0 may be set to be relatively short.

[0039] (Control Unit 31) The control unit 31 controls each component of the information processing device 3. For example, it acquires data from the storage unit 33 and outputs data to the input / output unit 32. In addition, as shown in Figure 3, the control unit 31 also functions as a depth information acquisition unit 311, a depth image feature point extraction unit 312, a depth image position estimation unit 313, an RGB image acquisition unit 314, an RGB image feature point extraction unit 315, an RGB image position estimation unit 316, and a 3D model extension unit 317. The depth information acquisition unit 311 and the RGB image acquisition unit 314 are configured to implement the acquisition means in this exemplary embodiment. The 3D model extension unit 317 is configured to implement the extension means in this exemplary embodiment. The depth image position estimation unit 313 and the RGB image position estimation unit 316 are configured to implement the estimation means in this exemplary embodiment.

[0040] The depth information acquisition unit 311 acquires depth information obtained by the depth sensor 4, which includes the target object within its sensing range. Furthermore, even if the target object is not within the sensing range, the depth information acquisition unit 311 acquires depth information related to the sensing range, which is obtained by the depth sensor 4. The depth information acquisition unit 311 supplies the acquired depth information to the depth image feature point extraction unit 312.

[0041] The depth image feature point extraction unit 312 performs a first feature point extraction process by referring to the depth information supplied from the depth information acquisition unit 311 and generates first two-dimensional data. The depth image feature point extraction unit 312 supplies the generated first two-dimensional data to the depth image position estimation unit 313. The first feature point extraction process is as described in the embodiment described above. An example of the process performed by the depth image feature point extraction unit 312 will be described later with different reference drawings.

[0042] The depth image position estimation unit 313 generates one or more candidate solutions regarding at least one of the position and orientation of the object in three-dimensional space by referring to the first two-dimensional data supplied from the depth image feature point extraction unit 312 and the 3D model 331 stored in the storage unit 33. The depth image position estimation unit 313 supplies the generated one or more candidate solutions to the RGB image position estimation unit 316. An example of the processing performed by the depth image position estimation unit 313 will be described later with different reference drawings.

[0043] The RGB image acquisition unit 314 acquires an RGB image (captured image) obtained by the RGB camera 5, which includes the target object in its field of view. The RGB image acquisition unit 314 supplies the acquired RGB image to the RGB image feature point extraction unit 315.

[0044] The RGB image feature point extraction unit 315 performs a second feature point extraction process by referring to the RGB image supplied by the RGB image acquisition unit 314 and generates second two-dimensional data. The RGB image feature point extraction unit 315 supplies the generated second two-dimensional data to the RGB image position estimation unit 316. The second feature point extraction process is as described in the exemplary embodiment described above. An example of the process performed by the RGB image feature point extraction unit 315 will be described later with different reference drawings.

[0045] The RGB image position estimation unit 316 refers to the second two-dimensional data supplied from the RGB image feature point extraction unit 315 and the 3D model 331 stored in the storage unit 33, and uses one or more candidate solutions supplied from the depth image position estimation unit 313 to derive one or more candidate solutions regarding the position and orientation of the object in three-dimensional space. The RGB image position estimation unit 316 supplies the derived candidate solutions to the 3D model extension unit 317.

[0046] Furthermore, the RGB image position estimation unit 316 refers to the second two-dimensional data supplied from the RGB image feature point extraction unit 315 and the 3D model 331 stored in the storage unit 33, and estimates the position and orientation of the object in three-dimensional space using one or more candidate solutions supplied from the depth image position estimation unit 313. The RGB image position estimation unit 316 supplies the estimated position and orientation of the object in three-dimensional space to the input / output unit 32. An example of the processing performed by the RGB image position estimation unit 316 will be described later.

[0047] The 3D model extension unit 317 extends the 3D model 331 with respect to each of the one or more candidate solutions derived by the RGB image position estimation unit 316, so as to improve the visibility of the 3D model 331. The visibility is as described in the exemplary embodiment 1 above. An example of the processing performed by the 3D model extension unit 317 will be described later.

[0048] (Example of a method for estimating the position and orientation of an object in three-dimensional space) Here, an example of how the RGB image position estimation unit 316 estimates the position and orientation of an object in three-dimensional space will be explained using Figure 4. Figure 4 shows the position of the camera CA2 that images the object RT, and the method by which the RGB image position estimation unit 316 estimates the position and orientation of the object in three-dimensional space according to this exemplary embodiment. The camera CA2 in Figure 4 is an example of the RGB camera 5 according to this exemplary embodiment.

[0049] In the example shown in Figure 4, camera CA2 is mounted on top of the backhoe 6 and photographs the object RT, which is the vessel of the dump truck 7. When camera CA2 images the object RT, camera CA2 outputs image P2. The RGB image position estimation unit 316 calculates the coordinates of the object RT on the global coordinate system (the position and orientation of the object RT in three-dimensional space) by moving and rotating the 3D model 331 based on the position parameters of the object RT included in image P2. Here, the position parameters represent the possible positions and orientations of the object RT. Examples of position parameters will be described later using different reference drawings.

[0050] (Processing flow executed by the information processing device 3) The processing flow executed by the information processing device 3 will be explained using Figures 5 to 7. Figures 5 and 6 are flowcharts illustrating an example of the processing flow executed by the information processing device 3 according to this exemplary embodiment. Figure 7 is a diagram showing examples of images referenced and generated in each process executed by the information processing device 3 according to this exemplary embodiment. In the example shown in Figure 7, a dump truck vessel is used as the object for explanation.

[0051] (Step S21) In step S21 of Figure 5, the control unit 31 acquires one or more 3D models 331. The 3D models 331 acquired by the control unit 31 are the initial 3D models 331_0 before they are expanded by the 3D model expansion unit 317. The information processing device 3 stores the acquired 3D models 331_0 in the storage unit 33.

[0052] The 3D model image P11 of Bessel in Figure 7 is an image showing the initial 3D model 331_0 of the object, Bessel. In the example in Figure 7, the initial 3D model 331_0 includes data about the edges of Bessel.

[0053] (Step S22) In step S22, the 3D model extension unit 317 extends one or more 3D models 331 acquired in step S21, or one or more 3D models 331 determined in step S26, which will be described later.

[0054] In this processing example, the 3D model 331 includes multiple vertices and one or more edges, and the one or more edges include: (i) Edges relating to depth and at least one of the image, which are definite edges whose length has been determined, (ii) Edges relating to depth and / or the image, which are effective edges whose length is undetermined, (iii) Undetermined edges that are candidates for extension, It includes at least one of the following. Here, an effective edge is, for example, an edge connected to a confirmed edge. An effective edge is an example of a candidate for extension of 3D model 331. Also, in this example, the initial 3D model 331_0 includes at least one confirmed edge or at least one effective edge.

[0055] Furthermore, the 3D model 331 may include invalid edges. Here, invalid edges are edges that are not used to estimate the position and orientation of the object, and one example is an edge that is not visible in the depth image or RGB image.

[0056] Step S22 includes steps S221 to S226. The 3D model extension unit 317 performs the processing of steps S221 to S226 for each of the one or more 3D models 331. In steps S221 to S226, the 3D model extension unit 317 extends the 3D model 331 by selecting effective edges to be extended from among the multiple effective edges included in the 3D model 331 and determining the length of the selected directed edges.

[0057] (Step S221) First, in step S221, the 3D model extension unit 317 determines whether the 3D model 331 to be extended contains multiple effective edges that are candidates for extension. If the 3D model 331 contains multiple effective edges (YES in step S221), the 3D model extension unit 317 proceeds to step S222. On the other hand, if the 3D model 331 contains only one effective edge (NO in step S221), the 3D model extension unit 317 skips steps S222 to S224 and proceeds to step S225. In other words, if the 3D model 331 to be extended contains multiple effective edges, the 3D model extension unit 317 selects the effective edge to be extended from the multiple effective edges by executing the following steps S222 to S224. On the other hand, if there is only one effective edge, the 3D model extension unit 317 performs the extension process shown in step S225 for that single effective edge without performing the processes in steps S222 to S224.

[0058] Here, "multiple effective edges included in 3D model 331" corresponds to "multiple extension candidates" as defined herein. In other words, the number of "multiple effective edges included in 3D model 331" corresponds to the number of "multiple extension candidates" as defined herein.

[0059] (Step S222・S223) In steps S222 to S223, the 3D model extension unit 317 calculates a first manifestation, which is the manifestation of the 3D model 331 with respect to depth, and a second manifestation, which is the manifestation of the 3D model 331 with respect to image. In step S222, the 3D model extension unit 317 calculates a first manifestation, which is the manifestation of depth, for each of the effective edges included in the 3D model 331. In step S223, the 3D model extension unit 317 calculates a second manifestation, which is the manifestation of image, for each of the effective edges included in the 3D model 331. In other words, in steps S222 to S223, the information processing device 3 calculates a first manifestation and a second manifestation for each of the multiple extension candidates (effective edges) of the 3D model 331. In the following explanation, when it is not necessary to distinguish between the first and second manifestations, they will simply be referred to as "manifestation." Specific examples of how to calculate the manifestation will be shown later, using different diagrams.

[0060] (Step S224) In step S224, the 3D model extension unit 317 selects an effective edge to be extended based on the manifestation level calculated for each of the effective edges that are candidates for extension. Here, the number of effective edges to be extended may be one or multiple. More specifically, as an example, the 3D model extension unit 317 selects the extension candidate with the largest integrated manifestation level obtained by integrating the first manifestation level and the second manifestation level. Here, as an example, the 3D model extension unit 317 uses the sum of the first manifestation level and the second manifestation level as the integrated manifestation level.

[0061] More specifically, the 3D model extension unit 317, as an example, selects the effective edge that has the greatest integrated manifestation level when one or more undetermined edges connected to an effective edge are promoted to effective edges from among the multiple effective edges included in the 3D model 331.

[0062] (Step S225) In step S225, the 3D model extension unit 317 extends the 3D model 331 so that its visibility is improved. More specifically, the information processing device 3 extends the 3D model 331 so that the integrated visibility obtained by integrating the first visibility and the second visibility is improved. More specifically, as an example, the 3D model extension unit 317 extends the 3D model 331 by promoting the effective edges selected in step S224 to confirmed edges and promoting one or more undetermined edges connected to the selected effective edges to effective edges.

[0063] In step S225, if there are multiple effective edges selected in step S224, the 3D model extension unit 317 performs extension processing for each of the selected effective edges. For example, if the 3D model extension unit 317 selects M effective edges (where M is an integer greater than or equal to 2) in step S224, it performs extension processing for each of the M effective edges. In this case, the 3D model extension unit 317 generates M 3D models 331 as extended 3D models for one 3D model 331.

[0064] (Step S226) In step S226, the 3D model extension unit 317 determines whether there are any 3D models 331 that have not been extended, that is, whether there are any 3D models 331 that are to be extended. If there are any 3D models 331 that have not been extended (yes in step S226), the 3D model extension unit 317 returns to the process in step S221 and performs the extension process on the next 3D model 331. On the other hand, if the extension process has been completed for all 3D models 331 that are to be extended (no in step S226), the 3D model extension unit 317 proceeds to the process in step S23.

[0065] The 3D model extension unit 317 executes steps S221 to S225 for each of the K (K is an integer greater than or equal to 1) 3D models 331 that are to be extended. For example, if the 3D model extension unit 317 extends each of the K 3D models 331 with M variations, then K × M 3D models 331 are obtained as the expanded 3D models 331.

[0066] (Step S23) In step S23, the depth image feature point extraction unit 312 and the depth image position estimation unit 313 derive one or more candidate solutions regarding the position and orientation of the object by performing a first estimation process using the depth information acquired by the depth information acquisition unit 311 and the 3D model 331 expanded by the 3D model expansion unit 317. At this time, the depth image position estimation unit 313 performs the first estimation process using, for example, the determined edges and effective edges included in the 3D model 331, and does not use the undetermined edges and invalid edges in the estimation process. Details of the process in step S23 will be described later with reference to different drawings.

[0067] (Step S24) In step S24, the RGB image feature point extraction unit 315 and the RGB image position estimation unit 316 use the RGB image acquired by the RGB image acquisition unit 314 and the 3D model 331 expanded by the 3D model expansion unit 317 to perform the first estimation process and the second estimation process. In this exemplary embodiment, the second estimation process by the RGB image position estimation unit 316 includes a process to derive one or more candidate solutions regarding the position and orientation of the object using each of the expanded 3D models 331. At this time, as an example, the RGB image position estimation unit 316 performs the first estimation process and the second estimation process using the determined edges and effective edges included in the 3D model 331, and does not use the undetermined edges and invalid edges in the estimation process. Details of the process in step S24 will be described later with reference to different drawings.

[0068] (Step S25) In step S25, the RGB image position estimation unit 316 determines whether to terminate the loop processing. For example, the RGB image position estimation unit 316 determines to continue the loop processing if there are still valid edges remaining and the processing time is less than a threshold. Alternatively, for example, the RGB image position estimation unit 316 may determine to terminate the loop processing if the difference between the 3D model 331 with the smallest difference between the image with edges extracted from the depth image and / or the image with edges extracted from the RGB image, and the 3D model 331 with the next smallest difference between the image with edges extracted from the depth image and / or the image with edges extracted from the RGB image, is greater than or equal to a predetermined threshold. Here, "difference between the image with edges extracted from the depth image and / or the image with edges extracted from the RGB image" means "at least one of the difference between the image with edges extracted from the depth image and the difference between the image with edges extracted from the RGB image." "The difference between the image with edges extracted from the depth image and / or the image with edges extracted from the RGB image" refers, for example, to a first difference which is the difference between the image with edges extracted from the depth image and the mapping of the 3D model 331, a second difference which is the difference between the image with edges extracted from the RGB image and the mapping of the 3D model 331, or the statistical value of the first difference and the second difference. Since a small difference between the two 3D models 331 is assumed to indicate that the two 3D models 331 are not sufficiently distinguishable, the accuracy of the estimation process related to the object can be improved by performing loop processing until the difference becomes sufficiently large.

[0069] Furthermore, as another example of the determination method, for example, a function distance(m*, m) is defined to define the distance between the first 3D model 331_m1 and the second 3D model 331_m2, and m* is the 3D model 331 with the smallest difference among the K 3D models 331, and e(m*) is the difference between the depth image and / or RGB image and the 3D model 331. In this case, the RGB image position estimation unit 316, e(m) - e(m*)+α×distance(m*, m) The loop processing may be terminated if the value exceeds a predetermined threshold. Here, α < 0. This determination rule includes the α × distance(m*, m) part, which takes into account the differences in position and orientation of the 3D model 331.

[0070] An example of the function distance(m*, m) is, for instance, distance(m1,m2)=α×|m1_translation-m2_translation|+β×|m1_rotation-m2_rotation|+γ×geo_distance(m1,m2) For example, m1_translation represents translation [x, y, z] and m1_rotation represents rotation [roll, pitch, yaw]. Also, geo_distance is defined as follows, where V is a function that returns the set of vertices included in the model.

number

[0071] If the loop processing is terminated in step S25 (YES in step S25), the RGB image position estimation unit 316 proceeds to the processing in step S27. On the other hand, if the loop processing is continued (NO in step S25), the RGB image position estimation unit 316 proceeds to the processing in step S26.

[0072] (Step S26) In step S26, the 3D model extension unit 317 determines the 3D model 331 to be extended based on one or more candidate solutions derived by the RGB image position estimation unit 316. That is, the 3D model extension unit 317 determines the 3D model 331 to be extended that corresponds to at least some of the one or more candidate solutions derived by the RGB image position estimation unit 316. As an example, the 3D model extension unit 317 selects K (K is a natural number of 1 or more) candidate solutions with small errors and determines the 3D model 331 to be extended that corresponds to each of the selected solutions.

[0073] For example, if M 3D models 331 (where M is the number of variations in the expansion process) are generated as expanded 3D models for each of the K 3D models 331, the number of expanded 3D models 331 will be K × M. In this case, in step S23, the depth image position estimation unit 313 derives a set of candidate solutions for the position and orientation of the object using the K × M 3D models 331, thereby obtaining K × M candidate solutions. In step S26, the 3D model expansion unit 317 selects K solutions with small errors from the set of K × M candidate solutions, and in step S225, it performs the expansion process on the K 3D models 331 corresponding to the selected K candidate solutions.

[0074] After completing the process in step S26, the 3D model extension unit 317 returns to the process in step S22 and executes the extension process of the 3D model 331 again. In other words, the control unit 31 repeats the processes in steps S22 to S26 multiple times. To put it another way, the control unit 31 repeats the extension process by the 3D model extension unit 317 and the second estimation process by the RGB image position estimation unit 316 multiple times.

[0075] (Step S27) In step S27, the control unit 31 selects at least one of the candidate solutions derived by the RGB image position estimation unit 316 and outputs it to the input / output unit 32.

[0076] (Example of derivation process for candidate solution groups) Next, a specific example of the processing in steps S23 to S24 of Figure 5 will be explained using Figure 6. The flowchart shown in Figure 6 is a flowchart of an example of the processing in steps S23 to S24 of Figure 5. In Figure 6, the information processing method S3 includes steps S31 to S51.

[0077] (Step S31) In step S31, the control unit 31 acquires the 3D model 331. The 3D model 331 acquired by the control unit 31 in step S31 includes the expanded 3D model 331 expanded by the 3D model expansion unit 317 in step S22 of Figure 5.

[0078] (Step S32) In step S32, the depth image position estimation unit 313 obtains a set of position parameters of the object to be evaluated. As described above, position parameters represent the possible positions and orientations of the object. One example of the set of position parameters of an object is the set of possible positions and orientations (set of position parameters) of Bessel.

[0079] (Step S33-S34) In step S33, the depth image position estimation unit 313 selects one unevaluated position parameter from the set of position parameters indicating the position and orientation of Bessel. In step S34, the depth image position estimation unit 313 moves and rotates the 3D model 331 stored in the memory unit 33 based on the selected position parameter.

[0080] (Step S35) In step S35, the depth image position estimation unit 313 maps the moved and rotated 3D model 331 onto a two-dimensional space and generates a mapping image. The mapping image generated by the depth image position estimation unit 313 is an image that shows the depth information of the 3D model 331.

[0081] (Step S36) In step S36, the depth image position estimation unit 313 extracts the contours (edges) of the object in the mapping image. As an example, the depth image position estimation unit 313 extracts the contours, which are feature points of the object, by applying a first feature point extraction process to the mapping image, and generates third two-dimensional data representing those contours. The third two-dimensional data generated by the depth image position estimation unit 313 is also called "template data".

[0082] (Step S37) In step S37, the depth information acquisition unit 311 acquires depth information obtained by the depth sensor 4, which includes the object within its sensing range. The depth information acquisition unit 311 then supplies the acquired depth information to the depth image feature point extraction unit 312.

[0083] The depth image feature point extraction unit 312 generates a depth image by referring to the depth information supplied from the depth image feature point extraction unit 312. As an example, the depth image feature point extraction unit 312 acquires depth information in which the object is included in the sensing range and depth information in which the object is not included in the sensing range, and generates a depth image in which the object is included and a depth image in which the object is not present.

[0084] The depth image feature point extraction unit 312 generates, as an example, a depth image P14 of the recognized object, which is a depth image in which the object RT is included in the sensing range, and a background depth image (not shown), which is a depth image in the case when the object RT is not present in the sensing range.

[0085] (Step S38) In step S38, the depth image feature point extraction unit 312 refers to the depth image and extracts the contour of the object. The data obtained by the depth image feature point extraction unit 312 from the contour of the object is the first two-dimensional data, also referred to as "depth edge" or "search data".

[0086] In the example shown in Figure 7, the depth image feature point extraction unit 312 first calculates the difference between the depth image P14 to be recognized and the background depth image (not shown), and generates a difference image P15, which is the difference information.

[0087] Next, the depth image feature point extraction unit 312 performs a first feature point extraction process by referring to the generated difference information and extracts one or more feature points included in the difference image. With this configuration, the information processing device 3 performs the extraction process of objects and feature points of objects included in the depth information by referring to depth information with a small amount of information, so that computation cost and computation time can be reduced.

[0088] In the example shown in Figure 7, the depth image feature point extraction unit 312 generates an image P16 by extracting edges OL2 from the difference image P15 using an edge extraction filter. Image P16 is the first two-dimensional data (depth edge or search data). The depth image feature point extraction unit 312 supplies the first two-dimensional data to the depth image position estimation unit 313.

[0089] Here, the depth image feature point extraction unit 312 may perform the first feature point extraction process by referring to the binarized difference information obtained by applying a binarization process to the difference information. With this configuration, the information processing device 3 refers to the binarized difference information obtained by applying a binarization process with less information, so computation cost and computation time can be reduced.

[0090] Steps S37 and S38 are examples of processes performed by the depth image feature point extraction unit 312. Steps S37 and S38 may be performed in parallel with steps S31 to S36, before steps S31 to S36, or after steps S31 to S36.

[0091] (Step S39) In step S39, the depth image position estimation unit 313 matches the template data (third 2D data) extracted in step S36 with the search data (first 2D data) supplied by the depth image feature point extraction unit 312 in step S38, and calculates the matching error. As an example, the depth image position estimation unit 313 calculates the matching error by performing a template matching process that references the third 2D data and the first 2D data.

[0092] Here, Chamfer Matching is given as an example of template matching processing, but this is not limited to this embodiment. Other examples of the depth image position estimation unit 313 include methods using PnP (Perspective n Point), ICP (Interactive Closest Point), and DCM (Directional Chamfer Matching) to calculate the matching error, but are not limited to these.

[0093] In the example shown in Figure 7, image P17 is shown as an image in which the search data image P16 and the template data applied to image P16 are superimposed. The depth image position estimation unit 313 calculates the error between the edge OL2 contained in image P16 and the template data as the matching error. The error calculated by the depth image position estimation unit 313 is also called the "matching error (depth)" to indicate that it is a matching error using depth information.

[0094] (Step S40) In step S40, the depth image position estimation unit 313 determines whether or not there are unevaluated position parameters. If it is determined in step S40 that there are unevaluated position parameters (step S40: yes), the depth image position estimation unit 313 returns to the process in step S33.

[0095] (Step S41) If it is determined in step S40 that there are no unevaluated position parameters (step S40: no), then in step S41, the depth image position estimation unit 313 selects up to N position parameters whose matching error (depth) is less than or equal to a predetermined threshold and has a small error, and sets these as N candidate solutions. Here, the depth image position estimation unit 313 may also select N position parameters with relatively small errors and set these as N candidate solutions. With this configuration, the information processing device 3 generates one or more candidate solutions by template matching processing that refers to first two-dimensional data obtained by referring to depth information which has less information than the RGB image, thus reducing computation cost and computation time. The depth image position estimation unit 313 supplies the N candidate solutions to the RGB image position estimation unit 316.

[0096] Steps S32 to S36 and S39 to S41 described above are examples of processes performed by the depth image position estimation unit 313.

[0097] (Step S42) In step S42, the RGB image position estimation unit 316 obtains N candidate solutions, which are position parameters, from the depth image position estimation unit 313, and uses these candidate solutions as the position parameters to be evaluated.

[0098] (Step S43) In step S43, the RGB image position estimation unit 316 selects one unevaluated position parameter from among the N position parameters.

[0099] (Step S44) In step S44, the RGB image position estimation unit 316 moves and rotates the 3D model 331 stored in the memory unit 33 based on the selected position parameters.

[0100] (Step S45) In step S45, the RGB image position estimation unit 316 maps the moved and rotated 3D model 331 onto a two-dimensional space and generates a mapping image. The mapping image generated by the RGB image position estimation unit 316 is characterized by being an image that includes texture information of the 3D model 331.

[0101] (Step S46) In step S46, the RGB image position estimation unit 316 extracts the contour of the object in the mapping image. As an example, the RGB image position estimation unit 316 extracts the contour (edge) of the object by applying a second feature point extraction process to the mapping image and generates a fourth two-dimensional data representing the contour. The contour extracted by the RGB image position estimation unit 316 may be rectangular. The fourth two-dimensional data generated by the RGB image position estimation unit 316 is also called "template data".

[0102] (Step S47) In step S47, the RGB image acquisition unit 314 acquires an RGB image in which the object obtained by the RGB camera 5 is included in the field of view. The RGB image acquisition unit 314 supplies the acquired RGB image to the RGB image feature point extraction unit 315.

[0103] (Step S48) In step S48, the RGB image feature point extraction unit 315 refers to the RGB image supplied from the RGB image acquisition unit 314, performs a second feature point extraction process, and generates second two-dimensional data.

[0104] In the example shown in Figure 7, the RGB image feature point extraction unit 315 extracts the rectangular contour of an object contained in the RGB image P18 as a feature point. A known method can be used as an example of how to extract the rectangular contour. The RGB image feature point extraction unit 315 generates image P19, which includes the extracted contour OL4, as a second two-dimensional data. Image P19 generated by the RGB image feature point extraction unit 315 is also referred to as "RGB edge" or "search data". The RGB image feature point extraction unit 315 supplies the generated second two-dimensional data to the RGB image position estimation unit 316.

[0105] The process in step S48 is an example of the process performed by the RGB image feature point extraction unit 315. Steps S47 and S48 may be performed in parallel with steps S42 to S46, before steps S42 to S46, or after steps S42 to S46.

[0106] (Step S49) In step S49, the RGB image position estimation unit 316 matches the template data (fourth 2D data) extracted in step S46 with the search data (second 2D data) supplied by the RGB image feature point extraction unit 315 in step S48, and calculates the matching error. As an example, the RGB image position estimation unit 316 calculates the matching error by a template matching process that references the fourth 2D data and the second 2D data. Here, Chamfer Matching can be cited as an example of a template matching process, but this is not limited to this embodiment. As other examples, the RGB image position estimation unit 316 may also use PnP, ICP, and DCM to calculate the matching error, but is not limited to these.

[0107] In the example shown in Figure 7, image P20 is shown as an image in which the search data image P19 and the template data applied to image P19 are superimposed. The RGB image position estimation unit 316 calculates the error between the contour OL4 contained in image P19 and the template data as the matching error. The error calculated by the RGB image position estimation unit 316 is also referred to as the "matching error (image)" to indicate that it is a matching error using an RGB image (image).

[0108] (Step S50) In step S50, the RGB image position estimation unit 316 determines whether or not there are unevaluated position parameters. If it is determined in step S50 that there are unevaluated position parameters (step S50: yes), the RGB image position estimation unit 316 returns to the process in step S43.

[0109] (Step S51) In step S51, the RGB image position estimation unit 316 calculates an overall error from the matching error (depth) and matching error (image) calculated for each position parameter, and selects the position parameter with the smallest overall error as the candidate solution. With this configuration, the information processing device 3 derives candidate solutions regarding the position and orientation of the object in three-dimensional space by template matching processing that refers to second two-dimensional data obtained by referring to an RGB image, which has a larger amount of information than depth information, thus enabling the deriving of suitable candidate solutions regarding the position and orientation of the object. The RGB image position estimation unit 316 supplies the selected parameters to the input / output unit 32.

[0110] In the example shown in Figure 7, image P17 is displayed, representing three candidate solutions corresponding to three templates with different positions and orientations of a single 3D model 331. Furthermore, for each of the three candidate solutions corresponding to image P17, the 3D model expansion unit 317 expands the 3D model 331 in step S22 of Figure 5, thereby obtaining three expanded 3D models 331.

[0111] As an example, the RGB image position estimation unit 316 can calculate the overall error e using the following formula (1), but this is not limited to this exemplary embodiment. e = wd × ed + wi × ei ... (1) Each variable in equation (1) represents the following: wd: weighting parameter wi: weighting parameter ed: Matching error (depth) ei: Matching error (image) In other words, the RGB image position estimation unit 316 uses as the sum of the product of the matching error (depth) ed calculated by the depth image position estimation unit 313 in step S39 and the weighted parameter wd, and the product of the matching error (image) ei calculated by the RGB image position estimation unit 316 in step S49 and the weighted parameter wi, as the overall error e.

[0112] As another example, the RGB image position estimation unit 316 can also calculate the overall error e using the following formula (2). e=βd×exp(αd×ed)+βi×exp(αi×ei)···(2) Each variable in equation (2) represents the following: βd: Weighted parameter βi: Weighted parameter αd: parameter αi: parameter ed: Matching error (depth) ei: Matching error (image)

[0113] Specifically, the RGB image position estimation unit 316 first calculates the exponential of the product of the matching error (depth) ed calculated by the depth image position estimation unit 313 in step S39 and the parameter αd. Subsequently, the RGB image position estimation unit 316 calculates the product (value d) of the calculated value and the weighted parameter βd.

[0114] Next, the RGB image position estimation unit 316 calculates the exponential of the product of the matching error (image) ei calculated by the RGB image position estimation unit 316 in step S49 and the parameter αi. Subsequently, the RGB image position estimation unit 316 calculates the product (value i) of the calculated value and the weighted parameter βi.

[0115] The RGB image position estimation unit 316 then uses the sum of value d and value i as the overall error e.

[0116] Here, the RGB image position estimation unit 316 may apply a data deletion process to the RGB image or the second two-dimensional data to delete data that is more than a predetermined distance away from the position indicated by the N candidate solutions (in other words, data that indicates that it is more than a predetermined distance away). In this case, the RGB image position estimation unit 316 may refer to the captured image or the second two-dimensional data after the data deletion process and calculate at least one of the position and orientation of the object in three-dimensional space. With this configuration, the information processing device 3 calculates at least one of the position and orientation of the object in three-dimensional space without processing data other than the object, thus reducing computation cost and computation time.

[0117] Steps S49 to S51 described above are an example of the processing performed by the RGB image position estimation unit 316.

[0118] Furthermore, in the flowchart shown in Figure 6, the information processing device 3 may be configured to reverse the order in which steps S37 to S39 and steps S47 to S49 are executed, and the RGB image position estimation unit 316 may execute steps S32 to S36 and steps S39 to S41 instead of the depth image position estimation unit 313, while the depth image position estimation unit 313 may execute steps S42 to S46 and steps S49 to S51 instead of the RGB image position estimation unit 316.

[0119] In other words, in step S47, the RGB image acquisition unit 314 acquires an image captured by the RGB camera 5 that includes the object in its field of view, and in step S39, the RGB image position estimation unit 316 generates one or more candidate solutions regarding at least one of the position and orientation of the object in three-dimensional space by referring to the second two-dimensional data obtained by the second feature point extraction process that references the captured image and the three-dimensional model.

[0120] Next, in step S37, the depth information acquisition unit 311 acquires depth information obtained by the depth sensor 4 which includes the object within its sensing range, and in step S49, the depth image position estimation unit 313 refers to the first two-dimensional data obtained by the first feature point extraction process which references the depth information and the three-dimensional model, and uses one or more candidate solutions to calculate at least one of the position and orientation of the object in three-dimensional space.

[0121] (Example of calculating the degree of manifestation) Next, a specific example of the manifestation level calculation process according to this exemplary embodiment will be described with reference to Figure 8. Figure 8 is a diagram showing a specific example of the manifestation level calculation method. In this exemplary embodiment, as described above, the 3D model expansion unit 317 gradually expands the 3D model 331 by repeatedly executing the expansion process starting from the initial 3D model 331.

[0122] Figure 8 illustrates the case where the integrated manifestation of a single 3D model 331 is calculated using three patterns e1 to e3. The 3D model 331 shown in Figure 8 includes one confirmed edge edg1 and three effective edges E11 to E13, which are effective edges edg2 connected to the confirmed edge edg1. The 3D model extension unit 317 calculates the first manifestation, the second manifestation, and the integrated manifestation for each of the three effective edges E11 to E13.

[0123] In this example, the first manifestation level for the effective edge egd2 is the number of undefined edges connected to the effective edge egd2 that are manifest in the depth image. The second manifestation level for the effective edge egd2 is the number of undefined edges connected to the effective edge egd2 that are manifest in the RGB image. The combined manifestation level is the sum of the first and second manifestation levels.

[0124] In calculating the first manifestation level, the 3D model extension unit 317 determines whether the effective edge edg2 becomes a depth edge (contour portion) when the 3D model 331 is mapped to a two-dimensional space. Specifically, as an example, the 3D model extension unit 317 maps the 3D model 331 to a two-dimensional space and determines whether the effective edge edg2 becomes a contour portion in relation to other mapped parts. However, the determination method is not limited to the example described above, and other methods may be used. For example, the 3D model extension unit 317 may make the determination based on the relationship between each edge and the 3D model 331.

[0125] On the other hand, in calculating the second degree of manifestation, the 3D model extension unit 317 determines whether the effective edge edg2 is extracted as an edge when the 3D model 331 is mapped to a two-dimensional space as an RGB image. At this time, the 3D model extension unit 317 may determine that the effective edge edg2, which is sandwiched between the faces of two triangles, is not manifest. However, the determination method is not limited to the example described above, and other methods may be used.

[0126] Patterns e1 to e3 in Figure 8 are specific examples of methods for calculating the integrated visibility for each of the effective edges E11 to E13. In pattern e1, two undefined edges E111 and E112 are connected to the effective edge E11. Of these two undefined edges E111 and E112, in the depth image, undefined edge E112 is in a position where it can be extracted, while in the RGB image, both undefined edges E111 and E112 are in a position where they can be extracted. Therefore, the first visibility for effective edge E11 is 1, and the second visibility is 2. In addition, the integrated visibility for effective edge E11 is 3.

[0127] Furthermore, in pattern e2, one undefined edge E121 is connected to the effective edge E12, but the undefined edge E121 is not in a position where it can be extracted in the depth image. On the other hand, in the RGB image, the undefined edge E121 is in a position where it can be extracted. Therefore, the first manifestation level for the effective edge E12 is 0, and the second manifestation level is 1. The combined manifestation level for the effective edge E12 is 3.

[0128] Furthermore, in pattern e3, one undetermined edge E131 is connected to the effective edge E13, but the undetermined edge E131 is located in a position where it can be extracted in both the depth image and the RGB image. Therefore, the first manifestation level for the effective edge E13 is 0, and the second manifestation level is 1. The combined manifestation level for the effective edge E13 is 3.

[0129] As described above, in the example in Figure 8, the integration manifestation levels of the effective edges E11, E12, and E13 are "3", "1", and "2", respectively. In step S224 of Figure 5, the 3D model extension unit 317 selects the effective edge E12, which has the highest integration manifestation level, as the extension target. In step S225, the 3D model extension unit 317 determines the length of the selected effective edge E12 and promotes the effective edge E12 to a determined edge edg1. The 3D model extension unit 317 also promotes the undetermined edge edg3, which is connected to the determined edge edg1 whose length has been determined, to an effective edge edg2.

[0130] In the example in Figure 8, the manifestation level for the effective edge egd2 is defined as the number of undefined edges connected to the effective edge egd2 that are visible in the depth image or RGB image. However, the manifestation level is not limited to the example described above. For example, the manifestation level may be a value that expresses the degree to which undefined edges are visible as a percentage. Alternatively, the manifestation level may be a value that shows the result of evaluating the degree of visibility of undefined edges on a pixel-by-pixel basis.

[0131] Furthermore, the degree of visibility is not limited to a value indicating the degree of visibility of the effective edges included in the 3D model 331, but may be other values. For example, the degree of visibility may be a value indicating the degree of visibility of the entire 3D model 331.

[0132] (Specific examples of 3D model extension) Figure 9 shows a specific example of the expansion of the 3D model 331. In the example in Figure 9, the 3D model expansion unit 317 selects the effective edge E11 as the target for expansion in the first expansion process and executes the expansion process of the 3D model 331. At this time, the 3D model expansion unit 317 expands the 3D model 331 by promoting the effective edge E11 to a confirmed edge and promoting the undetermined edges E22 and E32 connected to the effective edge E11 to effective edges.

[0133] Furthermore, in the second expansion process, the 3D model expansion unit 317 selects the effective edge E22 as the target for expansion and executes the expansion process for the 3D model 331. In the second expansion process, as in the first expansion process, the 3D model expansion unit 317 expands the 3D model 331 by promoting the effective edge E22 to a confirmed edge and promoting the undetermined edges E14 and E33 connected to the effective edge E22 to effective edges.

[0134] Furthermore, in the third expansion process, the 3D model expansion unit 317 selects the effective edge E13 as the expansion target and executes the expansion process for the 3D model 331. In the third expansion process, as in the first and second expansion processes, the 3D model expansion unit 317 expands the 3D model 331 by promoting the effective edge E13 to a confirmed edge and promoting the undetermined edge E21 connected to the selected effective edge E13 to an effective edge.

[0135] (Supplementary information regarding the degree of manifestation) The following supplementary information regarding the degree of manifestation will be explained with reference to Figures 10 to 16. Figure 10 is a diagram showing a specific example of the 3D model 331 used by the depth image position estimation unit 313 and the RGB image position estimation unit 316 (hereinafter referred to as "depth image position estimation unit 313, etc.") to derive candidate solutions. In Figure 10, 331_A, 331_B, and 331_C are models used to derive candidate solutions, and the length of edge E22 differs for each. In the example of Figure 10, the depth image position estimation unit 313, etc. estimates the position and orientation by varying the length of edge E22 and selects one or more candidates with small errors. At this time, the depth image position estimation unit 313, etc. performs estimation processing using the lengths of edges E14 and E33 connected to edge E22 as predefined minimum lengths. By using the lengths of the edges connected to edge E22 as minimum lengths, rather than using only edge E22, the accuracy of position and orientation estimation can be improved.

[0136] Figure 11 shows an example of the 3D model extension unit 317 mapping the 3D model 331 as a depth image into a two-dimensional space. When the 3D model extension unit 317 compares the 3D model 331 with a depth image, it maps the 3D model 331 into a two-dimensional space and determines whether each edge is part of the contour. When the 3D models 331_A, 331_B, and 331_C in Figure 10 are mapped into a two-dimensional space as depth images, edges E32 and E14 are not extracted as edges, as shown in Figure 11.

[0137] In other words, in the examples of Figures 10 and 11, performing the estimation process using edge E22 is equivalent to performing the estimation process using not only edge E22 but also other edges connected to edge E22, as shown in Figure 12. The amount of other edges available in such estimation processing (number of edges, the proportion of usable edges in a single edge, etc.) can also be expressed as the degree of manifestation according to this exemplary embodiment. Since the number of usable edges differs depending on which part of the 3D model 331 is used for the estimation processing, the estimation accuracy can be improved by selecting the expansion target in such a way that the number of usable edges increases (i.e., the degree of manifestation increases).

[0138] Figure 13 shows an example of the 3D model extension unit 317 mapping the 3D model 331 as an RGB image into a two-dimensional space. When the 3D model extension unit 317 compares the 3D model 331 with an RGB image, it maps the 3D model 331 into a two-dimensional space and determines whether each edge is extracted. When the 3D models 331_A, 331_B, and 331_C in Figure 10 are mapped into a two-dimensional space as RGB images, the undetermined edges E32 and E14, which were not extracted as edges in the depth image, are extracted as edges.

[0139] The second degree of visibility, which is the degree of visibility of the image of the 3D model 331, may be calculated based on whether or not it is hidden by other parts, or it may be calculated by other methods. For example, in the example of Figure 14, the 3D model extension 317 may determine that the edge E32 sandwiched between the faces of the two triangles is not visible.

[0140] In the exemplary embodiment described above, the 3D model extension unit 317 selects the next effective edge whose length to be determined (to be extended) from among the effective edges connected to the determined edge whose length has already been determined, as described above. For example, in the example in Figure 15, if edge E22 is a determined edge, the 3D model extension unit 317 selects the extension target from among the effective edges connected to the determined edge E22, and therefore cannot select edge E31 as the extension target. This is because the length of edge E11 has not yet been determined, and the relative relationship between edge E22 and edge E31 changes depending on the length of edge E11. In order to determine the length of an edge, it is necessary to select an edge whose relative relationship with other determined edges is determined, that is, an edge connected to a determined edge. In the example in Figure 15, the 3D model extension unit 317 evaluates the degree of manifestation in four patterns (number of effective edges): effective edges E11, E32, E14, and E33.

[0141] (Variations in methods for identifying effective edges) In the exemplary embodiment described above, the 3D model extension unit 317 promoted edges that met predetermined conditions from among the undetermined edges to effective edges. However, the method for determining effective edges is not limited to the example shown in the exemplary embodiment described above. The 3D model extension unit 317 may identify effective edges by other methods. For example, the 3D model extension unit 317 may generate a reception screen that accepts user input regarding the identification of effective edges, and identify the effective edges by referring to the user input.

[0142] Figure 16 shows an example of a reception screen generated by the 3D model extension unit 317. In Figure 16, the 3D model extension unit 317 displays reception screens SC1, SC2, etc., for identifying valid edges on a touch panel or the like connected to the input / output unit 32. The user performs an operation to specify valid edges using an input device connected to the input / output unit 32, and the 3D model extension unit 317 identifies the valid edges by referring to the user input received by the input / output unit 32.

[0143] (Grouping edges) Furthermore, in the process of selecting effective edges to be extended, the 3D model extension unit 317 may group the edges included in the 3D model 331. Since extending each edge in the 3D model 331 individually may be inefficient, edges that are known to have the same length may be grouped together. However, grouping is intended to impose the constraint that the lengths be the same, and it does not necessarily mean that the lengths can be determined simultaneously.

[0144] As a method for grouping edges, the 3D model extension unit 317 may, for example, group and manage edges that are parallel or substantially parallel to each other among the edges included in the 3D model 331. Alternatively, the 3D model extension unit 317 may generate a reception screen that accepts user input regarding the selection of the effective edges, and group and manage multiple effective edges by referring to the user input.

[0145] For example, in the 3D model 331 of Figure 17, the 3D model extension unit 317 groups edges E11, E12, E14, and E15, and also groups edges E13 and E16. Furthermore, the 3D model extension unit 317 groups edges E21 and E22, and also groups edges E31 to E34.

[0146] In this case, even if edge E22, which was an effective edge, is determined to be a confirmed edge, the undetermined edge E21, which belongs to the same group, cannot be treated as a confirmed edge because its position changes depending on the lengths of other edges (e.g., edges E12, E13, E15, E16, etc.). Only the length of the undetermined edge is determined. In this case, the undetermined edge E21, whose length has been determined, will be promoted to a confirmed edge without any estimation process when its adjacent edges E13 and E16 become confirmed edges.

[0147] Furthermore, when grouping edges, the 3D model extension unit 317 may select multiple effective edges belonging to the same group as targets for extension in the process of selecting effective edges to be extended from among multiple effective edges. For example, in the example in Figure 18, if the 3D model extension unit 317 selects a group including edges E11, E12, E14, and E15, the 3D model extension unit 317 can consider two effective edges E31 and E34 simultaneously. Also, for example, if a group including edges E31, E32, E33, and E34 is selected, the 3D model extension unit 317 can consider two effective edges E12 and E15 simultaneously.

[0148] (Effects of Information Processing Device 3) As explained above, the information processing device 3 according to this exemplary embodiment starts with an initial 3D model 331 corresponding to at least a part of the object, and gradually expands the 3D model 331 while performing position and orientation estimation processing, thereby constructing a more refined 3D model 331 in stages. In this case, if the model expansion is stopped once the model has been expanded to a certain extent in order to reduce the processing load, the order in which the 3D model 331 is expanded becomes important. This is because, in the position and orientation estimation processing, the 3D model 331 is mapped to 2D data viewed from a certain viewpoint, so there is no point in expanding parts that are not visible (blind spots). The information processing device 3 according to this exemplary embodiment checks in advance whether the part to be expanded is visible (manifested) when mapped to 2D data, and determines the part of the 3D model 331 to be expanded. This makes it possible to reduce the processing load related to the position and orientation estimation processing of the object.

[0149] Furthermore, the information processing device 3 according to this exemplary embodiment derives candidate solutions for the position and orientation of an object using a depth image. Since differences in texture (such as Bessel patterns) can be ignored in depth images, the estimation results of the depth image position estimation unit 313 are used in the RGB image position estimation unit 316. In other words, by using the depth image as a kind of filter, misrecognition in the RGB image position estimation unit 316 can be reduced.

[0150] Furthermore, in the information processing device 3 according to this exemplary embodiment, the 3D model extension unit 317 is configured to extend the 3D model 331 with respect to each of the one or more candidate solutions in such a way that the visibility of the 3D model 331 is improved. Therefore, according to the information processing device 3 according to this exemplary embodiment, the position and orientation of the object can be estimated with greater accuracy by estimating the position and orientation of the object using the 3D model 331 that has been extended to improve its visibility.

[0151] Furthermore, in the information processing device 3 according to this exemplary embodiment, the 3D model extension unit 317 calculates a first manifestation degree, which is the manifestation degree of the 3D model 331 with respect to depth, and a second manifestation degree, which is the manifestation degree of the 3D model 331 with respect to the image, and extends the 3D model 331 so as to improve the integrated manifestation degree obtained by integrating the first manifestation degree and the second manifestation degree. By estimating the position and orientation of an object using the 3D model 331 which has been extended by taking into account both the manifestation degree with respect to depth and the manifestation degree with respect to the image, the information processing device 3 according to this exemplary embodiment can estimate the position and orientation of an object with greater accuracy.

[0152] Furthermore, in the information processing device 3 according to this exemplary embodiment, the 3D model expansion unit 317 calculates the first manifestation level and the second manifestation level for each of the multiple expansion candidates of the 3D model 331, and selects the expansion candidate with the largest integrated manifestation level obtained by integrating the first manifestation level and the second manifestation level. By expanding the 3D model 331 by selecting the expansion candidate with the largest integrated manifestation level, the information processing device 3 according to this exemplary embodiment can estimate the position and orientation of the object with greater accuracy.

[0153] Furthermore, in the information processing device 3 according to this exemplary embodiment, the 3D model extension unit 317 uses the sum of the first manifestation level and the second manifestation level as the integrated manifestation level. By extending the 3D model 331 so that the sum of the first manifestation level and the second manifestation level is improved, the information processing device 3 according to this exemplary embodiment can estimate the position and orientation of the object with greater accuracy.

[0154] Furthermore, in the information processing device 3 according to this exemplary embodiment, the 3D model 331 includes the plurality of vertices and one or more edges, and the one or more edges include at least one of the following: a determined edge whose length is determined and relates to at least one of depth and / or an image; an effective edge whose length is undetermined and relates to at least one of depth and / or an image; and an undetermined edge which is an extension candidate edge. By extending the 3D model 331 to include a plurality of edges in such a way that the visibility of depth and / or an image is improved, the information processing device 3 according to this exemplary embodiment can estimate the position and orientation of an object more accurately using the 3D model 331 including a plurality of edges.

[0155] Furthermore, in the information processing device 3 according to this exemplary embodiment, the 3D model expansion unit 317 expands the 3D model 331 by selecting the effective edge that has the greatest integrated manifestation level among the multiple effective edges included in the 3D model 331, when one or more undetermined edges connected to the effective edge are promoted to effective edges, promoting the selected effective edge to a confirmed edge, and promoting one or more undetermined edges connected to the selected effective edge to effective edges. By expanding the 3D model 331 by determining the length of the effective edge that has the greatest integrated manifestation level among the multiple effective edges, the information processing device 3 according to this exemplary embodiment can estimate the position and orientation of an object with greater accuracy using a three-dimensional model including multiple edges.

[0156] Furthermore, in the information processing device 3 according to this exemplary embodiment, the 3D model extension unit 317 generates a reception screen that accepts user input regarding the identification of the effective edges, and identifies the effective edges by referring to the user input. As a result, the information processing device 3 according to this exemplary embodiment can estimate the position and orientation of an object in a way that better reflects the user's intent.

[0157] Furthermore, in the information processing device 3 according to this exemplary embodiment, the 3D model extension unit 317 generates a reception screen that accepts user input regarding the selection of the multiple effective edges, and groups and manages the multiple effective edges by referring to the user input. As a result, the information processing device 3 according to this exemplary embodiment can estimate the position and orientation of an object in a way that better reflects the user's intentions.

[0158] Furthermore, in the information processing device 3 according to this exemplary embodiment, the second estimation process by the depth image position estimation unit 313 and the second estimation process by the RGB image position estimation unit 316 include a process for deriving one or more candidate solutions regarding the position and orientation of the object using each expanded 3D model 331. The device includes a control unit 31 that repeats the expansion process by the 3D model expansion unit 317 and the second estimation process by the depth image position estimation unit 313 and / or the second estimation process by the RGB image position estimation unit 316 multiple times. By repeating the expansion process of the 3D model 331 and the second estimation process multiple times, the information processing device 3 according to this exemplary embodiment can estimate the position and orientation of the object using the 3D model 331 with greater accuracy.

[0159] [Examples of implementation using software] Some or all of the functions of the information processing devices 1 and 3 and the information processing system 100 may be implemented by hardware such as integrated circuits (IC chips) or by software.

[0160] In the latter case, the information processing devices 1 and 3, and the information processing system 100 are implemented, for example, by a computer that executes instructions for a program, which is software that implements each function. An example of such a computer (hereinafter referred to as computer C) is shown in Figure 19. Computer C comprises at least one processor C1 and at least one memory C2. The memory C2 stores a program P that causes computer C to operate as information processing devices 1 and 3, and the information processing system 100. In computer C, the processor C1 reads the program P from the memory C2 and executes it, thereby implementing each function of the information processing devices 1 and 3, and the information processing system 100.

[0161] Processor C1 can include, for example, a CPU (Central Processing Unit), GPU (Graphic Processing Unit), DSP (Digital Signal Processor), MPU (Micro Processing Unit), FPU (Floating Point Number Processing Unit), PPU (Physics Processing Unit), microcontroller, or a combination thereof. Memory C2 can include, for example, flash memory, HDD (Hard Disk Drive), SSD (Solid State Drive), or a combination thereof.

[0162] Computer C may also be equipped with RAM (Random Access Memory) for loading program P at runtime and for temporarily storing various data. Furthermore, computer C may be equipped with communication interfaces for sending and receiving data with other devices. Additionally, computer C may be equipped with input / output interfaces for connecting input / output devices such as keyboards, mice, displays, and printers.

[0163] Furthermore, program P can be recorded on a non-temporary, tangible recording medium M that is readable by computer C. Such a recording medium M could be, for example, tape, disk, card, semiconductor memory, or programmable logic circuitry. Computer C can acquire program P via such a recording medium M. Program P can also be transmitted via a transmission medium. Such a transmission medium could be, for example, a communication network or broadcast waves. Computer C can also acquire program P via such a transmission medium.

[0164] [Additional Note 1] The present invention is not limited to the embodiments described above, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means disclosed in the embodiments described above are also included in the technical scope of the present invention.

[0165] [Additional Note 2] Some or all of the embodiments described above may also be described as follows. However, the present invention is not limited to the embodiments described below.

[0166] (Note 1) An information processing device comprising: an acquisition means for acquiring sensing data obtained by a sensor that includes an object within its sensing range; an estimation means for deriving one or more candidate solutions regarding the position and orientation of the object by a first estimation process using the sensing data acquired by the acquisition means and a three-dimensional model; and an extension means for extending the three-dimensional model with respect to each of the one or more candidate solutions, wherein the estimation means estimates the position and orientation of the object by a second estimation process using each of the three-dimensional models extended by the extension means.

[0167] (Note 2) The information processing apparatus according to Appendix 1, wherein the extension means extends the three-dimensional model with respect to each of the one or more candidate solutions such that the degree of manifestation of the three-dimensional model is improved.

[0168] (Note 3) The information processing apparatus according to Appendix 2, wherein the extension means calculates a first manifestation, which is the manifestation with respect to the depth of the three-dimensional model, and a second manifestation, which is the manifestation with respect to the image of the three-dimensional model, and extends the three-dimensional model so that the integrated manifestation obtained by integrating the first manifestation and the second manifestation is improved.

[0169] (Note 4) The information processing device according to Appendix 3, wherein the extension means calculates a first manifestation level and a second manifestation level for each of the plurality of extension candidates of the three-dimensional model, and selects the extension candidate with the largest integrated manifestation level obtained by integrating the first manifestation level and the second manifestation level.

[0170] (Note 5) The information processing apparatus according to Appendix 3 or 4, wherein the extension means uses the sum of the first manifestation and the second manifestation as the integrated manifestation.

[0171] (Note 6) The information processing device according to any one of the appendices 3 to 5, wherein the three-dimensional model includes a plurality of vertices and one or more edges, and the one or more edges include at least one of the following: a determined edge relating to depth and / or an image whose length is determined; an effective edge relating to depth and / or an image whose length is not determined; and an undetermined edge which is an extension candidate edge.

[0172] (Note 7) The information processing device according to Appendix 6, wherein the extension means extends the 3D model by selecting the effective edge that has the greatest integrated manifestation when one or more undetermined edges connected to an effective edge are promoted to effective edges from among a plurality of effective edges included in the 3D model, promoting the selected effective edge to a confirmed edge, and promoting one or more undetermined edges connected to the selected effective edge to effective edges.

[0173] (Note 8) The information processing device according to Appendix 6, wherein the extension means generates a reception screen for receiving user input regarding the identification of the effective edge, and identifies the effective edge by referring to the user input.

[0174] (Note 9) The information processing device according to Appendix 6, wherein the extension means generates a reception screen for receiving user input regarding the selection of the effective edge, and groups and manages a plurality of the effective edges by referring to the user input.

[0175] (Note 10) The information processing apparatus according to any one of the appendices 1 to 9, wherein the second estimation process by the estimation means includes a process of deriving one or more candidate solutions regarding the position and orientation of the object using each of the expanded three-dimensional models, and the apparatus is further provided with a control means for repeating the expansion process by the expansion means and the second estimation process by the estimation means multiple times.

[0176] (Note 11) An information processing method comprising: acquiring sensing data obtained by a sensor that includes an object within its sensing range; deriving one or more candidate solutions regarding the position and orientation of the object by a first estimation process using the sensing data and a three-dimensional model; extending the three-dimensional model with respect to each of the one or more candidate solutions; and estimating the position and orientation of the object by a second estimation process using each of the extended three-dimensional models.

[0177] (Note 12) A program that causes a computer to function as an acquisition means for acquiring sensing data obtained by a sensor that includes an object within its sensing range; an estimation means for deriving one or more candidate solutions regarding the position and orientation of the object by performing a first estimation process using the sensing data acquired by the acquisition means and a three-dimensional model; and an extension means for extending the three-dimensional model with respect to each of the one or more candidate solutions, wherein the estimation means is a program that estimates the position and orientation of the object by performing a second estimation process using each of the three-dimensional models extended by the extension means.

[0178] [Additional Note 3] Some or all of the embodiments described above can also be expressed as follows:

[0179] An information processing device comprising at least one processor, the processor performing: an acquisition process for acquiring sensing data obtained by a sensor that includes an object within its sensing range; a derivation process for deriving one or more candidate solutions regarding the position and orientation of the object by a first estimation process using the sensing data acquired in the acquisition process and a three-dimensional model; an extension process for extending the three-dimensional model with respect to each of the one or more candidate solutions; and a position estimation process for estimating the position and orientation of the object by a second estimation process using each of the three-dimensional models extended by the extension process.

[0180] Furthermore, this information processing device may also be equipped with memory, and this memory may store a program that causes the processor to execute the acquisition process, the derivation process, the expansion process, and the position estimation process. This program may also be recorded on a computer-readable, non-temporary, tangible recording medium. [Explanation of symbols]

[0181] 1.3 Information Processing Devices 11 Acquisition Department 12 Estimation part 13. Expansion section 311 Depth information acquisition unit 312 Depth Image Feature Point Extraction Unit 313 Depth image position estimation unit 314 RGB Image Acquisition Unit 315 RGB Image Feature Point Extraction Unit 316 RGB Image Position Estimation Unit 317 3D Model Extension Section

Claims

1. An acquisition means for acquiring sensing data obtained by a sensor that includes an object within its sensing range, An estimation means that derives one or more candidate solutions regarding the position and orientation of the object by performing a first estimation process using the sensing data acquired by the acquisition means and a three-dimensional model, With respect to each of the one or more candidate solutions, an extension means for extending the three-dimensional model, Equipped with, The estimation means estimates the position and orientation of the object by a second estimation process using each of the three-dimensional models expanded by the expansion means, The aforementioned expansion means is A first manifestation level, which is the manifestation level with respect to the depth of the three-dimensional model, and a second manifestation level, which is the manifestation level with respect to the image of the three-dimensional model, are calculated, respectively. The three-dimensional model is extended so that the integrated manifestation obtained by integrating the first manifestation and the second manifestation is improved. Information processing device.

2. The aforementioned expansion means is For each of the multiple extension candidates of the three-dimensional model, the first manifestation level and the second manifestation level are calculated, respectively. Select the expansion candidate with the highest combined manifestation level obtained by integrating the first manifestation level and the second manifestation level. The information processing apparatus according to claim 1.

3. The expansion means uses the sum of the first manifestation level and the second manifestation level as the integrated manifestation level. The information processing apparatus according to claim 1 or 2.

4. The aforementioned three-dimensional model, It includes multiple vertices and one or more edges, The one or more edges mentioned above include: An edge relating to depth and / or an image, wherein the length of the edge is determined. An effective edge relating to depth and / or an image, the length of which is undetermined, and Undetermined edges are candidates for extension. It includes at least one of the following: The information processing apparatus according to claim 1 or 2.

5. The aforementioned expansion means is If, among the multiple effective edges included in the three-dimensional model, one or more undetermined edges connected to said effective edge are promoted to effective edges, then the effective edge that has the greatest integrated manifestation level with respect to said promoted effective edge is selected. The 3D model is expanded by promoting the selected effective edges to confirmed edges and promoting one or more undetermined edges connected to the selected effective edges to effective edges. The information processing apparatus according to claim 4.

6. The aforementioned expansion means is A reception screen is generated to receive user input regarding the identification of the effective edge. The effective edge is identified by referring to the user input. The information processing apparatus according to claim 4.

7. The aforementioned expansion means is A reception screen is generated to receive user input regarding the selection of the effective edge. Referencing the user input, the system groups and manages multiple valid edges. The information processing apparatus according to claim 4.

8. The second estimation process by the estimation means includes a process of deriving one or more candidate solutions regarding the position and orientation of the object using each of the expanded three-dimensional models. The system includes a control means that repeats the expansion process by the expansion means and the second estimation process by the estimation means multiple times. The information processing apparatus according to claim 1 or 2.

9. A computer, This involves acquiring sensing data obtained by a sensor that includes the target object within its sensing range, The first estimation process using the sensing data and the three-dimensional model derives one or more candidate solutions regarding the position and orientation of the object, With respect to each of the one or more candidate solutions, the three-dimensional model is extended, The position and orientation of the object are estimated by a second estimation process using each of the expanded 3D models. Includes, The aforementioned computer, A first manifestation level, which is the manifestation level with respect to the depth of the three-dimensional model, and a second manifestation level, which is the manifestation level with respect to the image of the three-dimensional model, are calculated, respectively. The three-dimensional model is extended so that the integrated manifestation obtained by integrating the first manifestation and the second manifestation is improved. Information processing methods.

10. Computers, An acquisition means for acquiring sensing data obtained by a sensor that includes an object within its sensing range, An estimation means that derives one or more candidate solutions regarding the position and orientation of the object by performing a first estimation process using the sensing data acquired by the acquisition means and a three-dimensional model, With respect to each of the one or more candidate solutions, an extension means for extending the three-dimensional model and It is a program that makes it function as such. The estimation means estimates the position and orientation of the object by a second estimation process using each of the three-dimensional models expanded by the expansion means, The aforementioned expansion means is A first manifestation level, which is the manifestation level with respect to the depth of the three-dimensional model, and a second manifestation level, which is the manifestation level with respect to the image of the three-dimensional model, are calculated, respectively. The three-dimensional model is extended so that the integrated manifestation obtained by integrating the first manifestation and the second manifestation is improved. program.

Images