Object matching device and object matching method

Abstract

To provide an apparatus and a method for collating objects with a light load.SOLUTION: In an object collating apparatus, a collation processing unit is equipped with: a data acquisition unit 1302 that acquires point cloud data of a product; a BB calculation unit 1303 that calculates a bounding box (BB) of the product based on the acquired point cloud data; a comparison unit 1304 that compares the BB of a product model with the BB of the product; a data extraction unit 1305 that, when it is determined that the difference in size is within a threshold value, extracts from the acquired point cloud data or polygons (hereinafter referred to as "point cloud data, etc.") the point cloud data, etc. corresponding to the BB of a component model at a position corresponding to a position relative to the component model; a SS calculation unit 1306 that calculates a cross-sectional silhouette shape (SS) of a component based on the extracted point cloud data, etc.; a comparison unit 1307 that compares the SS of the component model with the SS of the component; and a collation unit 1308 that determines whether the component is an object of the same type as the object corresponding to the component model based on the comparison result.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to an object matching device and an object matching method for matching objects. [Background technology]

[0002] Conventionally, a technology called mixed reality (MR) that combines the real world and virtual reality has been known. In MR, when visually adding information to an object, the object itself is not recognized, but a marker indicating the reference coordinates of the object is placed, and the marker is recognized using, for example, a QR code (registered trademark). Then, a means is used in which the information is displayed at a position of predetermined relative coordinates with respect to the recognized marker (see, for example, Patent Document 1).

[0003] As described above, conventionally, a marker indicating the reference position of an object is recognized without recognizing the object itself. Therefore, if the actual position of the object changes, the actual position of the object will deviate from the reference position indicated by the marker. Also, if the placement position of the marker deviates from the predetermined correct position, the position of the object will deviate from the reference position indicated by the marker. Therefore, when visually adding information to an object, the position of the information may deviate from the actual position of the object, which is not appropriate.

[0004] In contrast to this, techniques for recognizing objects themselves have been known for some time (see, for example, Patent Document 1). However, in conventional object recognition methods, objects are recognized by performing a three-dimensional matching process, which places a heavy load on the processing. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2005-293350 Summary of the Invention [Problem to be solved by the invention]

[0006] As described above, conventionally, the object itself is not recognized, but a marker indicating the reference position of the object is recognized. Therefore, if the actual position of the object changes, the actual position of the object will deviate from the reference position indicated by the marker. Therefore, when visually adding information to an object, the position of the information may deviate from the actual position of the object, which is not appropriate.

[0007] Therefore, in order to solve the above problem, it is necessary to recognize the object itself. However, in the past, recognizing the object itself imposes a heavy processing load.

[0008] The present disclosure has been made to solve the above-mentioned problems, and aims to provide an object matching device that can match objects with a lighter load than conventional devices. [Means for solving the problem]

[0009] The object matching device according to the present disclosure includes a data acquisition unit that acquires point cloud data of coordinates indicating a three-dimensional shape of a first object; a first bounding box calculation unit that calculates a bounding box of the first object based on the point cloud data of coordinates indicating the three-dimensional shape of the first object acquired by the data acquisition unit, with reference to a first reference axis that is a specific direction in the first object; a first comparison unit that compares the bounding box of the first object model with the bounding box of the first object calculated by the first bounding box calculation unit; and when the first comparison unit determines that the difference in size between the bounding box of the first object model and the bounding box of the first object is within a threshold, calculates a bounding box of a second object model included in the first object model and a relative position of the second object model in the first object model; and a data extraction unit that extracts, based on point cloud data of coordinates indicating the three-dimensional shape of the first object acquired by the data acquisition unit or a polygon generated from the point cloud data, point cloud data or polygons within a predetermined range including a range corresponding to a bounding box of the second object model at a position corresponding to the relative position from the point cloud data or polygons; a first cross-sectional silhouette shape calculation unit that calculates a cross-sectional silhouette shape of the second object on a first reference plane based on a first reference axis based on the point cloud data or polygons extracted by the data extraction unit; a second comparison unit that compares the cross-sectional silhouette shape of the second object model with the cross-sectional silhouette shape of the second object calculated by the first cross-sectional silhouette shape calculation unit; and a determination unit that determines whether the second object is of the same type as the object corresponding to the second object model based on a comparison result by the second comparison unit. Matching is performed by The present invention is characterized by comprising a matching unit. [Effects of the Invention]

[0010] According to the present disclosure, the above-described configuration makes it possible to match objects with a lighter load than conventional methods. [Brief explanation of the drawings]

[0011] [Figure 1]1 is a diagram illustrating an example of the configuration of an object matching device according to a first embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of the configuration of a database creation unit according to the first embodiment. [Figure 3] FIG. 2 is a diagram illustrating an example of the configuration of a matching processing unit according to the first embodiment. [Figure 4] 5 is a flowchart showing an example of a database creation operation performed by the object matching device according to the first embodiment. [Figure 5] FIG. 3 is a diagram showing an example of a bounding box of a product model calculated by the object matching device according to the first embodiment. [Figure 6] 6A to 6C are diagrams showing examples of bounding boxes of part models calculated by the object matching device according to the first embodiment. [Figure 7] 7A to 7C are diagrams showing examples of the cross-sectional silhouette shape of the part model shown in FIG. 6A, where FIG. 7A shows the cross-sectional silhouette shape in the XY plane, FIG. 7B shows the cross-sectional silhouette shape in the YZ plane, and FIG. 7C shows the cross-sectional silhouette shape in the XZ plane. [Figure 8] 5 is a flowchart showing an example of an object matching operation performed by the object matching device according to the first embodiment. [Figure 9] 4 is a diagram showing an example of the operation of calculating a bounding box of a product by the object matching device according to the first embodiment; FIG. [Figure 10] 4 is a diagram showing an example of an operation for extracting point cloud data by the object matching device according to the first embodiment. FIG. [Figure 11] 5 is a diagram showing an example of the operation of calculating a cross-sectional silhouette shape of a part by the object matching device according to the first embodiment. FIG. [Figure 12] 4A to 4C are diagrams illustrating an example of an operation performed by the object matching device according to the first embodiment to compare the cross-sectional silhouette shape of a part model with the cross-sectional silhouette shape of the part. [Figure 13] FIG. 10 is a diagram illustrating an example of the configuration of a database creation unit according to the second embodiment. [Figure 14] FIG. 10 is a diagram illustrating an example of the configuration of a matching processing unit according to the second embodiment. [Figure 15]FIG. 11 is a diagram illustrating an example of the configuration of a matching processing unit according to the third embodiment. [Figure 16] FIG. 11 is a diagram showing an example of a cross-sectional silhouette shape of a part with a missing part calculated by the object matching device according to the third embodiment. [Figure 17] 10 is a diagram showing an example of an operation of comparing the cross-sectional silhouette shape of a part model with the cross-sectional silhouette shape of a part, performed by the object matching device according to the third embodiment. FIG. [Figure 18] FIG. 10 is a diagram illustrating an example of the configuration of an object matching device according to a fourth embodiment. [Figure 19] FIG. 13 is a diagram illustrating an example of the configuration of a display processing unit according to the fourth embodiment. [Figure 20] 20A and 20B are diagrams illustrating examples of hardware configurations of object matching devices according to embodiments 1-4. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments will be described in detail with reference to the drawings. Embodiment 1 FIG. 1 is a diagram showing an example of the configuration of an object matching device 1 according to the first embodiment. The object matching device 1 is a device that matches objects. This object matching device 1 is applicable to the field of AR (Augmented Reality) in addition to MR, for example.

[0013] In the first embodiment, an example will be described in which the object to be matched by the object matching device 1 is a part (second object) included in a product (first object). One or more parts to be matched exist for a product.

[0014] As shown in FIG. 1, the object matching device 1 includes a data storage unit 11, a database creation unit 12, and a matching processing unit 13.

[0015] The data storage unit 11 stores various types of data handled by the object matching device 1.

[0016] The data storage unit 11 may be, for example, a non-volatile or volatile semiconductor memory such as a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable ROM), an EEPROM (Electrically EPROM), a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, or a DVD (Digital Versatile Disc).

[0017] 1 shows a case where the data storage unit 11 is provided inside the object matching device 1. However, this is not limiting, and the data storage unit 11 may be provided outside the object matching device 1, and the object matching device 1 may be configured to exchange data with the externally provided data storage unit 11.

[0018] The database creation unit 12 creates data related to object models used in matching by the matching processing unit 13. As shown in Fig. 2, the database creation unit 12 includes a bounding box calculation unit (second bounding box calculation unit) 1201 and a cross-sectional silhouette shape calculation unit (second cross-sectional silhouette shape calculation unit) 1202.

[0019] The bounding box calculation unit 1201 calculates a bounding box of the product model based on data indicating the three-dimensional shape of the product model, with the model reference axis (second reference axis) as the reference. The product model is a model corresponding to the above product. The model reference axis is a specific direction in the product model. For example, the bounding box calculation unit 1201 can use the short-side direction or long-side direction of the product model as the model reference axis.

[0020] Furthermore, the bounding box calculation unit 1201 calculates a bounding box of the part model based on the model reference axis, based on data indicating the three-dimensional shape of the product model. Note that the part model is a model corresponding to the above-mentioned part. Furthermore, the bounding box calculation unit 1201 calculates the relative position of the part model in the product model.

[0021] The cross-sectional silhouette shape calculation unit 1202 calculates the cross-sectional silhouette shape of the part model on the model reference plane (second reference plane) based on data indicating the three-dimensional shape of the product model. The model reference plane is a plane based on the model reference axis. For example, if the longitudinal direction of the product model is the X-axis, the cross-sectional silhouette shape calculation unit 1202 can use, for example, the XY plane, the YZ plane, or the XZ plane as the reference plane. Furthermore, it is preferable that the cross-sectional silhouette shape calculation unit 1202 calculates the cross-sectional silhouette shapes of the part model on each of a plurality of model reference planes.

[0022] The matching processing unit 13 matches objects using the data created by the database creation unit 12. As shown in Fig. 3 , the matching processing unit 13 includes a depth sensor 1301, a data acquisition unit 1302, a bounding box calculation unit (first bounding box calculation unit) 1303, a comparison unit (first comparison unit) 1304, a data extraction unit 1305, a cross-sectional silhouette shape calculation unit (first cross-sectional silhouette shape calculation unit) 1306, a comparison unit (second comparison unit) 1307, and a matching unit 1308.

[0023] The depth sensor 1301 measures point cloud data of coordinates that indicate the three-dimensional shape of a product that exists in a predetermined measurement area. The point cloud data of coordinates indicating the three-dimensional shape of the product measured by this depth sensor 1301 is transmitted to a data acquisition unit 1302 .

[0024] 1 shows a case where the depth sensor 1301 is provided inside the object matching device 1. However, this is not limiting, and the depth sensor 1301 may be provided outside the object matching device 1, and the object matching device 1 may be configured to exchange data with the depth sensor 1301 provided outside.

[0025] The data acquisition unit 1302 acquires point cloud data of coordinates that indicate the three-dimensional shape of the product measured by the depth sensor 1301. The depth sensor 1301 performs measurements on the measurement area. Therefore, the measurement results by the depth sensor 1301 usually include point cloud data of coordinates indicating the three-dimensional shape of the product present in the measurement area, as well as point cloud data of coordinates indicating the three-dimensional shape of objects other than the product present in the measurement area. The data acquisition unit 1302 then acquires all of the point cloud data. Examples of objects other than the product include the floor, walls, or a stand on which the product is placed that are present around the product.

[0026] The bounding box calculation unit 1303 calculates a bounding box of the product based on the point cloud data of coordinates indicating the three-dimensional shape of the product acquired by the data acquisition unit 1302, with the object reference axis (first reference axis) as the reference. The object reference axis is a specific direction in the product. The object reference axis and the model reference axis are in the same direction. For example, the bounding box calculation unit 1303 can use the short-side direction or long-side direction of the product as the object reference axis.

[0027] As described above, the point cloud data acquired by the data acquisition unit 1302 usually includes not only point cloud data of coordinates indicating the three-dimensional shape of the product, but also point cloud data of coordinates indicating the three-dimensional shapes of other objects existing around it. Therefore, first, the bounding box calculation unit 1303 removes the point cloud data of coordinates indicating the three-dimensional shape of the other object from the point cloud data acquired by the data acquisition unit 1302. Note that an existing method can be applied as a method for removing the point cloud data. Then, the bounding box calculation unit 1303 generates polygons based on the point cloud data after the removal. Note that existing methods can be applied to generate polygons. Then, the bounding box calculation unit 1303 calculates the bounding box of the product based on this polygon and with the object reference axis as the reference.

[0028] In the above description, the bounding box calculation unit 1303 generates polygons. However, generating polygons is not an essential process in the processing of the bounding box calculation unit 1303. In other words, the bounding box calculation unit 1303 may calculate the bounding box of the product directly from the point cloud data after the removal, using the object reference axis as a reference.

[0029] The comparison unit 1304 compares the bounding box of the product model calculated by the bounding box calculation unit 1201 with the bounding box of the product calculated by the bounding box calculation unit 1303 . At this time, the comparison unit 1304 determines whether the difference in size between the bounding box of the product model and the bounding box of the product is within a threshold value, which is set in advance as appropriate to a value that allows the product model and the product to be matched. The size is the area of ​​any plane (for example, the xy plane) in the bounding box, or the length in any direction in the bounding box.

[0030] When the comparison unit 1304 determines that the difference in size between the bounding box of the product model and the bounding box of the product is within a threshold, the data extraction unit 1305 extracts point cloud data within a predetermined range including the range corresponding to the bounding box of the part model at the position corresponding to the relative position from the point cloud data, based on the bounding box of the part model calculated by the bounding box calculation unit 1201 and the relative position of the part model in the product model, and the point cloud data of coordinates indicating the three-dimensional shape of the product acquired by the data acquisition unit 1302. That is, the data extraction unit 1305 extracts point cloud data within a predetermined range including a range where the part to be matched is thought to exist from the point cloud data acquired by the data acquisition unit 1302.

[0031] Alternatively, if the bounding box calculation unit 1303 generates polygons from point cloud data, the data extraction unit 1305 may extract polygons within a predetermined range that includes a range corresponding to the bounding box of the part model at a position that corresponds to the relative position, from the polygons instead of the point cloud data. In other words, the data extraction unit 1305 may extract polygons within a predetermined range that includes a range where the part to be matched is thought to exist, from the polygons generated by the bounding box calculation unit 1303.

[0032] The cross-sectional silhouette shape calculation unit 1306 calculates the cross-sectional silhouette shape of the part on the object reference plane (first reference plane) based on the point cloud data or polygons extracted by the data extraction unit 1305. The object reference plane is a plane based on the object reference axis. The object reference plane and the model reference plane are the same plane. For example, if the longitudinal direction of the product is set as the X-axis, which is the object reference axis, the cross-sectional silhouette shape calculation unit 1306 can use, for example, the XY plane, YZ plane, or XZ plane as the reference plane. Furthermore, it is preferable that the cross-sectional silhouette shape calculation unit 1306 calculates the cross-sectional silhouette shapes of the part on each of a plurality of object reference planes.

[0033] Here, when the data extraction unit 1305 extracts point cloud data, the cross-sectional silhouette shape calculation unit 1306 may calculate the cross-sectional silhouette shape of the part on the object reference plane directly from the point cloud data. Furthermore, when point cloud data is extracted by the data extraction unit 1305, the cross-sectional silhouette shape calculation unit 1306 may generate polygons from the point cloud data, and calculate the cross-sectional silhouette shape of the part on the object reference plane from the polygons. Furthermore, when polygons are extracted by the data extraction unit 1305, the cross-sectional silhouette shape calculation unit 1306 may calculate the cross-sectional silhouette shape of the part on the object reference plane from the polygons.

[0034] In addition, since the number of points in the polygons is thinned out compared to the point cloud data, the cross-sectional silhouette shape calculation unit 1306 calculates the two-dimensional cross-sectional silhouette shape from the polygons rather than the point cloud data, which is thought to lead to improved processing efficiency and a reduction in the processing load.

[0035] Furthermore, when generating polygons in the bounding box calculation unit 1303, the polygons are generated from point cloud data of coordinates indicating the three-dimensional shape of the product. On the other hand, when generating polygons in the cross-section silhouette shape calculation unit 1306, the polygons are generated from point cloud data within a predetermined range that includes the range in which the part to be matched is thought to exist. Therefore, generating polygons in the cross-section silhouette shape calculation unit 1306 requires less point cloud data than generating polygons in the bounding box calculation unit 1303, which is thought to lead to more efficient processing and a reduced processing load.

[0036] The comparison unit 1307 compares the cross-sectional silhouette shape of the part model calculated by the cross-sectional silhouette shape calculation unit 1202 with the cross-sectional silhouette shape of the part calculated by the cross-sectional silhouette shape calculation unit 1306. At this time, the comparison unit 1307 determines the degree of match between the cross-sectional silhouette shape of the part model and the cross-sectional silhouette shape of the part using an existing method such as pattern matching. If there are multiple pairs of object reference planes and model reference planes, the comparison unit 1307 compares the cross-sectional silhouette shapes for each corresponding object reference plane and model reference plane.

[0037] Based on the comparison result by the comparison unit 1307, the collation unit 1308 determines whether the part is of the same type as the part corresponding to the part model. At this time, for example, when the comparison unit 1307 determines that the matching rate between the cross-sectional silhouette shape of the part model and the cross-sectional silhouette shape of the part is equal to or greater than a threshold, the matching unit 1308 determines that the part is of the same type as the part corresponding to the part model. Note that the threshold is set in advance as appropriate to a value that allows matching between the part model and the part. Furthermore, when the comparison unit 1307 performs a comparison of cross-sectional silhouette shapes multiple times, the matching unit 1308 comprehensively evaluates the results of the multiple comparisons and determines whether the part is of the same type as the part corresponding to the part model.

[0038] 1 shows a case where a database creation unit 12 is provided in the object matching device 1. However, this database creation unit 12 is not an essential component of the object matching device 1, and does not have to be provided in the object matching device 1.

[0039] Next, an example of the operation of the object matching device 1 according to the first embodiment shown in Fig. 1 will be described. Here, the description will be given taking as an example a case where the object to be matched by the object matching device 1 according to the first embodiment is a part included in a product. First, an example of the operation of creating a database by the object matching device 1 according to the first embodiment shown in FIG. 1 will be described with reference to FIG. Moreover, the drawings shown in the following FIGS. 5 to 7 and 9 to 12 are images that are made by imitating the product in order to explain the function, and are different from the actual product images.

[0040] In an example of a database creation operation by the object matching device 1 according to embodiment 1 shown in FIG. 1, first, as shown in FIG. 4, the bounding box calculation unit 1201 calculates a bounding box of the product model based on data indicating the three-dimensional shape of the product model, with the model reference axis as the reference (step ST401).

[0041] Fig. 5 shows an example of a bounding box of a product model calculated by the bounding box calculation unit 1201. In Fig. 5, reference numeral 51 indicates the bounding box of the product model. Fig. 5 shows a case where the horizontal longitudinal direction of the product model is used as the model reference axis.

[0042] Furthermore, the bounding box calculation unit 1201 calculates a bounding box of the part model based on the data indicating the three-dimensional shape of the product model, with the model reference axis as the reference (step ST402).

[0043] Fig. 6 shows an example of a bounding box of a part model calculated by the bounding box calculation unit 1201. Fig. 6 shows a case where three part models to be matched exist within a product model. In Fig. 6, reference numerals 61 to 63 indicate the bounding boxes of the respective part models. Fig. 6 also shows a case where the horizontal longitudinal direction of the product model is used as the model reference axis.

[0044] Furthermore, the bounding box calculation unit 1201 calculates the relative position of the part model in the product model (step ST403). At this time, a position such as the center of gravity is appropriately set as the reference position of the product model, and the bounding box calculation unit 1201 calculates the relative position of the part model with respect to the reference position.

[0045] Next, the cross-sectional silhouette shape calculation unit 1202 calculates the cross-sectional silhouette shape of the part model on the model reference plane based on the data indicating the three-dimensional shape of the product model (step ST404).

[0046] Fig. 7 shows an example of the cross-sectional silhouette shape of a part model calculated by the cross-sectional silhouette shape calculation unit 1202. Fig. 7 shows the cross-sectional silhouette shape of the part model shown in Fig. 6A. Fig. 7 shows a case where the horizontal longitudinal direction of the product model is set as the X-axis, and three planes, the XY plane, the YZ plane, and the XZ plane, are used as reference planes.

[0047] Next, an example of an object matching operation performed by the object matching device 1 according to the first embodiment shown in Fig. 1 will be described with reference to Fig. 8. Note that the following description will be given taking as an example a case where the data extraction unit 1305 extracts point cloud data.

[0048] In an example of an object matching operation by the object matching device 1 according to embodiment 1 shown in FIG. 1, as shown in FIG. 8, first, the depth sensor 1301 measures point cloud data of coordinates indicating the three-dimensional shape of a product present in a predetermined measurement area (step ST801).

[0049] Next, the data acquisition unit 1302 acquires point cloud data of coordinates indicating the three-dimensional shape of the product measured by the depth sensor 1301 (step ST802). The depth sensor 1301 measures the measurement area. Therefore, the measurement results by the depth sensor 1301 usually include point cloud data of coordinates indicating the three-dimensional shapes of products present in the measurement area, as well as point cloud data of coordinates indicating the three-dimensional shapes of objects other than the products present in the measurement area. The data acquisition unit 1302 then acquires the above point cloud data all at once.

[0050] Next, the bounding box calculation unit 1303 calculates a bounding box of the product based on the point cloud data of coordinates indicating the three-dimensional shape of the product acquired by the data acquisition unit 1302, with the object reference axis as the reference (step ST803).

[0051] Fig. 9 shows an example of a bounding box of a product calculated by the bounding box calculation unit 1303. In Fig. 9, the diagram on the left shows the point cloud data acquired by the data acquisition unit 1302, and the diagram on the right shows the bounding box calculated after point cloud data of coordinates indicating the three-dimensional shapes of objects other than the product has been excluded from the point cloud data. In Fig. 9, for ease of explanation, the point cloud data is shown as a model. Also, in Fig. 9, reference numeral 91 denotes a bounding box of the product. Fig. 9 shows a case where the horizontal longitudinal direction of the product is used as the object reference axis.

[0052] As described above, the point cloud data acquired by the data acquisition unit 1302 typically includes not only point cloud data of coordinates indicating the three-dimensional shape of the product, but also point cloud data of coordinates indicating the three-dimensional shapes of other objects existing around it. Therefore, first, the bounding box calculation unit 1303 removes point cloud data of coordinates indicating the three-dimensional shapes of the other objects from the point cloud data acquired by the data acquisition unit 1302. In the example of Fig. 9, as shown in the diagram on the left, a product is placed on a stand, and objects other than the product include a floor, a wall, and a stand. Therefore, the bounding box calculation unit 1303 removes point cloud data of coordinates indicating the three-dimensional shapes of the floor, the wall, and the stand from the point cloud data acquired by the data acquisition unit 1302. Then, the bounding box calculation unit 1303 generates polygons based on the point cloud data after the removal. Then, the bounding box calculation unit 1303 calculates the bounding box of the product based on this polygon and with the object reference axis as the reference.

[0053] In the above description, the bounding box calculation unit 1303 generates polygons. However, generating polygons is not an essential process in the processing of the bounding box calculation unit 1303. In other words, the bounding box calculation unit 1303 may calculate the bounding box of the product directly from the point cloud data after the removal, using the object reference axis as a reference.

[0054] Next, the comparison unit 1304 compares the bounding box of the product model calculated by the bounding box calculation unit 1201 with the bounding box of the product calculated by the bounding box calculation unit 1303 (step ST804). At this time, the comparison unit 1304 determines whether the difference in size between the bounding box of the product model and the bounding box of the product is within a threshold value, which is set in advance as appropriate to a value that allows the product model and the product to be matched. The size is the area of ​​any plane (for example, the xy plane) in the bounding box, or the length in any direction in the bounding box.

[0055] Next, if the comparison unit 1304 determines that the difference in size between the bounding box of the product model and the bounding box of the product is within a threshold, the data extraction unit 1305 extracts point cloud data within a predetermined range including the range corresponding to the bounding box of the part model at the position corresponding to the relative position from the point cloud data, based on the bounding box of the part model calculated by the bounding box calculation unit 1201 and the relative position of the part model in the product model, and the point cloud data of coordinates indicating the three-dimensional shape of the product acquired by the data acquisition unit 1302 (step ST805).

[0056] That is, as shown in FIG. 10 , the data extraction unit 1305 extracts point cloud data within a predetermined range including a range where the part to be matched is thought to exist from the point cloud data acquired by the data acquisition unit 1302. At this time, as shown in the upper part of FIG. 10 , the data extraction unit 1305 applies the bounding box of a part model corresponding to the part to be matched and the relative position of the part model in the product model to the point cloud data acquired by the data acquisition unit 1302. Note that in the upper part of FIG. 10 , the diagram on the left shows the bounding box of the part model, and the diagram on the right shows the point cloud data acquired by the data acquisition unit 1302 (in the figure, point cloud data of coordinates indicating the 3D shapes of objects other than the product has been removed). Then, as shown in the lower part of FIG. 10 , the data extraction unit 1305 extracts point cloud data within a predetermined range including the applied range. Note that the size of the predetermined range may be the same as the size of the bounding box of the part model, but it is preferable to set it to be larger than the bounding box. 10 shows a case where point cloud data is extracted using the bounding box of the part model shown in FIG. 6A, but point cloud data is extracted in the same way for other parts. Also, in FIG. 10, for ease of explanation, the point cloud data is shown in place of the state of the model.

[0057] Next, the cross-sectional silhouette shape calculation unit 1306 calculates the cross-sectional silhouette shape of the part on the object reference plane based on the point cloud data extracted by the data extraction unit 1305 (step ST806).

[0058] Fig. 11 shows an example of the operation of calculating the cross-sectional silhouette shape of a part by the cross-sectional silhouette shape calculation unit 1306. Fig. 11 shows the calculation of the cross-sectional silhouette shape of the part from which point cloud data was extracted in Fig. 10. Fig. 11 also shows a case where the horizontal longitudinal direction of the product is set as the X axis and the XY plane is used as the reference plane. Furthermore, it is preferable that the cross-sectional silhouette shape calculation unit 1306 calculates the cross-sectional silhouette shapes of the part on each of a plurality of object reference planes.

[0059] Next, the comparing unit 1307 compares the cross-sectional silhouette shape of the part model calculated by the cross-sectional silhouette shape calculating unit 1202 with the cross-sectional silhouette shape of the part calculated by the cross-sectional silhouette shape calculating unit 1306 (step ST807). At this time, the comparing unit 1307 determines the degree of match between the cross-sectional silhouette shape of the part model and the cross-sectional silhouette shape of the part using an existing method such as pattern matching.

[0060] Fig. 12 shows an example of a comparison operation of cross-sectional silhouette shapes by the comparison unit 1307. In Fig. 12, the diagram on the left shows the cross-sectional silhouette shape of the part model shown in Fig. 7A, and the diagram on the right shows the cross-sectional silhouette shape of the part shown in Fig. 11.

[0061] Next, the collation section 1308 determines whether the part is of the same type as the part corresponding to the part model based on the comparison result by the comparison section 1307 (step ST808). At this time, for example, when the comparison unit 1307 determines that the matching rate between the cross-sectional silhouette shape of the part model and the cross-sectional silhouette shape of the part is equal to or greater than a threshold, the matching unit 1308 determines that the part is of the same type as the part corresponding to the part model. Note that the threshold is set in advance as appropriate to a value that allows matching between the part model and the part.

[0062] If there are multiple pairs of object reference planes and model reference planes, the comparison unit 1307 compares the cross-sectional silhouette shapes for each corresponding object reference plane and model reference plane. The matching unit 1308 then comprehensively evaluates the results of multiple comparisons by the comparison unit 1307 to determine whether the part is of the same type as the part corresponding to the part model. In this way, by having the comparison unit 1307 compare the cross-sectional silhouette shapes using various planes as references, the matching accuracy in the matching unit 1308 can be improved.

[0063] At this time, for example, the comparison unit 1307 first compares the cross-sectional silhouette shapes in a plane viewed from the front side. If the matching rate between the two cross-sectional silhouette shapes exceeds a predetermined threshold, the comparison unit 1307 then compares the cross-sectional silhouette shapes in a plane viewed from the side. If the matching rate between the two cross-sectional silhouette shapes exceeds a predetermined threshold, the comparison unit 1307 then compares the cross-sectional silhouette shapes in a plane viewed from the top side. If the matching rate between the two cross-sectional silhouette shapes exceeds a predetermined threshold, the matching unit 1308 determines that the part is of the same type as the part corresponding to the part model.

[0064] In this way, the object matching device 1 according to the first embodiment calculates a bounding box for point cloud data of coordinates indicating the three-dimensional shape of a product, and compares the bounding box of this product with the bounding box of a product model, thereby easily matching products. After performing the simple product matching, the object matching device 1 according to the first embodiment calculates cross-sectional silhouette shapes of components included in the product, and compares the cross-sectional silhouette shapes of these components with the cross-sectional silhouette shapes of the component models, thereby matching the components. In this case, it is preferable that the object matching device 1 according to the first embodiment compares the cross-sectional silhouette shapes on a plurality of reference planes. In this way, the object matching device 1 according to the first embodiment performs object comparison in two dimensions, which requires less processing, rather than in three dimensions, thereby enabling object matching with a lighter load than conventional methods. Furthermore, the object matching device 1 according to the first embodiment enables object matching at a higher speed than conventional methods.

[0065] As described above, according to the first embodiment, the object matching device 1 includes a data acquisition unit 1302 that acquires point cloud data of coordinates indicating the three-dimensional shape of a product, a bounding box calculation unit 1303 that calculates a bounding box of the product based on the point cloud data of coordinates indicating the three-dimensional shape of the product acquired by the data acquisition unit 1302, taking the object reference axis, which is a specific direction of the product, as a reference, and a comparison unit 1304 that compares the bounding box of the product model with the bounding box of the product calculated by the bounding box calculation unit 1303. When the comparison unit 1304 determines that the difference in size between the bounding box of the product model and the bounding box of the product is within a threshold, the comparison unit 1304 compares the bounding boxes of part models included in the product model and the relative positions of the part models in the product model, and the data acquisition unit 1303. The object matching device 1 according to the first embodiment includes: a data extraction unit 1305 that extracts, from the point cloud data or the polygons, point cloud data of coordinates indicating the three-dimensional shape of the product acquired by unit 1302 or a polygon generated from the point cloud data, point cloud data or polygons within a predetermined range including a range corresponding to the bounding box of the part model at a position corresponding to the relative position, a cross-sectional silhouette shape calculation unit 1306 that calculates a cross-sectional silhouette shape of the part on an object reference plane based on the object reference axis, based on the point cloud data or polygon extracted by data extraction unit 1305, a comparison unit 1307 that compares the cross-sectional silhouette shape of the part model with the cross-sectional silhouette shape of the part calculated by cross-sectional silhouette shape calculation unit 1306, and a matching unit 1308 that determines whether the part is an object of the same type as the object corresponding to the part model, based on the comparison result by comparison unit 1307. This enables object matching with a lighter load than conventional devices.

[0066] In the above description, the first object is a product and the second object is a part, but the first object and the second object are not limited to this.

[0067] Embodiment 2 In the first embodiment, an example has been described in which the object to be matched by the object matching device 1 is a part included in a product. However, the present invention is not limited to this, and the object to be matched by the object matching device 1 may also be a product. An example configuration of the object matching device 1 in this case will be described below.

[0068] Fig. 13 is a diagram showing an example of the configuration of the database creation unit 12 according to embodiment 2. In the example of the configuration of the database creation unit 12 according to embodiment 2 shown in Fig. 13, the bounding box calculation unit 1201 and the cross-section silhouette shape calculation unit 1202 of the example of the configuration of the database creation unit 12 according to embodiment 1 shown in Fig. 2 are changed to a bounding box calculation unit 1201b and a cross-section silhouette shape calculation unit 1202b. 14 is a diagram showing an example of the configuration of the matching processing unit 13 according to embodiment 2. In the example of the configuration of the matching processing unit 13 according to embodiment 2 shown in Fig. 14, the data extraction unit 1305 is removed from the example of the configuration of the matching processing unit 13 according to embodiment 1 shown in Fig. 3, and the cross-sectional silhouette shape calculation unit 1306, the comparison unit 1307, and the comparison unit 1308 are changed to a cross-sectional silhouette shape calculation unit 1306b, a comparison unit 1307b, and a comparison unit 1308b. Other configuration examples of the object matching device 1 according to embodiment 2 are similar to the configuration example of the object matching device 1 according to embodiment 1, so the same reference numerals are used and only the different parts will be described.

[0069] The bounding box calculation unit 1201b calculates the bounding box of the product model based on the data indicating the three-dimensional shape of the product model, with the model reference axes as the reference. The model reference axis is a specific direction in the product model. For example, the bounding box calculation unit 1201b can use the short-side direction or the long-side direction of the product model as the model reference axis.

[0070] On the other hand, the bounding box calculation unit 1201b differs from the bounding box calculation unit 1201 in the first embodiment in that it does not calculate the bounding box of the part model or the relative position of the part model in the product model.

[0071] Furthermore, the cross-sectional silhouette shape calculation unit 1202b calculates the cross-sectional silhouette shape of the product model on the model reference plane based on data indicating the three-dimensional shape of the product model. The model reference plane is a plane based on the model reference axis. For example, if the longitudinal direction of the product model is the X-axis, the cross-sectional silhouette shape calculation unit 1202 can use, for example, the XY plane, the YZ plane, or the XZ plane as the reference plane. Furthermore, it is preferable that the cross-sectional silhouette shape calculation unit 1202 calculates the cross-sectional silhouette shapes of the product model on each of a plurality of model reference planes.

[0072] Furthermore, if the comparison unit 1304 determines that the difference in size between the bounding box of the product model and the bounding box of the product is within a threshold, the cross-sectional silhouette shape calculation unit 1306b calculates the cross-sectional silhouette shape of the product on the object reference plane based on the point cloud data of coordinates indicating the three-dimensional shape of the product acquired by the data acquisition unit 1302. The object reference plane is a plane based on the object reference axis. The object reference plane and the model reference plane are the same plane. For example, if the longitudinal direction of the product is set as the X-axis, which is the object reference axis, the cross-sectional silhouette shape calculation unit 1306b can use, for example, the XY plane, YZ plane, or XZ plane as the reference plane. Furthermore, it is preferable that the cross-sectional silhouette shape calculation unit 1306b calculates the cross-sectional silhouette shapes of the product on each of a plurality of object reference planes.

[0073] Here, the cross-sectional silhouette shape calculation unit 1306b may calculate the cross-sectional silhouette shape of the part on the object reference plane directly from the point cloud data. Furthermore, the cross-sectional silhouette shape calculation unit 1306b may generate polygons from the point cloud data, and calculate the cross-sectional silhouette shape of the part on the object reference plane from the polygons.

[0074] Alternatively, if the bounding box calculation unit 1303 generates polygons from point cloud data, the cross-sectional silhouette shape calculation unit 1306b may calculate the cross-sectional silhouette shape of the product on the object reference plane based on the polygons rather than the point cloud data.

[0075] In addition, since the number of points in the polygons is thinned out compared to the point cloud data, the cross-sectional silhouette shape calculation unit 1306b calculates the two-dimensional cross-sectional silhouette shape from the polygons rather than the point cloud data, which is thought to lead to improved processing efficiency and a reduction in the processing load.

[0076] Furthermore, the comparison unit 1307b compares the cross-sectional silhouette shape of the product model calculated by the cross-sectional silhouette shape calculation unit 1202b with the cross-sectional silhouette shape of the product calculated by the cross-sectional silhouette shape calculation unit 1306b. At this time, the comparison unit 1307b determines the degree of match between the cross-sectional silhouette shape of the product model and the cross-sectional silhouette shape of the product using an existing method such as pattern matching. If there are multiple pairs of object reference planes and model reference planes, the comparison unit 1307b compares the cross-sectional silhouette shapes for each corresponding object reference plane and model reference plane.

[0077] The collation unit 1308b determines whether the product is of the same type as the product corresponding to the product model, based on the comparison result by the comparison unit 1307b. At this time, for example, when the comparison unit 1307b determines that the matching rate between the cross-sectional silhouette shape of the product model and the cross-sectional silhouette shape of the product is equal to or greater than a threshold, the matching unit 1308b determines that the product is of the same type as the product corresponding to the product model. Note that the threshold is set in advance as appropriate to a value that allows matching between the product model and the product. Furthermore, if the comparison unit 1307b performs a comparison of cross-sectional silhouette shapes multiple times, the matching unit 1308b comprehensively evaluates the results of the multiple comparisons and determines whether the product is of the same type as the product corresponding to the product model.

[0078] 15 shows a case where the database creation unit 12 is provided in the object matching device 1. However, the database creation unit 12 is not an essential component of the object matching device 1, and does not have to be provided in the object matching device 1.

[0079] In this way, the object matching device 1 according to the second embodiment calculates a bounding box for point cloud data of coordinates indicating the three-dimensional shape of the product, and compares the bounding box of this product with the bounding box of the product model, thereby performing simple product matching. After simple product matching is performed, the object matching device 1 according to the second embodiment calculates a cross-sectional silhouette shape of the product, and performs product matching by comparing the cross-sectional silhouette shape of this product with the cross-sectional silhouette shape of the product model. In this case, it is preferable that the object matching device 1 according to the second embodiment compares the cross-sectional silhouette shapes on a plurality of reference planes. In this way, the object matching device 1 according to the second embodiment performs object comparison in two dimensions, which requires less processing, rather than in three dimensions, thereby enabling object matching with a lighter load than conventional methods. Furthermore, the object matching device 1 according to the second embodiment enables object matching at a higher speed than conventional methods.

[0080] As described above, according to the second embodiment, the object matching device 1 includes a data acquisition unit 1302 that acquires point cloud data of coordinates indicating the three-dimensional shape of a product, a bounding box calculation unit 1303 that calculates a bounding box of the product based on the point cloud data of coordinates indicating the three-dimensional shape of the product acquired by the data acquisition unit 1302, with reference to an object reference axis that is a specific direction of the product, a comparison unit 1304 that compares the bounding box of the product model with the bounding box of the product calculated by the bounding box calculation unit 1303, and a comparison unit 1304 that compares the bounding box of the product model with the bounding box of the product calculated by the comparison unit 1304. If it is determined that the difference in size with the box is within a threshold, the object matching device 1 according to the second embodiment is further provided with: cross-sectional silhouette shape calculation unit 1306b that calculates a cross-sectional silhouette shape of the product on an object reference plane based on an object reference axis, based on point cloud data of coordinates indicating the three-dimensional shape of the product acquired by data acquisition unit 1302 or a polygon generated from the point cloud data; comparison unit 1307b that compares the cross-sectional silhouette shape of the product model with the cross-sectional silhouette shape of the product calculated by cross-sectional silhouette shape calculation unit 1306b; and matching unit 1308b that determines whether the product is an object of the same type as the object corresponding to the product model, based on the comparison result by comparison unit 1307b. This enables object matching with a lighter load than conventional methods.

[0081] Although the above description has been given taking the case where the first object is a product as an example, the first object is not limited to this.

[0082] Embodiment 3 The object matching device 1 according to the first embodiment has been described without taking into consideration missing point cloud data. However, in reality, it is not always possible to obtain all point cloud data, and some point cloud data may be missing. Therefore, an example configuration for solving this problem will be described for the object matching device 1 according to the third embodiment.

[0083] Fig. 15 is a diagram showing an example of the configuration of the matching processing unit 13 in embodiment 3. In the example of the configuration of the matching processing unit 13 in embodiment 3 shown in Fig. 15, the comparison unit 1307 in the example of the configuration of the matching processing unit 13 in embodiment 1 shown in Fig. 3 is changed to a comparison unit 1307c. Other configuration examples of the object matching device 1 according to embodiment 3 are similar to the configuration example of the object matching device 1 according to embodiment 1, so the same reference numerals are used and only the different parts will be described.

[0084] The comparison unit 1307c compares the cross-sectional silhouette shape of the part model calculated by the cross-sectional silhouette shape calculation unit 1202 with the cross-sectional silhouette shape of the part calculated by the cross-sectional silhouette shape calculation unit 1306. At this time, the comparison unit 1307 determines the degree of match between the cross-sectional silhouette shape of the part model and the cross-sectional silhouette shape of the part using an existing method such as pattern matching. Furthermore, the comparison unit 1307c compares the range of the cross-sectional silhouette shape of the part calculated by the cross-sectional silhouette shape calculation unit 1306, for which point cloud data or polygons have been obtained by the data extraction unit 1305. If there are a plurality of object reference planes and model reference planes, the comparison unit 1307c compares the cross-sectional silhouette shapes for each corresponding object reference plane and model reference plane.

[0085] For example, as shown in Fig. 16, it is assumed that the cross-sectional silhouette shape of a part calculated by the cross-sectional silhouette shape calculation unit 1306 is partially missing. In Fig. 16, reference numeral 161 denotes a missing region due to a missing point cloud data. In this case, if the comparison unit 1307c were to compare the cross-sectional silhouette shapes including the missing portion, for example, as shown in the upper part of Fig. 17, the matching rate would decrease, increasing the possibility of erroneous detection. Note that in the upper part of Fig. 17, the diagram on the left shows the cross-sectional silhouette shape of the partially missing part shown in Fig. 16, and the diagram on the right shows the cross-sectional silhouette shape of the part model shown in Fig. 7A.

[0086] Therefore, the comparison unit 1307c performs comparison by excluding from the comparison target the regions that are missing in the point cloud data as described above. For example, as shown in the lower part of Fig. 17, the comparison unit 1307c performs comparison by excluding from the comparison target the regions that correspond to the missing regions in the cross-sectional silhouette shape of the part model shown in Fig. 16, among the cross-sectional silhouette shapes of the part model shown in Fig. 7A. In this case, as shown in the lower part of Fig. 17, for example, an exclusion area of ​​a predetermined shape including the missing area is excluded from the cross-sectional silhouette shape of the part shown in Fig. 16 and the cross-sectional silhouette shape of the part model shown in Fig. 7A, and then the two are compared. Note that in Fig. 17, reference numeral 171 indicates the exclusion area, and Fig. 17 shows a case where the exclusion area is a rectangular area.

[0087] 16 and 17 are images that are created by imitating the product for the purpose of explaining the function, and are different from the actual product image.

[0088] In this way, in the object matching device 1 according to the third embodiment, when comparing cross-sectional silhouette shapes, the comparison is performed only within the range in which point cloud data or polygons have been obtained. This makes it possible to perform matching with high accuracy even when the point cloud data obtained by the data acquisition unit 1302 has missing data.

[0089] In the above description, the comparison unit 1307 is changed to the comparison unit 1307c in the object matching apparatus 1 according to embodiment 1. However, the present invention is not limited to this, and the comparison unit 1307b may be changed to the comparison unit 1307c in the object matching apparatus 1 according to embodiment 2, and the same effect as above can be obtained. In this case, the comparison unit 1307c compares the range of the cross-sectional silhouette shape of the product calculated by the cross-sectional silhouette shape calculation unit 1306b, for which point cloud data has been obtained by the data acquisition unit 1302.

[0090] Embodiment 4 In the first embodiment, a configuration related to object matching was described. In contrast, in the fourth embodiment, a configuration will be described in which additional information is visually displayed for an object that is determined to be of the same type as the model through matching.

[0091] Fig. 18 is a diagram showing an example of the configuration of an object matching device 1 according to embodiment 4. In the object matching device 1 according to embodiment 4 shown in Fig. 18, a display processing unit 14 is added to the object matching device 1 according to embodiment 1 shown in Fig. 1. Other configuration examples of the object matching device 1 according to the fourth embodiment are similar to the configuration example of the object matching device 1 according to the first embodiment, and only the different parts will be described.

[0092] The display processing unit 14 displays additional information based on the matching result by the matching processing unit 13. The additional information is information that is visually added to the object to be matched. As shown in FIG. 19 , the display processing unit 14 includes a position calculation unit (first position calculation unit) 1401, a position calculation unit (second position calculation unit) 1402, and an information addition unit 1403.

[0093] When the matching unit 1308 determines that a part is of the same type as the part corresponding to the part model, the position calculation unit 1401 calculates the reference position of the part. At this time, the position calculation unit 1401 calculates the reference position of the part based on the point cloud data extracted by the data extraction unit 1305. Note that the reference position is set in advance as appropriate to a position such as the center of gravity of the part.

[0094] The position calculation unit 1402 calculates the position at which to display the additional information based on the relative position of the additional information with respect to the part model and the reference position of the part calculated by the position calculation unit 1401. At this time, the position calculation unit 1402 calculates the position at which to display the additional information by adding the relative position of the additional information to the reference position of the part.

[0095] The information addition unit 1403 displays the additional information on a display device (not shown) such as MR glasses, based on data indicating the display content of the additional information and the position calculated by the position calculation unit 1402. At this time, the information addition unit 1403 displays the additional information at the position on the display device.

[0096] In this way, the object matching device 1 according to embodiment 4 displays additional information for an object based on the matching result of the object. As a result, the object matching device 1 according to embodiment 4 can display additional information in accordance with the actual position of the object, unlike conventional methods.

[0097] In the above description, a case has been shown in which display processing unit 14 is added to object matching device 1 according to embodiment 1. However, the present invention is not limited to this, and display processing unit 14 may be added to object matching device 1 according to embodiments 2 and 3, and the same effects as those described above can be obtained.

[0098] When the display processing unit 14 is added to the object matching device 1 according to the second embodiment, if the matching unit 1308 determines that the product is of the same type as the product corresponding to the product model, the position calculation unit 1401 calculates the reference position of the product. At this time, the position calculation unit 1401 calculates the reference position of the product based on the point cloud data acquired by the data acquisition unit 1302. The reference position is set in advance as appropriate to a position such as the center of gravity of the product. Furthermore, the position calculation unit 1402 calculates the position at which to display the additional information based on the relative position of the additional information with respect to the product model and the reference position of the product calculated by the position calculation unit 1401. At this time, the position calculation unit 1402 calculates the position at which to display the additional information by adding the relative position of the additional information to the reference position of the product.

[0099] Finally, an example of the hardware configuration of the object matching device 1 according to embodiments 1 to 4 will be described with reference to Fig. 20. Note that although an example of the hardware configuration of the object matching device 1 according to embodiment 1 will be described below, the same applies to the example of the hardware configuration of the object matching device 1 according to embodiments 2 to 4. Each function of the database creation unit 12 and the matching processing unit 13 in the object matching device 1 is realized by a processing circuit 501. The processing circuit 501 may be dedicated hardware as shown in Fig. 20A, or may be a CPU (also referred to as a central processing unit, processing unit, arithmetic unit, microprocessor, microcomputer, processor, or DSP (Digital Signal Processor)) 502 that executes a program stored in a memory 503 as shown in Fig. 20B.

[0100] When the processing circuit 501 is dedicated hardware, the processing circuit 501 corresponds to, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof. The functions of each unit of the database creation unit 12 and the matching processing unit 13 may be realized by the processing circuit 501 individually, or the functions of each unit may be realized by the processing circuit 501 collectively.

[0101] When the processing circuit 501 is a CPU 502, the functions of the database creation unit 12 and the matching processing unit 13 are realized by software, firmware, or a combination of software and firmware. The software and firmware are written as programs and stored in the memory 503. The processing circuit 501 realizes the functions of each unit by reading and executing the programs stored in the memory 503. That is, the object matching device 1 includes the memory 503 for storing programs that, when executed by the processing circuit 501, result in the execution of, for example, each step shown in FIGS. 4 and 8 . It can also be said that these programs cause a computer to execute the procedures and methods of the database creation unit 12 and the matching processing unit 13. Here, examples of the memory 503 include non-volatile or volatile semiconductor memories such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable ROM), and EEPROM (Electrically EPROM), magnetic disks, flexible disks, optical disks, compact disks, minidisks, and DVDs (Digital Versatile Discs).

[0102] It is also possible to realize some of the functions of the database creation unit 12 and the matching processing unit 13 with dedicated hardware and some with software or firmware. For example, the function of the database creation unit 12 can be realized by a processing circuit 501 as dedicated hardware, and the function of the matching processing unit 13 can be realized by the processing circuit 501 reading and executing a program stored in a memory 503.

[0103] In this way, the processing circuitry 501 can realize each of the above-described functions by hardware, software, firmware, or a combination thereof.

[0104] It should be noted that the embodiments may be freely combined, or any of the components in each embodiment may be modified, or any of the components in each embodiment may be omitted.

[0105] Various aspects of the present disclosure are summarized below as appendices.

[0106] (Appendix 1) a data acquisition unit that acquires point cloud data of coordinates that indicate a three-dimensional shape of the first object; a first bounding box calculation unit that calculates a bounding box of a first object based on point cloud data of coordinates indicating the three-dimensional shape of the first object acquired by the data acquisition unit, with a first reference axis that is a specific direction in the first object as a reference; a first comparison unit that compares a bounding box of a first object model with a bounding box of the first object calculated by the first bounding box calculation unit; a data extraction unit that, when the first comparison unit determines that the difference in size between the bounding box of the first object model and the bounding box of the first object is within a threshold, extracts point cloud data or polygons within a predetermined range including a range corresponding to the bounding box of the second object model at a position corresponding to the relative position from the point cloud data or polygons, based on the bounding box of a second object model included in the first object model and the relative position of the second object model in the first object model, and point cloud data of coordinates indicating the three-dimensional shape of the first object acquired by the data acquisition unit or polygons generated from the point cloud data; a first cross-sectional silhouette shape calculation unit that calculates a cross-sectional silhouette shape of a second object on a first reference plane based on the first reference axis, based on the point cloud data or polygons extracted by the data extraction unit; a second comparison unit that compares the cross-sectional silhouette shape of the second object model with the cross-sectional silhouette shape of the second object calculated by the first cross-sectional silhouette shape calculation unit; a matching unit that determines whether the second object is an object of the same type as the object corresponding to the second object model based on a comparison result by the second comparison unit; An object matching device comprising: (Appendix 2) a second bounding box calculation unit that calculates a bounding box of the first object model based on data indicating a three-dimensional shape of the first object model, with a second reference axis being a specific direction in the first object model as a reference; The first comparison unit performs the comparison using the bounding box of the first object model calculated by the second bounding box calculation unit. 2. The object matching device according to claim 1, (Appendix 3) a second cross-sectional silhouette shape calculation unit that calculates a cross-sectional silhouette shape of a second object model on a second reference plane based on the second reference axis, based on data indicating a three-dimensional shape of the first object model; The second comparison unit performs the comparison using the cross-sectional silhouette shape of the second object model calculated by the second cross-sectional silhouette shape calculation unit. 3. The object matching device according to claim 2. (Appendix 4) The second comparison unit compares the cross-sectional silhouette shape of the second object calculated by the first cross-sectional silhouette shape calculation unit with the range for which point cloud data is obtained by the data extraction unit. 4. The object matching device according to claim 1, wherein: (Appendix 5) a first position calculation unit that calculates a reference position of the second object when the matching unit determines that the second object is an object of the same type as the object corresponding to the second object model; a second position calculation unit that calculates a position at which the additional information is to be displayed based on a relative position of the additional information with respect to a second object model and a reference position of the second object calculated by the first position calculation unit; and and an information adding unit that displays the additional information based on data indicating the display content of the additional information and the position calculated by the second position calculating unit. 5. The object matching device according to claim 1, wherein: (Appendix 6) a data acquisition unit that acquires point cloud data of coordinates that indicate a three-dimensional shape of the first object; a first bounding box calculation unit that calculates a bounding box of a first object based on point cloud data of coordinates indicating the three-dimensional shape of the first object acquired by the data acquisition unit, with a first reference axis that is a specific direction in the first object as a reference; a first comparison unit that compares a bounding box of a first object model with a bounding box of the first object calculated by the first bounding box calculation unit; a first cross-sectional silhouette shape calculation unit that calculates a cross-sectional silhouette shape of the first object on a first reference plane based on the first reference axis, based on point cloud data of coordinates indicating the three-dimensional shape of the first object acquired by the data acquisition unit or polygons generated from the point cloud data, when the first comparison unit determines that the difference in size between the bounding box of the first object model and the bounding box of the first object is within a threshold value; a second comparison unit that compares the cross-sectional silhouette shape of the first object model with the cross-sectional silhouette shape of the first object calculated by the first cross-sectional silhouette shape calculation unit; a matching unit that determines whether the first object is an object of the same type as the object corresponding to the first object model based on a comparison result by the second comparing unit; An object matching device comprising: (Appendix 7) a second bounding box calculation unit that calculates a bounding box of the first object model based on data indicating a three-dimensional shape of the first object model, with a second reference axis being a specific direction in the first object model as a reference; The first comparison unit performs the comparison using the bounding box of the first object model calculated by the second bounding box calculation unit. 7. The object matching device according to claim 6, (Appendix 8) a second cross-sectional silhouette shape calculation unit that calculates a cross-sectional silhouette shape of the first object model on a second reference plane based on the second reference axis, based on data indicating a three-dimensional shape of the first object model; The second comparison unit performs comparison using the cross-sectional silhouette shape of the first object model calculated by the second cross-sectional silhouette shape calculation unit. 8. The object matching device according to claim 7, (Appendix 9) The second comparison unit compares the cross-sectional silhouette shape of the first object calculated by the first cross-sectional silhouette shape calculation unit with the range of point cloud data obtained by the data acquisition unit. 9. The object matching device according to claim 6, wherein: (Appendix 10) a first position calculation unit that calculates a reference position of the first object when the matching unit determines that the first object is an object of the same type as the object corresponding to the first object model; a second position calculation unit that calculates a position at which the additional information is to be displayed based on a relative position of the additional information with respect to a first object model and a reference position of the first object calculated by the first position calculation unit; and and an information adding unit that displays the additional information based on data indicating the display content of the additional information and the position calculated by the second position calculating unit. 10. The object matching device according to any one of Supplementary Note 6 to Supplementary Note 9, (Appendix 11) a step in which a data acquisition unit acquires point cloud data of coordinates indicating a three-dimensional shape of a first object; a first bounding box calculation unit calculating a bounding box of the first object based on point cloud data of coordinates indicating the three-dimensional shape of the first object acquired by the data acquisition unit, with a first reference axis being a specific direction of the first object as a reference; a step in which a first comparison unit compares a bounding box of a first object model with the bounding box of the first object calculated by the first bounding box calculation unit; a data extraction unit, when the first comparison unit determines that the difference in size between the bounding box of the first object model and the bounding box of the first object is within a threshold, extracting point cloud data or polygons within a predetermined range including a range corresponding to the bounding box of the second object model at a position corresponding to the relative position from the point cloud data or polygons based on the bounding box of a second object model included in the first object model and the relative position of the second object model in the first object model, and the point cloud data of coordinates indicating the three-dimensional shape of the first object acquired by the data acquisition unit or polygons generated from the point cloud data; a step in which a first cross-sectional silhouette shape calculation unit calculates a cross-sectional silhouette shape of a second object on a first reference plane based on the first reference axis, based on the point cloud data or polygons extracted by the data extraction unit; a step in which a second comparison unit compares a cross-sectional silhouette shape of a second object model with the cross-sectional silhouette shape of the second object calculated by the first cross-sectional silhouette shape calculation unit; a matching unit determining, based on a comparison result by the second comparing unit, whether the second object is an object of the same type as the object corresponding to the second object model; The object matching method has the following structure. (Appendix 12) a step in which a data acquisition unit acquires point cloud data of coordinates indicating a three-dimensional shape of a first object; a first bounding box calculation unit calculating a bounding box of the first object based on point cloud data of coordinates indicating the three-dimensional shape of the first object acquired by the data acquisition unit, with a first reference axis being a specific direction of the first object as a reference; a step in which a first comparison unit compares a bounding box of a first object model with the bounding box of the first object calculated by the first bounding box calculation unit; a first cross-sectional silhouette shape calculation unit, when it is determined by the first comparison unit that the difference in size between the bounding box of the first object model and the bounding box of the first object is within a threshold, calculating a cross-sectional silhouette shape of the first object on a first reference plane based on the first reference axis, based on point cloud data of coordinates indicating the three-dimensional shape of the first object acquired by the data acquisition unit or a polygon generated from the point cloud data; a step in which a second comparison unit compares a cross-sectional silhouette shape of a first object model with the cross-sectional silhouette shape of the first object calculated by the first cross-sectional silhouette shape calculation unit; a step in which a matching unit determines, based on a comparison result by the second comparison unit, whether the first object is an object of the same type as the object corresponding to the first object model; The object matching method has the following structure. [Explanation of symbols]

[0107] 1 object matching device, 11 data storage unit, 12 database creation unit, 13 matching processing unit, 14 display processing unit, 501 processing circuit, 502 CPU, 503 memory, 1201 bounding box calculation unit (second bounding box calculation unit), 1202 cross-sectional silhouette shape calculation unit (second cross-sectional silhouette shape calculation unit), 1301 depth sensor, 1302 data acquisition unit, 1303 bounding box calculation unit (first bounding box calculation unit), 1304 comparison unit (first comparison unit), 1305 data extraction unit, 1306 cross-sectional silhouette shape calculation unit (first cross-sectional silhouette shape calculation unit), 1307 comparison unit (second comparison unit), 1308 matching unit, 1401 position calculation unit (first position calculation unit), 1402 position calculation unit (second position calculation unit), 1403 information addition unit.

Claims

1. a data acquisition unit that acquires point cloud data of coordinates that indicate a three-dimensional shape of the first object; a first bounding box calculation unit that calculates a bounding box of a first object based on point cloud data of coordinates indicating a three-dimensional shape of the first object acquired by the data acquisition unit, with a first reference axis that is a specific direction in the first object as a reference; a first comparison unit that compares a bounding box of a first object model with a bounding box of the first object calculated by the first bounding box calculation unit; a data extraction unit that, when the first comparison unit determines that the difference in size between the bounding box of the first object model and the bounding box of the first object is within a threshold, extracts point cloud data or polygons within a predetermined range including a range corresponding to the bounding box of the second object model at a position corresponding to the relative position from the point cloud data or polygons, based on the bounding box of a second object model included in the first object model and the relative position of the second object model in the first object model, and point cloud data of coordinates indicating the three-dimensional shape of the first object acquired by the data acquisition unit or polygons generated from the point cloud data; a first cross-sectional silhouette shape calculation unit that calculates a cross-sectional silhouette shape of a second object on a first reference plane based on the first reference axis, based on the point cloud data or polygons extracted by the data extraction unit; a second comparison unit that compares the cross-sectional silhouette shape of the second object model with the cross-sectional silhouette shape of the second object calculated by the first cross-sectional silhouette shape calculation unit; a matching unit that performs matching by determining whether the second object is an object of the same type as the object corresponding to the second object model based on the comparison result by the second comparison unit; An object matching device comprising:

2. a second bounding box calculation unit that calculates a bounding box of the first object model based on data indicating a three-dimensional shape of the first object model, with a second reference axis being a specific direction in the first object model as a reference; The first comparison unit performs comparison using the bounding box of the first object model calculated by the second bounding box calculation unit.

2. The object matching device according to claim 1.

3. a second cross-sectional silhouette shape calculation unit that calculates a cross-sectional silhouette shape of the second object model on a second reference plane based on the second reference axis, based on data indicating a three-dimensional shape of the first object model; The second comparison unit performs comparison using the cross-sectional silhouette shape of the second object model calculated by the second cross-sectional silhouette shape calculation unit.

3. The object matching device according to claim 2.

4. The second comparison unit compares the cross-sectional silhouette shape of the second object model with the cross-sectional silhouette shape of the second object calculated by the first cross-sectional silhouette shape calculation unit after excluding an exclusion area of ​​a predetermined shape including a range from which point cloud data or polygons have not been obtained by the data extraction unit.

2. The object matching device according to claim 1.

5. a first position calculation unit that calculates a reference position of the second object when the matching unit determines that the second object is an object of the same type as the object corresponding to the second object model; and a second position calculation unit that calculates a position at which the additional information is to be displayed based on a relative position of the additional information with respect to the second object model and the reference position of the second object calculated by the first position calculation unit, and a display processing unit that displays the additional information based on data indicating display content of the additional information and the position calculated by the second position calculation unit.

2. The object matching device according to claim 1.

6. a data acquisition unit that acquires point cloud data of coordinates that indicate a three-dimensional shape of the first object; a first bounding box calculation unit that calculates a bounding box of a first object based on point cloud data of coordinates indicating a three-dimensional shape of the first object acquired by the data acquisition unit, with a first reference axis that is a specific direction in the first object as a reference; a first comparison unit that compares a bounding box of a first object model with a bounding box of the first object calculated by the first bounding box calculation unit; a first cross-sectional silhouette shape calculation unit that calculates a cross-sectional silhouette shape of the first object on a first reference plane based on the first reference axis, based on point cloud data of coordinates indicating the three-dimensional shape of the first object acquired by the data acquisition unit or polygons generated from the point cloud data, when the first comparison unit determines that the difference in size between the bounding box of the first object model and the bounding box of the first object is within a threshold value; a second comparison unit that compares the cross-sectional silhouette shape of the first object model with the cross-sectional silhouette shape of the first object calculated by the first cross-sectional silhouette shape calculation unit; a matching unit that performs matching by determining whether the first object is an object of the same type as the object corresponding to the first object model based on the comparison result by the second comparison unit; An object matching device comprising:

7. a second bounding box calculation unit that calculates a bounding box of the first object model based on data indicating a three-dimensional shape of the first object model, with a second reference axis being a specific direction in the first object model as a reference; The first comparison unit performs comparison using the bounding box of the first object model calculated by the second bounding box calculation unit.

7. The object matching device according to claim 6.

8. a second cross-sectional silhouette shape calculation unit that calculates a cross-sectional silhouette shape of the first object model on a second reference plane based on the second reference axis, based on data indicating a three-dimensional shape of the first object model; The second comparison unit performs comparison using the cross-sectional silhouette shape of the first object model calculated by the second cross-sectional silhouette shape calculation unit.

8. The object matching device according to claim 7.

9. The second comparison unit compares the cross-sectional silhouette shape of the second object model with the cross-sectional silhouette shape of the second object calculated by the first cross-sectional silhouette shape calculation unit after excluding an exclusion area of ​​a predetermined shape including a range for which point cloud data has not been obtained by the data acquisition unit.

7. The object matching device according to claim 6.

10. The display device includes a first position calculation unit that calculates a reference position of the first object when the matching unit determines that the first object is an object of the same type as the object corresponding to the first object model, and a second position calculation unit that calculates a position at which the additional information is to be displayed based on a relative position of the additional information with respect to the first object model and the reference position of the first object calculated by the first position calculation unit, and a display processing unit that displays the additional information based on data indicating display content of the additional information and the position calculated by the second position calculation unit.

7. The object matching device according to claim 6.

11. a step in which a data acquisition unit acquires point cloud data of coordinates indicating a three-dimensional shape of a first object; a first bounding box calculation unit calculating a bounding box of the first object based on point cloud data of coordinates indicating the three-dimensional shape of the first object acquired by the data acquisition unit, with a first reference axis being a specific direction of the first object as a reference; a step in which a first comparison unit compares a bounding box of a first object model with the bounding box of the first object calculated by the first bounding box calculation unit; a data extraction unit, when the first comparison unit determines that the difference in size between the bounding box of the first object model and the bounding box of the first object is within a threshold, extracting point cloud data or polygons within a predetermined range including a range corresponding to the bounding box of the second object model at a position corresponding to the relative position from the point cloud data or polygons based on the bounding box of a second object model included in the first object model and the relative position of the second object model in the first object model, and point cloud data of coordinates indicating the three-dimensional shape of the first object acquired by the data acquisition unit or polygons generated from the point cloud data; a step in which a first cross-sectional silhouette shape calculation unit calculates a cross-sectional silhouette shape of a second object on a first reference plane based on the first reference axis, based on the point cloud data or the polygon extracted by the data extraction unit; a step in which a second comparison unit compares a cross-sectional silhouette shape of a second object model with the cross-sectional silhouette shape of the second object calculated by the first cross-sectional silhouette shape calculation unit; a matching unit performing matching by determining whether the second object is an object of the same type as the object corresponding to the second object model based on the comparison result by the second comparison unit; The object matching method has the following structure.

12. a step in which a data acquisition unit acquires point cloud data of coordinates indicating a three-dimensional shape of a first object; a first bounding box calculation unit calculating a bounding box of the first object based on point cloud data of coordinates indicating the three-dimensional shape of the first object acquired by the data acquisition unit, with a first reference axis being a specific direction of the first object as a reference; a step in which a first comparison unit compares a bounding box of a first object model with the bounding box of the first object calculated by the first bounding box calculation unit; a first cross-sectional silhouette shape calculation unit calculating a cross-sectional silhouette shape of the first object on a first reference plane based on the first reference axis, based on point cloud data of coordinates indicating the three-dimensional shape of the first object acquired by the data acquisition unit or polygons generated from the point cloud data, when the first comparison unit determines that the difference in size between the bounding box of the first object model and the bounding box of the first object is within a threshold value; a step in which a second comparison unit compares a cross-sectional silhouette shape of a first object model with the cross-sectional silhouette shape of the first object calculated by the first cross-sectional silhouette shape calculation unit; a matching unit performing matching by determining whether the first object is an object of the same type as the object corresponding to the first object model based on the comparison result by the second comparison unit; The object matching method has the following structure.

Images