Template generation device, matching system, matching device, template generation method, matching method, and program

Abstract

To provide a technique capable of improving accuracy of verification when a template of an object is verified with an image.SOLUTION: A template creation device for creating a template used in a verification device for verifying a measurement image representing results of measuring a measurement range including a predetermined object with a template indicating a feature quantity of the predetermined object includes: first generation means for generating a projection image representing the predetermined object as a two-dimensional image based on a three-dimensional model of the predetermined object; feature quantity acquiring means for acquiring a feature quantity corrected according to a degree of attention of each region of the three-dimensional model based on the three-dimensional model or the projection image; and second generation means for generating the template indicating the feature quantity corresponding to the projection image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a template generating device, a matching system, a matching device, a template generating method, a matching method, and a program. [Background technology]

[0002] The position and orientation of an object are recognized by comparing a template that indicates the characteristics of the object in each orientation with an image (measurement image) acquired by an imaging device capturing (measuring) the object.

[0003] In Patent Document 1, a template indicating the shape of an object is stored, and the largest plane of the object in a measurement image is matched with the template, thereby making the matching process for recognizing the target object more efficient. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2018-097889 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, while the technique disclosed in Patent Document 1 can improve the efficiency of the matching process by matching the largest plane with the template, the accuracy of the matching decreases when, for example, part of the plane is hidden by an obstruction or the like and cannot be captured by the imaging device.

[0006] Therefore, an object of the present invention is to provide a technique that can improve the accuracy of matching when matching an object template with an image. [Means for solving the problem]

[0007] In order to achieve the above object, the present invention employs the following configuration.

[0008] That is, a template generation device according to one aspect of the present invention is a template generation device that generates a template to be used in a matching device that matches a measurement image showing the results of measuring a measurement range including a specified object with a template indicating features of the specified object, the template generation device comprising: a first generation means that generates a projection image that shows the specified object as a two-dimensional image based on a three-dimensional model of the specified object; a feature acquisition means that acquires, based on the three-dimensional model or the projection image, a feature corrected in accordance with the attention level of each area of ​​the three-dimensional model; and a second generation means that generates the template indicating the feature corresponding to the projection image.

[0009] With this configuration, the matching device that uses the template generated by the template generating device can match the template with an appropriate feature amount according to the attention level of the region, thereby enabling highly accurate matching between the template and the measurement image.

[0010] The above template generating device may further include a setting means for setting a first region in the three-dimensional model that is less noticeable than other regions, and the feature amount acquiring means may acquire a feature amount in which the feature of the first region is suppressed. According to this, a matching device using a template generated by the template generating device may emphasize the region in the template that is less noticeable. Therefore, the template and the measurement image can be matched with high accuracy.

[0011] In the above template generating device, the setting means may set an area designated by a user as the first area.

[0012] In the template generating device, the setting means may set, as the first region, a region in the predetermined object where a ratio of regions having the same characteristics in the predetermined object is equal to or greater than a first value. This makes it possible to deemphasize regions that are commonly present in the predetermined object in matching. Therefore, it is possible to match the template and the measurement image with high accuracy.

[0013] In the above template generating device, the setting means may set a second region in the three-dimensional model that is more noticeable than the first region, and the feature acquiring means may acquire a feature that emphasizes the feature of the second region. In this way, a matching device using the template generated by the template generating device can perform matching by placing emphasis on the more noticeable region of the template. Therefore, the template and the measurement image can be matched with high accuracy.

[0014] In the template generating device, the setting means may set a region having a concave-convex shape as the second region. This allows the region having a concave-convex shape, which is generally a characteristic region, to be emphasized in matching. Therefore, the template and the measurement image can be matched with high accuracy.

[0015] In the template generating device, the setting means may set, as the second region, a region in the predetermined object where a ratio of regions having the same characteristics in the predetermined object is equal to or less than a second value. This allows emphasis to be placed on characteristic regions in the predetermined object in matching. Therefore, the template and the measurement image can be matched with high accuracy.

[0016] In the above template generating device, the measurement image may be a distance image in which each pixel represents a distance to a subject.

[0017] In the above template generating device, the first generating means may generate a plurality of the projection images each expressing the predetermined object in a different posture as a two-dimensional image, and the second generating means may generate the template for each of the plurality of projection images. This makes it possible to generate templates of the predetermined object seen from various viewpoints, thereby enabling the matching device to perform matching using templates of many patterns.

[0018] The present invention may be a comparison system comprising: the template generating device; and a comparison device that compares the template generated by the template generating device with a measurement image that represents a result of measuring a measurement range.

[0019] In the above matching system, the matching device may include an estimation means for estimating a position and orientation of the predetermined object based on the measurement image, a determination means for determining an overlapping region in which another object overlaps the predetermined object in the measurement image based on the position and orientation of the predetermined object estimated by the estimation means, and an update means for updating the template so as to suppress features of a region of the template corresponding to the overlapping region. This allows the matching device to deemphasize a region in which the predetermined object is hidden by another object (a so-called occlusion region), which should not be emphasized in matching. Therefore, the template and the measurement image can be matched with high accuracy.

[0020] A matching device according to one aspect of the present invention is characterized by having an acquisition means for acquiring a measurement image showing a result of measuring a measurement range including a predetermined object and a template showing a feature amount of the predetermined object, an estimation means for estimating a position and orientation of the predetermined object based on the measurement image, a determination means for determining an overlapping area in which another object overlaps the predetermined object in the measurement image based on the position and orientation of the predetermined object estimated by the estimation means, an update means for updating the template so as to suppress the features of an area of ​​the template corresponding to the overlapping area, and a matching means for matching the template updated by the update means with the measurement image. According to this, the matching device can deemphasize an area (so-called occlusion area) where the predetermined object is hidden by another object, which should not be emphasized in matching. Therefore, the template and the measurement image can be matched with high accuracy.

[0021] The present invention may be understood as an apparatus having at least a part of the above means, or as an electronic device, a control system, an information processing system, an information processing device, a recognition device, or a recognition system. The present invention may also be understood as a control method, a template generation method, a matching method, or a recognition method including at least a part of the above processing. The present invention may also be understood as a program for realizing such a method, or a recording medium (storage medium) on which the program is non-temporarily recorded. Note that the above means and processing can be combined with each other as much as possible to constitute the present invention. Effect of the Invention

[0022] According to the present invention, when matching an object template with an image, matching accuracy is improved. [Brief description of the drawings]

[0023] [Figure 1] FIG. 1 is a diagram illustrating a collation system according to the first embodiment. [Diagram 2] FIG. 2 is a configuration diagram of the template generating device and the matching device according to the first embodiment. [Diagram 3] FIG. 3 is a flowchart of the process of the template generating device according to the first embodiment. [Figure 4] FIG. 4 is a flowchart of the process of the collation device according to the first embodiment. [Diagram 5] FIG. 5 is a diagram illustrating a verification system according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings.

[0025] <Application Examples> In the following, a matching system 1 that matches a template indicating the feature amount of an object based on a 3D model or 2D image of the object with an image (measurement image) obtained by measuring the object with an imaging device obtains a template with the feature amount corrected according to the attention level. For example, the matching system 1 corrects the feature amount so as to suppress the feature of an area with low attention level, and corrects the feature amount so as to emphasize the feature of an area with high attention level. Then, the matching system 1 calculates the result of matching the template with the measurement image.

[0026] Here, a region with high attention is, for example, a region with uneven shape, a region colored with multiple colors, or other distinctive region. A region with low attention is a region where there are many other regions with similar characteristics, a region where other objects are likely to overlap and hide the region, etc. These regions can be set by a user's instruction (operation; designation).

[0027] According to this, the matching system 1 can perform matching without placing emphasis on areas of the template with low attention. Therefore, the matching system 1 can appropriately match the measurement image with the template by matching using areas with high attention without placing emphasis on areas with low attention. Therefore, the template and the measurement image can be matched with high accuracy.

[0028] <Embodiment 1> [Matching system configuration] The configuration of a matching system 1 according to the first embodiment will be described with reference to Fig. 1. The matching system 1 recognizes the position and orientation of the object 2 in the measurement image by matching a template for the object 2 with a measurement image obtained by measuring the object 2. The matching system 1 has an image sensor 10, a template generating device 20, a matching device 30, and a storage device 40. Note that while the position and orientation refers to the position and orientation in this embodiment, it may be the position or the orientation as long as no technical contradiction occurs.

[0029] The imaging sensor 10 acquires a measurement image by measuring a measurement range including the object 2. In this embodiment, the imaging sensor 10 is a three-dimensional sensor that acquires an image of the subject captured through a left lens and an image of the subject captured through a right lens, and acquires a distance image as a measurement image by comparing the difference between the two images (left and right images). In the distance image, each pixel indicates the distance from the imaging sensor 10 to the subject. The imaging sensor 10 may acquire the distance image by any method such as triangulation measurement or a ToF (Time of Flight) method. The measurement image may be a temperature image in which each pixel indicates the temperature of the subject, or may be a normal optical image (an image expressing the color and brightness of the subject).

[0030] The template generating device 20 generates a template indicating feature quantities of the object 2 based on a 3D model of the object 2 measured in advance or a 3D model of the object 2 used when designing the object 2. The 3D model (3D data) may be data that represents the object 2 using point cloud data.

[0031] The matching device 30 performs matching processing using the measurement image acquired by the image sensor 10 and the template generated by the template generation device 20. Furthermore, the matching device 30 recognizes the current position and orientation of the object 2 based on the matching result. Therefore, the matching device 30 can also be said to be a recognition device that recognizes the position and orientation of the object 2.

[0032] The storage device 40 stores (records) the 3D model of the object 2, the template generated by the template generating device 20, the matching result obtained by the matching device 30, and / or information on the position and orientation of the object 2 recognized by the matching device 30. The storage device 40 may be a server having a hard disk (HDD) or memory (RAM; Random Access Memory), or the like. The storage device 40 may also be a storage medium that is removable from the template generating device 20 and the matching device 30. The storage device 40 stores a plurality of templates that represent feature amounts of the object 2 in different orientations (the object 2 viewed from different viewpoints).

[0033] (Internal configuration of the template generation device) The internal configuration of the template generating device 20 will be described with reference to Fig. 2. The template generating device 20 has a control unit 201, an information acquiring unit 202, a region setting unit 203, a projection image generating unit 204, a feature amount calculating unit 205, a template generating unit 206, and an information output unit 207.

[0034] The control unit 201 controls each functional unit of the template generating device 20 according to a program non-temporarily stored in a storage medium.

[0035] The information acquisition unit 202 acquires a three-dimensional model of the object 2. The information acquisition unit 202 may acquire the three-dimensional model of the object 2 from the storage device 40 or from another external device. The information acquisition unit 202 also acquires imaging parameters of the image sensor 10 (camera focal length, center coordinates of the image, and lens distortion correction coefficient).

[0036] The region setting unit 203 sets regions with high attention levels (attention regions) and regions with low attention levels (non-attention regions) in the three-dimensional model.

[0037] The projection image generating unit 204 generates a projection image by converting a three-dimensional model into a two-dimensional image. Specifically, the projection image generating unit 204 generates a projection image that represents the object 2 in each orientation when measured by the image sensor 10 using a two-dimensional image. At this time, since the measurement image changes depending on the imaging parameters of the image sensor 10, the projection image generating unit 204 generates a projection image corrected by the imaging parameters.

[0038] The feature amount calculation unit 205 (feature amount acquisition unit) calculates (acquires) the feature amount of each pixel (each region) in the projection image based on the three-dimensional model or the projection image. Here, the feature amount can be an edge feature amount (edge ​​direction histogram) or a normal feature amount (normal direction histogram). However, the feature amount is not limited to this, and may be distance information, temperature information, or color information. Then, when a region with high attention (attention region) and a region with low attention (non-attention region) are set, the feature amount calculation unit 205 acquires the feature amount of the region corrected according to the attention level.

[0039] The template generating unit 206 generates a template which is a two-dimensional image in which each pixel indicates the feature amount calculated (corrected) by the feature amount calculating unit 205. The template generating unit 206 also adds, to the template, information on the posture of the object 2 corresponding to the projected image on which the template is based.

[0040] Information output unit 207 outputs the template generated by template generation unit 206 to storage device 40.

[0041] In this embodiment, the projection image generating unit 204 generates a plurality of projection images from a three-dimensional model. At this time, the plurality of projection images are images in which the object 2 in different postures (the object 2 viewed from different viewpoints) is represented by two-dimensional images. Thereafter, the feature amount calculating unit 205 calculates the feature amount for each of the plurality of projection images. Then, the template generating unit 206 generates a template for each of the plurality of projection images, and the information output unit 207 outputs the plurality of templates to the storage device 40.

[0042] (Internal configuration of the verification device) The internal configuration of the matching device 30 will be described with reference to Fig. 2. The matching device 30 has a control unit 301, an image acquisition unit 302, an information acquisition unit 303, a feature calculation unit 304, an area setting unit 305, a template update unit 306, a matching unit 307, a recognition unit 308, and a result output unit 309.

[0043] The control unit 301 controls each functional unit of the verification device 30 according to a program non-temporarily stored in a storage medium.

[0044] The image acquiring section 302 acquires a measurement image from the imaging sensor 10. Note that the image acquiring section 302 does not need to acquire a measurement image from the imaging sensor 10, and may acquire a measurement image stored in the storage device 40, for example.

[0045] The information acquisition unit 303 acquires a plurality of templates from the storage device 40 .

[0046] The feature amount calculation unit 304 calculates the feature amount of each pixel (each region) of the measurement image.

[0047] The region setting unit 305 determines a region of the object 2 in the measurement image that is hidden by other objects (occlusion region; overlapping region) according to the position and orientation of the object 2 roughly estimated by the control unit 301 (recognition unit 308). Then, the region setting unit 305 sets the occlusion region as a non-attention region having a low degree of attention.

[0048] The template update unit 306 updates the feature amounts in the multiple templates. Specifically, the template update unit 306 corrects (updates) the feature amounts in the templates for the regions set by the region setting unit 305 as non-attention regions (occlusion regions).

[0049] The matching unit 307 matches each of the multiple templates with the measurement image. Specifically, the matching unit 307 calculates the degree of match (similarity) between the feature amount of the template and the feature amount of the measurement image. If the evaluation value is greater than a predetermined value, the matching unit 307 determines that the matching is successful. On the other hand, if the evaluation value is equal to or less than the predetermined value, the matching unit 307 determines that the matching is unsuccessful. Note that any matching method using the feature amount may be used.

[0050] The recognition unit 308 recognizes the position and orientation of the object 2 at the time of capturing an image of the object 2 from the template when the matching unit 307 determines that matching has been successful. Specifically, the recognition unit 308 can recognize the position and orientation of the object 2 at the time of capturing an image, based on the information on the position and orientation of the object 2 added to the template. Furthermore, as will be described later, the recognition unit 308 may recognize the position and orientation of the object 2 in more detail by fitting a three-dimensional model of the object 2, the position and orientation of which is set as an initial value using the template, to the object 2 in the measurement image.

[0051] The result output unit 309 outputs the matching result and the recognition result of the position and orientation of the object 2 to the storage device 40 or an external device. For example, if the result output unit 309 outputs the recognition result to a robot for driving the object 2, the robot can perform control to place the object 2 in a predetermined position and orientation according to the position and orientation of the object 2. The result output unit 309 may also output information on the updated template to the storage device 40, etc.

[0052] Furthermore, the template generating device 20 and the matching device 30 can be configured by a computer including, for example, a CPU (processor), a memory, a storage, and the like. In this case, the configuration shown in FIG. 2 is realized by loading a program stored in the storage into the memory and executing the program by the CPU. Such a computer may be a general-purpose computer such as a personal computer, a server computer, a tablet terminal, or a smartphone, or may be an embedded computer such as an on-board computer. Alternatively, all or a part of the configuration shown in FIG. 2 may be configured by ASIC, FPGA, or the like. Alternatively, all or a part of the configuration shown in FIG. 2 may be realized by cloud computing or distributed computing.

[0053] Furthermore, the template generating device 20 may have each functional unit of the matching device 30. That is, the template generating device 20 may generate (update) a template according to a measurement image, and may perform matching and recognition processing.

[0054] [Template generator processing] The process of template generating device 20 will be described in detail with reference to the flowchart of Fig. 3. Each process of the flowchart of Fig. 3 is performed by control unit 201 operating according to a program. This is how it is achieved.

[0055] In step S1001, the control unit 201 controls the information acquisition unit 202 to acquire a three-dimensional model of the object 2 and imaging parameters of the imaging sensor 10. Here, the three-dimensional model may be a three-dimensional model obtained by measuring the object 2 in advance, as described above, or a three-dimensional model (CAD data) of the object 2 used when designing the object 2.

[0056] In step S1002, the control unit 201 determines whether or not it is possible to identify an attention region or a non-attention region in the object 2. For example, if the user has input information on the attention region or the non-attention region in advance, the control unit 201 determines that it is possible to identify the attention region or the non-attention region in the object 2. If it is determined that it is possible to identify the attention region or the non-attention region in the object 2, the process proceeds to step S1003, and if not, the process proceeds to step S1005.

[0057] Also, in step S1002, the control unit 201 may determine whether or not it is possible to identify an attention region or a non-attention region based on the three-dimensional model of the object 2. Specifically, the control unit 201 may determine that it is possible to identify an attention region or a non-attention region if there is a region to be set as an attention region or a non-attention region. For example, if there is a region with a special feature (unique feature) such as an uneven shape in the three-dimensional model of the object 2, the control unit 201 should set the region as an attention region. Alternatively, if the area (surface area) of a region having a certain first feature in the three-dimensional model of the object 2 is a first ratio or more (e.g., 70% or more) of the area (surface area) of the entire region of the object 2 of the three-dimensional model, the control unit 201 should set the region having the first feature as a non-attention region. Furthermore, if the area of ​​a region having a certain second feature in the three-dimensional model of the object 2 is a second ratio or less (e.g., 10% or less) of the area of ​​the entire region of the object 2 of the three-dimensional model, the control unit 201 should set the region having the second feature as an attention region.

[0058] In step S1003, the control unit 201 controls the region setting unit 203 to set a region of interest and / or a region of non-interest. Specifically, if information on a region of interest or a region of non-interest is input in advance by the user, the region setting unit 203 sets the region of interest or a region of non-interest according to the information. Also, if the control unit 201 determines a region to be set as a region of interest or a region of non-interest in step S1002, the region setting unit 203 sets the region as a region of interest or a region of non-interest. Note that the region setting unit 203 may set a degree of attention (importance) for each region in the projection image instead of setting a region of interest or a region of non-interest.

[0059] In step S1004, the control unit 201 controls the projection image generation unit 204 to generate, based on the three-dimensional model, a plurality of two-dimensional images corresponding to each posture of the object 2 as projection images. That is, the plurality of projection images are two-dimensional images of the object 2 viewed from each viewpoint around it. Step S1005 is similar to step S1004.

[0060] In step S1006, the control unit 201 controls the feature amount calculation unit 205 to extract (acquire) a feature amount for each pixel (region) of each projection image based on the 3D model or the projection image. Here, the feature amount may be a normal feature amount representing a normal vector or an edge feature amount representing an edge vector, but may be any feature amount.

[0061] In step S1006, the feature amount calculation unit 205 acquires (calculates) feature amounts corrected according to the attention level of each region of the three-dimensional model. Specifically, the feature amount calculation unit 205 acquires feature amounts for non-attention regions by performing correction (filter processing) to suppress features extracted from the three-dimensional model or the projected image. For example, the feature amount calculation unit 205 The feature amount is corrected by multiplying the feature amount extracted from the model or the projected image by a predetermined value (a positive number less than 1) corresponding to the attention level, or by obtaining the average feature amount of the area surrounding the non-attention area. Alternatively, the feature amount calculation unit 205 may extract the feature amount from the non-attention area after performing a blurring process on the non-attention area in the three-dimensional model.

[0062] Here, for example, when edge features are used in matching between a template and a measurement image, matching processing is performed by associating regions (positions) that are similar to each other and have prominent (strong edge strength) edge features. However, if the edge features of a non-attention region of the template are prominent, there is a possibility that a region (position) in the measurement image that is not a region (position) that should be associated with the non-attention region of the template may be associated with the non-attention region of the template. In other words, there is a possibility that matching accuracy may decrease. In contrast, as in this embodiment, if the feature calculation unit 205 performs correction (filter processing) to suppress features of the non-attention region and acquires the feature, the edge feature of the non-attention region is reduced. Therefore, the possibility that the edge feature of the non-attention region is used for matching is reduced, or the possibility that the edge feature affects matching is reduced. In other words, the accuracy of matching using a template is improved.

[0063] On the other hand, the feature amount calculation unit 205 may acquire the feature amount by performing correction so as to emphasize the feature extracted from the three-dimensional model or the projection image for the region of interest. According to this, for example, when an edge feature amount is used for matching, the feature amount of the region of interest is more likely to be used in matching. Therefore, matching using a feature amount suitable for matching becomes possible, and the accuracy of matching using a template is improved. Note that the degree of suppression or emphasis of the feature may be a degree according to the degree of attention.

[0064] In step S1007, the control unit 201 controls the feature amount calculation unit 205 to extract (acquire) feature amounts for each pixel (region) of each projection image based on the three-dimensional model or projection image, without making correction according to the attention level.

[0065] In step S1008, the control unit 201 controls the template generating unit 206 to generate a template for each of the multiple projection images. Specifically, the template generating unit 206 generates, as a template, a two-dimensional image in which each pixel indicates the feature amount acquired in step S1006 for each of the multiple projection images.

[0066] In step S1009, similarly to step S1008, the control unit 201 controls the template generating unit 206 to generate different templates for each of the multiple projection images.

[0067] In step S1010, the control unit 201 controls the information output unit 207 to output the plurality of templates generated by the template generation unit 206 to the storage device 40. As a result, the plurality of templates for the object 2 are stored in the storage device 40.

[0068] Note that after step S1004, the control unit 201 may set (determine) a non-attention area and an attention area for each of the multiple projection images according to the ratio of the area of ​​an area having the same characteristic to the area of ​​the object 2 appearing in the projection image. That is, the control unit 201 may set the area having a first characteristic as a non-attention area if the area of ​​an area having a certain first characteristic in the object 2 appearing in the projection image is equal to or greater than a first ratio (e.g., 70% or more) of the area of ​​the entire area of ​​the object 2 appearing in the projection image. Furthermore, the control unit 201 may set the area having the second characteristic as a attention area if the area of ​​an area having a certain second characteristic in the object 2 appearing in the projection image is equal to or less than a second ratio (e.g., 10% or less) of the area of ​​the entire area of ​​the object 2 appearing in the projection image. may be set to the region.

[0069] [Processing of the verification device] The process of the collation device 30 will be described in detail with reference to the flowchart of Fig. 4. Each process in the flowchart of Fig. 4 is realized by the control unit 301 operating according to a program.

[0070] In step S2001, the control unit 301 controls the image acquisition unit 302 to acquire a measurement image from the image sensor 10. Specifically, the image sensor 10 identifies corresponding pixels between the two left and right images and calculates the difference in position of the corresponding pixels. Then, the image sensor 10 measures the distance to the subject using a triangulation technique based on the difference in position of the corresponding pixels and the difference in position of the left and right lenses. This allows the image sensor 10 to acquire a measurement image that is, for example, a distance image having point cloud data.

[0071] Furthermore, the control unit 301 controls the information acquisition unit 303 to acquire a plurality of templates for the object 2 from the storage device 40. Each of the plurality of templates may be a template whose feature amount has been corrected in the template generation device 20, or a template whose feature amount has not been corrected in the template generation device 20. Furthermore, the control unit 301 may control the image acquisition unit 302 to acquire imaging parameters of the imaging sensor 10, or may control the information acquisition unit 303 to acquire a three-dimensional model of the object 2.

[0072] In step S2002, the control unit 301 controls the feature amount calculation unit 304 to calculate the feature amount from the measurement image (distance image).

[0073] In step S2003, the control unit 301 estimates the position and orientation of the object 2. For example, the control unit 301 controls the matching unit 307 and the recognition unit 308 to perform matching on the measurement image using a template that does not take into account information on non-attention areas or attention areas (feature amounts are not corrected), and estimates the position and orientation of the object 2. The control unit 301 may estimate the position and orientation of the object 2 based on information on the past position and orientation of the object 2. Here, since it is sufficient to roughly estimate the position and orientation of the object 2, any method may be used for estimation.

[0074] In step S2004, the control unit 301 determines whether or not there is an area in the measurement image where the object 2 is hidden by another object (occlusion area; overlapping area). The control unit 301 can determine the occlusion area based on preset position information of objects around the object 2 and the estimated position and orientation of the object 2. If it is determined that there is an occlusion area, the process proceeds to step S2005, and if not, the process proceeds to S2007.

[0075] In step S2005, the control unit 301 controls the region setting unit 305 to set the occlusion region determined in step S2004 as a non-attention region.

[0076] In step S2006, the control unit 301 controls the template update unit 306 to update the template. Specifically, the template update unit 306 corrects (updates) the feature amount for each template so as to suppress the features of the occlusion region (non-attention region) by treating the occlusion region as a non-attention region.

[0077] In this way, in steps S2005 and S2006, the region setting unit 305 sets a non-attention region in the template, and the template update unit 306 corrects the feature amount of the non-attention region of the template. However, in steps S2005 and S2006, the region setting unit 305 may set a non-attention region in the 3D model, and the template update unit 306 may acquire a new corrected feature amount from the non-attention region of the 3D model.

[0078] The processes of steps S2003 to S2006 do not necessarily have to be performed, and the process may proceed to step S2007 when the process of step S2002 ends.

[0079] In step S2007, the control unit 301 controls the matching unit 307 to match each of the multiple templates with the measurement image. Here, the matching unit 307 obtains a degree of match (similarity) by matching (comparing) the feature amount of the measurement image with the feature amount of the template. For example, the matching unit 307 may obtain the reciprocal 1 / Sum(D) of the total sum Sum(D) of the differences D of the feature amounts of corresponding pixels between the template and the measurement image as the degree of match.

[0080] In step S2008, the control unit 301 controls the recognition unit 308 to recognize the position and orientation of the object 2. Specifically, the recognition unit 308 recognizes the orientation corresponding to the template with the highest degree of match acquired by the matching unit 307 as the orientation of the object 2, and recognizes the position of the object 2 based on the position in the measurement image that most closely matches the template.

[0081] Furthermore, in order to recognize the position and orientation of the object 2 recognized by the template with higher accuracy, the recognition unit 308 may recognize the detailed position and orientation of the object 2 by fitting a three-dimensional model to the object 2 in the measurement image using the position and orientation of the object 2 recognized by the template as an initial value. In this case, the recognition unit 308 can fit the three-dimensional model to the measurement image by calculating the correspondence between each point of the three-dimensional model and each point of the object 2 in the measurement image. For example, an ICP (Iterative Closest Point) algorithm can be used to calculate the correspondence between each point.

[0082] In step S2009, the control unit 301 controls the result output unit 309 to output the information on the position and orientation of the object 2 to the storage device 40. As a result, the information on the position and orientation of the object 2 is stored in the storage device 40.

[0083] According to this embodiment, the matching device 30 can perform matching using a template whose feature amount has been corrected according to the attention level. Therefore, the matching device 30 can give more importance to important areas (areas that require attention) in matching and less importance to unimportant areas in matching, so that a more accurate matching result can be obtained. Therefore, the matching device 30 can recognize the position and orientation of the object 2 with higher accuracy.

[0084] <Embodiment 2> In the second embodiment, a matching system 1 (robot control system) will be described in which a robot 60 controls the position and orientation of an object 3 to connect the object 3 to an object 2, as shown in Fig. 5. The matching system 1 has a robot control device 50 and a robot 60 in addition to the configuration of the matching system 1. In this embodiment, the configurations of the image sensor 10, the template generating device 20, the matching device 30, and the storage device 40 are the same as those in the first embodiment.

[0085] The robot control device 50 controls the posture of the robot 60 based on the recognition result of the position and posture of the object 2 (object 2 and grasped object 3). The robot control device 50 controls the robot 60 so as to connect the grasped object 3 grasped by the grasper 61 of the robot 60 to the object 2.

[0086] The posture of the robot 60 is controlled by the robot control device 50. The robot 60 has a gripping unit 61 that grips the object 3. The image sensor 10 is connected (fixed) to a part of the gripping unit 61. That is, in this embodiment, the image sensor 10 is connected (fixed) to the gripping unit 61 in response to a change in the posture of the robot 60. This changes the position and orientation of the image sensor 10. However, the position and orientation of the image sensor 10 may be fixed regardless of the orientation of the robot 60.

[0087] [Template generator processing] In the second embodiment, the template generating device 20 generates a template for the object 2 in the same manner as in the first embodiment. Then, the template generating device 20 executes the same processes (processes shown in the flowchart in FIG. 3 ) as those for generating a template for the object 2 for the object 3, as well. In this way, the template generating device 20 generates a template for the object 3.

[0088] [Processing of the verification device] In addition, since the processing of the matching device 30 differs from that of the first embodiment, the processing of the matching device 30 according to the second embodiment will be described in detail with reference to the flowchart of Fig. 4. Note that steps S2003 and S2004 will be described below. Regarding steps S2005 to S2009, while only "object 2" is processed in the first embodiment, it is sufficient to perform processing for "both object 2 and grasped object 3", and therefore the description thereof will be omitted. Furthermore, regarding steps S2001 and S2002, since they are the same as the steps with the same reference numerals in the first embodiment, the description thereof will be omitted.

[0089] In step S2003, the control unit 301 estimates the positions and orientations of the object 2 and the object to be grasped 3 by the same method as in the first embodiment.

[0090] In step S2004, the control unit 301 determines whether or not there is a region (occlusion region; overlap region) between the object 2 and the held object 3 in the measurement image where one is hidden by the other. Here, even if there is a region in the measurement image where either the object 2 or the held object 3 is hidden by the holding unit 61, the control unit 301 determines the region as an occlusion region. If an occlusion region is present, the process proceeds to step S2005, and if an occlusion region is not present, the process proceeds to step S2007.

[0091] In this embodiment, the robot control device 50 needs to control the posture of the robot 60 so that the object 2 and the object to be held 3 are connected. Therefore, the process in FIG. 4 (steps S2001 to S2009) is repeated until the collation device 30 determines that the object 2 and the object to be held 3 are connected based on the recognition results of the positions and postures of the object 2 and the object to be held 3.

[0092] In this embodiment, the matching device 30 can determine (set) an occlusion region (non-attention region) occurring between the object 2 and the held object 3 based on the estimated positions and orientations of the object 2 and the held object 3. This makes it possible to prevent the occlusion region occurring between the object 2 and the held object 3 from being given too much importance in matching. Therefore, the matching device 30 can recognize the position and orientation of the object 2 with higher accuracy. Furthermore, the matching system 1 (object movement system) can connect the held object 3 to the object 2 with high accuracy.

[0093] In the above-described embodiments, the matching system 1 sets the non-attention area and the attention area (attention level of each area) based on the user's instruction, a three-dimensional model, an occlusion area, etc., but the non-attention area and the attention area may be set by other methods. For example, the matching system 1 may use the non-attention area and the attention area when recognizing the position and orientation of the object 2 that is approximate to the actual position and orientation of the object 2, among the results of recognizing the position and orientation of the object 2 by randomly setting the non-attention area and the attention area multiple times (comprehensively), for subsequent recognition.

[0094] The interpretation of the claims is not limited solely to the matters described in the embodiment. The interpretation of the claims includes the scope of claims that are described in a way that a person skilled in the art can recognize that the problem to be solved by the invention can be solved, taking into account the common general technical knowledge at the time of filing.

[0095] (Appendix 1) A template generating device (20) that generates a template used in a matching device (30) that matches a measurement image showing a result of measuring a measurement range including a predetermined object with a template that indicates a feature amount of the predetermined object, the template generating device (20) comprising: a first generating means (204) for generating a projected image representing the predetermined object as a two-dimensional image based on a three-dimensional model of the predetermined object; A feature acquisition means (205) for acquiring a feature corrected according to the attention level of each area of ​​the three-dimensional model based on the three-dimensional model or the projected image; A second generating means (206) for generating the template indicating the feature amount corresponding to the projection image; A template generating device (20) comprising:

[0096] (Appendix 2) acquisition means (302, 303) for acquiring a measurement image representing a result of measuring a measurement range including a predetermined object and a template indicating a feature amount of the predetermined object; An estimation means (301) for estimating a position and orientation of the predetermined object based on the measurement image; a determination means (301) for determining an overlapping area in which another object overlaps with the predetermined object in the measurement image based on the position and orientation of the predetermined object estimated by the estimation means (301); an updating means (306) for updating the template so as to suppress features of a region of the template corresponding to the overlap region; a comparison means (308) for comparing the template updated by the update means (306) with the measurement image; A collation device (30) comprising:

[0097] (Appendix 3) A template generation method for generating a template used in a matching device (30) that matches a measurement image showing a result of measuring a measurement range including a predetermined object with a template indicating a feature amount of the predetermined object, comprising the steps of: A first generation step (S1004) of generating a projection image that represents the predetermined object as a two-dimensional image based on a three-dimensional model of the predetermined object; A feature acquisition step (S1006) of acquiring a feature corrected according to the attention level of each area of ​​the three-dimensional model based on the three-dimensional model or the projected image; A second generation step (S1008) of generating the template indicating the feature amount corresponding to the projection image; 13. A template generating method comprising:

[0098] (Appendix 4) An acquisition step (S2001) of acquiring a measurement image representing a result of measuring a measurement range including a predetermined object and a template indicating a feature amount of the predetermined object; an estimation step (S2003) of estimating a position and orientation of the predetermined object based on the measurement image; a determining step (S2004) of determining an overlapping area in which another object overlaps with the predetermined object in the measurement image based on the position and orientation of the predetermined object estimated in the estimating step; an updating step (S2006) of updating the template so as to suppress features of the region of the template corresponding to the overlap region; a matching step (S2007) of matching the template updated in the updating step with the measurement image; A matching method comprising: [Explanation of symbols]

[0099] 1: Matching system, 2: Object, 10: Imaging sensor, 20: Template generating device, 30: Collation device, 40: Storage device, 201: Control device, 202: Information acquisition unit, 203: area setting unit, 204: projection image generating unit, 205: feature amount calculating unit, 206: template generation unit, 207: information output unit, 301: control unit, 302: image acquisition unit, 303: information acquisition unit, 304: feature amount calculation unit, 305: area setting unit, 306: template update unit, 307: matching unit, 308: Recognition section, 309: Result output section

Claims

1. 1. A template generating device that generates a template used in a matching device that matches a measurement image that represents a result of measuring a measurement range including a predetermined object with a template that indicates a feature amount of the predetermined object, the template generating device comprising: a first generating means for generating a projection image representing the predetermined object as a two-dimensional image based on a three-dimensional model of the predetermined object; a feature acquisition means for acquiring a feature corrected according to the attention level of each area of ​​the three-dimensional model based on the three-dimensional model or the projected image; a second generating means for generating the template indicating the feature amount corresponding to the projection image; A template generating device comprising:

2. The method further includes a setting unit for setting a first region in the three-dimensional model that is less noticeable than other regions, the feature acquisition means acquires a feature in which the feature of the first region is suppressed; 2. The template generating apparatus according to claim 1.

3. the setting means sets an area designated by a user as the first area; 3. The template generating apparatus according to claim 2.

4. the setting means sets, as the first region, a region in the predetermined object in which a ratio of regions having the same characteristic in the predetermined object is equal to or greater than a first value; 4. The template generating device according to claim 2 or 3.

5. the setting means sets a second region in the three-dimensional model that is more noticeable than the first region; the feature amount acquiring means acquires a feature amount that emphasizes the feature of the second region; 5. The template generating device according to claim 2, wherein the template generating device generates a template.

6. The setting means sets a region having a concave and convex shape as the second region.

6. The template generating apparatus according to claim 5.

7. the setting means sets, as the second region, a region in the predetermined object in which a ratio of regions having the same characteristic in the predetermined object is equal to or less than a second value; 7. The template generating device according to claim 5 or 6.

8. The measurement image is a distance image in which each pixel represents a distance to a subject.

8. The template generating device according to claim 1,

9. The first generating means generates a plurality of the projection images each representing the predetermined object in a different posture as a two-dimensional image, The second generating means generates the template for each of the plurality of projection images.

9. The template generating device according to claim 1,

10. A template generating device according to any one of claims 1 to 9; a matching device that matches the template generated by the template generating device with the measurement image; A collation system comprising:

11. The collation device includes: an estimation means for estimating a position and orientation of the predetermined object based on the measurement image; a determination means for determining an overlapping area in which another object overlaps with the predetermined object in the measurement image based on the position and orientation of the predetermined object estimated by the estimation means; an updating means for updating the template so as to suppress features of a region of the template corresponding to the overlap region; having The collation system according to claim 10 .

12. an acquisition means for acquiring a measurement image representing a result of measuring a measurement range including a predetermined object and a template indicating a feature amount of the predetermined object; an estimation means for estimating a position and orientation of the predetermined object based on the measurement image; a determination means for determining an overlapping area in which another object overlaps with the predetermined object in the measurement image based on the position and orientation of the predetermined object estimated by the estimation means; an updating means for updating the template so as to suppress features of a region of the template corresponding to the overlap region; a comparison means for comparing the template updated by the update means with the measurement image; A collation device comprising:

13. 1. A template generation method for generating a template used in a matching device that matches a measurement image showing a result of measuring a measurement range including a predetermined object with a template indicating a feature amount of the predetermined object, the method comprising: a first generation step of generating a projection image representing the predetermined object as a two-dimensional image based on a three-dimensional model of the predetermined object; a feature amount acquisition step of acquiring a feature amount corrected according to the attention level of each region of the three-dimensional model based on the three-dimensional model or the projected image; a second generation step of generating the template indicating the feature amount corresponding to the projection image; A template generating method using a template generating device comprising:

14. A measurement image showing the result of measuring a measurement range including a predetermined object and a feature amount of the predetermined object are An acquisition step of acquiring a template representing the an estimation step of estimating a position and orientation of the predetermined object based on the measurement image; a determination step of determining an overlapping area in which another object overlaps with the predetermined object in the measurement image based on the position and orientation of the predetermined object estimated in the estimation step; updating the template to suppress features of a region of the template that corresponds to the overlap region; a matching step of matching the template updated in the updating step with the measurement image; A collation method using a collation device, comprising:

15. A program for causing a computer to execute each step of the template generating method according to claim 13.

16. A program for causing a computer to execute each step of the collation method according to claim 14.

Images