Object detection device

Abstract

The present invention provides an object detection device comprising: a model transformation unit that applies one or more transformation processes to a model image and a model feature to generate one or more transformed model images and one or more transformed model features; a first characteristic calculation unit that calculates a first characteristic related to a characteristic of a feature extraction process by a feature extraction unit, on the basis of a transformed model feature and a third image feature extracted from the transformed model image by the feature extraction unit; and an evaluation index calculation unit that calculates an evaluation index for the feature extraction process on the basis of one or more first characteristics calculated by the first characteristic calculation unit on the basis of one or more third image features and one or more transformed model features, wherein the feature extraction unit has a plurality of feature extraction processes, the first characteristic calculation unit calculates the one or more first characteristics on the basis of the third image feature and the transformed model feature for each of the plurality of feature extraction processes, and the evaluation index calculation unit calculates the evaluation index for determining the feature extraction process to apply to the detection process by the detection unit for each of the plurality of feature extraction processes.

Description

Object detection device

[0001] The present disclosure relates to an object detection device.

[0002] There is a known robot system that detects the position and orientation of an object based on an image captured by a visual sensor and has the robot perform a task on the object. In the detection process, image features that represent specific parts of the object are extracted from an image of the object whose position and orientation are known using a feature extraction process, and the image features are registered together with the position and orientation. For an object whose position and orientation is unknown, image features that represent specific parts of the object are similarly extracted from the captured image using a feature extraction process, and the position and orientation of the object are identified based on the amount of change in the position and orientation in the image obtained by comparing them with the registered image features (model features).

[0003] Patent Documents 1 and 2 describe devices that identify the position and orientation of an object by performing a matching process using edge points.

[0004] JP 2019-211903 A JP 2017-91079 A

[0005] The image features used for matching vary depending on the feature extraction process applied. For example, even with the same edge feature extraction process, the edge features extracted by a Sobel filter and a Laplacian filter differ. Furthermore, if the imaging conditions change and the appearance of the object changes, the image features output by the feature extraction process will also change. What is needed is an object detection device that can determine a feature extraction process that can stably detect objects even when the imaging conditions change.

[0006] One aspect of the present disclosure is an object detection device including: a feature extraction unit that extracts image features from an image; a model image receiving unit that receives a first image of an object whose position and orientation are known as a model image; a model feature storage unit that stores the first image features extracted from the model image by the feature extraction unit as model features; and a detection unit that identifies the position and orientation of the object by comparing second image features extracted by the feature extraction unit from a second image of the object whose position and orientation are unknown with the model features, wherein the object detection device further includes a model conversion unit that applies one or more conversion processes to the model image and the model features to generate one or more converted model images and one or more converted model features, respectively; an evaluation index calculation unit that calculates an evaluation index of the feature extraction processing based on one or more first characteristics calculated by the first characteristic calculation unit based on the one or more third image features and the one or more transformation model features, wherein the feature extraction unit has a plurality of feature extraction processes, the first characteristic calculation unit calculates the one or more first characteristics for each of the plurality of feature extraction processes based on the third image feature and the transformation model feature, and the evaluation index calculation unit calculates the evaluation index for each of the plurality of feature extraction processes to determine the feature extraction processing to be applied to the detection processing of the detection unit.

[0007] These and other objects, features and advantages of the present invention will become more apparent from the detailed description of exemplary embodiments of the invention illustrated in the accompanying drawings.

[0008] 6A is a diagram showing the device configuration of a robot system including an object detection device according to an embodiment; FIG. 6B is a functional block diagram of a robot control device; FIG. 6C is a diagram showing the result of applying a Sobel filter to an image of an object; FIG. 6D is a diagram showing the result of applying a Laplacian filter to an image of an object; FIG. 6E is a data flow diagram showing the flow of data in a feature extraction process determination process; FIG. 6F is a diagram showing an example of a conversion process; FIG. 6G is a flowchart showing the feature extraction process determination process together with FIG. 6B; FIG. 6H is a flowchart showing the feature extraction process determination process together with FIG. 6A; FIG. 6H is a diagram showing an example of a list of evaluation indices; FIG. 6I is a diagram showing an example of processing using a Shi-Tomasi corner detector;

[0009] Next, embodiments of the present disclosure will be described with reference to the drawings. In the drawings, like components or functional parts are designated by like reference numerals. The scales of these drawings have been changed appropriately to facilitate understanding. Furthermore, the embodiment shown in the drawings is one example for implementing the present invention, and the present invention is not limited to the illustrated embodiment.

[0010] FIG. 1 is a diagram showing the equipment configuration of a robot system 100 including an object detection device 60 according to one embodiment. As shown in FIG. 1, the robot system 100 includes a robot 10, a robot control device 50 that controls the robot 10, a visual sensor 70, and a teaching device 40 connected to the robot control device 50. The visual sensor 70 is mounted on the tip of the arm of the robot 10. The visual sensor 70 is connected to the robot control device 50 and operates under control of the robot control device 50. The robot control device 50 functions as an object detection device 60 that detects the position and orientation of an object 1 placed on, for example, a workbench 2 based on an image captured by the visual sensor 70. The robot control device 50 can cause the robot 10 to perform a predetermined task on the object 1 based on the detected position and orientation of the object 1.

[0011] As an example, the robot 10 is a six-axis vertical articulated robot. However, various types of robots, such as a horizontal articulated robot, a parallel link robot, or a dual-arm robot, may be used as the robot 10 depending on the work target. The robot 10 can perform a desired task using an end effector attached to the wrist. The end effector is an external device that can be replaced depending on the application, such as a hand, a welding gun, or a tool. FIG. 1 shows an example in which a hand 33 is used as the end effector.

[0012] The visual sensor 70 may be a camera that captures two-dimensional images such as grayscale images or color images, or a stereo camera or three-dimensional sensor that can acquire distance images or three-dimensional point clouds. In this embodiment, the visual sensor 70 is assumed to be a camera that captures two-dimensional images. The robot control device 50 stores model data of the object and can perform a detection process to identify the position and orientation of the object by matching the image of the object in the captured image with the model data (pattern matching). In this embodiment, the visual sensor 70 is assumed to be calibrated, and the robot control device 50 is assumed to have calibration data that defines the relative positional relationship between the visual sensor 70 and the robot 10. This allows positions in the image captured by the visual sensor 70 to be converted to positions in a coordinate system (e.g., a robot coordinate system) fixed to the workspace.

[0013] The robot control device 50 controls the operation of the robot 10 in accordance with an operation program or commands from the teaching device 40. The robot control device 50 may have a hardware configuration as a general computer having a processor, memory (ROM, RAM, non-volatile memory, etc.), a storage device 152 (see FIG. 2), an operation unit, an input / output interface, a network interface, etc.

[0014] The teaching device 40 is used as an operation terminal for teaching the robot 10, performing various settings, displaying information, etc. A teaching operation panel, a tablet terminal, etc. may be used as the teaching device 40. The teaching device 40 may have a hardware configuration as a general computer having a processor, memory (ROM, RAM, non-volatile memory, etc.), a storage device, an operation unit, a display unit 41 (see FIG. 2 ), an input / output interface, a network interface, etc. The display unit 41 includes, for example, a liquid crystal display as a display device.

[0015] 2 is a functional block diagram of the robot control device 50. As shown in FIG. 2, the robot control device 50 has a function as an object detection device 60 in addition to a function for controlling the robot 10.

[0016] The robot control device 50 includes an operation control unit 151 and a storage device 152. The storage device 152 is a storage device formed, for example, of a non-volatile memory or a hard disk drive. The storage device 152 stores a robot program for controlling the robot 10, a program (vision program) for performing image processing such as workpiece detection based on images captured by the visual sensor 70, calibration data, various setting information, and the like.

[0017] The operation control unit 151 controls the operation of the robot in accordance with the robot program or commands from the teaching device 40. The robot control device 50 includes a servo control unit (not shown) that executes servo control of the servo motors of the respective axes in accordance with commands for the respective axes generated by the operation control unit 151.

[0018] The object detection device 60 has a function of detecting an object from an image captured by the visual sensor 70. In a detection process for detecting the position and orientation of an object, typically, feature processing such as a filter is performed on the input image to extract image features (such as edge points) of the object in the input image, and the extracted image features are compared with pre-prepared model data (image features) of the object to identify the position and orientation of the object in the input image. In a system for handling objects such as that shown in FIG. 1 , a filter for extracting edge features of the object may be used as a feature extraction process. However, even for the same edge feature extraction process, there are multiple types of filters, such as a Sobel filter and a Laplacian filter. The object detection device 60 according to this embodiment provides a function for automatically determining a feature extraction process suitable for reliably detecting a given object.

[0019] 2 , the object detection device 60 includes a visual sensor control unit 161, an image acquisition unit 162, a detection unit 163, a feature extraction unit 164, a model image reception unit 165, a matching area reception unit 166, a model feature storage unit 167, a model conversion unit 168, a stability calculation unit (first characteristic calculation unit) 169, a sensitivity calculation unit (second characteristic calculation unit) 170, an evaluation index calculation unit 171, and a display control unit 172. These functional blocks may be realized by a processor of the robot control device 50 executing a program. In this case, the components of the object detection device 60 in FIG. 2 correspond to the processor of the robot control device 50.

[0020] The visual sensor control unit 161 controls the operation of the visual sensor 70. For example, the visual sensor control unit 161 can control the visual sensor 70 in accordance with commands for the visual sensor 70 in an operation program. The image acquisition unit 162 has a function of acquiring image information obtained by the visual sensor 70 capturing an image within its field of view. In this embodiment, the image acquisition unit 162 acquires a two-dimensional image from the visual sensor 70.

[0021] The detection unit 163 can perform a detection process to identify the position and orientation of an object by comparing image features (hereinafter, these image features are referred to as the "second image") extracted by the feature extraction unit 164 from an image captured of an object whose position and orientation are unknown with model features of the object prepared in advance. The model features are, for example, image features extracted by the feature extraction unit 164 from an image captured of an object whose position and orientation are known.

[0022] The feature extraction unit 164 has multiple types of feature extraction processes for extracting image features from an image. The feature extraction processes include, for example, multiple filters for extracting edge features. The multiple types of filters include, for example, a Sobel filter and a Laplacian filter. Both the Sobel filter and the Laplacian filter are filters for extracting edge features of an object. However, these filters have different properties. The Sobel filter determines whether a pixel is an edge point based on the edge strength represented by the first derivative value in the x direction (horizontal direction) and the first derivative value in the y direction (vertical direction) of the image. On the other hand, the Laplacian filter determines whether a pixel is an edge point based on the second derivative value of the image. These differences in filtering properties result in different edge features as a result of the filtering process.

[0023] Here, examples of processing results when a Sobel filter and a Laplacian filter are applied to a model image of the same object are shown in Figures 3A (image GI1) and 3B (image GI2). The model image used here is image IG11 shown on the left side of Figure 5. However, in image IG11, details of the cast surface of object 90 are omitted. In Figures 3A (image GI1) and 3B (image GI2), edge points extracted by the filters are represented by black dots. As can be seen from Figures 3A (image GI1) and 3B (image GI2), the edge features (edge ​​points) extracted by each filter are different. In the processing result using the Laplacian filter in Figure 3B (image GI2), it can be seen that edge points are output well in the arc-shaped step portion 91 of object 90. In this case, it can be said that a Laplacian filter is suitable for matching the arc-shaped step portion 91 as a feature of the object to identify the position and orientation of the object.

[0024] In this way, the image features used in the detection process (matching) change depending on the feature extraction process applied. Furthermore, even when the same object is imaged, the appearance of the object changes depending on the object's properties and imaging conditions (including lighting, the angle at which the object is captured, etc.). Therefore, it is necessary to consider that the output results of filters such as those shown in FIGS. 3A and 3B also change depending on the object's properties and imaging conditions. As described below, the object detection device 60 according to this embodiment is configured to evaluate the properties of the feature extraction process for a certain object in response to changes in imaging conditions, and to determine a suitable feature extraction process to apply to the object.

[0025] The model image receiving unit 165 receives, as a model image, an image of an object whose position and orientation are known (hereinafter, this image will also be referred to as a "first image").

[0026] The matching area receiving unit 166 provides a function for receiving input specifying an area in the model image to be used for matching with the object. The matching area may be a part of the model image, or may be the entire model image. For example, the matching area receiving unit 166 receives a user operation for specifying an area in the model image to be used for matching with the object. For example, the matching area receiving unit 166 may display a captured image of the object on the display unit 41 of the teaching device 40 and provide a graphical user interface that receives an operation for specifying a matching area on the captured image using a pointing device or the like.

[0027] The model feature storage unit 167 stores, as model features, image features included in the matching region among the image features extracted from the model image by the feature extraction unit 164 (hereinafter also referred to as "first image features").

[0028] The model transformation unit 168 provides a function of applying one or more transformation processes to input image data. The transformation processes include one or more of a brightness change and a projective transformation. The projective transformation includes rotation, scale transformation, and the like. The model transformation unit 168 can output a "transformed model image" obtained by applying a transformation process to a model image. The model transformation unit 168 can also output "transformed model features" obtained by applying a transformation process to model features.

[0029] The image feature obtained by applying the feature extraction process to the transformed model image is referred to as the "third image feature."

[0030] The stability calculation unit 169 calculates a first characteristic related to the characteristics of the feature extraction process based on the third image characteristic and the transformation model characteristic extracted from the transformation model image by the feature extraction unit 164. The third image characteristic represents an image (transformed model image) obtained by applying a transformation (brightness transformation, projection transformation, etc.) by the model transformation unit to the model image, i.e., an image characteristic obtained by applying the feature extraction process to an image in which the appearance of the object has changed. Therefore, by comparing the transformation model characteristic and the third image characteristic, information representing the characteristics of the feature extraction process in response to changes in the image can be derived. The stability calculation unit 169 calculates, as the first characteristic, an index representing the stability of the feature extraction process in response to changes in the image. Hereinafter, this index will also be simply referred to as stability.

[0031] The sensitivity calculation unit 170 calculates a second characteristic related to the characteristics of the feature extraction process based on the third image characteristic and the transformation model characteristic extracted from the transformation model image by the feature extraction unit 164. The sensitivity calculation unit 170 calculates, as the second characteristic, an index representing the sensitivity of the feature extraction process by the feature extraction unit to changes in the image. Hereinafter, this index will also be simply referred to as sensitivity.

[0032] The evaluation index calculation unit 171 can calculate an evaluation index for determining a feature extraction process based on one or more stabilities calculated based on the third image feature and the transformation model feature. The evaluation index calculation unit 171 can also calculate an evaluation index for determining a feature extraction process based on one or more stabilities and one or more sensitivities.

[0033] The general flow of processing in the object detection device 60 from inputting a model image to calculating an evaluation index for determining a feature extraction process will be described with reference to the data flow diagram of FIG.

[0034] (Step 1) The user inputs a model image 201 obtained by capturing an object whose position and orientation are known. (Step 2) The user specifies an area on the model image 201 to be used for matching.

[0035] (Step 3) The object detection device 60 performs the following processes (step 3-1) to (step 3-4) for each feature extraction process of the plurality of feature extraction processes 164A. (Step 3-1) The feature extraction process is applied to the model image 201. (Step 3-2) The image feature in the matching area in the model image 201 is registered as the model feature 202 of the selected feature extraction process. (Step 3-3) The object detection device 60 performs the following processes (a) to (d) for each conversion process of the plurality of conversion processes 168A. (a) The conversion process is applied to the model image 201 to obtain a converted model image 203. (b) The feature extraction process is applied to the converted model image 203 to obtain a third image feature 204. (c) The conversion process is applied to the model feature 202 to obtain a converted model feature 205. (d) Stability and sensitivity are calculated (reference numeral 169A) from the image features (third image features 204) extracted from the transformation model image 203 and the transformed model features (transformation model features 205), and stored as the stability and sensitivity of the currently selected transformation process. (Step 3-4) Next, an evaluation index for determining a feature extraction process is calculated (reference numeral 171A) from multiple sets of stabilities and sensitivities corresponding to multiple transformation processes. (Step 4) Based on the evaluation index calculated for each feature extraction process in Step 3 above, the object detection device 60 determines the feature extraction process to be applied to the detection process. For example, the evaluation index calculation unit 171 may determine the feature extraction process with the highest evaluation index as the feature extraction process to be used for the detection process.

[0036] An example of the transformation process by the model transformation unit 168 is shown in Fig. 5. In Fig. 5, a transformed image IG12 is generated by applying projective transformation (including rotation and scale transformation) to a model image IG11 obtained by capturing an image of an object 90.

[0037] A specific example of the calculation of stability by the stability calculation unit 169 will be described. The stability calculation unit 169 calculates stability by the following process. Here, it is assumed that the image features are edge points. (Step k1) A correspondence relationship between the transformed model feature and the edge points included in each of the third image features extracted from the transformed model image is determined. (Step k2) The stability of the model feature is calculated based on the proportion of edge points included in the transformed model feature whose difference from the corresponding edge points included in the third image feature is smaller than a specific value.

[0038] In the above step k1, the correspondence between the edge points included in the transformation model feature and the third image feature may be determined, for example, using the following rule: (Rule) Define the edge points in the transformation model feature and the third image feature as points consisting of two variables: their position and the direction of the brightness gradient. Then, determine the edge points in the third image feature whose distance from an edge point in the transformation model feature (distance in a space defined by the two variables (position, brightness gradient)) is within a predetermined threshold as the edge points corresponding to the edge points of the transformation model feature.

[0039] In the above step k2, the stability calculation unit 169 calculates the stability as follows, for example: ｔｏｔａｌ The total number of edge points that appear in the third image feature and whose distance from the corresponding point of the transformation model feature is smaller than a specific value is defined as G １ In this case, the stability calculation unit 169 may calculate the stability as follows: (stability)=G １ / H ｔｏｔａｌ

[0040] From the above, stability can be said to be the rate at which the corresponding image feature of the third image feature is similar to the transformation model feature. In other words, stability is an index representing how many edge points are extracted at the target position (the position of the corresponding point in the transformation model feature) in the third image feature. In this way, the stability calculation unit 169 provides an index representing the stability of the image feature against changes (transformation) in the image, with respect to the image feature extracted by the feature extraction process to be evaluated.

[0041] A specific example of the calculation of sensitivity by the sensitivity calculation unit 170 will be described. The sensitivity calculation unit 170 calculates the sensitivity by the following process. Here, it is assumed that the image features are edge points. (Step m1) A correspondence relationship between the edge points included in the transformation model feature and each of the third image features extracted from the transformation model image is determined. (Step m2) The sensitivity of the model feature is calculated based on the proportion of image features whose difference from the corresponding edge points included in the transformation model feature is smaller than a specific value, among the edge points included in the third image feature.

[0042] The determination of corresponding points in the above step m1 is the same as in the above step k1.

[0043] In step m2, the sensitivity calculation unit 170 calculates the sensitivity as follows, for example: ｔｏｔａｌ Among them, the total number of edge points whose distance from the corresponding point of the transformation model feature is smaller than a specific value is defined as G １ In this case, the sensitivity calculation unit may calculate the sensitivity as follows: (sensitivity)=G １ / G ｔｏｔａｌ

[0044] From the above, sensitivity can be said to be an index that indicates how few image features other than those similar to the transformation model features are present in the third image features. In other words, sensitivity is an index that indicates the degree to which the image features extracted by the feature extraction process to be evaluated react sensitively to only the model features even if there is a change (transformation) in the image.

[0045] For example, the number of edge points that appear in the third image feature (G ｔｏｔａｌ ) increases, the likelihood that edge points will appear at the same positions of edge points of the transformation model feature increases, and stability tends to increase, but sensitivity tends to decrease.

[0046] 6A and 6B are flowcharts showing a process for determining a feature extraction process (feature extraction process determination process) executed in the object detection device 60. This process is executed under the control of a processor.

[0047] First, the model image receiving unit 165 receives an input of a model image acquired by capturing an image of an object whose position and orientation are known using the visual sensor 70 (step S1). Next, the matching area receiving unit 166 receives an operation by the user to specify a matching area on the model image corresponding to an area from which image features are to be extracted (step S2).

[0048] The series of processes from step S4 to S16 is repeatedly executed for each feature extraction process (loop 1 of step S3). The model feature storage unit 167 applies the feature extraction process to the model image (step S4). Then, the model feature storage unit 167 saves the image features in the matching region of the model image as model features for the feature extraction process currently selected in loop 1 (step S5).

[0049] The series of processes from steps S7 to S15 are repeated a predetermined number of times while changing the brightness conversion parameter (loop 2 of step S6), and the series of processes from steps S8 to S15 are repeated a predetermined number of times while switching the projection transformation parameter (loop 3 of step S7).

[0050] In step S8, the model transformation unit 168 applies a process to change the brightness of the model image. Furthermore, the model transformation unit 168 applies a projective transformation to the model image whose brightness has been changed (step S9). The projective transformation includes, for example, rotation and scale transformation of the model image.

[0051] Next, the feature extraction unit 164 applies the selected feature extraction process to the model image (transformed model image) to which the brightness change and projective transformation have been applied (step S10).

[0052] 6B , the model transformation unit 168 then applies projective transformation to the model features resulting from the selected feature extraction process (step S11). The stability calculation unit 169 then calculates the above-mentioned stability from the image features extracted from the transformed model image (third image features) and the model features to which projective transformation has been applied (transformed model features) (step S12). The stability calculation unit 169 then stores the calculated stability as one of the stabilities of the selected feature extraction process.

[0053] Next, in step S14, the sensitivity calculation unit 170 calculates the above-mentioned sensitivity from the image feature (third image feature) extracted from the transformation model image and the transformation model feature, and then stores the calculated sensitivity as one of the sensitivities for the selected feature extraction process (step S15).

[0054] The loop processing of Loop 2 and Loop 3 is executed for a plurality of brightness modification parameters and projection transformation parameters, and the number of stability and sensitivity values ​​generated is equal to the number of brightness modification parameters multiplied by the number of projection transformation parameters. When the loop processing of Loop 2 ends, the process proceeds to Step S16. In Step S16, the evaluation index calculation unit 171 calculates the evaluation index for the currently selected feature extraction process from the set of stability and set of sensitivity generated for that feature extraction process.

[0055] For example, suppose that n stabilities ST(i) and sensitivities SE(i) have been calculated for each selected feature extraction process (i is 1 to n). In this case, the evaluation index calculation unit 171 may use the average value, weighted harmonic mean, or other statistical quantity for the stabilities ST(1) to ST(n) and sensitivities SE(1) to SE(n) as the evaluation index for the selected feature extraction process.

[0056] When the loop processing by loop 1 is completed, the evaluation index calculation unit 171 determines, for example, the feature extraction process with the highest evaluation index among the evaluation indexes calculated for each of the multiple feature extraction processes as the feature extraction process to be used for the detection process (step S17).

[0057] In the above-described embodiment, by applying multiple types of transformations (brightness transformation, projection transformation) to calculate multiple stability and sensitivity values ​​and then calculating the evaluation index based on these, it becomes possible to select an appropriate feature extraction process and apply it to detection even in situations where the appearance, quality of the image, etc. of the object changes due to various environmental factors.

[0058] The feature extraction process determination process is configured to calculate an evaluation index for determining the feature extraction process based on the stability of image features against changes in the image. By applying the feature extraction process determined in this way to the detection process, stable detection becomes possible. Because the detection process is stabilized, it also becomes easier to adjust the detection parameters (detection score, etc.).

[0059] The feature extraction process determination process is configured to calculate an evaluation index for determining the feature extraction process based on the sensitivity of image features to changes in the image. By applying the feature extraction process determined in this way to the detection process, it becomes possible to stabilize detection. In this case, it is also possible to shorten the detection process time.

[0060] The feature extraction process determination process is configured to be able to calculate an evaluation index for determining the feature extraction process using a single image (model image), which has the advantage of reducing the user's effort required for the feature extraction process determination process.

[0061] In the embodiment described above, the evaluation index calculation unit 171 is configured to calculate an evaluation index for determining a feature extraction process based on stability and sensitivity, but the evaluation index calculation unit 171 may also calculate an evaluation index based on stability. In this case, it is possible to determine a feature extraction process that can stably perform detection even when an image changes. Alternatively, the evaluation index calculation unit 171 may calculate an evaluation index based on sensitivity. In this case, it is also possible to determine a feature extraction process that can provide stable detection.

[0062] In the above-described embodiment, when one or more pairs of stability and sensitivity are obtained by one or more transformation processes for a certain feature extraction process, a statistical quantity such as a weighted harmonic mean of those pairs is used as the evaluation index for that feature extraction process. As another operational example, the evaluation index calculation unit 171 may calculate an evaluation index for each of one or more transformation processes (i.e., for each pair of one or more third image features and one or more transformation model features) and determine the minimum value of the one or more calculated evaluation indexes as the evaluation index for the feature extraction process. By calculating an evaluation index for each feature extraction process in this manner and determining the feature extraction process to be used for detection, it is possible to determine a feature extraction process that can perform stable detection even in the worst case scenario where the object is poorly captured.

[0063] The display control unit 172 may operate to display the evaluation index for each feature extraction process calculated by the evaluation index calculation unit 171. FIG. 7 shows an example in which a list 250 showing the evaluation indexes for each feature extraction process (filter) is displayed on the display unit 41 by the display control unit 172. By displaying the evaluation indexes in this manner, the user can recognize which feature extraction process is suitable. By referring to this list 250, the user can also select the feature extraction process to be applied to detection. The list 250 may function as a user interface that accepts a user operation to select a feature extraction process to be used for detection from multiple feature extraction processes.

[0064] In the above embodiment, a filter that performs edge detection has been described as an example of a feature extraction process. However, the above-described method for determining a feature extraction process can be applied to various types of feature extraction processes. For example, a corner detector that detects corners in an image can be used. Examples of corner detectors include the Harris corner detector, the Shi-Tomasi corner detector, and the FAST (Features from Accelerated Segment Test) corner detector.

[0065] The Harris corner detector and Shi-Tomasi corner detector detect corners at points where the amount of change (E in the following equation (1)) when the image position is shifted is large. In the following equation (1), I(x, y) is the luminance of the pixel in the image, I(x + u, y + v) is the luminance of the pixel in the shifted image, and w(x, y) is the window function.

[0066]

[0067] The Harris corner detector and the Shi-Tomasi corner detector determine whether a corner exists from the eigenvalues ​​of M in the following equations (2) and (3), which are simplified versions of the above equations, but the determination methods are different.

[0068]

[0069]

[0070] An example of the results of processing an image of an object using the Shi-Tomasi corner detector is shown in Figure 8. Image IG20 shown in Figure 8 shows the results of applying the Shi-Tomasi corner detector to an image of an object 190 (part of the object is labeled 190). In image IG20, the detected corners CP are represented by black dots (only part of the object is labeled CP).

[0071] The FAST corner detector classifies whether a candidate point is actually a corner based on the difference in brightness values ​​of a 16-pixel circle. If the difference is continuously greater or smaller than a threshold, it is considered a corner. The FAST corner detector is said to be able to operate at high speed.

[0072] Due to these differences in the detection methods used by each corner detector, it is believed that, as with edge detectors, the extraction of image features by a corner detector is also affected by changes in the appearance of the object. Therefore, it is possible to calculate the stability and sensitivity described above, obtain evaluation indices, and determine a corner detector that provides stable detection. Therefore, the various processes used to determine the feature extraction process in the above-described embodiments can also be applied to corner detectors.

[0073] As described above, according to this embodiment, it is possible to determine a feature extraction process that can perform stable detection even in situations where the imaging conditions change.

[0074] In the above-described embodiment, an example of a configuration in which the object detection device is realized as a function of the robot control device is shown, but the object detection device can also be realized as an independent device separate from the robot control device. For example, the object detection device may be configured as an information processing device such as a personal computer connected to the robot control device.

[0075] The functional blocks of the robot control device shown in FIG. 2 may be realized by the processor of the robot control device executing various software stored in a storage device, or may be realized by a configuration mainly based on hardware such as an ASIC (Application Specific Integrated Circuit).

[0076] The programs that execute various processes such as the feature extraction process and determination process in the above-described embodiments can be recorded on various computer-readable recording media (e.g., semiconductor memories such as ROM, EEPROM, and flash memory, magnetic recording media, and optical discs such as CD-ROM and DVD-ROM).

[0077] Although the present disclosure has been described in detail, the present disclosure is not limited to the individual embodiments described above. Various additions, substitutions, modifications, partial deletions, etc. are possible in these embodiments without departing from the gist of the present disclosure or the spirit of the present disclosure derived from the content of the claims and their equivalents. These embodiments can also be implemented in combination. For example, in the above-described embodiments, the order of each operation and the order of each process are shown as examples and are not limited to these. The same applies when numerical values ​​or mathematical expressions are used in the description of the above-described embodiments.

[0078] The following additional notes are provided regarding the above embodiment and modifications. (Supplementary Note 1) An object detection device (60) comprising: a feature extraction unit (164) that extracts image features from an image; a model image reception unit (165) that receives, as a model image, a first image obtained by capturing an object whose position and orientation are known; a model feature storage unit (167) that stores, as a model feature, the first image feature extracted from the model image by the feature extraction unit; and a detection unit (163) that identifies the position and orientation of the object by comparing a second image feature extracted by the feature extraction unit from a second image obtained by capturing the object whose position and orientation are unknown with the model feature, wherein the object detection device (60) further comprises: a model conversion unit (168) that applies one or more conversion processes to the model image and the model feature to generate one or more converted model images and one or more converted model features, respectively; and a first characteristic calculation unit (169) that calculates a first characteristic related to the characteristic of the feature extraction process by the feature extraction unit, based on a third image feature extracted from the converted model image by the feature extraction unit and the converted model feature. and an evaluation index calculation unit (171) that calculates an evaluation index of the feature extraction process based on one or more first characteristics calculated by the first characteristic calculation unit based on the one or more third image features and the one or more transformation model features, wherein the feature extraction unit (164) has a plurality of feature extraction processes, the first characteristic calculation unit (169) calculates the one or more first characteristics for each of the plurality of feature extraction processes based on the third image feature and the transformation model feature, and the evaluation index calculation unit (171) calculates the evaluation index for each of the plurality of feature extraction processes to determine the feature extraction process to be applied to the detection process of the detection unit. (Supplementary Note 2) The object detection device (60) according to Supplementary Note 1, further comprising a matching area acceptance unit (166) that accepts part or all of the object captured in the model image as a matching area, and the model feature storage unit (167) stores image features included in the matching area from the first image features as the model features.(Supplementary Note 3) The object detection device (60) according to Supplementary Note 1 or 2, wherein the evaluation index calculation unit (171) determines a feature extraction process to be used by the feature extraction unit in the detection process by the detection unit (163) based on the evaluation index calculated for each of the plurality of feature extraction processes. (Supplementary Note 4) The object detection device (60) according to Supplementary Note 3, wherein the evaluation index calculation unit (171) determines a feature extraction process having the highest evaluation index among the evaluation indexes calculated for each of the plurality of feature extraction processes as the feature extraction process to be used by the feature extraction unit (164) in the detection process by the detection unit (163). (Supplementary Note 5) The object detection device (60) according to any one of Supplementary Notes 1 to 4, wherein the one or more conversion processes by the model conversion unit (168) include a process of converting brightness of the model image. (Supplementary Note 6) The object detection device (60) according to any one of Supplements 1 to 5, wherein the one or more transformation processes by the model transformation unit (168) include performing projective transformation on each of the model image and the model features. (Supplementary Note 7) The object detection device (60) according to any one of Supplements 1 to 6, wherein the first characteristic calculated by the first characteristic calculation unit (169) represents stability of the feature extraction process against changes in an image. (Supplementary Note 8) The object detection device (60) according to Supplementary Note 7, wherein the first characteristic calculation unit (169) determines a correspondence relationship between image features included in the transformation model features and the third image features, and calculates the first characteristic based on a proportion of image features included in the transformation model features that differ from corresponding image features included in the third image features by less than a specific value. (Supplementary Note 9) The object detection device (60) according to any one of Supplementary Notes 1 to 8, further comprising a second characteristic calculation unit (170) that calculates a second characteristic that indicates sensitivity to image changes of the feature extraction process by the feature extraction unit (164) based on the third image feature and the transformation model feature, and the evaluation index calculation unit (171) calculates the evaluation index for each of the plurality of feature extraction processes based on the one or more first characteristics and one or more second characteristics calculated by the second characteristic calculation unit (170) based on the one or more third image features and the one or more transformation model features.(Supplementary Note 10) The object detection device (60) according to Supplementary Note 9, wherein the second characteristic calculation unit (170) determines a correspondence relationship between the image features included in each of the transformation model features and the third image features, and calculates the second characteristic based on a proportion of image features included in the third image features that have a difference from a corresponding image feature included in the transformation model features that is smaller than a specific value. (Supplementary Note 11) The object detection device (60) according to Supplementary Note 9 or 10, wherein the evaluation index calculation unit (171) calculates, for each of the plurality of feature extraction processes, a weighted harmonic mean of the one or more first characteristics and the one or more second characteristics as the evaluation index. (Supplementary Note 12) The object detection device (60) according to any one of Supplements 1 to 11, wherein the evaluation index calculation unit (171) calculates, for each of the plurality of feature extraction processes, an index for determining the feature extraction process for each pair of the one or more third image features and the one or more transformation model features, and sets the minimum value of the indexes calculated for each pair as the evaluation index of the feature extraction process. (Supplementary Note 13) The object detection device (60) according to any one of Supplementary Notes 1 to 12, further comprising a display control unit (172) that displays the evaluation index calculated for each of the plurality of feature extraction processes on a display device.

[0079] 10 Robot 33 Hand 40 Teaching device 41 Display unit 50 Robot control device 60 Object detection device 70 Visual sensor 100 Robot system 151 Operation control unit 161 Visual sensor control unit 162 Image acquisition unit 163 Detection unit 164 Feature extraction unit 165 Model image reception unit 166 Matching area reception unit 167 Model feature storage unit 168 Model conversion unit 169 Stability calculation unit 170 Sensitivity calculation unit 171 Evaluation index calculation unit 172 Display control unit 201 Model image 202 Model feature 203 Transformed model image 204 Third image feature 205 Transformed model feature

Claims

1. a feature extraction unit that extracts image features from the image; a model image receiving unit that receives, as a model image, a first image obtained by capturing an object whose position and orientation are known; a model feature storage unit that stores the first image feature extracted from the model image by the feature extraction unit as a model feature; a detection unit that identifies a position and orientation of the object by comparing second image features extracted by the feature extraction unit from a second image capturing the object, the position and orientation of which are unknown, with the model features, The object detection device further comprises: a model transformation unit that applies one or more transformation processes to the model image and the model feature to generate one or more transformed model images and one or more transformed model features, respectively; a first characteristic calculation unit that calculates a first characteristic related to the characteristics of the feature extraction processing by the feature extraction unit based on the third image feature extracted from the transformation model image by the feature extraction unit and the transformation model feature; an evaluation index calculation unit that calculates an evaluation index of the feature extraction processing based on one or more first characteristics calculated by the first characteristic calculation unit based on the one or more third image features and the one or more transformation model features, the feature extraction unit has a plurality of feature extraction processes, the first characteristic calculation unit calculates the one or more first characteristics based on the third image feature and the transformation model feature for each of the plurality of feature extraction processes; The object detection device, wherein the evaluation index calculation unit calculates the evaluation index for determining which feature extraction process to apply to the detection process of the detection unit, for each of the plurality of feature extraction processes.

2. a matching area receiving unit that receives a part or all of the object shown in the model image as a matching area; The object detection device according to claim 1 , wherein the model feature storage unit stores, as the model feature, an image feature included in the matching region among the first image features.

3. The object detection device according to claim 1 , wherein the evaluation index calculation unit determines the feature extraction process to be used by the feature extraction unit in the detection process by the detection unit, based on the evaluation index calculated for each of the plurality of feature extraction processes.

4. 4. The object detection device according to claim 3, wherein the evaluation index calculation unit determines the feature extraction process having the highest evaluation index among the evaluation indexes calculated for each of the plurality of feature extraction processes as the feature extraction process to be used by the feature extraction unit in the detection process by the detection unit.

5. The object detection device according to claim 1 , wherein the one or more transformation processes performed by the model transformation unit include a process of transforming brightness of the model image.

6. The object detection device according to claim 1 , wherein the one or more transformation processes performed by the model transformation unit include performing a projective transformation on each of the model image and the model feature.

7. The object detection device according to claim 1 , wherein the first characteristic calculated by the first characteristic calculation unit represents stability of the feature extraction process against changes in an image.

8. The first characteristic calculation unit determining a correspondence between the transformation model features and the image features included in each of the third image features; The object detection device according to claim 7 , wherein the first characteristic is calculated based on a proportion of image features included in the transformation model features whose difference from corresponding image features included in the third image features is smaller than a specific value.

9. a second characteristic calculation unit that calculates a second characteristic representing sensitivity of the feature extraction process performed by the feature extraction unit to a change in an image based on the third image characteristic and the transformation model characteristic; 5. The object detection device according to claim 1, wherein the evaluation index calculation unit calculates the evaluation index for each of the plurality of feature extraction processes based on the one or more first characteristics and one or more second characteristics calculated by the second characteristic calculation unit based on the one or more third image features and the one or more transformation model features.

10. The second characteristic calculation unit determining a correspondence between the transformation model features and the image features included in each of the third image features; The object detection device according to claim 9 , wherein the second characteristic is calculated based on the proportion of image features included in the third image features whose difference from the corresponding image features included in the transformation model features is smaller than a specific value.

11. The object detection device according to claim 9 , wherein the evaluation index calculation unit calculates, for each of the plurality of feature extraction processes, a weighted harmonic mean of the one or more first characteristics and the one or more second characteristics as the evaluation index.

12. The evaluation index calculation unit, for each of the plurality of feature extraction processes, calculating an index for determining the feature extraction process for each pair of the one or more third image features and the one or more transformation model features; The object detection device according to claim 1 , wherein a minimum value of the indices calculated for the respective sets is set as an evaluation index for the feature extraction process.

13. The object detection device according to claim 1 , further comprising a display control unit that displays the evaluation index calculated for each of the plurality of feature extraction processes on a display device.