Image processing device, image processing method, and program

Abstract

An image processing device (1) uses a matching model to perform matching processing for identifying the arrangement of a detection target in a scene image, the image processing device comprising: a scene image acquisition unit (10) for acquiring a scene image; a disturbance occurrence determination unit (11) for estimating the type of disturbance included in the scene image; a model selection unit (12) for selecting, from among a plurality of candidates, one or more matching models to be used for the matching processing, on the basis of the type of disturbance estimated for the scene image; and a matching processing unit (15) for performing matching processing using the scene image and the matching model(s) selected for the scene image.

Description

Image processing device, image processing method and program

[0001] The present disclosure relates to an image processing device, an image processing method, and a program that perform matching processing.

[0002] In the field of image processing, pattern matching is known as a method for detecting the position of a target object in an image. In pattern matching, the feature values ​​of a pre-prepared matching model are compared with the feature values ​​of the target image to detect the position of the target object in the target image. A matching model is generated by cutting out an area around the target object from an image captured under ideal conditions. Since detection accuracy varies depending on which area is cut out from the image and what feature values ​​are used for matching, it is important to use an appropriate matching model.

[0003] Patent Literature 1 discloses a method for creating a plurality of provisional models, which are provisional matching models, using a reference image in which a detection target object is recorded, and determining which matching model to actually use from among the plurality of provisional models. In the method disclosed in Patent Literature 1, one matching model to use is determined based on matching scores obtained when pattern matching is performed with various combinations using each of the plurality of provisional models and each of a plurality of images in which the detection target object is recorded.

[0004] Japanese Patent Application Laid-Open No. 2020-77397

[0005] However, the technology disclosed in Patent Document 1 has a problem in that detection accuracy may decrease if a captured image contains disturbances. Types of disturbances include, for example, lighting changes, noise, and blur. Some features used in pattern matching can be extracted even when an image contains noise, but extraction becomes difficult due to lighting changes. Other features can be extracted even when lighting changes occur, but extraction becomes difficult when the image contains noise. With the technology disclosed in Patent Document 1, if an image used to determine a matching model contains multiple types of disturbances, the matching score of a provisional model that retains only features that are robust against all of the multiple types of disturbances will be high, and the amount of information contained in the matching model will be reduced, resulting in decreased detection accuracy.

[0006] The present disclosure has been made in view of the above, and aims to provide an image processing device that can suppress a decrease in detection accuracy even when an image includes disturbances.

[0007] In order to solve the above-mentioned problems and achieve the objective, the image processing device of the present disclosure is an image processing device that performs matching processing to identify the position of a detection target object in a scene image using a matching model, and is characterized by comprising: a scene image acquisition unit that acquires a scene image; a disturbance occurrence determination unit that estimates the type of disturbance contained in the scene image; a model selection unit that selects one or more matching models to be used for the matching processing from a plurality of candidates based on the type of disturbance estimated for the scene image; and a matching processing unit that performs matching processing using the scene image and the matching model selected for the scene image.

[0008] According to the present disclosure, it is possible to suppress a decrease in detection accuracy even when disturbances are included in an image.

[0009] 1 is a diagram showing an example of a system including an image processing device according to a first embodiment; an explanatory diagram of a matching model used by the image processing device; a diagram showing the functional configuration of the image processing device shown in FIG. 1; an explanatory diagram of a sub-region used by the image processing device; a flowchart for explaining a matching model generation procedure by the model generation unit; an explanatory diagram of the functions of the model generation unit; an explanatory diagram of classification processing performed by the model generation unit; an explanatory diagram of the definition of variance in the first embodiment; an explanatory diagram of the relationship between image rotation and extracted feature amounts; a flowchart for explaining a generation procedure of a matching model that takes into account the variation in feature amounts caused by image rotation; a flowchart for explaining a generation procedure of a matching model; a diagram showing an example of a model feature amount used in the generation procedure of the matching model shown in FIG. 11; a diagram showing an example of a model feature amount list created by the model generation unit; a diagram showing an example of a matching point generated by the model generation unit; a flowchart for explaining the procedure of matching processing; a diagram showing an example of a list generated by the matching processing unit; a diagram showing an example of a scene feature amount list generated by the matching processing unit; a scene image FIG. 1 is a diagram showing a first example of scene features extracted from an image; FIG. 2 is a diagram showing a second example of scene features extracted from a scene image; FIG. 3 is an explanatory diagram of transformation parameters; FIG. 4 is a diagram showing an error between a model matching point and a scene matching point; FIG. 5 is a diagram showing a functional configuration of an image processing device according to a first modified example of embodiment 1; FIG. 6 is a diagram showing a functional configuration of an image processing device according to a second modified example of embodiment 1;

[0010] An image processing device, an image processing method, and a program according to an embodiment of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0011] First Embodiment. FIG. 1 is a diagram illustrating an example of a system including an image processing device 1 according to a first embodiment. The image processing device 1 performs a matching process to identify the location of the detection target object 3 in an image including the detection target object 3 captured by an imaging device 2. The "location of the detection target object 3" refers to the position and orientation of the detection target object 3, and can be expressed, for example, by a position from a reference position and a rotation angle from a reference line. The image processing device 1 can output the identified location of the detection target object 3 to a work device 4. The work device 4 performs an operation determined according to the location of the detection target object 3. The work device 4 is, for example, an industrial robot, a driving device such as a servo mechanism, or the like. The detection target object 3 is placed on a transport device such as a conveyor 5 and moves in accordance with the movement of the conveyor 5. The imaging device 2 is fixed so as to capture an image of the surface of the conveyor 5 on which the detection target object 3 is placed. As the conveyor 5 moves, the detection target object 3 changes in the image captured by the imaging device 2.

[0012] The image processing device 1 performs matching processing using a matching model. The matching processing is processing for identifying the arrangement of a detection target object 3 in an image captured by the image capturing device 2. Information used to identify the arrangement of the detection target object 3 is called a matching model.

[0013] FIG. 2 is an explanatory diagram of a matching model 60 used by the image processing device 1. The matching model 60 is generated, for example, using a model image 63, which is an image obtained by cropping a partial region 62 from an image 61 captured of a detection target object 3. Furthermore, model feature quantities 65 extracted from a model shape 64 included in the model image 63 are used to generate the matching model 60. The model shape 64 may be, for example, a portion of the detection target object 3, or a marker attached to the captured object. Note that, although the model shape 64 is a single block here, the model shape 64 may also be composed of multiple blocks. A model reference position 66 is set within the matching model 60, and the matching process determines the position of the model reference position 66 in a scene image, which is an image of the detection target captured by the image capture device, and how many degrees the matching model 60 has rotated relative to the registered state. In FIG. 2, the top left point of the model image 63 is set as the model reference position 66.

[0014] Fig. 3 is a diagram showing the functional configuration of the image processing device 1 shown in Fig. 1. The image processing device 1 has a scene image acquisition unit 10, a disturbance occurrence determination unit 11, a model selection unit 12, a storage unit 13, a model generation unit 14, and a matching processing unit 15.

[0015] The scene image acquisition unit 10 acquires a scene image using an imaging device. The scene image is an image that is a target for searching for the detection target object 3 in the matching process. The scene image is obtained by using the imaging device to capture an image of an area where the detection target object 3 may be present. The scene image and the model image 63 may be grayscale images or color images. The scene image acquisition unit 10 outputs the acquired scene image to the disturbance occurrence determination unit 11.

[0016] The disturbance occurrence determination unit 11 estimates the type of disturbance included in the scene image output by the scene image acquisition unit 10. The disturbance occurrence determination unit 11 can acquire disturbance information, which is information used to estimate the type of disturbance included in the scene image, from the scene image.

[0017] For example, the disturbance occurrence determination unit 11 can acquire information about the brightness of the scene image as disturbance information for determining whether the scene image includes an illumination change. The information about brightness may be, for example, the maximum and minimum brightness values ​​of the scene image, or the difference between the maximum and minimum brightness values. For example, the disturbance occurrence determination unit 11 can apply a smoothing filter to the scene image to reduce the influence of noise, acquire the maximum and minimum brightness values ​​of the image, and record the acquired values ​​as disturbance information.

[0018] Furthermore, the disturbance occurrence determination unit 11 can acquire information about blur in the scene image as disturbance information for determining whether or not the scene image contains blur. The blur may include out-of-focus blur caused by a lens and camera shake blur caused by camera shake. The disturbance occurrence determination unit 11 can, for example, calculate a power spectrum of the scene image using a fast Fourier transform, and record the maximum frequency at which the power spectrum exceeds a certain intensity as disturbance information.

[0019] Furthermore, the disturbance occurrence determination unit 11 can acquire information about noise in the scene image as disturbance information for determining whether or not the scene image contains noise. The noise information is, for example, an S / N (Signal to Noise) ratio that indicates the intensity of noise contained in the scene image. The disturbance occurrence determination unit 11 can calculate the S / N ratio by using, for example, an image obtained by applying a smoothing filter to the scene image, and record the calculated S / N ratio value as disturbance information.

[0020] Although illumination change, blur, and noise have been cited here as types of disturbance, the present invention is not limited to these examples, and determination may be made for other disturbances as well. Disturbance information is also an example. The disturbance occurrence determination unit 11 can output the type of disturbance and the disturbance information as estimation results to each of the model selection unit 12 and the matching processing unit 15. The disturbance occurrence determination unit 11 may also output information indicating "no disturbance included" as the estimation result.

[0021] The disturbance occurrence determination unit 11 may acquire disturbance information from the entire scene image, or may divide the scene image into multiple sub-regions, acquire disturbance information for each sub-region, and output an estimation result for each sub-region.

[0022] FIG. 4 is an explanatory diagram of sub-regions 71-1, 71-2, and 71-3 used by the image processing device 1. The image processing device 1 can divide a scene image 70 into a plurality of sub-regions 71-1, 71-2, and 71-3. Hereinafter, when there is no need to distinguish between the sub-regions 71-1, 71-2, and 71-3, they will be simply referred to as sub-regions 71. The sub-regions 71 may be set so that multiple sub-regions 71 do not overlap, or, as in the example shown in FIG. 4, multiple sub-regions 71 may be set so that multiple sub-regions 71 overlap. Note that in FIG. 4, the sub-regions 71 are set so that adjacent sub-regions 71-1 and 71-2, and sub-regions 71-2 and 71-3 overlap in the horizontal direction, but they can also be set so that adjacent sub-regions 71 overlap in the vertical direction.

[0023] Returning to the description of FIG. 3 , the model selection unit 12 selects a matching model to be used in the matching processing unit 15 using the estimation result of the disturbance occurrence determination unit 11. Matching model candidates are assumed to have been generated in advance by the model generation unit 14 and stored in the storage unit 13. The procedure for generating a matching model will be described later. For each matching model candidate, multiple intervals are set for the value of the disturbance information of each disturbance type used by the disturbance occurrence determination unit 11, and a matching model corresponding to each interval is associated with it. For example, for a certain type of disturbance, intervals such as "less than 1," "1 or greater but less than 3," and "3 or greater but less than 5" are set so that the values ​​of the disturbance information do not overlap, and a matching model corresponding to each interval is associated with it in advance. In this case, the model selection unit 12 selects one matching model associated with an interval that includes the value of the disturbance information recorded by the disturbance occurrence determination unit 11. The model selection unit 12 does not select a common matching model for multiple scene images 70, but selects a matching model for each scene image 70. At this time, if the disturbance occurrence determination unit 11 outputs an estimation result for each scene image 70, the model selection unit 12 can select one matching model for each scene image 70. Alternatively, if the disturbance occurrence determination unit 11 outputs an estimation result for each sub-region 71, the model selection unit 12 can select one matching model for each sub-region 71 of the scene image 70. The model selection unit 12 obtains the selected matching model from the storage unit 13 and passes it to the matching processing unit 15.

[0024] The matching processing unit 15 performs matching processing to identify the arrangement of the detection target object 3 in the scene image 70 using the matching model selected by the model selection unit 12, and outputs information indicating the arrangement of the detection target object 3 in the scene image 70. The information indicating the arrangement of the detection target object 3 in the scene image 70 is, for example, information indicating the position and orientation of the detection target object 3, and can be expressed, for example, by position information of a model reference position 66 in the scene image 70 and rotation angle information based on the state in which the matching model 60 is registered. The matching processing unit 15 can output the position information and rotation angle information as processing results.

[0025] The position information and rotation angle information output by the matching processing unit 15 are input to the work device 4 shown in Fig. 1. The work device 4 can use the input position information and rotation angle information to perform a task related to the detection target object 3, such as a picking task.

[0026] Here, the procedure for generating a matching model in the model generation unit 14 will be described.

[0027] 5 is a flowchart for explaining the procedure for generating a matching model by the model generation unit 14. The model generation unit 14 acquires an image to be used for cutting out a model image (step S11). Here, the model generation unit 14 uses an image of the detection target object 3, the position of which is to be detected, captured by a system separate from the device at runtime. The image acquired here is preferably one that has been captured under shooting conditions with little disturbance and that allow the detection target object 3 to be clearly captured.

[0028] Next, the model generation unit 14 cuts out a portion to be used as a model image from the acquired image (step S12). The model generation unit 14 also extracts model features to be used for generating a matching model from the model image (step S13). For example, the model generation unit 14 can extract model features using a method based on geometric pattern matching. Specifically, an example will be described in which edge information included in the model image is used as the model features. The edge information is information that indicates the shape of the detection target object 3 in the model image, and here, the edge information is represented by geometric figures such as lines and circles. Therefore, the model features extracted by the model generation unit 14 are one or more geometric figures that indicate the shape of the detection target object 3.

[0029] First, the model generation unit 14 extracts only the contour portion from the model image using a known method such as the Canny algorithm, and generates contour information indicating the contour portion. Next, the model generation unit 14 extracts geometric figures such as lines and circles from the contour information using a known method such as the Houph transform. The model generation unit 14 extracts the extracted geometric figures as model features.

[0030] The model generation unit 14 generates multiple pseudo model images by adding a pseudo disturbance to the model image (step S14). If the added disturbance is noise, the model generation unit 14 may add Gaussian noise with different variance magnitudes to the model image in order to increase the variety of images with different noise levels. In this case, it is desirable that the model generation unit 14 generate multiple pseudo model images for one variance magnitude, taking randomness into consideration.

[0031] Furthermore, when the disturbance to be added is blur, in order to generate pseudo model images with different degrees of blur, the model generation unit 14 may apply Gaussian filters of various kernel sizes to the model image to generate multiple pseudo model images.

[0032] The type of disturbance applied by the model generating unit 14 may be a single type or a combination of multiple types.

[0033] The model generating unit 14 extracts features as pseudo model features from the pseudo model image in the same manner as the method used to extract the model features in step S13 (step S15).

[0034] Fig. 6 is an explanatory diagram of the function of the model generation unit 14. The model generation unit 14 generates a model image in step S12 of Fig. 5 and extracts model features in step S13 of Fig. 5. It also generates a pseudo model image in step S14 of Fig. 5 and extracts pseudo model features in step S15 of Fig. 5. Because the pseudo model image is obtained by adding a disturbance to the model image, the pseudo model features are affected by the disturbance and therefore partially differ from the model features, as shown in Fig. 6.

[0035] Returning to the description of Fig. 5, the model generating unit 14 classifies the pseudo model features, and calculates and records the variance indicating the degree of variation in the features (step S16).

[0036] 7 is an explanatory diagram of the classification process performed by the model generation unit 14. The model generation unit 14 compares the model features with the pseudo model features and classifies the model features into three types of features: (1) model features that match the pseudo model features, (2) model features that do not exist in the pseudo model features, and (3) model features that do not match but have similar pseudo model features. Here, the model features are geometric figures such as lines and circles included in edge information extracted from the model image. In other words, the above three types of classification can be expressed as: (1) geometric figures extracted from the model image that match the geometric figures extracted from the pseudo model image, (2) geometric figures extracted from the model image that do not exist in the pseudo model image, and (3) geometric figures extracted from the model image that do not match the geometric figures in the pseudo model image but have geometric figures similar to the pseudo model image. Here, a similar geometric figure is, for example, a geometric figure that is slightly displaced.

[0037] By using the model feature values ​​classified as (1) to generate a matching model, it is possible to generate a matching model that is robust against disturbances. However, depending on the strength of the disturbance, there may be a small number of model feature values ​​in (1), resulting in insufficient information for identifying the shape from within the scene image. For this reason, the model generation unit 14 generates a matching model using model feature values ​​classified as (3) in addition to model feature values ​​in (1).

[0038] When model features classified as (3) are used to generate a matching model, variations in the features occur, and therefore, when determining that the features are from the same part, it is necessary to have a range that takes the variations into account.

[0039] 8 is an explanatory diagram of the definition of variance in the first embodiment. One of the lines in the model feature is called a model line Lm1. Of the pseudo model feature extracted from a certain pseudo model image, the one corresponding to the position of the model line Lm1 is called a pseudo model line Lm1-α. Since the pseudo model image is generated by adding a known disturbance to the model image and using known transformation parameters, it is easy to associate the model line Lm1 with the pseudo model line Lm1-α.

[0040] A perpendicular line to the model line Lm1 is drawn from each of endpoints P1 and P2 of the model line Lm1 toward the pseudo model line Lm1-α, and the length of the perpendicular line extending from endpoint P1 is defined as d1, and the length of the perpendicular line extending from endpoint P2 is defined as d2. Specifically, the "length of the perpendicular line extending from endpoint P1" corresponds to the length from endpoint P1 to the intersection of the pseudo model line Lm1-α and the perpendicular line, and the "length of the perpendicular line extending from endpoint P2" corresponds to the length from endpoint P2 to the intersection of the pseudo model line Lm1-α and the perpendicular line.

[0041] Furthermore, among the pseudo model features extracted from a pseudo model image different from the pseudo model image from which the pseudo model line Lm1-α was extracted, the one corresponding to the position of the model line Lm1 is designated as the pseudo model line Lm2-α. As with the pseudo model line Lm1-α, for the pseudo model line Lm2-α, perpendicular lines to the model line Lm1 are drawn from each of the end points P1 and P2 toward the pseudo model line Lm2-α, and the length of the perpendicular line extending from the end point P1 toward the pseudo model line Lm2-α is designated as d3, and the length of the perpendicular line extending from the end point P2 toward the pseudo model line Lm2-α is designated as d4. For simplicity, while FIG. 8 shows a case where there are two pseudo model images, in reality the model generation unit 14 performs similar processing on all of the pseudo model images used to generate the matching model.

[0042] The variance of the model line Lm1 is the sum of the lengths of the longest of the two perpendicular lines pointing to each of the pseudo model lines Lm1-α and Lm2-α. In the example of Figure 8, since d1 < d2 and d3 < d4, the variance amount v1 = d2 + d4.

[0043] Returning to the explanation of Fig. 5, the model generation unit 14 generates a matching model based on the model feature amounts (step S18). Details of step S18 will be described later.

[0044] In Fig. 5, an example of generating a pseudo model image by adding noise and blur as disturbances has been described, but when variations in feature values ​​occur due to image rotation, the procedure is partially different from Fig. 5 and will be described below. Note that image rotation is also included as a disturbance.

[0045] This section explains the variation in feature values ​​that occurs due to image rotation. First, edge detection detects pixels whose brightness gradient is above a threshold, which adds discretization error to the contour of an actual object. Furthermore, the Houph transform detects shapes using a voting format with arbitrary angle and position increment parameters, so the detected values ​​contain discretization error derived from the voting parameters. The probabilistic Houph transform, a faster version of the Houph transform, outputs pixels that are believed to be the endpoints of lines, and is affected by discretization error derived from pixels. In particular, when detecting lines that are diagonal to the row or column direction of pixels in an image, if an attempt is made to detect a line that is longer than a certain length, the discretization error may result in the line being detected as multiple lines split along the way.

[0046] As mentioned above, when a scene image is rotated relative to a model image, differences may occur in the number and values ​​of feature quantities extracted from the model image and the scene image, even though they show the same location. These differences can cause a decrease in the accuracy of position detection.

[0047] 9 is a diagram illustrating the relationship between image rotation and extracted feature amounts. Here, pseudo model images are generated by rotating a certain model image by 0 degrees, 1 degree, and 4 degrees, and the pseudo model feature amounts extracted from the generated pseudo model images are shown, along with the pseudo model feature amounts obtained by reverse rotation. Comparing the pseudo model feature amounts after reverse rotation reveals that the lengths and slopes of the lines included in the pseudo model feature amounts differ in some areas.

[0048] Fig. 10 is a flowchart for explaining the procedure for generating a matching model that takes into account variations in feature values ​​caused by image rotation. Here, it is assumed that the process shown in Fig. 10 is executed after the process shown in Fig. 5, and that a model image has already been cut out from the original image and model feature values ​​have already been extracted.

[0049] The model generation unit 14 acquires a model image (step S12α). Here, the model image acquired is the one extracted from the original image in step S12 of FIG. 5. The model generation unit 14 rotates the model image at specified angle intervals to generate multiple pseudo model images (step S14α). The model generation unit 14 then extracts pseudo model features from the pseudo model images at each rotation angle (step S15α). The method for extracting the pseudo model features is similar to the method for extracting model features from the model image in step S13 of FIG. 5, for example.

[0050] The model generation unit 14 reversely rotates the extracted pseudo model features by a specified angle (step S17). Specifically, the model generation unit 14 reversely rotates the pseudo model features by the same angle as the model image was rotated when generating the pseudo model image in step S14α. For example, the model generation unit 14 rotates the pseudo model features extracted from a pseudo model image obtained by rotating the model image by 4 degrees by −4 degrees in step S17.

[0051] It is desirable that the rotation angles used in steps S14α and S17 be set in increments of angle finer than the recognition accuracy of the rotation angle required for position detection.

[0052] Next, the model generation unit 14 classifies the pseudo model features using a procedure similar to that described in step S16 of Figure 5, and classifies the model features into three types of features: (1) model features that match the pseudo model features, (2) model features that do not exist in the pseudo model features, and (3) model features that do not match but have similar pseudo model features, and calculates and records the variance of model features classified as (3) (step S16α).

[0053] Furthermore, processing errors may occur depending on the detection target object 3. For example, if the contour of the detection target object 3 is not a smooth straight line, a straight line on the drawing may be interrupted midway on the matching model. Regarding variations due to individual differences during processing, the accuracy conditions during processing and statistical results of multiple detection target objects 3 may be used to understand the variations in feature amounts.

[0054] In other words, the variance is a value indicating the degree of deviation that may occur due to processing errors or disturbances for each geometric figure of the model feature.

[0055] Finally, the model generating unit 14 generates a matching model based on the model feature amounts, the pseudo model feature amounts, and the variance calculated in step S16α (step S18).

[0056] Here, the process of generating a matching model in step S18 in FIGS. 5 and 10 will be described with reference to FIGS. 11 to 14. FIG.

[0057] 11 is a flowchart for explaining the procedure for generating a matching model. The model generation unit 14 first selects one reference line from the model features (step S181). It is preferable to select a reference line with a small variance.

[0058] Fig. 12 is a diagram showing an example of model features used in the procedure for generating the matching model shown in Fig. 11. Fig. 12 shows IDs (identifiers) L0 to L15 assigned to each of the geometric figures included in the model features. The reference line selected in step S181 is designated as L0. Note that, since all of the geometric figures are straight lines, this ID will be referred to as a line ID.

[0059] Returning to the description of Fig. 11, the model generation unit 14 lists the relative positions from the reference line L0 of the geometric figures other than the reference line L0 among the geometric figures that are model features indicated by the respective line IDs (step S182).

[0060] FIG. 13 is a diagram showing an example of a model feature list created by the model generation unit 14. The relative positions listed in step S182 can be expressed, for example, as the distance from the reference line L0 for straight lines parallel to the reference line L0, i.e., L1, L4, L5, L8, L9, L12, and L13 in the example of FIG. 12 . Furthermore, as for straight lines not parallel to the reference line L0, i.e., L2, L3, L6, L7, L10, L11, L14, and L15 in the example of FIG. 12 , the relative positions can be expressed as the angle between the reference line L0 and the straight line. Furthermore, although not present in the example of FIG. 12 , if the geometric figure is a circle, the distance between the center of the circle and the reference line can be recorded as the relative position.

[0061] Next, the model generation unit 14 sets an allowable variation for each model feature corresponding to each row of the model feature list (step S183). The allowable variation is a parameter used in the matching process described below. The allowable variation is a value indicating how much a scene feature, which is a feature extracted from a scene image, can deviate from a model feature of a matching model in the matching process and still be determined to be a corresponding feature. The smaller the value of the allowable variation, the smaller the allowable deviation.

[0062] The model generation unit 14 generates a matching model from the model feature (step S184). In the first embodiment, the intersections of the model lines and the centers of the model circles included in the model feature as shown in Fig. 14 are used as the matching model. These points are called matching points.

[0063] 14 is a diagram showing an example of matching points generated by the model generation unit 14. In this example, all model feature amounts are made up of straight lines, so the intersections of the model straight lines become matching points.

[0064] Here, we will explain the matching process of the image processing device 1. At the start of the matching process, it is assumed that the matching model has been generated by the model generation unit 14 according to the procedure described above, and that the matching model has been stored in the storage unit 13.

[0065] Here, an example is shown in which the variance of the feature amounts stored in the model generation unit 14 is used in the matching process. The matching processing unit 15 searches for a shape that is the same as the matching model in the scene image. Specifically, the matching processing unit 15 detects straight lines from the scene image and sets the detected straight lines as candidates for the reference line. Using the candidate reference lines included in the scene image as a position reference, if the scene image contains more than the required number of shapes other than the reference line L0 included in the matching model, the matching processing unit 15 determines that the candidate straight lines are the straight lines that correspond to the reference line L0. Through this process, the detection target object 3 is roughly positioned in the image.

[0066] Here, an index is needed to determine whether the points included in the matching model and the points created from the circles and lines detected in the scene image form the same shape. The more rigorous the index is with the model, the less likely it is that an object with a similar shape other than the detection target object 3 will be mistakenly identified as the detection target object 3. However, the probability of identifying the detection target object 3 as not being the detection target object 3 due to individual differences and discretization errors increases. Therefore, if the judgment index is made uniformly strict, it is possible to prevent erroneous recognition of similar shapes, but it is more likely that the desired detection target object 3 will not be detected due to individual differences and discretization errors. If the judgment index is made uniformly lenient, it will reduce the number of missed detection target object 3 positions, but it will increase the number of candidate positions for rough positioning, resulting in a longer search time. Therefore, in embodiment 1, the judgment conditions are changed for each feature, allowing both advantages to be enjoyed.

[0067] 15 is a flowchart illustrating the procedure of the matching process. First, the scene image acquisition unit 10 of the image processing device 1 acquires a scene image (step S21). The scene image acquired by the scene image acquisition unit 10 is output to the disturbance occurrence determination unit 11. The disturbance occurrence determination unit 11 identifies the type of disturbance in the scene image (step S22). The model selection unit 12 selects one matching model according to the type of disturbance identified by the disturbance occurrence determination unit 11 (step S23). Note that if the type of disturbance is identified for each divided region in the scene image in step S22, one matching model will be selected for each divided region in step S23. If the type of disturbance is identified for each scene image in step S22, one matching model will be selected for each scene image in step S23.

[0068] Next, the matching processing unit 15 extracts scene features from the scene image (step S24). The method by which the matching processing unit 15 extracts scene features is, for example, the same as the method by which the model generation unit 14 extracts model features from the model image, and geometric figures such as lines and circles are extracted as features. The matching processing unit 15 lists the scene features extracted from the scene image and generates a list such as that shown in FIG.

[0069] FIG. 16 is a diagram showing an example of a list generated by the matching processing unit 15. The list shown in FIG. 16 includes information indicating the positions of geometric figures extracted from a scene image. Here, since all of the extracted geometric figures are straight lines, information identifying each feature represented by a straight line is referred to as a line ID. As shown in FIG. 16, the matching processing unit 15 generates a list including line IDs identifying each scene line, which is a straight line extracted from a scene image as a scene feature, and position information for each of the two end points of the scene line, "point 1" and "point 2." If a scene circle, which is a circle extracted as a scene feature, is present, the matching processing unit 15 generates a list including the position of the center point of the scene circle.

[0070] Returning to the description of Fig. 15, the matching processing unit 15 then creates a list of the positional relationships between sets of scene lines (step S25). The list created here is referred to as a scene feature list.

[0071] FIG. 17 is a diagram showing an example of a scene feature list generated by the matching processing unit 15. The scene feature list indicates the positional relationship between a pair of scene features. The matching processing unit 15 selects two scene features from the list shown in FIG. 16 and calculates and records the angle between the scene features. The scene feature list includes two line IDs, "line ID 1" and "line ID 2," that indicate a combination of a pair of scene features. If the scene line indicated by "line ID 1" and the scene line indicated by "line ID 2" are not parallel, the matching processing unit 15 records the angle between the two scene lines in the scene feature list. Furthermore, if the scene line indicated by "line ID 1" and the scene line indicated by "line ID 2" are parallel, the matching processing unit 15 calculates the distance between the two scene lines and records it in the scene feature list.

[0072] 15 , the matching processing unit 15 first selects one candidate for the reference line from the scene feature (step S26). Specifically, the matching processing unit 15 selects one candidate for the straight line corresponding to the reference line L0 of the model feature from the scene feature.

[0073] Next, the matching processing unit 15 determines whether the selected candidate scene line is "line ID 1" in the scene feature list, and by checking whether the group of lines corresponding to "line ID 2" includes more than the required number of lines listed in the model feature list, determines whether the remaining model features other than the reference line are present in the scene image in more than the required number (step S27).

[0074] If the scene image does not contain enough model features (step S27: No), the matching processing unit 15 returns to step S26 and reselects candidates for the reference line. By steps S26 and S27, the matching processing unit 15 can identify a scene straight line corresponding to the reference line of the model features and perform rough positioning of the matching model within the scene image.

[0075] When associating a model feature list with a scene feature list, it is easy to determine if there are no errors in the extraction of scene features. However, in reality, errors occur in the extraction of features, and therefore, even when model features and scene features are superimposed, they usually do not match exactly. If no deviation is allowed in the association between model features and scene features, it is impossible to find a location in the scene image that corresponds to the matching model. Therefore, a tolerance range for deviation is set, but the likelihood of deviation varies for each feature, i.e., for each geometric figure such as a line or a circle. Therefore, when associating a model feature list with a scene feature list, the threshold value used to determine whether features correspond is set to a value corresponding to the variance recorded by the model generation unit 14. Specifically, the greater the variance, the wider the tolerance range for deviation during determination.

[0076] For example, if you want to determine the corresponding line for a model line L1 that is parallel to the reference line L0 at a distance d1, and the variance of the model line L1 recorded when the model was generated is v1 = Δd1, then the scene line that falls within the range of d1 ± Δd1 from the reference line in the scene image is considered to be the line that matches the model line L1.

[0077] Furthermore, when it is desired to determine the corresponding line for a model line L2 at an angle θ2 from the reference line L0, the matching processing unit 15 first calculates the allowable angle. Assume that the variance v2 recorded when the model was generated was v2 = Δd2. The length of the model line L2 from the end points P1 and P2 of the model line L2 is expressed by the following formula (1):

[0078]

[0079] The angle variation Δθ2 can be calculated using the following equation (2).

[0080]

[0081] Using the calculated angle variation Δθ2, the matching processing unit 15 can consider a line whose angle with the reference line L0 is within the range of θ2±Δθ2 to be a line that matches the model line L2.

[0082] Fig. 18 is a diagram showing a first example of scene features extracted from a scene image. In Fig. 18, dashed lines indicate scene features, solid lines indicate model features, and gray areas indicate areas where it is determined that the scene features and model features match. In the example shown in Fig. 18, all scene features fall within the gray areas, so it is determined that the scene image contains more than the required number of straight lines listed in the model feature list. In this case, the determination in step S27 of Fig. 15 is "Yes."

[0083] 19 is a diagram showing a second example of scene features extracted from a scene image. In FIG. 19 , as in FIG. 18 , dashed lines indicate scene features, solid lines indicate model features, and gray areas indicate areas where it is determined that the scene features and the model features match. In the example shown in FIG. 19 , a portion of the scene line of the scene feature deviates from the gray area, so it is determined that the model features and the scene features do not match. In this case, since it is determined that the scene features are insufficient in step S27 of FIG. 15 , the result is "No," and the process returns to step S26, where another scene line is reselected from the scene features.

[0084] Returning to the description of Fig. 15, when the rough correspondence is completed and "Yes" is selected in step S27, the matching processing unit 15 calculates matching points in the scene image (step S28).

[0085] A matching point on the matching model is designated pm_i. Where N is the number of matching points, i is an integer between 1 and N. During rough positioning, the geometric figures contained in the scene features are associated with the geometric figures contained in the model features, so it is also possible to calculate and associate matching points in the scene image. A matching point in the scene image is designated ps_i. Here again, i is an integer between 1 and N. ps_i is designated the matching point on the scene image that corresponds to pm_i. In the following description, a matching point on the matching model is referred to as a model matching point, and a matching point on the scene image is referred to as a scene matching point.

[0086] Finally, transformation parameters that indicate the position and orientation of the matching model within the scene image are calculated (step S29).

[0087] 20 is an explanatory diagram of transformation parameters. Here, the transformation parameters are defined as coordinates (X, Y) indicating the position of a model reference position defined in a matching model within a scene image, and a rotation angle θ within a plane, and represent the positional relationship between the scene image and the model image. The transformation parameters can also be said to be information indicating the arrangement of the detection target object 3 within the scene image. The transformation parameters may include parameters indicating a scale ratio and distortion. The distortion indicates the degree of distortion of the detection target object 3 on the two-dimensional image due to rotation in the depth direction.

[0088] In general, the transformation parameters are optimized to minimize the error between the scene matching points and the model matching points, specifically, to minimize the sum of squares of the distances between corresponding matching points.

[0089] FIG. 21 is a diagram showing the error between the model matching points and the scene matching points.

[0090] However, when optimizing the sum of squared distances, the optimization result is significantly affected by points with large errors. The effect is small when there are many matching points and few outliers, but becomes significant when the number of matching points is limited.

[0091] Therefore, the matching processing unit 15 uses the sum of squares of weighted distances, which reduces the weight of errors for geometric shapes with a high probability of outliers occurring, that is, geometric shapes with large variations in feature values, as the evaluation function C. In this case, the evaluation function C is expressed by the following formula (3).

[0092]

[0093] The coordinates of the model matching point pm_i are (x m i , y m i ), the coordinates of the scene matching point ps_i are (x s i , y s i ), the following equation (4) holds using the transformation parameters:

[0094]

[0095] When equation (4) is arranged as many times as the number of matching points, the relationship y=Ax holds, using matrix y shown in equation (5), matrix A shown in equation (6), and matrix x shown in equation (7).

[0096]

[0097]

[0098]

[0099] In this case, by using a diagonal matrix W relating to the weight of the evaluation function C as shown in the following formula (8), and formulas (9) and (10), the transformation parameter x* that minimizes the evaluation function C can be calculated using the following formula (11).

[0100]

[0101]

[0102]

[0103]

[0104] Here, it is assumed that the model matching point pm_i is the intersection of the model lines Li1 and Li2. In this case, the variance of the model line Li1 is expressed as v ｉ １ , the variance of the model line Li2 is v ｉ ２ Then, the weight of the evaluation function w i can be calculated using, for example, the following equation (12).

[0105]

[0106] The matching processing unit 15 may also calculate a scale factor based on the ratio between the length of a straight line in the model image and the length of a straight line in the scene image that corresponds to this straight line, and use this as a conversion parameter.

[0107] In addition, the matching processing unit 15 may estimate the parameters of the affine transformation using matching points contained in the respective straight lines between the scene image and the model image, and calculate distortion-related parameters from the estimated parameters as a solution, which may be used as the transformation parameters.

[0108] Here, a description will be given of a modification of the first embodiment. In Fig. 3, the model generation unit 14 generates a model image using an image of the detection target object 3 captured by a system separate from the device at runtime, but a scene image captured by the scene image acquisition unit 10 may be used, or a simulation image generated on a simulator from a CAD (Computer Aided Design) model of the detection target object 3 may be used.

[0109] 22 is a diagram showing the functional configuration of an image processing device 1A according to a first modified example of embodiment 1. The image processing device 1A includes a scene image acquisition unit 10, a disturbance occurrence determination unit 11, a model selection unit 12, a storage unit 13, a model generation unit 14, a matching processing unit 15, and a disturbance simulation unit 16. That is, the image processing device 1A further includes the disturbance simulation unit 16 in addition to the configuration of the image processing device 1. The following mainly describes the parts that differ from the image processing device 1.

[0110] The disturbance simulation unit 16 generates a simulation image by adding a pseudo-disturbance to the scene image acquired by the scene image acquisition unit 10, and outputs the generated simulation image to the model generation unit 14. The model generation unit 14 generates a matching model to be used in the matching processing unit 15, using the simulation image generated by the disturbance simulation unit 16.

[0111] According to the image processing device 1A, the scene image acquisition unit 10 can generate multiple simulation images from a single scene image actually captured using an imaging device, thereby reducing the number of scene images actually captured. This reduces the startup time of the image processing device 1A. Furthermore, since the effects of individual differences in the detection target object 3 and the surface texture of the detection target object 3 can be removed from the matching model, a matching model that is robust to individual differences can be generated, thereby improving the recognition rate. While the example in which a simulation image is generated from a scene image and used to generate a matching model has been described, the model generation unit 14 may also be configured to use the scene image acquired by the scene image acquisition unit 10 as is.

[0112] 23 is a diagram showing the functional configuration of an image processing device 1B according to a second modified example of embodiment 1. The image processing device 1B includes a scene image acquisition unit 10, a disturbance occurrence determination unit 11, a model selection unit 12, a storage unit 13, a model generation unit 14, a matching processing unit 15, a simulation condition setting unit 17, and a rendering unit 18. That is, the image processing device 1B further includes the simulation condition setting unit 17 and the rendering unit 18 in addition to the configuration of the image processing device 1. The following mainly describes the parts that are different from the image processing device 1.

[0113] The image processing device 1B has a function of generating a simulation image on a simulator using a CAD model of the detection target object 3. A method for generating the simulation image will be described. A 3D (Dimensions) CAD model or a 2D CAD model of the detection target object 3 is stored in the storage unit 13 of the image processing device 1B. The simulation condition setting unit 17 sets simulation conditions including at least one of information on the imaging equipment in the simulation, information on the lighting equipment, information on the detection target object 3, and information on the simulation environment. The information on the imaging equipment may include the specifications of the imaging equipment and information on the installation environment. For example, the information on the imaging equipment may include the angle of view, working distance, installation angle, and resolution of the imaging equipment. The information on the lighting equipment may include the specifications of the lighting equipment and information on the installation environment. For example, the information on the lighting equipment may include the illuminance, wavelength, installation angle, and so on of the lighting equipment. Furthermore, the image processing device 1B may use the CAD model of the detection target object 3 stored in the storage unit 13 to confirm in advance whether the detection target object 3 fits within the field of view of the imaging environment in the simulation. The information on the detection target objects 3 may include, for example, color information, size, and number of models of the detection target objects 3 to be used, of the detection target objects 3, using a CAD model stored in the storage unit 13. The environmental information in the simulation may include a light source environment other than lighting equipment, such as diffuse reflection or ambient light, and a supply method of the detection target objects 3. Possible supply methods of the detection target objects 3 include a bulk pile state in which the detection target objects 3 are arranged completely irregularly, and an aligned state in which the detection target objects 3 are arranged in advance. The image processing device 1B may check the supply state in the simulation using the CAD model.

[0114] The rendering unit 18 has a simulator function that simulates a scene based on the CAD model of the detection target object 3 and the simulation conditions set by the simulation condition setting unit 17. The rendering unit 18 can generate a simulation image, which is a grayscale image or an RGB (Red, Green, Blue) image, based on the simulated scene. The simulation image generated by the rendering unit 18 is not limited to the above content and may be other data such as a distance image as needed. By changing the relative position between the detection target object 3 and the light source or the intensity of the light source in the simulator, it is possible to change the shooting pattern more easily than by actually capturing a scene image, and it is also possible to exclude from the matching model the influence of individual differences in the detection target object 3 and the surface texture of the detection target object 3.

[0115] As described above, the image processing device 1 according to the first embodiment is an image processing device 1 that performs matching processing to identify the location of a detection target object 3 in a scene image using a matching model, and is characterized by including a scene image acquisition unit 10 that acquires a scene image, a disturbance occurrence determination unit 11 that estimates the type of disturbance included in the scene image, a model selection unit 12 that selects one or more matching models to be used for the matching processing from a plurality of candidates based on the type of disturbance estimated for the scene image, and a matching processing unit 15 that performs matching processing using the scene image and the matching model selected for the scene image. As a result, even if a disturbance is included in the scene image, the location of the detection target object 3 is identified using a matching model corresponding to the disturbance included in the scene image, and therefore it is possible to suppress a decrease in the detection accuracy of the detection target object 3.

[0116] Furthermore, the disturbance occurrence determination unit 11 estimates disturbance information indicating the degree of disturbance in addition to the type of disturbance, and the model selection unit 12 can select a matching model based on the type of disturbance and the disturbance information. This makes it possible to perform matching processing using an appropriate matching model depending on the degree of disturbance, and more reliably suppresses deterioration in detection accuracy.

[0117] Furthermore, the matching model is generated based on model features, which are features extracted from a model image obtained by photographing the detection target object 3, and the matching processing unit 15 can perform matching processing based on variations in features extracted from one or more pseudo model images generated by adding disturbances to the model image. More specifically, the matching processing unit 15 is characterized by setting the magnitude of error to be tolerated when performing matching processing for each model feature, depending on the variations in the feature. This makes it possible to change the allowable error range in accordance with the likelihood of error in the feature, thereby more reliably suppressing deterioration in detection accuracy.

[0118] Furthermore, the disturbance occurrence determination unit 11 can estimate the type of disturbance for each divided region obtained by dividing the scene image into a plurality of regions, and the model selection unit 12 can select a matching model for each divided region. This makes it possible to select a matching model according to the type of disturbance even when a disturbance occurs in only a part of the region in the scene image, thereby more reliably suppressing a decrease in detection accuracy.

[0119] The matching processing unit 15 selects a candidate geometric shape from the geometric shapes contained in the scene image that corresponds to a reference shape, which is one of the geometric shapes contained in the matching model, and can position the matching model within the scene image by matching the geometric shape contained in the matching model with the geometric shape contained in the scene image.

[0120] The image processing device 1A according to the first modification further includes a disturbance simulation unit 16 that generates a simulation image in which a pseudo-disturbance is added to a scene image, and a model generation unit 14 that generates a matching model to be used by the matching processing unit using the simulation image generated by the disturbance simulation unit 16. This makes it possible to reduce the number of scene images captured by the actual image capture device, thereby shortening the start-up time of the image processing device 1A. Furthermore, by being able to exclude the influence of individual differences in the detection target object 3 and the surface texture of the detection target object 3 from the matching model, it is possible to generate a matching model that is robust to individual differences, thereby improving detection accuracy.

[0121] Moreover, the image processing device 1B according to the second modified example further includes a simulation condition setting unit 17 that sets simulation conditions including at least one of information on imaging equipment in the simulation, information on lighting equipment, information on an object to be detected, and environmental information in the simulation, a rendering unit 18 that performs a simulation based on the simulation conditions and generates a simulation image, and a model generation unit 14 that generates a matching model to be used by the matching processing unit 15 using the simulation image generated by the rendering unit 18. This makes it possible to simulate the user's actual environment in the simulation and automatically determine a matching model based on the automatically generated simulation image, thereby reducing the data collection cost when generating a matching model.

[0122] Second Embodiment In a second embodiment, a configuration will be described that can also handle the occurrence of a disturbance at runtime that was not anticipated when the model was generated.

[0123] 24 is a diagram showing the functional configuration of an image processing device 1C according to embodiment 2. The image processing device 1C has a scene image acquisition unit 10, a disturbance occurrence determination unit 11, a model selection unit 12, a storage unit 13, a matching processing unit 15, a matching execution determination unit 19, and a shooting environment correction unit 20. In the following, detailed description of the same parts as those of the image processing device 1 according to embodiment 1 will be omitted, and the following mainly describes the parts that are different from the image processing device 1.

[0124] The model selection unit 12 selects a matching model based on the type of disturbance and the disturbance information. For example, in the case of noise, the storage unit 13 is provided with a first model for an S / N ratio range of V1 or more but less than V2, a second model for an S / N ratio range of V2 or more but less than V3, and a third model for an S / N ratio range of V3 or more but less than V4. Here, V1 to V4 are positive real numbers that satisfy the relationship V4 > V3 > V2 > V1 > 1 [dB]. In this case, for example, if the S / N ratio of the scene image is V5 and the relationship V3 > V5 > V2 is satisfied, the second model is selected. If the S / N ratio of the scene image is V6 and V6 < V1, no corresponding matching model exists, and matching processing cannot be performed.

[0125] When there is a lot of noise or the contrast is extremely low, it is difficult to improve the recognition rate by simply improving the matching model generation process, and it is necessary to reconsider the shooting environment of the scene image.

[0126] Therefore, the matching execution determination unit 19 determines whether or not to execute matching based on the scene image, the type of disturbance, and the disturbance information output by the disturbance occurrence determination unit 11. If the matching execution determination unit 19 determines to execute matching, it causes the model selection unit 12 to select a matching model, and if it determines not to execute matching, it causes the shooting environment correction unit 20 to correct the shooting environment of the scene image.

[0127] The imaging environment correction unit 20 adjusts parameters related to the imaging device 2, such as exposure time and camera sensitivity, and parameters such as the illuminance of lighting equipment installed in the imaging environment. The imaging environment correction unit 20 may also prompt the user of the image processing device 1C to change the vision sensor or optical lens to be used, using a display screen, audio, or the like.

[0128] 25 is a flowchart for explaining the operation of the matching execution determination unit 19. First, the matching execution determination unit 19 uses the scene image acquisition unit 10 to acquire a scene image from the photographing device 2 (step S31).

[0129] Next, the matching execution determination unit 19 determines whether the noise is equal to or less than a threshold based on the disturbance information related to the noise (step S32). For example, the matching execution determination unit 19 may check whether the S / N ratio value included in the disturbance information is within a range considered in the generated matching model. If the noise is equal to or less than the threshold (step S32: Yes), the matching execution determination unit 19 records the degree of noise indicated by the disturbance information, for example, the S / N ratio, as a noise index for use in the model selection unit 12 (step S33).

[0130] Next, the matching execution determination unit 19 determines whether the specified contrast is equal to or greater than a threshold based on the disturbance information related to blur (step S34). For example, the matching execution determination unit 19 may check whether a power spectrum calculated using a fast Fourier transform included in the disturbance information is within a range considered in the generated matching model. If the specified contrast is equal to or greater than the threshold (step S34: Yes), the matching execution determination unit 19 records the degree of blur indicated by the disturbance information, for example, the value of the power spectrum, as a blur index for use in the model selection unit 12 (step S35).

[0131] Next, the matching execution determination unit 19 determines whether the specified luminance value is within a threshold based on the disturbance information related to luminance (step S36). For example, the matching execution determination unit 19 may check whether the difference between the maximum and minimum luminance values ​​of the image included in the disturbance information is within the range considered in the generated matching model. If the specified luminance value is within the threshold (step S36: Yes), the matching execution determination unit 19 records the degree of luminance indicated by the disturbance information, for example, the difference between the maximum and minimum luminance values, as a luminance index (step S37).

[0132] If the noise is not below the threshold (step S32: No), if the specified contrast is not above the threshold (step S34: No), or if the specified brightness value is not within the threshold (step S36: No), the matching execution determination unit 19 instructs the shooting environment correction unit 20 to adjust the shooting environment, proceeds to processing by the shooting environment correction unit 20 (step S38), and terminates the processing of the matching execution determination unit 19.

[0133] Furthermore, after executing step S37, there is no need to adjust the shooting environment, so the matching execution determination unit 19 inputs the recorded disturbance index to the model selection unit 12 (step S39), and the processing of the matching execution determination unit 19 ends.

[0134] The combination and order of disturbances that the matching execution determination unit 19 determines during processing are not limited to the example shown in the second embodiment, and can be changed as necessary.

[0135] As described above, according to the second embodiment, the image processing device 1C further includes a matching execution determination unit 19 that determines whether to execute a matching process based on disturbance information indicating the estimated type and degree of disturbance, and a shooting environment correction unit 20 that adjusts the shooting environment in which a scene image is captured based on the determination result of the matching execution determination unit 19. This allows the shooting environment to be corrected as needed, thereby mitigating disturbances contained in the scene image and improving the recognition rate. Furthermore, matching model candidates are prepared in advance in association with the degree of disturbance. If there is no matching model candidate associated with the degree of disturbance of the scene image, the matching execution determination unit 19 determines not to execute the matching process, and the shooting environment correction unit 20 adjusts the shooting environment. As a result, if the degree of disturbance can be addressed by using an appropriate matching model, the matching process is performed using the appropriate matching model. Only when a disturbance that cannot be addressed by the matching model occurs is the shooting environment adjusted and the scene image reacquired. This reduces the time required for adjusting the shooting environment and improves the recognition rate.

[0136] Third Embodiment In the third embodiment, an example of performing an appearance inspection of a detection target object 3 will be described. FIG. 26 is a diagram showing the functional configuration of an image processing device 1D according to the third embodiment. The image processing device 1D has a scene image acquisition unit 10, a disturbance occurrence determination unit 11, a model selection unit 12, a storage unit 13, a model generation unit 14, a matching processing unit 15, and an appearance inspection unit 21. Below, detailed description of the functions of the image processing device 1D that are the same as those of the image processing device 1 will be omitted, and the following mainly describes the parts that are different from the image processing device 1.

[0137] The appearance inspection unit 21 determines whether the detection target object 3, the location of which has been identified by the matching processing unit 15, satisfies the inspection standard. The appearance inspection unit 21 performs the appearance inspection by utilizing the model feature amounts described in the first embodiment.

[0138] 27 is a flowchart for explaining the operation of the appearance inspection unit 21. First, the appearance inspection unit 21 extracts scene features from a scene image (step S24) and lists the positional relationships of pairs of scene lines included in the scene features (step S25). Note that the processes of steps S24 and S25 are the same as steps S24 and S25 in FIG. 15, and if the process shown in FIG. 15 has been executed in advance by the matching processing unit 15, the scene feature list generated by the matching processing unit 15 can be reused.

[0139] Next, the visual inspection unit 21 selects one reference line candidate from the scene features (step S26). The visual inspection unit 21 determines whether the remaining inspection model features exist in the scene image (step S41). Note that here, the number of model features for visual inspection is different from that used for position detection, but the same model features as used for position detection may also be used. The processing of step S41 can be performed using a threshold value that takes into account the variability of the model features in a manner similar to step S27 of FIG. 15. The threshold values ​​used in this case may be different between position detection and visual inspection. For example, the reliability of the visual inspection can be ensured by setting the threshold value for visual inspection to a stricter value than the threshold value for position detection.

[0140] If the inspection model feature exists in the scene image (step S41: Yes), the appearance inspection unit 21 assigns an appearance inspection OK label (step S42) and ends the process.

[0141] If there is an inspection model feature that does not exist in the scene image (step S41: No), the appearance inspection unit 21 determines whether all the reference line candidates have been selected (step S43), and if all the reference line candidates have been selected (step S43: Yes), it assigns an appearance inspection NG label (step S44) and ends the process.If there is a reference line candidate that has not been selected (step S43: No), the appearance inspection unit 21 returns to the process of step S26 and reselects a reference line candidate.

[0142] The visual inspection OK label indicates that the visual inspection was successful. Success here means that it can be assumed that there is no damage to the detection target object 3. The visual inspection may also include inspections other than for damage.

[0143] Next, a modified example of the third embodiment will be described. In this modified example, the model feature amount is not used for the appearance inspection, and a matching model for the appearance inspection that is different from that used for position detection is used. This is intended for cases where, for example, different characters are printed on parts of the same shape, or where color information or pattern information of the detection target object 3 is used as the basis for the appearance inspection.

[0144] 28 is a diagram showing the functional configuration of an image processing device 1E according to a modification of Embodiment 3. The image processing device 1E includes a scene image acquisition unit 10, a disturbance occurrence determination unit 11, a model selection unit 12, a storage unit 13, a model generation unit 14, a matching processing unit 15, an appearance inspection unit 21, and an appearance inspection model generation unit 22.

[0145] The procedure for position detection in the image processing device 1E is the same as that in embodiment 1. The matching processing unit 15 executes the matching process to identify the location of the detection target object 3 in the scene image, and then outputs the identified position of the detection target object 3 to the appearance inspection unit 21.

[0146] The matching model for appearance inspection can be, for example, a model for detecting a mark written at a specific position on the detection target object 3. The appearance inspection unit 21 uses the matching model for appearance inspection generated by the appearance inspection model generation unit 22 and the position of the detection target object 3 output by the matching processing unit 15 to identify the position on the image where the mark for appearance inspection is written, and compares that portion with the matching model for appearance inspection to determine the inspection result.

[0147] As described above, the image processing device 1D according to the third embodiment includes the model generation unit 14 that generates a matching model based on model feature amounts extracted from a model image, and the appearance inspection unit 21 that determines whether or not the detection target object 3 detected in the matching process satisfies the inspection standard by utilizing the model feature amounts extracted by the model generation unit 14. This makes it possible to automate the appearance inspection, thereby reducing the time required for the inspection work.

[0148] Moreover, the image processing device 1E according to the modified example of the third embodiment further includes an appearance inspection model generation unit 22 that generates a matching model for appearance inspection to determine whether or not the detection target object 3 detected by the matching process satisfies the inspection standard. This makes it possible to handle, for example, cases where different characters are printed on parts of the same shape, and also makes it possible to perform appearance inspection based on information that is not used during position detection, such as color information or pattern information of the detection target object 3.

[0149] Fourth Embodiment In the fourth embodiment, a model generation process that can be performed even when the number of generated matching models is limited will be described. Note that, since the configuration of the image processing device in the fourth embodiment is the same as the image processing device 1 according to the first embodiment shown in FIG. 3, the reference numerals shown in FIG. 3 will be used in the description.

[0150] The model generation unit 14 generates a matching model by preparing a plurality of pseudo model images in which pseudo disturbances are added to a model image. In the first embodiment, an example was given in which pseudo disturbances are added for each of noise and blur, but it is also possible to create a pseudo model image that combines multiple types of disturbances. However, combining multiple types of disturbances increases the number of disturbance patterns, which poses a problem in that the number of matching models to be stored in the storage unit 13 increases. Increasing the types of disturbances used further exponentially increases the number of matching models.

[0151] For example, in the first embodiment, the S / N ratio is divided into three levels to generate matching models, but the range division is ad hoc. If disturbances other than the S / N ratio are also classified independently into three levels, the number of matching models increases as the number of types of disturbances increases.

[0152] For this reason, by integrating similar models among the many matching models generated for each combination of disturbance information values, the number of matching models can be reduced while suppressing a decrease in the recognition rate. Because it is difficult for a user to determine which matching models to integrate, in the fourth embodiment, the model generation unit 14 of the image processing device 1 automatically generates a specified number of matching models using a classifier based on machine learning.

[0153] One well-known classification method for classifying data into two or more classes is the k-means method. The k-means method is an unsupervised method and is an algorithm for creating any specified k number of clusters based on feature vectors.

[0154] In the fourth embodiment, the model generation unit 14 calculates clustering features from pseudo model images, performs class classification on the clustering features, and generates a matching model for the set of pseudo model images after clustering.

[0155] 29 is a flowchart for explaining the operation of the image processing device 1 according to the fourth embodiment. The processes from step S11 to step S15 are the same as those in the first embodiment, and therefore detailed explanations thereof will be omitted.

[0156] After extracting the pseudo model features from the pseudo model image, the model generation unit 14 calculates the distance between the pseudo model features and the model features using the model feature list (step S191).

[0157] Here, a method for calculating the distance between the pseudo model feature amount and the model feature amount will be described.

[0158] Fig. 30 is a diagram showing an example of model features. In Fig. 30, for the sake of explanation, an ID of each model straight line included in the model features is assigned. Fig. 31 is a diagram showing an example of pseudo model features. As in Fig. 30, for the sake of explanation, an ID of each pseudo model straight line included in the pseudo model features is assigned in Fig. 31 as well.

[0159] For example, when the model features shown in FIG. 30 and the pseudo model features shown in FIG. 31 are acquired, the model generation unit 14 first lists the model features using the same method as in the first embodiment.

[0160] FIG. 32 is a diagram showing an example of a list generated in the fourth embodiment. The model generation unit 14 first lists the line IDs identifying each geometric figure included in the model feature and records the positional relationship between the model feature indicated by each line ID and the reference line L0. This process generates columns 1 to 4 in FIG. 32. Next, the model generation unit 14 calculates the distance between the model feature indicated by each line ID and the corresponding pseudo model feature. For example, in the examples of FIGS. 30 and 31, the distance between feature values ​​determined to be in the same position is calculated, such as the distance between the model line L1 in FIG. 30 and the pseudo model line L1-α in FIG. 31, and the distance between the model line L2 and the pseudo model line L2-α. The method of calculating the distance between feature values ​​is also the same as in the first embodiment.

[0161] Next, the model generation unit 14 creates clustering features from the distances calculated in step S191 (step S192). The clustering features can be, for example, vectors that list the distances of each feature. In the example of Figure 32, the vectors in the fifth column of the table, surrounded by a bold frame, are used.

[0162] 30 to 32, there may be a case where a feature exists in the model feature but does not correspond to the pseudo model feature, as in L6. A positive value, such as "10," that is sufficiently larger than the other elements is assigned to the element corresponding to the vector in this row.

[0163] The model generation unit 14 calculates the clustering features for all of the prepared pseudo model images. Then, the model generation unit 14 uses the set of clustering features to classify the images into k classes using a classification method, for example, the k-means method.

[0164] FIG. 33 shows six different pseudo model images classified into three classes in the fourth embodiment.

[0165] Returning to the description of FIG. 29 , the model generation unit 14 classifies the pseudo model features using the clustering features and records the variances (step S16). The model generation unit 14 generates a matching model for each class after classification (step S18). The matching model generation method is also the same as in the first embodiment. However, because the clustering features are vectorized quantities representing the distance between the model features and the pseudo model features, the variances of each feature can be calculated from the clustering features without repeating the processing of the model generation unit 14.

[0166] Since the classification based on the above clustering feature does not correspond to disturbance information such as the amount of blur and noise intensity, the model selection unit 12 cannot select a matching model based on the value of the disturbance information as it is.

[0167] Therefore, for example, a first method is to select the generated matching models in order and repeat the matching process until matching is successful or all matching models have been selected.

[0168] As a second method, after performing class classification based on clustering features, additional supervised class classification is performed using disturbance information values ​​as features and classified class numbers as labels. Here, a classifier based on clustering features is referred to as classifier #1, and a classifier that receives a feature vector created by arranging expected disturbance information values ​​such as the amount of blur and noise intensity as input and outputs the class number classified by classifier #1 as output is referred to as classifier #2. During matching processing, the model selection unit 12 inputs a feature vector created by calculating disturbance information values ​​for a scene image to classifier #2, thereby determining which matching model corresponds to which class number created by classifier #1 to use. The feature vector created by arranging disturbance information values, which is input to classifier #2, can be created during matching processing, making it possible to select an appropriate matching model even from a matching model set reduced using classifier #1.

[0169] As described above, according to the fourth embodiment, the model generation unit 14 performs class classification on a plurality of sets of generated matching models to classify the matching models into classes, and can generate a new matching model by integrating the matching models classified for each class. This makes it possible to reduce the number of matching models while suppressing a decrease in the recognition rate. By reducing the number of matching models stored in the storage unit 13, the capacity of the storage unit 13 can be reduced.

[0170] Here, the hardware configuration of the image processing devices 1, 1A to 1E according to the first to fourth embodiments will be described. The image processing devices 1, 1A to 1E according to the present embodiments are realized, for example, by a computer system. The image processing devices 1, 1A to 1E may be realized by one computer system or by multiple computer systems. For example, the image processing devices 1, 1A to 1E may be realized by a cloud system. In a cloud system, the division between the computer system hardware and devices such as servers for each function can be arbitrarily set. For example, one computer system may have the functions of multiple devices, or multiple computer systems may have the functions of one device.

[0171] An example of the configuration of a computer system that realizes the image processing devices 1, 1A to 1E will be described. Fig. 34 is a diagram showing an example of the configuration of a computer system that realizes the image processing devices 1, 1A to 1E according to embodiments 1 to 4. As shown in Fig. 34, this computer system includes a control unit 101, an input unit 102, a storage unit 103, a display unit 104, a communication unit 105, and an output unit 106, which are connected via a system bus 107.

[0172] In FIG. 34 , the control unit 101 is, for example, a CPU (Central Processing Unit). The control unit 101 executes an image processing program that describes the processes to be performed by the image processing devices 1, 1A to 1E of this embodiment. The input unit 102 is composed of, for example, a keyboard, a mouse, etc., and is used by a user of the computer system to input various information. The memory unit 103 includes various memories such as RAM (Random Access Memory) and ROM (Read Only Memory) and a storage device such as a hard disk, and stores programs to be executed by the control unit 101, necessary data obtained during processing, etc. The memory unit 103 is also used as a temporary storage area for programs. The display unit 104 is composed of, for example, an LCD (Liquid Crystal Display) panel, and displays various screens to the user of the computer system. The communication unit 105 is, for example, a communication circuit that performs communication processing. The communication unit 105 may be composed of multiple communication circuits corresponding to multiple communication methods. The output unit 106 is an output interface that outputs data to external devices such as a printer or external storage device.

[0173] Note that Fig. 34 is just an example, and the configuration of the computer system is not limited to the example of Fig. 34. For example, the computer system may not include the output unit 106. Furthermore, when the image processing devices 1, 1A to 1E are realized by multiple computer systems, not all of these computer systems may be the computer systems shown in Fig. 34. For example, some computer systems may not include at least one of the display unit 104, output unit 106, and input unit 102 shown in Fig. 34.

[0174] Here, an example of the operation of a computer system until an image processing program describing the processing of image processing devices 1, 1A to 1E of this embodiment is ready to be executed will be described. In a computer system having the above configuration, for example, the image processing program is installed in storage unit 103 from a CD-ROM or DVD-ROM inserted in a CD (Compact Disc)-ROM drive or DVD (Digital Versatile Disc)-ROM drive (not shown). Then, when the image processing program is executed, the image processing program read from storage unit 103 is stored in an area that serves as the main storage device of storage unit 103. In this state, control unit 101 executes the processing of image processing devices 1, 1A to 1E of this embodiment in accordance with the image processing program stored in storage unit 103.

[0175] In the above description, a program describing the processing in the image processing device 1, 1A to 1E is provided using a CD-ROM or DVD-ROM as a recording medium, but this is not limited to this. Depending on the configuration of the computer system, the capacity of the program to be provided, etc., it is also possible to use a program provided via a transmission medium such as the Internet via the communication unit 105, for example.

[0176] The image processing program of this embodiment is a program that causes a computer to perform a matching process that identifies the position of a detection target object in a scene image using a matching model, and causes the computer to execute the following steps: acquiring a scene image; estimating the type of disturbance contained in the scene image; selecting one or more matching models to be used for the matching process from a plurality of candidates based on the type of disturbance estimated for the scene image; and performing the matching process using the scene image and the matching model selected for the scene image.

[0177] Note that the division of functions in the image processing device 1 shown in Fig. 3 is one example, and as long as the image processing device 1 can perform the above-described operations, the division of functions in the image processing device 1 is not limited to the example shown in Fig. 3. The same applies to the image processing devices 1A to 1E.

[0178] The configurations shown in the above embodiments are merely examples, and may be combined with other known technologies, or different embodiments may be combined with each other. It is also possible to omit or modify parts of the configurations as long as they do not deviate from the gist of the invention.

[0179] REFERENCE SIGNS LIST 1, 1A to 1E image processing device, 2 imaging device, 3 object to be detected, 4 work equipment, 10 scene image acquisition unit, 11 disturbance occurrence determination unit, 12 model selection unit, 13 memory unit, 14 model generation unit, 15 matching processing unit, 16 disturbance simulation unit, 17 simulation condition setting unit, 18 rendering unit, 19 matching execution determination unit, 20 imaging environment correction unit, 21 appearance inspection unit, 22 appearance inspection model generation unit, 101 control unit, 102 input unit, 103 memory unit, 104 display unit, 105 communication unit, 106 output unit, 107 system bus.

Claims

1. An image processing device that performs a matching process to specify a location of a detection target object in a scene image using a matching model, a scene image acquisition unit for acquiring the scene image; a disturbance occurrence determination unit that estimates a type of disturbance included in the scene image; a model selection unit that selects one or more matching models to be used in a matching process from among a plurality of candidates based on the type of disturbance estimated for the scene image; a matching processing unit that performs the matching process using the scene image and the matching model selected for the scene image; Equipped with 2. An image processing device according to claim 1, wherein the plurality of candidates include a matching model based on a pseudo model image obtained by applying image rotation as a pseudo disturbance to a model image obtained by photographing the detection target object.

2. The disturbance occurrence determination unit estimates disturbance information indicating a degree of the disturbance in addition to the type of the disturbance, The image processing apparatus according to claim 1 , wherein the model selection unit selects the matching model based on the type of the disturbance and the disturbance information.

3. the matching model is generated based on a model feature that is a feature extracted from the model image; The image processing device according to claim 1 , characterized in that the matching processing unit performs the matching processing based on the variation in feature amounts extracted from one or more pseudo model images generated by adding the pseudo disturbance to the model image.

4. 4. The image processing apparatus according to claim 3, wherein the matching processing unit sets, for each of the model features, a magnitude of an error that is permitted when performing the matching processing in accordance with a variation in the features.

5. the disturbance occurrence determination unit estimates a type of the disturbance for each of a plurality of divided regions obtained by dividing the scene image into a plurality of regions; The image processing apparatus according to claim 1 , wherein the model selection unit selects the matching model for each of the divided regions.

6. The image processing device according to claim 1, characterized in that the matching processing unit selects a candidate geometric shape from among the geometric shapes contained in the scene image, which corresponds to a reference shape, which is one of the geometric shapes contained in the matching model, and positions the matching model within the scene image by matching the geometric shape contained in the matching model with the geometric shape contained in the scene image.

7. a disturbance simulation unit for generating a simulation image by adding the pseudo disturbance to the scene image; a model generation unit that generates the matching model to be used by the matching processing unit, using the simulation image generated by the disturbance simulation unit; The image processing device according to claim 1 , further comprising:

8. a simulation condition setting unit that sets simulation conditions including at least one of information on a photographing device, information on a lighting device, information on a detection target object, and information on an environment in the simulation; a rendering unit that performs a simulation based on the simulation conditions and generates a simulation image; a model generation unit that generates the matching model to be used by the matching processing unit, using the simulation image generated by the rendering unit; The image processing device according to claim 1 , further comprising:

9. a matching execution determination unit that determines whether or not to execute the matching process based on disturbance information indicating the estimated type and degree of the disturbance; a photographing environment correction unit that adjusts a photographing environment in which the scene image is photographed based on a determination result of the matching execution determination unit; The image processing device according to claim 1 , further comprising:

10. Candidates for the matching model are prepared in advance in association with the degree of the disturbance; The image processing device according to claim 9, characterized in that if there is no candidate matching model corresponding to the degree of disturbance of the scene image, the matching execution determination unit determines not to execute the matching process, and the shooting environment correction unit adjusts the shooting environment.

11. a model generation unit that generates the matching model based on model features extracted from a model image; an appearance inspection unit that uses the model feature amount extracted by the model generation unit to determine whether or not a detection target object detected in the matching process satisfies an inspection standard; and The image processing device according to claim 1 , further comprising:

12. a visual inspection model generation unit that generates a matching model for visual inspection to determine whether or not the detection target object detected by the matching process satisfies an inspection standard; The image processing device according to claim 1 , further comprising:

13. a model generation unit that generates the matching model based on model features extracted from a model image, classifies a set of the generated matching models into classes by performing class classification, and generates a new matching model by integrating the matching models classified for each class; The image processing device according to claim 1 , further comprising:

14. An image processing method in which an image processing device performs a matching process to specify a location of a detection target object in a scene image using a matching model, the method comprising: acquiring the scene image; estimating a type of disturbance included in the scene image; selecting one or more matching models to be used in a matching process from among a plurality of candidates based on the type of disturbance estimated for the scene image; performing the matching process using the scene image and the matching model selected for the scene image; Including, The image processing method according to claim 1, wherein the plurality of candidates include a matching model based on a pseudo model image obtained by applying image rotation as a pseudo disturbance to a model image of the detection target object.

15. A program for causing a computer to perform a matching process for identifying a location of a detection target object in a scene image by using a matching model, the program comprising: The computer includes: acquiring the scene image; estimating a type of disturbance included in the scene image; selecting one or more matching models to be used in a matching process from among a plurality of candidates based on the type of disturbance estimated for the scene image; performing the matching process using the scene image and the matching model selected for the scene image; Run the command, The program, wherein the plurality of candidates include a matching model based on a pseudo model image in which image rotation is applied as a pseudo disturbance to a model image of the detection target object.