Image processing device, image processing method, and image processing program
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- FANUC LTD
- Filing Date
- 2023-02-16
- Publication Date
- 2026-08-04
Smart Images
Figure 0007900591000001 
Figure 0007900591000002 
Figure 0007900591000003
Abstract
Description
Technical Field
[0001] The present disclosure relates to an image processing apparatus, an image processing method, and an image processing program having a function of setting a coordinate system for a template image used in template matching.
Background Art
[0002] Template matching is one of the useful methods for searching for feature portions included in a template image from an input image. For example, by performing template matching on a captured image obtained by capturing a work attached to a robot, a function of automatically measuring the coordinate system of the work as viewed from the mechanical interface coordinate system of the robot is known (Patent Document 1).
Prior Art Documents
Patent Documents
[0003]
Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0004] The template image used in template matching is manually captured by a user. Therefore, the position and orientation of the work area included in the template image may deviate from the predefined position and orientation. In order to cope with this, the user has manually set the coordinate system of the work in the template image. However, the manual coordinate system setting work is time-consuming and depends on the user, so the quality as a template image is difficult to be stabilized. Therefore, it is desired to provide a function of automatically setting the coordinate system of the template image.
Means for Solving the Problems
[0005] The image processing apparatus according to this disclosure comprises a storage unit for storing data relating to a template image for extracting a symmetrical work region from an input image by template matching, a criterion identification unit for identifying a criterion for the symmetry of the work region, and a coordinate system setting unit for setting the coordinate system of the template image based on the criterion. The origin of the coordinate system is set to the rotational symmetry center of the work region. The reference identification unit includes a matching rate calculation unit that calculates multiple matching rates between the template image and multiple other template images obtained by rotating the template image by multiple rotation angles, and a symmetry center identification unit that identifies the symmetry center based on the multiple matching rates. . [Brief explanation of the drawing]
[0006] [Figure 1] Figure 1 shows a robot system including an image processing device according to this embodiment. [Figure 2] Figure 2 shows the template coordinate system set for the template image by the image processing device according to this embodiment. [Figure 3] Figure 3 is a hardware configuration diagram of the image processing device shown in Figure 1. [Figure 4] Figure 4 is a functional block diagram of the image processing device shown in Figure 1. [Figure 5] Figure 5 is a supplementary diagram illustrating the image creation section of Figure 4. [Figure 6] Figure 6 is a supplementary diagram illustrating the matching rate calculation unit in Figure 4. [Figure 7] Figure 7 shows an example of weighting coefficients. [Figure 8] Figure 8 is a supplementary diagram illustrating the coordinate transformation unit, symmetry center identification unit, and coordinate system setting unit of Figure 4. [Figure 9] Figure 9 shows the first and second template images when the matching score reaches its maximum value. [Figure 10] Figure 10 is a supplementary diagram illustrating the coordinate value transformation unit, symmetry axis identification unit, and coordinate system setting unit of Figure 4. [Figure 11] Figure 11 shows the first and third template images when the second matching rate is at its maximum value. [Figure 12] Figure 12 is a flowchart showing the procedure for setting the template coordinate system by the image processing device according to this embodiment. [Figure 13] Figure 13 is a flowchart showing the procedure for setting the origin of the template coordinate system in Figure 12. [Figure 14] Figure 14 is a flowchart showing the procedure for setting the coordinate axes of the template coordinate system in Figure 12. [Modes for carrying out the invention]
[0007] The image processing apparatus according to this embodiment will be described below with reference to the drawings. In the following description, components having substantially the same function and configuration will be denoted by the same reference numerals, and redundant explanations will be given only when necessary.
[0008] In this embodiment, terms are defined as follows: Template image: This is an image used for template matching to identify the position and orientation of a workpiece from an input image. The template image represents the area of the workpiece to be extracted (hereinafter referred to as the workpiece area). In this embodiment, the workpiece to be extracted has a triangular workpiece area. Of course, the workpiece area is not limited to a triangular shape, and can be any shape that has rotational symmetry and line symmetry, with the center of rotational symmetry located on the axis of symmetry of line symmetry. First template image: This is an image generated from the template image, with its resolution, brightness, etc., adjusted for calculations in the template coordinate system. Of course, the first template image may be the same image as the template image. Second template image: This is an image generated from the first template image, obtained by rotating the first template image. The second template image, along with the first template image, is used in template matching performed when setting the origin of the template coordinate system. Third template image: This image is generated from the first template image by inverting and rotating it. The third template image, along with the first template image, is used in template matching performed when setting the coordinate axes of the template coordinate system. Template coordinate system (Σt): This is the coordinate system that is ultimately set for the template image by the image processing device according to this embodiment. Since it is based on the work area, it is also called the work coordinate system. First Coordinate System (Σ1): This is the coordinate system set for the first template image. Typically, the origin O1 is set in the upper left corner of the first template image, the X1 axis is set along the vertical direction, and the Y1 axis is set along the horizontal direction. Second coordinate system (Σ2): This is the coordinate system set in the second template image, which is a rotation of the first coordinate system. Third coordinate system (Σ3): This is the coordinate system set for the third template image. When the first coordinate system is given as a right-handed system, the third coordinate system is given as a left-handed system. The first matching rate is a parameter that represents the evaluation value of the image similarity between the first template image and the second template image. The first matching rate is maximized when the work area represented in the first template image and the work area represented in the second template image overlap. Theoretically, the maximum value is 100%. Matching Score: This parameter represents the first matching rate, taking into account the rotation angle of the second template image. It is calculated by multiplying the first matching rate by a weighting coefficient that uses the rotation angle of the second template image as a variable. Second Matching Rate: This parameter represents the evaluation value of the image similarity between the third template image and the first template image. The second matching rate reaches its maximum value when the work area represented by the first template image and the work area represented by the third template image overlap. Theoretically, the maximum value is 100%.
[0009] FIG. 1 is a diagram showing a robot system 1 including an image processing apparatus 5 according to the present embodiment. As shown in FIG. 1, the robot system 1 includes a robot 2 that grips a workpiece W, a control device 3 that controls the robot 2, a camera 4 that captures the workpiece W gripped by the robot 2, and an image processing apparatus 5 that identifies a template coordinate system as information regarding the position and orientation of the workpiece W by performing template matching on a captured image of the workpiece W captured by the camera 4 and outputs the information to the control device 3. The functions of the image processing apparatus 5 according to the present embodiment may be implemented as one function of the control device 3.
[0010] One feature of the image processing apparatus 5 according to the present embodiment is that it has a function of automatically setting a template coordinate system (workpiece coordinate system) Σt serving as a workpiece reference for a template image used for template matching. As shown in FIG. 2, the image processing apparatus 5 performs template matching using a template image in which a template coordinate system Σt is set for a captured image U, and outputs to the control device 3 the position of the origin Ot of the template coordinate system Σt as the position of the workpiece W and the directions of the respective axes (Xt axis, Yt axis) of the template coordinate system Σt as the orientation of the workpiece W. Thereby, the control device 3 can identify the position and orientation of the workpiece W as seen from the mechanical interface coordinate system of the robot 2, and can control the robot 2 based on the position and orientation of the workpiece W.
[0011] Typically, the image processing apparatus 5 according to the present embodiment is configured as follows. As shown in FIG. 3, the image processing apparatus 5 is configured by connecting hardware such as a communication device 8 and a storage device 9 to a processor 6 such as a CPU. The communication device 8 is realized by a communication module compliant with an arbitrary communication standard, and transmits and receives various data to and from the control device 3 and the camera 4 under the control of the processor 6. The storage device 9 is realized by an HDD, an SSD, or the like. The storage device 9 stores an operation program of the robot 2 and an image processing program for setting the template coordinate system Σt in the template image.
[0012] As shown in FIG. 4, when the image processing program stored in the storage device 9 is executed by the processor 6, the image processing apparatus 5 functions as a data reception unit 50, a storage unit 51, a reference specifying unit 52, and a coordinate system setting unit 60.
[0013] The data reception unit 50 is realized by the function of the communication device 8 shown in FIG. 3. The data reception unit 50 receives data regarding a captured image of the work W from the camera 4. This captured image corresponds to the template image. The data regarding the template image is stored in the storage unit 51. The photographing of the template image and the setting process of the template coordinate system Σt for the template image are performed before actually operating the robot system 1.
[0014] The storage unit 51 stores data regarding the template image. The storage unit 51 is realized by the function of the storage device 9 shown in FIG. 3. The data regarding the template image is stored in an arbitrary format such as raster data representing an image by dots (points), which is represented by file formats such as jpg and bmp, vector data representing an image by lines, which is represented by file formats such as eps and ai, and point group data represented by a collection of points having spatial coordinate value information (x, y, z) and color information (R, G, B), which is represented by a file format such as xyz.
[0015] The reference specifying unit 52 specifies the symmetry reference of the work area included in the template image. More specifically, the reference specifying unit 52 specifies the center of rotational symmetry of the work area and the symmetry axis of line symmetry of the work area as the symmetry reference of the work area included in the template image.
[0016] The coordinate system setting unit 60 sets the template coordinate system Σt for the template image based on the symmetry criteria of the work area identified by the reference identification unit 52. Specifically, the coordinate system setting unit 60 sets the origin Ot of the template coordinate system Σt at the rotational symmetry center of the work area identified by the reference identification unit 52, and sets the Xt axis of the template coordinate system Σt at the line symmetry axis of the work area identified by the reference identification unit 52. The coordinate system setting unit 60 also sets the Yt axis, which passes through the origin Ot of the template coordinate system Σt and is orthogonal to the Xt axis. The orientation of the Xt and Yt axes is set according to predefined rules. For example, the coordinate system setting unit 60 sets the Xt axis so that its positive direction is the upward direction of the template image. When the Xt axis is parallel to the left-right direction of the template image, the coordinate system setting unit 60 sets the Yt axis so that its positive direction is the upward direction of the template image. Details of the coordinate system setting unit 60 will be described later.
[0017] The reference identification unit 52 includes an image creation unit 53, a matching rate calculation unit 54, a matching score calculation unit 55, an extraction unit 56, a coordinate value transformation unit 57, a symmetry center identification unit 58, and a symmetry axis identification unit 59.
[0018] The image creation unit 53 creates the first template image, the second template image, and the third template image from the template image. Details of the image creation unit 53 will be described later.
[0019] The matching rate calculation unit 54 calculates multiple first matching rates between the first template image and the second template image by template matching. Each of the multiple first matching rates differs in at least one of the shift amount of the second template image relative to the first template image and the rotation angle of the second template image. Similarly, the matching rate calculation unit 54 calculates multiple second matching rates between the first template image and the third template image by template matching. Each of the multiple second matching rates differs in at least one of the shift amount of the third template image relative to the first template image and the rotation angle of the third template image.
[0020] Typically, the matching rate is calculated for the overlapping area of two images. Any known method can be used to calculate the matching rate. For example, the matching rate is calculated as the ratio of the number of matching feature points to the total number of feature points included in the overlapping area of the two images. Typically, a single feature point is given by a single pixel, and the total number of feature points corresponds to the number of pixels included in the overlapping area of the two images. Feature points are considered to match when the least squared sum of the brightness values of two overlapping feature points (pixels) is less than a preset threshold. Details of the matching rate calculation unit 54 will be described later.
[0021] Note that a single feature point may be represented by a single set of multiple pixels, such as 2x2 or 3x3. Also, the total number of feature points may be less than the total number of pixels in the overlapping area of the two images. If the feature points of the two images do not overlap, any method such as interpolation may be used to calculate the matching rate.
[0022] The matching score calculation unit 55 calculates multiple matching scores by multiplying multiple first matching rates by weighting coefficients. Details of the matching score calculation unit 55 will be described later.
[0023] The extraction unit 56 extracts the maximum value from a plurality of matching scores calculated by the matching score calculation unit 55. Similarly, the extraction unit 56 extracts the maximum value from a plurality of second matching rates calculated by the matching rate calculation unit 54.
[0024] The coordinate value transformation unit 57 transforms the coordinate values of feature points on the second template image, represented in the second coordinate system Σ2, into the first coordinate system Σ1 so that the coordinate values of feature points on the first template image where the matching score extracted by the extraction unit 56 shows the maximum value can be compared with the coordinate values of feature points on the second template image. Similarly, the coordinate value transformation unit 57 transforms the coordinate values of feature points on the third template image, represented in the third coordinate system Σ3, into the first coordinate system Σ1 so that the coordinate values of feature points on the first template image where the second matching rate extracted by the extraction unit 56 shows the maximum value can be compared with the coordinate values of feature points on the third template image.
[0025] The symmetry center identification unit 58 compares the coordinate values of multiple feature points on the first template image represented in the first coordinate system Σ1 with the coordinate values of multiple feature points on the second template image converted from the second coordinate system Σ2 to the first coordinate system Σ1, and identifies the feature points whose coordinate values match as the symmetry centers of the work region. In this embodiment, matching coordinate values does not only mean that the coordinate value of one feature point is exactly the same as the coordinate value of the other feature point, but may also include cases where the coordinate value of one feature point is within a predetermined margin range.
[0026] The symmetry axis identification unit 59 compares the coordinate values of multiple feature points on the first template image represented in the first coordinate system Σ1 with the coordinate values of multiple feature points on the third template image transformed from the third coordinate system Σ3 to the first coordinate system Σ1, and identifies a straight line passing through multiple feature points whose coordinate values match as the symmetry axis of the work region.
[0027] The image creation unit 53 will be described below with reference to Figure 5. As shown in Figure 5, for example, the image creation unit 53 creates a first template image T1 obtained by adjusting parameters such as resolution and brightness for image processing from a template image T, multiple second template images T2 obtained by rotating the first template image T1 by multiple rotation angles θs1, and multiple third template images T3 obtained by flipping the first template image T1 around the X1 axis and rotating it by multiple rotation angles θs2.
[0028] The matching rate calculation unit 54 will be described below with reference to Figure 6. The first matching rate and the second matching rate are calculated using the same method. Here, the first matching rate will be used as an example. The matching rate calculation unit 54 repeatedly calculates the first matching rate while changing at least one of the rotation angle θs1 of the second template image T2 and the shift amount ΔS1 of the second template image T2 relative to the first template image T1. For example, the rotation unit of the rotation angle θs1 is set to 5 degrees. Also, the shift amounts in the X1 axis direction and the Y1 axis direction are set to a distance narrower than the pixel spacing. Of course, the shift amounts in the X1 axis direction and the Y1 axis direction may be set to integer multiples such as 1, 2, or 3 times the pixel spacing.
[0029] As shown in Figure 6, for example, the matching rate calculation unit 54 calculates a first matching rate M11 (ΔS11, θs1) between the first template image T1 and the second template image T2 at a position where the second template image T2, which is obtained by rotating the first template image T1 by a rotation angle θs1, is moved relative to the first template image T1 by a shift amount ΔS11 (ΔSx11, ΔSy11). Similarly, the matching rate calculation unit 54 calculates a first matching rate M12 (ΔS12, θs1) between the first template image T1 and the second template image T2 at a position where the second template image T2, which is obtained by rotating the first template image T1 by a rotation angle θs1, is moved relative to the first template image T1 by a shift amount ΔS12 (ΔSx12, ΔSy12).
[0030] The matching score calculation unit 55 will be described below with reference to Figures 6 and 7. As shown in Figure 6, for example, the matching score calculation unit 55 calculates the matching score Ms11(ΔS11, θs1) by multiplying the first matching rate M11(ΔS11, θs1) between the first template image T1 and the second template image T2 by the weighting coefficient Wf(θs1). Similarly, the matching score Ms12(ΔS12, θs1) is calculated by multiplying the first matching rate M12(ΔS12, θs1) between the first template image T1 and the second template image T2 by the weighting coefficient Wf(θs1).
[0031] The weighting coefficient is set to be small at 0 degrees (360 degrees) and large at 180 degrees. For example, as shown in Figure 7, the weighting coefficient is given by a linear equation (solid line in Figure 7) or a stepped equation (dotted line in Figure 7) where the value is '0' when the rotation angle θs1 is 0 degrees (360 degrees) and the value is '1' when it is 180 degrees.
[0032] As described below, the calculation of the origin coordinate value of the template coordinate system Σt is performed using the first template image and the second template image that maximize the matching score. Since the second template image is obtained by rotating the first template image by an angle θs1, the closer the rotation angle θs1 is to 0 (360) degrees, the smaller the change in position between the first template image T1 and the second template image T2, which can easily generate noise in the calculation and potentially reduce calculation accuracy.
[0033] For example, when the work area is represented as a square, the matching rate between the first template image and the second template image is maximized when the rotation angle θs1 is 90 degrees, 180 degrees, 270 degrees, or 360 degrees. However, by multiplying the matching rate by a weighting coefficient as shown in Figure 7, the matching score can be maximized when the rotation angle θs1 is 180 degrees. By using the second template image with a rotation angle θs1 of 180 degrees in the calculation of the coordinate values of the origin of the template coordinate system Σt, it is possible to reduce noise in the calculation and improve the accuracy of the calculation compared to using the second template image with a rotation angle θs1 of 90 degrees, 270 degrees, or 360 degrees.
[0034] The coordinate value transformation unit 57 will be described below with reference to Figures 8 to 11. As shown in Figure 8, for example, the coordinate value transformation unit 57 transforms the coordinate values of multiple feature points Pf represented in the second coordinate system Σ2 on the second template image T2 when the first matching score is maximized into the same first coordinate system Σ1 as the multiple feature points Pe on the first template image. The amount of change in the position of the second template image T2 relative to the first template image T1 when the matching score is maximized is used for the transformation of coordinate values from the second coordinate system Σ2 to the first coordinate system Σ1.
[0035] As shown in Figure 9, the change in position of the second template image T2 relative to the first template image T1 is represented by the shift amount ΔS1 (ΔSx1, ΔSy1) of the second template image T2 relative to the first template image T1 and the rotation angle θs1 of the second template image T2. The coordinate value transformation unit 57 transforms the coordinate values of the feature points Pf on the second template image, represented in the second coordinate system Σ2, into the coordinate values of the second template image T2 before rotation, i.e., the first coordinate system Σ1, using the shift amount ΔS1 and the rotation angle θs1.
[0036] As shown in Figure 10, the coordinate value transformation unit 57 transforms the coordinate values of multiple feature points Pg represented in the third coordinate system Σ3 on the third template image T3 when the second matching rate is maximized into the same first coordinate system Σ1 as the multiple feature points Pe on the first template image. The amount of change in the position of the third template image T3 relative to the first template image T1 when the second matching rate is maximized is used for the transformation of coordinate values from the third coordinate system Σ3 to the first coordinate system Σ1.
[0037] As shown in Figure 11, the change in position of the third template image T3 relative to the first template image T1 when the second matching rate is maximized is represented by the shift amount ΔS2 (ΔSx2, ΔSy2) of the third template image T3 relative to the first template image T1 and the rotation angle θs2 of the third template image T3. The coordinate value transformation unit 57 transforms the coordinate values of the feature points Pg on the third template image T3, represented in the third coordinate system Σ3, into the coordinate values of the third template image T3 before rotation using the shift amount ΔS2 and the rotation angle θs2. Furthermore, by inverting the polarity of the Y3 coordinate value, the coordinate values of the feature points Pg on the third template image T3, represented in the third coordinate system Σ3, are transformed into the first coordinate system Σ1.
[0038] The symmetry center identification unit 58 and the coordinate system setting unit 60 will be described below with reference to Figures 8 and 10. As shown in Figure 8, the coordinate value transformation unit 57 represents multiple feature points Pe on the first template image T1 and multiple feature points Pf on the second template image T2 in the same first coordinate system Σ1. The symmetry center identification unit 58 compares the coordinate values of the multiple feature points Pe on the first template image T1 and the multiple feature points Pf on the second template image T2 that correspond to each of the multiple feature points Pe on the first template image T1, and identifies feature points Ph that have the same coordinate values. Feature points Ph that have the same coordinate values are points whose position has not changed before and after the rotation of the first template image T1, and correspond to the rotational symmetry center of the work area W1. The coordinate system setting unit 60 sets the coordinate values of feature points Ph that have the same coordinate values to the coordinate values of the origin Ot of the template coordinate system Σt.
[0039] Furthermore, the feature point Pf on the second template image T2 that corresponds to the feature point Pe on the first template image T1 means that the feature point identification information is the same. The feature point identification information is, for example, the pixel number assigned to the pixel, and is common to the first template image, the second template image, and the third template image. The symmetry center identification unit 58 and the symmetry axis identification unit 59 compare the coordinate values of two pixels having the same pixel number and identify pixels that have the same coordinate values.
[0040] As shown in Figure 10, the coordinate value transformation unit 57 represents multiple feature points Pe on the first template image T1 and multiple feature points Pg on the third template image T3 in the same first coordinate system Σ1. The symmetry axis identification unit 59 compares the coordinate values of multiple feature points Pe on the first template image T1 and multiple feature points Pg on the third template image T3 that correspond to each of the multiple feature points Pe on the first template image T1, and identifies multiple feature points Pi1 to Pi6 that have the same coordinate values. The line passing through feature points Pi1 to Pi6 that have the same coordinate values is a line whose position and orientation do not change before and after the inversion and rotation of the first template image T1, and corresponds to the axis of symmetry of the line symmetry of the work area W1. The coordinate system setting unit 60 sets the line passing through feature points Pi1 to Pi6 that have the same coordinate values as the Xt axis of the template coordinate system Σt.
[0041] The process of setting the template coordinate system Σt for a template image by the image processing device 5 according to this embodiment will be explained below with reference to Figure 12. As shown in Figure 12, the image processing device 5 uses the rotational symmetry of the work area to set the origin Ot of the template coordinate system Σt (step S1), and uses the line symmetry of the work area to set the coordinate axes (Xt axis, Yt axis) of the template coordinate system Σt (step S2).
[0042] The method for setting the origin Ot of the template coordinate system Σt will be explained below with reference to Figure 13. As shown in Figure 13, the image processing device 5 reads data related to the template image from the storage unit 51 (step S101) and creates a first template image (step S102). The rotation angle θs1 is initialized (θs1 = 5 degrees) (step S103). Next, the image processing device 5 creates a second template image by rotating the first template image by the rotation angle θs1 (step S104), and while translating the second template image in parallel with the first template image, calculates the first matching rate at each position of the second template image relative to the first template image by template matching (step S105), and calculates the matching score by multiplying the first matching rate calculated in step S105 by a weighting coefficient (step S106). The processes in steps S104 to S106 are repeated, for example, every time the rotation angle θs1 changes by 5 degrees, until the rotation angle θs1 reaches 360 degrees (step S107; No, step S108). When the rotation angle θs1 reaches 360 degrees (step S107; Yes), the image processing device 5 extracts the maximum value of the matching score from the multiple matching scores (step S109), obtains the shift amount ΔS1 of the second template image relative to the first template image and the rotation angle θs1 of the second template image when the extracted matching score shows the maximum value (step S110), and uses the shift amount ΔS1 and rotation angle θs1 obtained in step S109 to convert the coordinate values of the feature points on the second template image when the matching score is maximum from the second coordinate system Σ2 to the first coordinate system Σ1 (step S111). Finally, the image processing device 5 compares the coordinate values of corresponding feature points in the first template image and the second template image, and sets the coordinate values of the feature points whose coordinate values match to the coordinate values of the origin Ot of the template coordinate system Σt (step S112).
[0043] The method for setting the coordinate axes of the template coordinate system Σt will be explained below with reference to Figure 14. As shown in Figure 14, the image processing device 5 reads data related to the template image from the storage unit 51 (step S201) and creates a first template image (step S202). The rotation angle θs2 is initialized (θs2 = 5 degrees) (step S203). Next, the image processing device 5 inverts the first template image and creates a third template image by further rotating it by the rotation angle θs2 (step S204). While translating the third template image in parallel with respect to the first template image, the second matching rate at each position of the third template image relative to the first template image is calculated by template matching (step S205). The processes of steps S204 and S205 are repeated, for example, every time the rotation angle θs2 changes by 5 degrees, until the rotation angle θs2 reaches 360 degrees (step S206; No, step S207). When the rotation angle θs2 reaches 360 degrees (step S206; Yes), the image processing device 5 extracts the maximum value of the second matching rate from a plurality of second matching rates (step S208), obtains the shift amount ΔS2 and rotation angle θs2 of the third template image relative to the first template image when the extracted second matching rate is at its maximum (step S209), and uses the shift amount ΔS2 and rotation angle θs2 obtained in step S209 to convert the coordinate values of the feature points on the third template image when the second matching rate is at its maximum from the third coordinate system Σ3 to the first coordinate system Σ1 (step S210). Next, the image processing device 5 compares the coordinate values of corresponding feature points in the first template image and the third template image, and sets a straight line passing through multiple feature points with matching coordinate values as the Xt axis of the template coordinate system Σt (step S211). Furthermore, the image processing device 5 sets a line that is orthogonal to the Xt axis of the template coordinate system Σt and passes through the origin Ot as the Yt axis of the template coordinate system Σt (S212).
[0044] According to the image processing device 5 of this embodiment, a template coordinate system can be automatically set for a template image by utilizing the symmetry of the work area. The template coordinate system automatically set for a template image corresponds to a work coordinate system based on the work area. In template matching for images captured by the camera 4, the image processing device 5 can output a work-referenced template coordinate system to the control device 3 in the same way, even when using multiple types of template images with different work area positions and orientations. The user does not need to manually set a work-referenced template coordinate system for template images captured by the camera 4, saving the user effort. Furthermore, it is possible to avoid situations where the template coordinate system is set to an offset position and orientation from the position and orientation it should originally be set to, due to user-initiated settings, thereby stabilizing the quality of the work position and orientation data output from the image processing device 5 to the control device 3. In addition, when capturing a template image with a camera, the user does not need to consider the position and orientation of the work area as long as the work area is included, saving the effort required to capture the template image. In other words, according to the image processing device 5 of this embodiment, the user's effort required to set the coordinate system for a template image can be reduced.
[0045] The image processing device 5 according to this embodiment has both the function of setting the origin of the template coordinate system with respect to the template image, and the function of setting each axis of the template coordinate system. However, the image processing device 5 may be configured to have only one of these functions. For example, if the work area is circular, the image processing device 5 can set the origin of the template coordinate system at the center of symmetry of the work area and set each axis of the template coordinate system according to any rule, by having at least the function of setting the origin of the template coordinate system.
[0046] One feature of the image processing apparatus 5 according to this embodiment is that it sets a template coordinate system for a template image by utilizing the symmetry of the work area, such as rotational symmetry and line symmetry of the work area. Therefore, as long as it is possible to set a template coordinate system for a template image by utilizing the symmetry of the work area, the method is not limited to this embodiment.
[0047] One feature of the image processing apparatus 5 according to this embodiment is that it utilizes the rotational symmetry of the work region to set the origin of the template coordinate system at the center of symmetry of the rotational symmetry of the work region. Therefore, as long as it is possible to set the origin of the template coordinate system at the center of symmetry of the rotational symmetry of the work region, the method is not limited to this embodiment.
[0048] The image processing apparatus 5 according to this embodiment is characterized by using the line symmetry of the work area to set one axis of the template coordinate system on the axis of symmetry of the line symmetry of the work area. Therefore, as long as it is possible to set one axis of the template coordinate system on the axis of symmetry of the line symmetry of the work area, the method is not limited to this embodiment.
[0049] One feature of the image processing apparatus 5 according to this embodiment is that it sets a template coordinate system for a template image using template matching. For example, if the image to be matched with the template image (referred to as the detection image) is fixed, even if the position and orientation of the work area included in the template image change, the positional relationship between the work area included in the template image and the work area included in the detection image remains fixed when the matching rate reaches a predetermined value (including maximum and minimum values). By utilizing the fact that the positional relationship between the work area included in the template image and the work area included in the detection image does not change when the matching rate reaches a predetermined value, a coordinate system can be set for the template coordinates.
[0050] Various data, such as the coordinate system setting program, stored in the storage device 9 may be distributed to users by being recorded on removable media, or by being downloaded to the control device 2 via a network.
[0051] Further details regarding this embodiment and its modifications are disclosed below. (Note 1) The image processing device 5 includes a storage unit 51 that stores data relating to a template image for extracting a symmetrical work region from an input image by template matching, a criterion identification unit 52 that identifies a criterion for the symmetry of the work region, and a coordinate system setting unit 60 that sets the coordinate system of the template image based on the criterion. (Note 2) In the image processing device 5 described in Appendix 1, the origin of the coordinate system is set to the rotational symmetry center of the work region. (Note 3) In the image processing apparatus 5 described in Appendix 2, the reference identification unit 52 includes a matching rate calculation unit 54 that calculates multiple matching rates between a template image and multiple other template images obtained by rotating the template image by multiple rotation angles, and a symmetry center identification unit 58 that identifies a symmetry center based on the multiple matching rates. (Note 4) In the image processing apparatus 5 described in Appendix 3, the reference identification unit 52 further includes an extraction unit 56 that extracts the maximum value from among a plurality of matching rates or a plurality of values derived from each of the plurality of matching rates. The symmetry center identification unit 58 identifies the symmetry center using the template image and another template image corresponding to the maximum value. (Note 5) In the image processing apparatus 5 described in Appendix 1 or Appendix 2, the X-axis or Y-axis of the coordinate system is set to the axis of symmetry of the line symmetry of the work area. (Note 6) In the image processing apparatus 5 described in Appendix 5, the reference identification unit 52 includes a matching rate calculation unit 54 that calculates multiple matching rates between a template image and multiple other template images obtained by rotating the template image by multiple rotation angles, and a symmetry axis identification unit 59 that identifies the axis of symmetry based on the multiple matching rates. (Note 7) In the image processing apparatus 5 described in Appendix 6, the reference identification unit 52 further includes an extraction unit 56 that extracts the maximum value from among a plurality of matching rates or a plurality of values derived from each of the plurality of matching rates. The symmetry axis identification unit 59 identifies the symmetry axis using a template image and another template image corresponding to the maximum value. (Note 8) The image processing program provides a computer that stores data about a template image for extracting a symmetrical work region from an input image using template matching, and implements means for identifying a criterion for the symmetry of the work region and means for setting the coordinate system of the template image based on the criterion. (Note 9) An image processing device 5, which has means for storing data relating to a template image for extracting a symmetrical work region from an input image by template matching, performs an image processing method comprising the steps of: identifying a criterion for the symmetry of the work region; and setting the coordinate system of the template image based on the criterion. (Note 10) The image processing device 5 includes a storage unit 51 that stores data relating to a first template image for extracting a work area from an input image by template matching, a matching rate calculation unit 54 that calculates the matching rate between the first template image and a plurality of second template images generated from the first template image, and a coordinate system setting unit 60 that sets the coordinate system of the first template image based on a plurality of matching rates between the first template image and the plurality of second template images. (Note 11) In the image processing device 5 described in Appendix 10, the multiple second template images are images obtained by rotating the first template image, and the coordinate system setting unit 60 sets the origin of the coordinate system based on multiple matching rates. (Note 12) The image processing device 5 described in Appendix 11 further includes an extraction unit 56 that extracts the maximum value from among a plurality of matching rates or a plurality of values derived from each of the plurality of matching rates. The coordinate system setting unit 60 sets the origin using a template image and another template image corresponding to the maximum value. (Note 13) In the image processing device 5 described in Appendix 10, the multiple second template images are images obtained by inverting and rotating the first template image, and the coordinate system setting unit 60 sets the X axis or Y axis of the coordinate system based on multiple matching rates. (Note 14) The image processing device 5 described in Appendix 13 further includes an extraction unit 56 that extracts the maximum value from among a plurality of matching rates or a plurality of values derived from each of the plurality of matching rates. The coordinate system setting unit 60 sets the X axis or Y axis of the coordinate system using the first template image and the second template image corresponding to the maximum value. (Note 15) The image processing program provides a computer that stores data relating to a first template image for extracting a symmetrical work region from an input image by template matching, a means for calculating multiple matching rates between the first template image and multiple second template images generated from the first template image, and a means for setting the coordinate system of the first template image based on the multiple matching rates. (Note 16) An image processing device 5, which has means for storing data relating to a first template image for extracting a symmetrical work region from an input image by template matching, performs an image processing method comprising the steps of calculating multiple matching rates between a first template image and a plurality of second template images generated from the first template image, and setting the coordinate system of the first template image based on the plurality of matching rates.
[0052] While embodiments of this disclosure have been described in detail, this disclosure is not limited to the individual embodiments described above. These embodiments can be added, replaced, modified, partially deleted, etc., in any way that does not depart from the spirit of the invention or the idea and spirit of the invention derived from the claims and their equivalents. For example, the order of operations and processes in the embodiments described above are shown as examples only and are not limited thereto. The same applies when numerical values or mathematical formulas are used in the description of the embodiments described above. [Explanation of symbols]
[0053] 1...Robot system, 2...Robot, 3...Control device, 4...Camera, 5...Image processing device, 6...Processor, 50...Data receiving unit, 51...Storage unit, 52...Reference identification unit, 53...Image creation unit, 54...Matching rate calculation unit, 55...Matching score calculation unit, 56...Extraction unit, 57...Coordinate value transformation unit, 58...Symmetry center identification unit, 59...Symmetry axis identification unit, 60...Coordinate system setting unit.
Claims
1. A storage unit that stores data related to a template image for extracting a symmetrical work region from an input image using template matching, A criterion identification unit that identifies the symmetry criteria of the work region, The system comprises a coordinate system setting unit that sets the coordinate system of the template image based on the aforementioned criteria, The origin of the coordinate system is set to the rotational symmetry center of the work region. The aforementioned standard specification unit is, A matching rate calculation unit calculates multiple matching rates between the template image and multiple other template images obtained by rotating the template image by multiple rotation angles, An image processing apparatus having a symmetry center identification unit that identifies the symmetry center based on the plurality of matching rates.
2. The aforementioned standard specification unit is, The system further includes an extraction unit that extracts the maximum value from the plurality of matching rates or the plurality of values derived from the plurality of matching rates, The symmetry center identification unit identifies the symmetry center using the template image and the other template image corresponding to the maximum value. The image processing apparatus according to claim 1.
3. A storage unit that stores data relating to a template image for extracting a symmetrical work region from an input image by template matching, A criterion identification unit that identifies the symmetry criteria of the work region, The system comprises a coordinate system setting unit that sets the coordinate system of the template image based on the aforementioned criteria, The X-axis or Y-axis of the aforementioned coordinate system is set to the axis of symmetry of the line symmetry of the work area, The aforementioned standard specification unit is, A matching rate calculation unit calculates multiple matching rates between the template image and multiple other template images obtained by rotating the template image by multiple rotation angles, An image processing apparatus having a symmetry axis identification unit that identifies the symmetry axis based on the plurality of matching rates.
4. The aforementioned standard specification unit is, The system further includes an extraction unit that extracts the maximum value from the plurality of matching rates or the plurality of values derived from the plurality of matching rates, The symmetry axis identification unit identifies the symmetry axis using the template image and the other template image corresponding to the maximum value. The image processing apparatus according to claim 3.
5. A computer that stores data about template images for extracting symmetrical work regions from input images using template matching, A criterion identification means for identifying the symmetry criteria of the work area, An image processing program that implements means for setting the coordinate system of the template image based on the aforementioned criteria, The origin of the coordinate system is set to the rotational symmetry center of the work region. The aforementioned standard identification means is, means for calculating multiple matching rates between the template image and multiple other template images obtained by rotating the template image by multiple rotation angles, An image processing program having means for identifying the center of symmetry based on the aforementioned multiple matching rates.
6. A computer that stores data relating to a template image for extracting a symmetrical work region from an input image by template matching, A criterion identification means for identifying the symmetry criteria of the work area, An image processing program that implements means for setting the coordinate system of the template image based on the aforementioned criteria, The X-axis or Y-axis of the aforementioned coordinate system is set to the axis of symmetry of the line symmetry of the work area, The aforementioned standard identification means is, means for calculating multiple matching rates between the template image and multiple other template images obtained by rotating the template image by multiple rotation angles, An image processing program having means for identifying the axis of symmetry based on the plurality of matching rates.
7. An image processing method performed by an image processing device having means for storing data relating to a template image for extracting a symmetrical work region from an input image by template matching, A first step of identifying the symmetry criteria for the work area, The system comprises a second step of setting the coordinate system of the template image based on the aforementioned criteria, The origin of the coordinate system is set to the rotational symmetry center of the work region. The first step is, A step of calculating multiple matching rates between the template image and multiple other template images obtained by rotating the template image by multiple rotation angles, An image processing method comprising the step of identifying the center of symmetry based on the plurality of matching rates.
8. An image processing method performed by an image processing apparatus having means for storing data relating to a template image for extracting a symmetrical work region from an input image by template matching, A first step of identifying the symmetry criteria for the work area, The system comprises a second step of setting the coordinate system of the template image based on the aforementioned criteria, The X-axis or Y-axis of the aforementioned coordinate system is set to the axis of symmetry of the line symmetry of the work area, The first step is, A step of calculating multiple matching rates between the template image and multiple other template images obtained by rotating the template image by multiple rotation angles, An image processing method comprising the step of identifying the axis of symmetry based on the plurality of matching rates.
9. A storage unit that stores data relating to a first template image for extracting a work area from an input image by template matching, A matching rate calculation unit calculates the matching rate between the first template image and a plurality of second template images generated from the first template image, A coordinate system setting unit sets the coordinate system of the first template image based on a plurality of matching rates between the first template image and the plurality of second template images, An image processing apparatus comprising the following:
10. The aforementioned plurality of second template images are images obtained by rotating the first template image. The coordinate system setting unit sets the origin of the coordinate system based on the plurality of matching rates. The image processing apparatus according to claim 9.
11. The system further comprises an extraction unit that extracts the maximum value from among the plurality of matching rates or a plurality of values derived from each of the plurality of matching rates, The coordinate system setting unit sets the origin using the first template image and the second template image corresponding to the maximum value. The image processing apparatus according to claim 10.
12. The aforementioned plurality of second template images are images obtained by inverting and rotating the first template image. The coordinate system setting unit sets the X-axis or Y-axis of the coordinate system based on the plurality of matching rates. The image processing apparatus according to claim 9.
13. The system further comprises an extraction unit that extracts the maximum value from among the plurality of matching rates or a plurality of values derived from each of the plurality of matching rates, The coordinate system setting unit sets the X-axis or Y-axis of the coordinate system using the first template image and the second template image corresponding to the maximum value. The image processing apparatus according to claim 12.
14. A computer that stores data relating to a first template image for extracting a symmetrical work region from an input image by template matching, means for calculating multiple matching rates between the first template image and a plurality of second template images generated from the first template image, means for setting the coordinate system of the first template image based on the plurality of matching rates, An image processing program to achieve this.
15. An image processing method performed by an image processing device having means for storing data relating to a first template image for extracting a symmetrical work region from an input image by template matching, A step of calculating multiple matching rates between the first template image and a plurality of second template images generated from the first template image, A step of setting the coordinate system of the first template image based on the plurality of matching rates, An image processing method comprising the following.