Image processing system and image processing methods

Abstract

Image processing system (100) which features: an industrial machine (10); a visual sensor (70); a visual pattern for determining a relative positional relationship between the visual sensor (70) and the visual pattern placed at a detection target point; an image processing unit (202) configured to determine a relative positional relationship between the visual sensor (70) and the visual pattern from an image of the visual pattern captured by the visual sensor (70); and a focusing process control unit (502) configured to cause the visual sensor (70) to focus on the visual pattern by causing the visual sensor (70) to capture an image of the visual pattern while changing a relative position between the visual sensor (70) and the visual pattern in a predetermined direction with the visual pattern as the basis on the determined relative position relationship, and determining a degree of focusing of the visual pattern in a state in which the visual pattern is captured in a field of view of the visual sensor (70), wherein the visual sensor (70) is attached to a predetermined movable part of the industrial machine (10), and the focusing process control unit (502) moves the visual sensor (70) into a focused position in which the visual sensor (70) focuses on the visual pattern by causing the visual sensor (70) to capture an image of the visual pattern while moving the visual sensor (70) through the industrial machine (10) in the predetermined direction based on the relative position relationship, and determines a degree of focusing of the visual pattern in a state in which the visual pattern is captured in a field of view of the visual sensor (70), and wherein the visual sensor (70) is calibrated, and in order to align the visual sensor (70) in the specified direction with the visual pattern as a basis, the focusing process control unit (502) uses calibration data which represents a position and orientation of the visual sensor (70) with a specified reference position in the industrial machine (10) as a basis.

Description

[0001] The present invention relates to an image processing system and to an image processing method.

[0002] Systems are known that are designed to mount a visual sensor on a moving part of an industrial robot or other industrial machine and to perform visual inspection, position detection, and similar operations on a workpiece using image processing (see, for example, PTL 1). To mount a visual sensor on a moving part of an industrial robot or similar machine and to perform position detection and similar operations on a workpiece, it is necessary to determine a relative positional relationship between the coordinate system of the visual sensor and a standard coordinate system defined for the industrial robot or similar machine; that is, to calibrate the visual sensor. Various techniques are known in this field as technologies for calibrating a visual sensor (e.g., PTL 2 to 5).

[0003] In a robot system, as described above, adjustments must be made to focus a camera on a target position. In such a case, the distance between a target object and the camera is generally adjusted by a user rotating a focusing ring on the camera or by the user moving the robot using a programming handheld device. In contrast, PTL 5 describes that “an image processing system SYS determines a normal line V with respect to a defined measuring point Wp on a given workpiece W according to a placement state of the workpiece W and varies the position and orientation of a two-dimensional camera 310 such that the determined normal line V coincides with the optical axis of the two-dimensional camera 310 (S1).”By varying the distance between the two-dimensional camera 310 and the measuring point Wp so that the defined normal line V coincides with the optical axis of the two-dimensional camera 310, the image processing system SYS focuses the two-dimensional camera 310 on the measuring point Wp." (Summary). [PTL 1] JP 2018 – 091 774 A [PTL 2] JP 2014 - 128 845 A [PTL 3] JP H8 – 210 816 A [PTL 4] JP 2018 - 192 569 A [PTL 5] JP 2018 - 194 542 A

[0004] EP 3 221 095 B1 discloses a robot comprising a command receiving unit that receives a calibration start command and an arm that changes a positional relationship between a marker indicating a reference point and a capture unit when the calibration start command is received, wherein the calibration of a coordinate system of the capture unit and a coordinate system of the robot is performed on the basis of an image in which the marker is captured by the capture unit after the positional relationship between the capture unit and the marker has changed.

[0005] EP 2 783 814 A2 discloses a robot system comprising a robot arm, a camera, a calibration device with a marker enabling image recognition, and a calibration device configured to derive a correlation between camera coordinates (which are coordinates in a photographed image) and robot coordinates using the robot arm as a reference. The robot arm is configured to assume a posture corresponding to the relative position of the camera with respect to the marker.

[0006] US 2016 / 0354929A1 discloses a robot comprising a local coordinate system derivation section that derives a local coordinate system with two axes parallel to a work plane and orthogonal to each other, based on an image in which markers representing three or more points on the non-horizontal work plane are captured, and a control parameter capture section that captures control parameters via the local coordinate system.

[0007] While PTL 5 describes a configuration for performing focusing by adjusting the distance between a camera and a specific workpiece, the configuration in PTL 5 requires complex processing, such as processing to specify the shape of the workpiece, processing to determine the workpiece's placement status, and processing to establish a normal line for the workpiece, as well as settings for these processing steps. Furthermore, the workpiece can have various shapes. Therefore, it is an object of the invention to potentially improve the acquisition of workpiece placement information, the acquisition of a normal line, and / or the calculation of a focusing degree.

[0008] This task is solved in each case by the characteristics of the independent claims.

[0009] One aspect of the present disclosure relates to an image processing system comprising: an industrial machine; a visual sensor; a visual pattern for determining a relative positional relationship between the visual sensor and the visual pattern, which is placed at a recognition target point; an image processing unit configured to determine a relative positional relationship between the visual sensor and the visual pattern from an image of the visual pattern captured by the visual sensor;and a focusing process control unit designed to cause the visual sensor to focus on the visual pattern by causing the visual sensor to capture an image of the visual pattern while changing a relative position between the visual sensor and the visual pattern in a predetermined direction with the visual pattern as the basis on the determined relative position relationship, and determining a degree of focusing of the visual pattern in a state in which the visual pattern is captured in a field of view of the visual sensor; wherein the visual sensor is attached to a predetermined moving part of the industrial machine, and the focusing process control unit moves the visual sensor into a focused position in which the visual sensor focuses on the visual pattern by causing the visual sensor to capture an image of the visual pattern while moving the visual sensor through the industrial machine in the predetermined direction based on the relative positional relationship, and determines a degree of focusing of the visual pattern in a state in which the visual pattern is captured in a field of view of the visual sensor, and wherein the visual sensor is calibrated, and in order to align the visual sensor in the specified direction using the visual pattern as a basis, the focusing process control unit uses calibration data that represents a position and orientation of the visual sensor with a specified reference position in the industrial machine as a basis.

[0010] Another aspect of the present disclosure relates to an image processing method that includes: determining a relative positional relationship between a visual sensor and a visual pattern located at a detection target point from an image of the visual pattern captured by the visual sensor; and causing the visual sensor to focus on the visual pattern by causing the visual sensor to capture an image of the visual pattern while changing a relative position between the visual sensor and the visual pattern in a predetermined direction based on the determined relative positional relationship, with the visual pattern as the basis; and determining a degree of focus of the visual pattern in a state in which the visual pattern is captured in a field of view of the visual sensor. wherein the visual sensor is attached to a predetermined moving part of an industrial machine, and a focusing process control unit moves the visual sensor into a focused position in which the visual sensor focuses on the visual pattern by causing the visual sensor to capture an image of the visual pattern while moving the visual sensor through the industrial machine in the predetermined direction based on the relative positional relationship, and determines a degree of focusing of the visual pattern in a state in which the visual pattern is captured in a field of view of the visual sensor, and wherein the visual sensor is calibrated, and in order to align the visual sensor in the specified direction using the visual pattern as a basis, the focusing process control unit uses calibration data that represents a position and orientation of the visual sensor with a specified reference position in the industrial machine as a basis.

[0011] Placing a visual pattern at a detection target point allows a visual sensor to automatically focus on the point where the visual pattern is placed.

[0012] The object, features and advantages and other tasks, features and advantages will become more apparent from the detailed description of typical embodiments of the present invention, which are illustrated in the accompanying drawings. Fig. Figure 1 is a graphical representation depicting a device configuration of a robot system (an image processing system) according to one embodiment. Fig. Figure 2 is a graphical representation depicting schematic hardware configurations of a control device, a visual sensor, and a robot control device. Fig. Figure 3 is a graphical representation of functional blocks that depicts functional configurations of the visual sensor control device and the robot control device. Fig. Figure 4 is a top view of a calibration device. Fig. 5 is a process plan that represents a focusing process. Fig. Figure 6 is a graphic representation that shows an example of image data of a license plate captured by a camera.

[0013] Next, an embodiment of the present disclosure will be described with reference to the drawings. In the referenced drawings, similar components or functional parts are labelled with similar symbols. For ease of understanding, the drawings may use different scales. Furthermore, a configuration shown in a drawing is an example of an implementation of the present invention, and the present invention is not limited to the configuration shown.

[0014] Fig. Figure 1 is a graphical representation depicting a device configuration of a robot system 100 according to one embodiment. As shown in Fig. As shown in Figure 1, the robot system 100 comprises an industrial robot (hereinafter referred to as the robot) 10 as an industrial machine, a robot control device 50 that controls the robot 10, a visual sensor 70, and a control device 20 of a visual sensor that controls the visual sensor 70. The robot system 100 is a system for detecting the position of a workpiece based on an image of the workpiece captured by the visual sensor 70 and for performing predefined tasks such as inspection, handling, and machining of the workpiece. Although the robot 10 in the example is in Fig. If 1 is a vertical articulated robot, a different type of robot can be used.

[0015] The control device 20 of the visual sensor and the robot control device 50 are connected via a data transmission interface and can exchange various types of information.

[0016] A tool 31, acting as an end effector, is attached to the tip surface of a flange 11 on the arm tip of the robot 10. The tool 31 is, for example, a hand for gripping a workpiece. The robot 10 performs a predefined task, such as handling a workpiece, according to the control provided by the robot control device 50. The visual sensor 70 is attached to a support part of the tool 31 (i.e., a predefined movable part of the robot 10).

[0017] The visual sensor 70 is controlled by the visual sensor's control device 20 and captures an image of an image capture target, such as a workpiece. An ordinary two-dimensional camera can be used as the visual sensor 70, or a three-dimensional sensor, such as a stereo camera, can be used. The visual sensor 70 includes an image sensor (such as a CCD or a CMOS) that receives a subject image and performs a photoelectric conversion on the image, an optical lens that condenses the subject image onto the image sensor, and the like. The visual sensor 70 transmits captured image data to the visual sensor's control device 20. It should be noted that Fig. 1 an image M1 of a calibration device J captured by the visual sensor 70 in a Fig. 1 represents the depicted state.

[0018] Fig. Figure 2 is a graphical representation depicting schematic hardware configurations of the visual sensor control device 20 and the robot control device 50. The visual sensor control device 20 can be configured as an ordinary computer in which a processor 21 is connected via a bus to a memory 22 (including a ROM, a RAM, non-volatile memory, and the like), an input / output interface 23 for data transmission to an external device, and the like, as shown in Figure 2. Fig. Figure 2 shows that the robot control device 50 can be configured as an ordinary computer, in which a processor 51 is connected via a bus to a memory 52 (including a ROM, a RAM, non-volatile memory, and the like), an input / output interface 53 for data transmission with an external device, an operator unit 54 comprising various operating switches and the like. The robot control device 50 can also be connected to a teaching device (such as a programming handheld) for teaching the robot 10.Although the present embodiment shows a device configuration in which the control device 20 of the visual sensor and the robot control device 50 are separate devices, the control device 20 of the visual sensor and the robot control device 50 can be configured to be the same device. For example, functions of the control device 20 of the visual sensor can be integrated into the robot control device 50.

[0019] A robot coordinate system C1 is defined for the robot 10. The robot coordinate system C1 is a control coordinate system for controlling the operation of each moving element in the robot 10 and is fixed in three-dimensional space. Although, in the present embodiment, it is assumed by way of example that the origin of the robot coordinate system C1 is defined at the center of a base part of the robot 10, the robot coordinate system C1 can be defined such that it has a different position and orientation.

[0020] A flange coordinate system C2 is defined at the tip face of the flange 11. The flange coordinate system C2 is a control coordinate system for controlling the position and orientation of the flange 11 within the robot coordinate system C1. Although, for the present embodiment, the origin of the flange coordinate system C2 is assumed to be located at the center of the tip face of the flange 11, and a z-axis of the system is defined to coincide with the central axis of the flange 11, the flange coordinate system C2 can be defined to have a different position and orientation.When a wrist (the tool 31) of the robot 10 is moved, the robot controller 50 (the processor 51) sets the flange coordinate system C2 in the robot coordinate system C1 and controls a servo motor of each joint in the robot 10 so that the flange 11 is placed in the position and orientation specified by the set flange coordinate system C2. In this way, the robot controller 50 can position the flange 11 (the tool 31) in any position and orientation in the robot coordinate system C1.

[0021] A sensor coordinate system C3 is defined for the visual sensor 70. The sensor coordinate system C3 is a coordinate system for defining the coordinates of each pixel in image data acquired by the visual sensor 70 and is defined with respect to the visual sensor 70 such that the origin of the system is located in the center of a light-receiving surface (or an optical lens) of the image sensor in the visual sensor 70, an x-axis and a y-axis of the system are placed parallel to a lateral direction and a longitudinal direction of the image sensor, and the z-axis of the system coincides with the line of sight (the optical axis) of the visual sensor 70.

[0022] As described in detail below, the robot system 100 is designed as an image processing system that performs a focusing process by moving the visual sensor 70 in a predetermined direction with respect to a visual pattern. This process determines the relative positional relationship between the visual sensor 70 and the visual pattern, which is positioned at a detection target point of the visual sensor 70, by capturing an image of the visual pattern. According to the present embodiment, the detection target point of the visual sensor 70 is the upper surface of a base 60 on which a workpiece is mounted, and the visual pattern is formed on a calibration device J. The visual pattern simply needs to have a visually recognizable shape, such as a pattern using color or an engraved marking (an irregularity).The visual pattern can be formed directly at the detection target position (the upper surface of the base 60). A visual pattern includes various visual representations such as a pattern, a marking, a label, an identification mark, and a symbol. Using the visual pattern to obtain a relative position relationship with the visual sensor 70 enables highly precise acquisition of this relationship. In particular, the precision can be improved compared to a configuration where a relative position relationship is determined between a camera and a specific workpiece, as in the PTL 5 mentioned above.

[0023] The following description focuses on one function of the robot system 100 as such an image processing system.

[0024] Fig. Figure 3 is a graphical representation of functional blocks that depicts functional configurations of the control device 20 of the visual sensor and the robot control device 50, when attention is focused on the function of the robot system 100 as an image processing system. As in Fig. As shown in Figure 3, the control device 20 of the visual sensor includes an image processing unit 202, which performs image processing on an input image 201 captured by the visual sensor 70, and a calibration data storage unit 204, which stores calibration data that determines a relative position of the visual sensor 70 (of the sensor coordinate system C3) with respect to a standard coordinate system (the robot coordinate system C1 or the flange coordinate system C2) that is specified for the robot 10.

[0025] As a function for providing for the handling of a workpiece and the like, the image processing unit 202, for example, has a function for recognizing a workpiece from an input image using a model pattern and for recognizing the position and orientation of the workpiece. Furthermore, according to the present embodiment, the image processing unit 202 includes a focus degree calculation unit 203, which calculates a focus degree of a target object captured in a captured image.

[0026] The calibration data storage unit 204 stores calibration data for the visual sensor 70. The calibration data storage unit 204 is, for example, a non-volatile memory (such as an EEPROM). The calibration data for the visual sensor 70 includes internal parameters and one external parameter of the visual sensor 70 (the camera). The internal parameters include optical data such as lens distortion and focal length. The external parameter is the position of the visual sensor 70 with a predefined reference position (such as the position of the flange 11) as a basis.

[0027] A geometric transformation characteristic within the camera and a geometric relationship between a three-dimensional space in which an object is located and a two-dimensional image plane are determined using the calibration data. Furthermore, the three-dimensional spatial positions of the visual sensor 70 and the calibration device J can be uniquely determined from a characteristic of the calibration device J and an image of the calibration device J captured by the visual sensor 70. In other words, the position and orientation of the calibration device, based on the position of the visual sensor 70, can be determined from information about the calibration device J in an image captured by the calibrated visual sensor 70.

[0028] Fig. Figure 4 is a top view of the calibration device J. Various calibration devices known in the relevant field can be used as the calibration device J, which allow the position and orientation of a calibration device to be determined from an image captured by the visual sensor 70, using the position of the visual sensor 70 as a basis. In the case of the calibration device J in Fig. 4. This is a device that enables the acquisition of information required for the calibration of the visual sensor 70 by capturing an image of a dot pattern placed on a plane using the visual sensor 70, and which fulfills the following three requirements: (1) a grid spacing of the dot pattern is known, (2) a certain number or more grid points are present, and (3) a grid point corresponding to each grid point is uniquely determinable. Without being limited to a device in which a characteristic such as a predetermined dot pattern is placed on a two-dimensional plane, as in Fig. As shown in Figure 4, the calibration device J can be a device in which a characteristic is placed on a three-dimensional solid and which only provides three-dimensional position information, including position information in a vertical direction (the direction of an arrow 91 in Fig. 1) must enable in addition to two-dimensional position information (the x-direction and the y-direction). Furthermore, the calibration device J can be the same device used when the calibration data of the visual sensor 70 are acquired, or it can be a different one. It should be noted that the internal parameters of the aforementioned calibration data are used to calculate the position and orientation of a dot pattern, based on the position of the visual sensor 70 and an image of the dot pattern captured by the visual sensor 70. Although, in the present embodiment, the calibration device J, which includes the dot pattern, is as shown in Fig. 4 shown, as a target object used to determine the position and orientation of the target object as viewed from an image acquisition device, any target object type can be used instead that allows the position and orientation of the target object to be determined as viewed from the image acquisition device (the visual sensor 70).

[0029] The calibration device J according to the present embodiment is attached to a target location on which the visual sensor 70 is focused (i.e., the upper surface of the base 60).

[0030] As in Fig. As shown in Figure 3, the robot control device 50 includes an operating control unit 501, which controls the operation of the robot 10 according to an operating program. Furthermore, according to the present embodiment, the robot control device 50 includes a focusing process control unit 502, which controls the movement of the robot 10 based on a focus degree calculated by the focus degree calculation unit 203 and the movement of the visual sensor 70 into a focused position.

[0031] It should be noted that a functional block in the control device 20 of the visual sensor or the robot control device 50, which is in Fig. 3 shown, may be provided by the execution of different types of software stored in memory by the processor (the CPU) in the control device 20 of the visual sensor or the robot control device 50, or may be provided by a configuration that is mainly based on hardware such as an application-specific integrated circuit (ASIC).

[0032] Fig. Figure 5 is a flowchart depicting the sequence of a focusing operation (an image processing procedure) by the focusing operation control unit. It is assumed that calibration data for the visual sensor 70 is previously stored in the calibration data storage unit 204. The focusing operation in Fig. 5 is executed under control by the processor 51 in the robot control device 50.

[0033] First, the focusing process control unit 502 calculates and obtains a positional relationship between the calibration device J and the visual sensor 70 based on an image of the calibration device J captured by the visual sensor 70. In this case, the focusing process control unit 502 uses the calibration data stored in the calibration data storage unit J.

[0034] Using the relative positional relationship between the visual sensor 70 and the calibration device J obtained in step S1, the focusing process control unit 502 acquires an image of the calibration device J while moving the visual sensor 70 so that the optical axis of the visual sensor 70 is parallel to the normal direction of the upper surface of the calibration device J (the direction of arrow 91 in Fig. 1) proceeds (step S2). At this point, the focusing process control unit 502 causes the focusing degree calculation unit 203 to calculate a focusing degree of an image of a target object captured in the image (i.e., of points of the calibration device J) (step S3).

[0035] Subsequently, the focusing process control unit 502 determines whether the focusing degree obtained in step S3 is the highest focusing degree (step S4) by comparing the focusing degree calculated by the focusing degree calculation unit 203 with a reference value (focusing degree reference value) as the value when the focusing degree of the points of the calibration device J is highest. The reference value used here (focusing degree reference value) can be previously stored by the robot control device 50 (focusing process control unit 502); or the focusing process control unit 502 can, for example, perform a process to move the visual sensor 70 within a predefined range of motion and to set the highest focusing degree as the focusing degree reference value.If the focus level is not determined to be at its highest (S4: NO), the Focusing Operation Control Unit 502 continues processing to check the focus level while moving the visual sensor 70 (steps S2 to S4). If the focus level is determined to be at its highest (S4: YES), the Focusing Operation Control Unit 502 records the position of the visual sensor 70 when the focus level is at its highest (focused position) as the image acquisition position (step S5).

[0036] The focus level calculation unit 203 can employ various calculation techniques known in the relevant field, such as a phase difference detection method and a contrast detection method, as techniques for calculating a focus level. For example, if it is determined that the reliability of a focus level detected by a phase difference detection method is low, the focus level calculation unit 203 can use a technique for selecting a pixel area in which the phase difference detection method is used, based on a detection result from a contrast detection method, and for calculating a focus level by the phase difference detection method using a signal in the selected pixel area, as described in the unexamined Japanese patent publication (Kokai) No. 2013-29803 A.The focus degree calculation unit 203 can obtain data used to calculate a focus degree from a functional element for calculating a focus in the visual sensor 70.

[0037] A predefined operating range, within which the visual sensor 70 is moved during processing in steps S2 to S4, can be determined based on an adjustment range of the camera's focal length. For example, the robot 10 can be controlled to move the visual sensor 70 within an image acquisition distance range determined by the focusing range of a focusing ring on the visual sensor 70. Since the positional relationship between the visual sensor 70 and the calibration device J is known by step S1, such motion control of the visual sensor 70 is enabled.

[0038] The image acquisition position recorded in step S5 is used as the image acquisition position for positioning the visual sensor 70 when a predefined task, such as a visual inspection of a workpiece placed on the upper surface of the base 60, is performed. In this way, the focusing process in Fig. 5. Automatic positioning of the visual sensor 70 at a position where the sensor focuses on a position where the calibration device J is attached. In other words, focusing the visual sensor 70 can be carried out automatically and efficiently.

[0039] As an example of a prior calibration of the visual sensor 70 (i.e., for a case where the control device 20 of the visual sensor holds prior calibration data), it has been described above; however, if no calibration of the visual sensor 70 is performed (i.e., in a case where the control device 20 of the visual sensor does not hold any calibration data), the robot control device 50 and the control device 20 of the visual sensor can acquire calibration data using various calibration techniques known in the relevant field (such as in the PTL 2 to 5 mentioned above).

[0040] As described above, the calibration device J is used in the focusing control process in Fig. 5 according to the present embodiment, and therefore calibration at the point can be easily carried out using the calibration device J when the focusing control process is in Fig. 5. Although a method for calibrating a camera is itself a generally known technique and therefore its description is omitted, such a method is described in detail, for example, in "An Efficient and Accurate Camera Calibration Technique for 3D Machine Vision" by Roger Y. Tsai (CVPR, pp. 364–374, 1986 IEEE). As a specific means, a camera can be calibrated by causing a device, such as the one described in Fig. 4. The calibration device J shown (a dot pattern plate on which points are formed in a known geometric placement) intersects the optical axis of the camera at right angles and a measurement is carried out at the positions of two known points.

[0041] An example of an image processing technique for determining the position of the sensor coordinate system C3 is described, based on a standard coordinate system defined for the robot 10 (the robot coordinate system C1 or the flange coordinate system C2). The focusing process control unit 502 can also be configured to perform this function. Making the position of the sensor coordinate system C3 known with respect to the standard coordinate system defined for the robot 10 using the technique described here enables the optical axis of the visual sensor 70 to be directed in a predetermined direction (such as a normal direction) with respect to the calibration device J during the focusing process mentioned above.

[0042] In this example, data on the position of the visual sensor 70 in the standard coordinate system are obtained based on image data of a license plate ID detected by the visual sensor 70. Fig. Figure 6 presents an example of the ID identifier. The ID identifier is located at a target recognition position (the upper surface of base 60) and consists of a circle C and straight lines D and E perpendicular to each other. The ID identifier, for example, only needs to have a visually recognizable shape, such as a pattern formed on the upper surface of base 60 using color or an engraved mark (an irregularity).

[0043] The robot control device 50 (the focusing process control unit 502) determines the position of the sensor coordinate system C3 using the standard coordinate system defined for the robot 10 as a basis according to the following procedures. (A1) In a state where the visual sensor 70 is positioned by the robot control device 50 in an initial position PS0 and an initial position OR0, an image of the license plate ID is captured and image data JD0 of the license plate ID is acquired such that the license plate ID enters the field of view of the visual sensor 70. It is assumed that an image (JD n ), as in Fig. 6 is shown, and is won. (A2) The image processing unit 202 acquires coordinates (x n , y n ) an intersection F from an image of the image JD n The recorded license plate ID is used as the position of the license plate ID, and the area of ​​a circle C is determined as the size IS. n(Unit: pixels) of the license plate ID. The image processing unit 202 obtains a size RS (unit: mm) of the license plate ID in a real space, a focal length FD of the optical lens of the visual sensor 70, a size SS (unit: mm / pixel) of the image sensor as previously stored data. (A3) The image processing unit 202 acquires a vector (X n , Y n , Z n ) using the obtained coordinates (x n , y n ), the gained size IS n , the obtained size RS, the obtained focal length FD, and the obtained size SS. It should be noted that X n from an equation X n = x n × IS n × SS / RS can be determined. Y n Can Y be derived from an equation? n = Y n × IS n × SS / RS are determined. Z n can Z be derived from an equation n = IS n × SS × FD / RS can be determined. For the vector (Xn , Y n , Z n ) is a vector from the visual sensor 70 (i.e., the origin of the sensor coordinate system C3) to the identifier ID (in particular, the intersection F), if the image data JD n are recorded and are data that indicate a relative position (or coordinates in the sensor coordinate system C3) of the license plate ID in relation to the visual sensor 70. (A4) Similarly, the image processing unit 202 obtains a vector from the visual sensor 70 to the license plate ID from an image JD1 of the license plate ID, which has been captured at a position PS1 obtained by moving the visual sensor 70 from the initial position by a predetermined distance δx in the direction of the x-axis of the flange coordinate system and in position OR0, when the image data JD1 is captured. (A5) Similarly, the image processing unit 202 obtains a vector from the visual sensor 70 to the license plate ID from an image JD2 of the license plate ID, which has been captured at a position PS2 obtained by moving the visual sensor 70 from the initial position by a predetermined distance δy in the direction of the y-axis of the flange coordinate system and in position OR0, when the image data JD2 is captured. (A6) Similarly, the image processing unit 202 obtains a vector from the visual sensor 70 to the license plate ID from an image JD3 of the license plate ID, which has been captured at a position PS3 obtained by moving the visual sensor 70 from the initial position by a predetermined distance δz in the direction of the z-axis of the flange coordinate system and in position OR0, when the image data JD3 is captured. (A7) From the data described above, the image processing unit 202 obtains a rotation matrix that represents a position (W, P, R) of the visual sensor 70 (of the sensor coordinate system C3) in the flange coordinate system C2, by the following equation (1). [Math 1] M1=((X1−X0) / δx(X2−X0) / δy(X3−X0) / δz(Y1−Y0) / δx(Y2−Y0) / δy(Y3−Y0) / δz(Z1−Z0) / δx(Z2−Z0) / δy(Z3−Z0) / δz)

[0044] The robot control device 50 (the focusing process control unit 502) can further be configured to determine the position of the sensor coordinate system C3 using the standard coordinate system defined for the robot 10 as a basis. The procedures for this process are described below. (B1) The robot control device 50 first establishes a reference coordinate system C4 in the flange coordinate system C2 at the initial position PS0 and the initial position OR0. According to the present embodiment, the robot control device 50 establishes the reference coordinate system C4 in the flange coordinate system C2 such that the origin of the reference coordinate system C4 is located at the origin of the flange coordinate system C2 and the position (the direction of each axis) of the reference coordinate system C4 corresponds to the position (W, P, R) obtained by the aforementioned procedure. Accordingly, the directions of the x-axis, y-axis, and z-axis of the reference coordinate system C4 are parallel to the x-axis, y-axis, and z-axis, respectively, of the sensor coordinate system C3. (B2) Next, the robot control device 50 operates the robot 10 and places the visual sensor 70 (i.e., the flange 11) in a position PS4 and a position OR1 by rotating the visual sensor 70 about the z-axis of the reference coordinate system C4 from the initial position PS0 and the initial position OR0 by a position change θ1 (first position change). (B3) The image processing unit 202 operates the visual sensor 70 and captures an image of the license plate ID and obtains relative position data (X4, Y4, Z4) of the license plate ID in relation to the visual sensor 70 at that time by a technique similar to that used to determine the position mentioned above. (B4) Next, the robot control device 50 operates the robot 10 and places the visual sensor 70 in a position PS5 and a position OR2 by rotating the visual sensor 70 about the x-axis or the y-axis (i.e., an axis orthogonal to the direction of the line of sight) of the reference coordinate system C4 from the initial position PS0 and the initial position OR0 by a position change θ2 (first position change). (B5) The image processing unit 202 operates the visual sensor 70 and captures an image of the license plate ID and obtains relative position data (X5, Y5, Z5) of the license plate ID in relation to the visual sensor 70 at that time by a technique similar to that used to determine the position mentioned above.

[0045] If a vector from the origin of the reference coordinate system C4 in the flange coordinate system C2 (the origin of the MIF coordinate system C2 according to the present embodiment) to the origin of the sensor coordinate system C3, whose position is unknown, is designated by (ΔX1, ΔY1, ΔZ1), the following equations (2) and (3) apply. [Math. 2] (cos θ1−sin θ1sin θ1cos θ1)⋅(X0+ΔX1Y0+ΔY1)=(X4+ΔX1Y4+ΔY1) [Math. 3] cos θ2⋅Y0−sin θ2⋅(Z0+ΔZ1)=Y5

[0046] By solving the above-mentioned equations (2) and (3), the robot control device 50 can estimate the vector (ΔX1, ΔY1, ΔZ1) from the origin of the reference coordinate system C4 in the flange coordinate system C2 to the origin of the unknown sensor coordinate system C3.

[0047] As described above, the identifier ID is a visual pattern that allows a determination of a relative positional relationship between the visual sensor 70 and the identifier ID, and therefore the identifier ID can be used instead of the calibration device J used in the focusing process mentioned above.

[0048] As described above, according to the present embodiment, placing a visual pattern at a detection target point can cause the visual sensor to automatically focus on the point where the visual pattern is located. In other words, focusing the visual sensor can be performed automatically and efficiently.

[0049] Although the present invention has been described above using the typical embodiments, it is apparent to those skilled in the art that modifications and various other changes, omissions or additions can be made to the embodiments mentioned above without deviating from the scope of the present invention.

[0050] The configuration described in the embodiment mentioned above is applicable to a focusing process in various industrial machines equipped with a visual sensor on a moving part.

[0051] Although the in Fig. While the device configuration example shown in 1 is a configuration example in which the visual sensor is attached to the robot and the calibration device is placed in a fixed position, a configuration can be used in which the visual sensor is fixed as a stationary camera in a work area and the calibration device (the visual pattern) is attached to the robot's tool. A configuration of functional blocks similar to that shown in Fig. The configuration of function blocks shown in Figure 3 can also be applied in this case. In this case, the focusing process control unit 502 causes the visual sensor to focus on the calibration device (the visual pattern) by causing the visual sensor to capture an image of the visual pattern while moving the visual pattern relative to the visual sensor (while changing the relative position between the visual sensor and the visual pattern in a predetermined direction using the visual pattern as a basis) based on the determined relative position relationship and determining a degree of focus of the visual pattern in a state where the visual pattern is captured in the field of view of the visual sensor.

[0052] The placement of the functional blocks in the control device 20 of the visual sensor and the robot control device 50 in the in Fig. Figure 2, a graphical representation of function blocks, is an example, and various modified examples can be used for the placement of the function blocks. For example, at least some of the functions can be placed as an image processing unit 202 on the side of the robot control device 50.

[0053] Although a focused position can be determined by comparing a focus level with a focus level reference value and determining whether the two match (step S4), in which in Fig. Instead of using the focusing process shown in section 5, a focused position can be obtained by moving the camera within a predetermined operating range and searching for a position where a maximum value of a focus level is obtained.

[0054] A program for performing various types of processing, including the focusing process according to the embodiment mentioned above, can be recorded in various computer-readable recording media (such as semiconductor memories including ROM, EEPROM and flash memory, magnetic recording media and optical discs including CD-ROM and DVD-ROM). 10 robots 11 Flange 20 Control device of the visual sensor 21 processor 22 storage 23 Input / Output Interface 31 tools 50 robot control device 51 processor 52 storage 53 Input / Output Interface 54 Control unit 60 base 70 visual sensor 100 robot systems 201 Input image 202 Image processing unit 203 Focus level calculation unit 204 Calibration data storage unit

Claims

[1] Image processing system (100) comprising: an industrial machine (10); a visual sensor (70); a visual pattern for determining a relative positional relationship between the visual sensor (70) and the visual pattern placed at a detection target point; an image processing unit (202) configured to determine a relative positional relationship between the visual sensor (70) and the visual pattern from an image of the visual pattern captured by the visual sensor (70); and a focusing process control unit (502) configured to cause the visual sensor (70) to focus on the visual pattern by causing the visual sensor (70) to capture an image of the visual pattern while changing a relative position between the visual sensor (70) and the visual pattern in a predetermined direction with the visual pattern as the basis on the determined relative position relationship, and determining a degree of focusing of the visual pattern in a state in which the visual pattern is captured in a field of view of the visual sensor (70), wherein the visual sensor (70) is attached to a predetermined movable part of the industrial machine (10), and the focusing process control unit (502) moves the visual sensor (70) into a focused position in which the visual sensor (70) focuses on the visual pattern by causing the visual sensor (70) to capture an image of the visual pattern while moving the visual sensor (70) through the industrial machine (10) in the predetermined direction based on the relative position relationship, and determines a degree of focusing of the visual pattern in a state in which the visual pattern is captured in a field of view of the visual sensor (70), and wherein the visual sensor (70) is calibrated, and in order to align the visual sensor (70) in the specified direction with the visual pattern as a basis, the focusing process control unit (502) uses calibration data which represents a position and orientation of the visual sensor (70) with a specified reference position in the industrial machine (10) as a basis. [2] Image processing system (100) according to claim 1, wherein the focusing process control unit (502) changes a relative position between the visual sensor (70) and the visual pattern in a predetermined direction with respect to a plane on which the visual pattern is formed. [3] Image processing system (100) according to claim 1 or 2, wherein the focusing process control unit (502) moves the visual sensor (70) in a predetermined operating range with respect to the visual pattern and moves the visual sensor (70) into the focused position by comparing a focus level reference value, when a focus level of the visual pattern is at its highest, with a focus level obtained during the movement of the visual sensor (70). [4] Image processing system (100) according to any one of claims 1 to 3, wherein the industrial machine (10) is an industrial robot (10). [5] Image processing method which features: Determining a relative positional relationship between a visual sensor (70) and a visual pattern located at a detection target point, from an image of the visual pattern captured by the visual sensor (70); and Causing the visual sensor (70) to focus on the visual pattern by causing the visual sensor (70) to capture an image of the visual pattern while changing a relative position between the visual sensor (70) and the visual pattern in a predetermined direction with the visual pattern as the basis on the determined relative position relationship, and determining a degree of focus of the visual pattern in a state in which the visual pattern is captured in a field of view of the visual sensor (70), wherein the visual sensor (70) is attached to a predetermined moving part of an industrial machine (10), and a focusing process control unit (502) moves the visual sensor (70) into a focused position in which the visual sensor (70) focuses on the visual pattern by causing the visual sensor (70) to capture an image of the visual pattern while moving the visual sensor (70) through the industrial machine (10) in the predetermined direction based on the relative position relationship, and determines a degree of focusing of the visual pattern in a state in which the visual pattern is captured in a field of view of the visual sensor (70), and wherein the visual sensor (70) is calibrated, and in order to align the visual sensor (70) in the specified direction with the visual pattern as a basis, the focusing process control unit (502) uses calibration data which represents a position and orientation of the visual sensor (70) with a specified reference position in the industrial machine (10) as a basis.

Images