Image processing system and image processing method

Abstract

A focus state of a vision sensor is simply obtained in an image processing system using the vision sensor. An image processing system (100) includes an industrial machine, a vision sensor, a vision pattern configured at a detection target site and used to determine a relative positional relationship between the vision sensor and the vision pattern, an image processing section (202) that determines the relative positional relationship between the vision sensor and the vision pattern based on an image of the vision pattern captured by the vision sensor, and a focus operation control section (502) that changes the relative position between the vision sensor and the vision pattern in a predetermined direction based on the determined relative positional relationship with the vision pattern as a reference, and captures the vision pattern by the vision sensor to obtain a focus degree of the vision pattern, thereby focusing the vision sensor on the vision pattern while the vision pattern is reflected in a field of view of the vision sensor.

Description

Technical Field

[0001] This invention relates to image processing systems and image processing methods. Background Technology

[0002] A system is known in which a vision sensor is mounted on the movable part of an industrial robot or other industrial machinery to perform visual inspection, position detection, etc., of a workpiece through image processing (see, for example, Patent Document 1). In order to mount a vision sensor on the movable part of an industrial robot or the like for workpiece position detection, it is necessary to pre-determine the relative positional relationship of the vision sensor's coordinate system with respect to a reference coordinate system set in the industrial robot or the like; that is, the vision sensor needs to be calibrated. Various methods for calibrating vision sensors are known in the art (for example, Patent Documents 2-5).

[0003] In robot systems like those described above, adjustments are required to align the camera's focus with the position of the object being inspected. In such cases, the user typically adjusts the camera's focus by rotating the camera's focus adjustment ring or by operating a teach pendant to move the robot and thus adjust the distance between the object and the camera. On the other hand, Patent Document 5 describes a process where "the image processing system SYS determines the normal V of the measurement point Wp of the set workpiece W based on the determined configuration of the workpiece W, and changes the position and orientation of the 2D camera 310 in a manner that the determined normal V aligns with the optical axis of the 2D camera 310 (S1). The image processing system SYS changes the distance between the 2D camera 310 and the measurement point Wp in a manner that the determined normal V aligns with the optical axis of the 2D camera 310, thereby aligning the focus of the 2D camera 310 with the measurement point Wp" (excerpt from the specification).

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Application Publication No. 2018-91774

[0007] Patent Document 2: Japanese Patent Application Publication No. 2014-128845

[0008] Patent Document 3: Japanese Patent Application Publication No. 8-210816

[0009] Patent Document 4: Japanese Patent Application Publication No. 2018-192569

[0010] Patent Document 5: Japanese Patent Application Publication No. 2018-194542 Summary of the Invention

[0011] The problem that the invention aims to solve

[0012] Patent Document 5 describes a structure for adjusting focus by adjusting the distance between a camera and a specific workpiece. However, the structure in Patent Document 5 requires complex processing or settings for specifying the shape of the workpiece, determining the configuration state of the workpiece, and calculating the normal to the workpiece. Furthermore, since the workpiece may have various shapes, there may be situations where it is impossible to accurately obtain the workpiece's configuration information, the normal, or calculate the focus.

[0013] Methods for solving problems

[0014] One aspect of this disclosure is an image processing system comprising: industrial machinery; a vision sensor; a vision pattern disposed on a detection object, the vision pattern being used to determine the relative positional relationship between the vision sensor and the vision pattern; an image processing unit that determines the relative positional relationship between the vision sensor and the vision pattern based on an image obtained by the vision sensor capturing the vision pattern; and a focusing action control unit that, while the vision pattern is reflected in the field of view of the vision sensor, changes the relative position between the vision sensor and the vision pattern in a predetermined direction based on the determined relative positional relationship, and calculates the focus degree of the vision pattern by capturing the vision pattern by the vision sensor, thereby causing the vision sensor to focus on the vision pattern.

[0015] Another aspect of this disclosure is an image processing method that determines the relative positional relationship between a visual sensor and a visual pattern based on an image obtained by capturing a visual pattern disposed on a detection object area using a visual sensor. With the visual pattern reflected in the field of view of the visual sensor, the relative position of the visual sensor and the visual pattern is changed in a predetermined direction based on the determined relative positional relationship, and the focus of the visual pattern is calculated by capturing the visual pattern using the visual sensor, thereby enabling the visual sensor to focus on the visual pattern.

[0016] Invention Effects

[0017] By placing a visual pattern on the object being detected, the visual sensor can automatically focus on the area where the visual pattern is placed.

[0018] These objects, features, and advantages of the invention will become more apparent from the detailed description of typical embodiments of the invention shown in the accompanying drawings. Attached Figure Description

[0019] Figure 1 This is a diagram illustrating the device structure of a robot system (image processing system) according to one embodiment.

[0020] Figure 2 This is a diagram showing the schematic structure of the hardware for the vision sensor control device and the robot control device.

[0021] Figure 3 This is a functional block diagram representing the functional structure of the vision sensor control device and the robot control device.

[0022] Figure 4 This is a top view of the calibration fixture.

[0023] Figure 5 This is a flowchart representing the focusing action.

[0024] Figure 6 This is an example of image data obtained by representing the camera's shooting parameters. Detailed Implementation

[0025] Next, embodiments of the present disclosure will be described with reference to the accompanying drawings. In the drawings, the same reference numerals are used to denote the same components or functional parts. The scales of these drawings have been appropriately altered for ease of understanding. Furthermore, the embodiments shown in the drawings are examples for carrying out the invention, and the invention is not limited to the illustrated embodiments.

[0026] Figure 1 This is a diagram illustrating the device structure of a robot system 100 according to one embodiment. (See diagram for example.) Figure 1 As shown, the robot system 100 includes: an industrial robot (hereinafter referred to as robot) 10 as industrial machinery; a robot control device 50 for controlling robot 10; a vision sensor 70; and a vision sensor control device 20 for controlling vision sensor 70. The robot system 100 is a system for identifying the position of a workpiece based on images of the workpiece captured by the vision sensor 70, and performing predetermined tasks such as workpiece inspection, manipulation, and processing. Robot 10 in Figure 1 The example used is a vertical joint robot, but other types of robots can also be used.

[0027] The vision sensor control device 20 and the robot control device 50 are connected via a communication interface and can exchange various information.

[0028] A tool 31, serving as an end effector, is mounted on the front end face of the flange 11 at the front end of the arm of the robot 10. The tool 31 is, for example, a hand that holds a workpiece. The robot 10 performs predetermined tasks such as manipulating the workpiece under the control of the robot control device 50. A vision sensor 70 is mounted on the support of the tool 31 (i.e., a predetermined movable part of the robot 10).

[0029] The vision sensor 70, controlled by the vision sensor control device 20, captures images of objects such as workpieces. The vision sensor 70 can be a general 2D camera or a 3D sensor such as a stereo camera. The vision sensor 70 includes an image sensor (CCD, CMOS, etc.) that receives and photoelectrically converts the image of the object, and an optical lens that focuses the image of the object onto the image sensor. The vision sensor 70 sends the captured image data to the vision sensor control device 20. Furthermore, in... Figure 1 The text shows that the location is in the middle. Figure 1 The image M1 is obtained by the visual sensor 70 capturing the state of the calibration fixture J.

[0030] Figure 2 This is a diagram showing the schematic structure of the hardware of the vision sensor control device 20 and the robot control device 50. (See diagram for example.) Figure 2 As shown, the vision sensor control device 20 may also have a general computer structure, where a memory (including ROM, RAM, non-volatile memory, etc.) 22 and an input / output interface 23 for communicating with external devices are connected to the processor 21 via a bus. Similarly, the robot control device 50 may also have a general computer structure, where a memory (including ROM, RAM, non-volatile memory, etc.) 52, an input / output interface 53 for communicating with external devices, and an operation unit 54 including various operation switches are connected to the processor 51 via a bus. Furthermore, a teaching pendant (teaching operation panel, etc.) for teaching the robot 10 may be further connected to the robot control device 50. In this embodiment, the vision sensor control device 20 and the robot control device 50 are configured as different devices, but they can also be configured as the same device. For example, the functionality of the vision sensor control device 20 may be installed within the robot control device 50.

[0031] A robot coordinate system C1 is established in the robot 10. The robot coordinate system C1 is a control coordinate system used to control the movements of the various movable elements of the robot 10, and it is fixed in 3D space. In this embodiment, as an example, the origin of the robot coordinate system C1 is set at the center of the base of the robot 10, but the robot coordinate system C1 can also be set to a different position or posture.

[0032] A flange coordinate system C2 is set on the front end face of flange 11. Flange coordinate system C2 is a control coordinate system used to control the position and orientation of flange 11 in robot coordinate system C1. In this embodiment, as an example, flange coordinate system C2 is set with its origin located at the center of the front end face of flange 11, and its z-axis is aligned with the central axis of flange 11. However, flange coordinate system C2 can also be set to different positions and orientations. When the wrist (tool 31) of robot 10 is moved, robot control device 50 (processor 51) sets flange coordinate system C2 in robot coordinate system C1 and controls the servo motors of each joint of robot 10 to configure flange 11 to the position and orientation represented by the set flange coordinate system C2. In this way, robot control device 50 can position flange 11 (tool 31) to any position and orientation in robot coordinate system C1.

[0033] A sensor coordinate system C3 is set in the vision sensor 70. The sensor coordinate system C3 is a coordinate system that defines the coordinates of each pixel in the image data captured by the vision sensor 70. The vision sensor 70 is set in the following manner: the origin of the sensor coordinate system is set at the center of the light-receiving surface (or optical lens) of the imaging sensor of the vision sensor 70, the x-axis and y-axis of the sensor coordinate system are set to be parallel to the horizontal and vertical directions of the imaging sensor, and the z-axis of the sensor coordinate system is aligned with the line of sight (optical axis) of the vision sensor 70.

[0034] As detailed below, the robot system 100 is configured as an image processing system that captures a visual pattern on the detection object location of the vision sensor 70, i.e., a visual pattern used to determine the relative positional relationship between the vision sensor 70 and the visual pattern. This causes the vision sensor 70 to move in a predetermined direction relative to the visual pattern to perform a focusing action. In this embodiment, the detection object location of the vision sensor 70 is the upper surface of the stage 60 on which the workpiece is mounted, and the visual pattern is formed on the calibration fixture J. The visual pattern can be any visually recognizable form, such as a painted pattern or an engraving (embossing). Alternatively, the visual pattern can be formed directly on the detection object location (the upper surface of the stage 60). In this specification, the term "visual pattern" includes patterns, markings, indicators, identification marks, symbols, and various other visual displays. By using a visual pattern to obtain the relative positional relationship with the vision sensor 70, the relative positional relationship can be obtained with high precision. In particular, compared to the structure described in Patent Document 5 above, which determines the relative positional relationship between a camera and a specific workpiece, accuracy can be improved.

[0035] The following description focuses on the function of such an image processing system in the robot system 100.

[0036] Figure 3 This is a functional block diagram illustrating the functional structure of the vision sensor control device 20 and the robot control device 50, considering their function as an image processing system within the robot system 100. (Example) Figure 3 As shown, the vision sensor control device 20 includes: an image processing unit 202 that performs image processing on the input image 201 captured by the vision sensor 70; and a calibration data storage unit 204 that stores calibration data that determines the relative position of the vision sensor 70 (sensor coordinate system C3) relative to a reference coordinate system (robot coordinate system C1 or flange coordinate system C2) set on the robot 10.

[0037] The image processing unit 202 performs functions such as manipulating a workpiece, and for example, it has the function of detecting a workpiece from an input image using a model pattern and detecting the position and posture of the workpiece. Furthermore, the image processing unit 202 of this embodiment includes a focus calculation unit 203 for calculating the focus of an object captured in an image.

[0038] The calibration data storage unit 204 stores the calibration data of the vision sensor 70. The calibration data storage unit 204 is, for example, a non-volatile memory (EEPROM, etc.). The calibration data of the vision sensor 70 includes both internal and external parameters of the vision sensor 70 (camera). The internal parameters include optical data such as lens distortion and focal length. The external parameters are the position of the vision sensor 70 relative to a predetermined reference position (e.g., the position of flange 11).

[0039] By utilizing calibration data, the geometric transformation characteristics inside the camera and the geometric relationship between the three-dimensional space of the object and the two-dimensional image plane are determined. Furthermore, based on the image of the calibration fixture J captured by the vision sensor 70 according to the characteristics of the calibration fixture J, the three-dimensional spatial position of the vision sensor 70 and the calibration fixture J can be uniquely determined. That is, based on the information of the calibration fixture J in the captured image using the calibrated vision sensor 70, the position and orientation of the calibration fixture, with the position of the vision sensor 70 as a reference, can be determined.

[0040] Figure 4 This is a top view of the calibration fixture J. As the calibration fixture J, various calibration fixtures known in the art can be used, whose position and orientation can be determined based on the image captured by the vision sensor 70 with reference to the position of the vision sensor 70. Figure 4The calibration fixture J is a fixture capable of acquiring the information required for the calibration of the vision sensor 70 by photographing a dot pattern arranged on a plane using the vision sensor 70, and satisfies the following three requirements: (1) the grid spacing of the dot pattern is known; (2) there exists a constant number of grid points; and (3) it is possible to uniquely determine which grid point each grid point corresponds to. The calibration fixture J is not limited to Figure 4 The calibration fixture shown can be a two-dimensional plane with predetermined dot patterns or other features, or it can be a three-dimensional solid with features, as long as it can obtain not only two-dimensional position information (X direction, Y direction) but also height direction information. Figure 1 A calibration fixture for the three-dimensional position information (arrow 91 direction) can be used. This calibration fixture J can be the same as or different from the calibration fixture used when acquiring calibration data from the vision sensor 70. Furthermore, in order to calculate the position and pose of the dot pattern based on the position of the vision sensor 70, the internal parameters of the aforementioned calibration data are used. In addition, in this embodiment, an object with [missing information] is used as the object for determining the position and pose of the object observed from the imaging device. Figure 4 Such a dot pattern calibration fixture J, but it can also be replaced by all types of objects that can determine the position and orientation of the object observed from the imaging device (visual sensor 70).

[0041] In this embodiment, the calibration fixture J is positioned at the location of the object that the vision sensor 70 focuses on (i.e., the upper surface of the stage 60).

[0042] like Figure 3 As shown, the robot control device 50 includes an action control unit 501 that controls the actions of the robot 10 according to an action program. Furthermore, the robot control device 50 of this embodiment includes a focusing action control unit 502, which controls the robot 10 to move and the vision sensor 70 to the focusing position based on the focusing degree calculated by the focusing degree calculation unit 203.

[0043] also, Figure 3 The functional blocks of the visual sensor control device 20 or robot control device 50 shown can be implemented by the processor (CPU) of the visual sensor control device 20 or robot control device 50 executing various software stored in the memory, or they can be implemented by a structure based on hardware such as ASIC (Application Specific Integrated Circuit).

[0044] Figure 5This is a flowchart illustrating the focusing action (image processing method) of the focusing action control unit. Furthermore, it is assumed here that the calibration data of the vision sensor 70 is pre-stored in the calibration data storage unit 204. Figure 5 The focusing action is performed under the control of the processor 51 of the robot control device 50.

[0045] First, the focusing motion control unit 502 calculates and obtains the positional relationship between the calibration fixture J and the vision sensor 70 based on the image captured by the vision sensor 70. In this case, the focusing motion control unit 502 uses calibration data stored in the calibration data storage unit J.

[0046] Using the relative positional relationship between the vision sensor 70 and the calibration fixture J obtained in step S1, the focusing action control unit 502, while aligning the optical axis of the vision sensor 70 with the normal direction of the upper surface of the calibration fixture J, performs the focusing action control unit 502. Figure 1 The vision sensor 70 is moved in a parallel manner (in the direction of the middle arrow 91) while the calibration fixture J is photographed (step S2). At this time, the focus motion control unit 502 causes the focus calculation unit 203 to calculate the focus of the image of the object (i.e., the point of the calibration fixture J) reflected in the image (step S3).

[0047] Then, the focusing motion control unit 502 compares the focusing degree calculated by the focusing degree calculation unit 203 with a reference value (focus degree reference value) that represents the highest focusing degree at the point of calibration fixture J, and determines whether the focusing degree obtained in step S3 is the highest focusing degree (step S4). The reference value (focus degree reference value) used here can be pre-stored by the robot control device 50 (focusing motion control unit 502), or the focusing motion control unit 502 can perform an action such as moving the vision sensor 70 within a predetermined movement range to set the highest focusing degree as the focusing degree reference value. If the focusing degree is not determined to be the highest (S4: No), the focusing motion control unit 502 continues to perform the process of checking the focusing degree while moving the vision sensor 70 (steps S2-S4). If the focusing degree is determined to be the highest (S4: Yes), the focusing motion control unit 502 records the position of the vision sensor 70 (focus position) when the focusing degree is the highest as the shooting position (step S5).

[0048] As a method for calculating focus, the focus calculation unit 203 can employ various calculation methods known in the art, such as phase difference detection and contrast detection. As an example, the focus calculation unit 203 can also use a method as described in Japanese Patent Application Publication No. 2013-29803: when it is determined that the reliability of the focus detected by the phase difference detection method is low, a pixel range for which the phase difference detection method is used is selected based on the detection result of the contrast detection method, and the signal of the selected pixel range is used to calculate the focus based on the phase difference detection method. The focus calculation unit 203 can acquire data for calculating focus from the functional elements for calculating focus within the vision sensor 70.

[0049] In steps S2 to S4, the predetermined range of motion for moving the vision sensor 70 can also be determined based on the focal length adjustment range of the camera. For example, the robot 10 can be controlled to move the vision sensor 70 within a shooting distance determined by the focal length adjustment range of the focus adjustment ring of the vision sensor 70. Since the positional relationship between the vision sensor 70 and the calibration fixture J is known through step S1, such movement control of the vision sensor 70 is possible.

[0050] The shooting position recorded in step S5 is used to position the vision sensor 70 for pre-defined operations such as visual inspection of a workpiece placed on the upper surface of the stage 60. Thus, according to... Figure 5 The focusing action automatically positions the vision sensor 70 at a position where the focus coincides with the location where the calibration fixture J is set. In other words, it can automatically and efficiently adjust the focus of the vision sensor 70.

[0051] The above examples illustrate the case where the calibration of the vision sensor 70 has been performed in advance (i.e., the vision sensor control device 20 has stored the calibration data in advance). However, in the case where the calibration of the vision sensor 70 has not been performed (i.e., the vision sensor control device 20 has not stored the calibration data), the robot control device 50 and the vision sensor control device 20 may also use various calibration methods known in the art (e.g., the aforementioned Patent Documents 2-5) to obtain the calibration data.

[0052] As described above, in this embodiment, the configuration is as follows: Figure 5 The focus control operation uses calibration fixture J, therefore, during the... Figure 5During focus control operations, it is also easy to perform on-site calibration using the calibration fixture J. The method for calibrating a camera is itself a well-known technique, therefore its description is omitted; however, it is detailed, for example, in Roger Y. Tsai's "An Efficient and Accurate Camera Calibration Technique for 3D Dachine Vision" (CVPR, pp. 364-374, 1986 IEEE). As a specific means, it enables... Figure 4 The calibration fixture J shown (a dot pattern plate with dots arranged in a known geometric configuration) is orthogonal to the optical axis of the camera, and measurements are taken at two known locations, thereby enabling camera calibration.

[0053] Here, an example of an image processing method for determining the orientation of a sensor coordinate system C3 based on a reference coordinate system (robot coordinate system C1 or flange coordinate system C2) set on the robot 10 will be described. The focusing motion control unit 502 may also be configured to perform this function. By using the method described here to make the orientation of the sensor coordinate system C3 known relative to the reference coordinate system set on the robot 10, the optical axis of the vision sensor 70 can be oriented in a predetermined direction (e.g., the normal direction) relative to the calibration fixture J during the focusing motion described above.

[0054] In this example, the pose data of the visual sensor 70 in the reference coordinate system is obtained based on the image data of the indicator ID captured by the visual sensor 70. Figure 6 Here is an example of an indicator ID. The indicator ID is set at the location of the object being detected (the upper surface of the platform 60) and consists of a circular line C and two mutually orthogonal straight lines D and E. The indicator ID can be a visually identifiable shape, such as a pattern created by paint or an engraving (embossing) formed on the upper surface of the platform 60.

[0055] The robot control device 50 (focusing motion control unit 502) calculates the posture of the sensor coordinate system C3 based on the reference coordinate system set on the robot 10 in the following order.

[0056] (A1) The robot control device 50 configures the vision sensor 70 in its initial position PS0 and initial pose OR0 by having the indicator ID enter the field of view of the vision sensor 70, and then captures the indicator ID to obtain the image data JD0 of the indicator ID. Assuming that the image data JD0 of the indicator ID is obtained... Figure 6 The image shown (JD) n ).

[0057] (A2) Image processing unit 202 processes the image JD. n From the image of the indicator ID, obtain the coordinates (x, y) of the intersection point F.n y n The area of ​​circle C is used as the dimension IS of the indicator ID, representing the position of the indicator ID. n (Unit: pixel). The image processing unit 202 acquires the size RS (unit: mm) of the indicator ID in the actual space, the focal length FD of the optical lens of the vision sensor 70, and the size SS (unit: mm / pixel) of the imaging sensor as pre-saved data.

[0058] (A3) The image processing unit 202 uses the acquired coordinates (xn, yn), size ISN, size RS, focal length FD, and size SS to acquire a vector (Xn, Yn, Zn). Here, Xn can be calculated using the formula Xn = xn × ISN × SS / RS. Yn can be calculated using the formula Yn = yn × ISN × SS / RS. Zn can be calculated using the formula Zn = ISN × SS × FD / RS. This vector (Xn, Yn, Zn) is a vector from the visual sensor 70 (i.e., the origin of the sensor coordinate system C3) when image data JDn is captured to the index ID (specifically, the intersection point F), representing the relative position of the index ID with respect to the visual sensor 70 (or, the coordinates of the sensor coordinate system C3).

[0059] (A4) Similarly, the image processing unit 202 obtains the vector from the visual sensor 70 to the index ID when the image data JD1 is captured, based on the position PS1 after the visual sensor 70 has been translated a predetermined distance δx along the x-axis of the flange coordinate system from the initial position and the image JD1 obtained by the posture OR0.

[0060] (A5) Similarly, the image processing unit 202 obtains the vector from the visual sensor 70 to the index ID when the image data JD2 is captured, based on the position PS2 after the visual sensor 70 is translated a predetermined distance δy along the y-axis direction of the flange coordinate system from the initial position and the posture OR0 to capture the index ID.

[0061] (A6) Similarly, the image processing unit 202 obtains the vector from the visual sensor 70 to the index ID when the image data JD3 is captured, based on the position PS3 after the visual sensor 70 is translated a predetermined distance δz along the z-axis direction of the flange coordinate system from the initial position and the posture OR0 to capture the index ID.

[0062] (A7) Based on the above data, the image processing unit 202 obtains the rotation matrix representing the pose (W, P, R) of the vision sensor 70 (sensor coordinate system C3) in the flange coordinate system C2 by the following formula (1).

[0063] [Formula 1]

[0064]

[0065] The robot control device 50 (focusing motion control unit 502) can also be configured to determine the position of the sensor coordinate system C3 based on the reference coordinate system set on the robot 10. The sequence of actions in this case is shown below.

[0066] (B1) The robot control device 50 first sets a reference coordinate system C4 in the flange coordinate system C2 under the initial position PS0 and the initial posture OR0. In this embodiment, the robot control device 50 sets the reference coordinate system C4 in the flange coordinate system C2 such that the origin of the reference coordinate system is located at the origin of the flange coordinate system C2, and the posture (direction of each axis) of the reference coordinate system is consistent with the posture (W, P, R) obtained through the above steps. Therefore, the directions of the x-axis, y-axis, and z-axis of the reference coordinate system C4 are parallel to the x-axis, y-axis, and z-axis of the sensor coordinate system C3, respectively.

[0067] (B2) Next, the robot control device 50 causes the robot 10 to move, causing the vision sensor 70 (i.e., flange 11) to rotate from the initial position PS0 and the initial posture OR0 about the z-axis of the reference coordinate system C4 by a posture change θ1 (first posture change), thereby being positioned at position PS4 and posture OR1.

[0068] (B3) The image processing unit 202 uses the same method as when the above posture is obtained to activate the vision sensor 70 to capture the index ID and obtain the relative position data (X4, Y4, Z4) of the index ID relative to the vision sensor 70 at this time.

[0069] (B4) Next, the robot control device 50 causes the robot 10 to move, causing the vision sensor 70 to rotate from the initial position PS0 and the initial posture OR0 about the x-axis or y-axis of the reference coordinate system C4 (i.e., the axis orthogonal to the direction of the line of sight) by a posture change amount θ2 (first posture change amount), thereby being positioned at position PS5 and posture OR2.

[0070] (B5) The image processing unit 202 uses the same method as when the above posture was obtained to activate the vision sensor 70 to capture the index ID and obtain the relative position data (X5, Y5, Z5) of the index ID relative to the vision sensor 70 at this time.

[0071] If the vector in the flange coordinate system C2 from the origin of the reference coordinate system C4 (in this embodiment, the origin of the MIF coordinate system C2) to the origin of the sensor coordinate system C3 whose position is unknown is set as (ΔX1, ΔY1, ΔZ1), then the following equations (2) and (3) hold true.

[0072] [Formula 2]

[0073]

[0074] [Formula 3]

[0075] cosθ2·Y0-sinθ2·( Z 0+ΔZ1)=Y5 …(3)

[0076] By solving equations (2) and (3) above, the robot control device 50 can estimate the vector (ΔX1, ΔY1, ΔZ1) in the flange coordinate system C2 from the origin of the reference coordinate system C4 to the origin of the unknown sensor coordinate system C3.

[0077] As explained above, the index ID is a visual pattern that can determine the relative positional relationship between the vision sensor 70 and the index ID, and therefore the index ID can be used in place of the calibration fixture J used in the focusing action described above.

[0078] As explained above, according to this embodiment, by placing a visual pattern on the detection target area, the visual sensor can automatically focus on the area where the visual pattern is placed. That is, the focus adjustment of the visual sensor can be performed automatically and efficiently.

[0079] The present invention has been described above using typical embodiments. However, those skilled in the art will understand that various modifications, omissions, and additions can be made to the above embodiments without departing from the scope of the present invention.

[0080] The structure described in the above embodiments can be applied to focus adjustment operations in various industrial machines with vision sensors mounted on movable parts.

[0081] exist Figure 1 The illustrated device structure example shows a configuration where a vision sensor is mounted on a robot and a calibration fixture is positioned in a fixed location. However, it could also be a tool structure where the vision sensor is a fixed camera fixed in the workspace, and the calibration fixture (visual pattern) is mounted on the robot. In this case, it can also be applied to… Figure 3 The functional block structure shown is the same as the functional block structure. In this case, when the calibration fixture (visual pattern) is reflected in the field of view of the vision sensor, the focusing action control unit 502 moves the visual pattern relative to the vision sensor according to the determined relative position relationship (while changing the relative position of the vision sensor and the visual pattern in a predetermined direction based on the visual pattern), and determines the focus of the visual pattern by taking a picture of the visual pattern with the vision sensor, thereby making the vision sensor focus on the visual pattern.

[0082] Figure 2 The configuration of the functional blocks of the vision sensor control device 20 and the robot control device 50 in the functional block diagram shown is an example, and the configuration of the functional blocks can have various variations. For example, at least a part of the function of the image processing unit 202 can also be configured on the robot control device 50 side.

[0083] exist Figure 5 In the focusing action shown, the focus position is determined by comparing whether the focus degree is consistent with the focus degree reference value (step S4). However, instead of this structure, the focus position can be obtained by moving the camera within a predetermined range of motion to search for the position of the focus degree peak.

[0084] The program used to perform the focusing action and other various processes in the above embodiments can be recorded in various computer-readable recording media (such as ROM, EEPROM, flash memory and other semiconductor memories, magnetic recording media, CD-ROM, DVD-ROM and other optical discs).

[0085] Explanation of reference numerals in the attached figures

[0086] 10 robots

[0087] 11 flanges

[0088] 20 vision sensor control devices

[0089] 21 processor

[0090] 22 memory

[0091] 23 Input / Output Interfaces

[0092] 31 tools

[0093] 50 robot control devices

[0094] 51 processor

[0095] 52 memory

[0096] 53 Input / Output Interfaces

[0097] 54 Operations Department

[0098] 60 units

[0099] 70 vision sensors

[0100] 100 Robot Systems

[0101] 201 Input Images

[0102] 202 Image Processing Department

[0103] 203 Focusing Power Calculation Unit

[0104] 204 Calibration Data Storage Unit.

Claims

1. An image processing system, characterized in that, The image processing system includes: Industrial machinery; Visual sensors; A visual pattern is disposed on the detection object, which is the upper surface of the stage on which the workpiece is mounted, and the visual pattern is used to determine the relative positional relationship between the visual sensor and the visual pattern. An image processing unit determines the relative positional relationship between the visual sensor and the visual pattern based on an image obtained by the visual sensor capturing the visual pattern. as well as The focusing action control unit, when the visual pattern is reflected in the field of view of the visual sensor, changes the relative position of the visual sensor and the visual pattern in a predetermined direction based on the determined relative positional relationship, and calculates the focus degree of the visual pattern by capturing the visual pattern with the visual sensor, thereby causing the visual sensor to focus on the visual pattern. The vision sensor is mounted on a predetermined movable part of the industrial machinery. When the visual pattern is reflected in the field of view of the vision sensor, the focusing action control unit moves the vision sensor in the predetermined direction based on the relative positional relationship using the industrial machinery, and calculates the focus degree of the visual pattern by capturing the visual pattern with the vision sensor, thereby moving the vision sensor to a focusing position focused on the visual pattern. The vision sensor has been calibrated using the vision pattern. The focusing control unit uses calibration data to orient the vision sensor in a predetermined direction based on the vision pattern. This calibration data represents the position and orientation of the vision sensor relative to a predetermined reference position of the industrial machinery. The focusing position is used to locate the shooting position of the vision sensor when performing a predetermined operation on a workpiece placed on the upper surface of the stage.

2. The image processing system according to claim 1, characterized in that, The focusing action control unit changes the relative position of the visual sensor and the visual pattern in a predetermined direction relative to the surface on which the visual pattern is formed.

3. The image processing system according to claim 1 or 2, characterized in that, The focusing motion control unit moves the visual sensor relative to the visual pattern within a predetermined motion range, compares the focus reference value when the focus of the visual pattern is at its highest with the focus obtained while moving the visual sensor, thereby moving the visual sensor to the focusing position.

4. The image processing system according to claim 1 or 2, characterized in that, The industrial machinery mentioned is an industrial robot.

5. An image processing method executed in an image processing system, characterized in that, Includes the following steps: The relative positional relationship between the visual sensor and the visual pattern is determined by the image obtained by capturing the visual pattern disposed on the detection object part using a visual sensor, wherein the detection object part is the upper surface of the stage on which the workpiece is mounted. as well as With the visual pattern projected into the field of view of the visual sensor, based on the determined relative positional relationship, the relative position of the visual sensor and the visual pattern is changed in a predetermined direction relative to the visual pattern. Furthermore, the visual sensor captures an image of the visual pattern to determine its focus, thereby enabling the visual sensor to focus on the visual pattern. The vision sensor is mounted on a predetermined movable part of the industrial machinery. In the step of focusing the vision sensor on the visual pattern, while the visual pattern is reflected in the field of view of the vision sensor, the vision sensor is moved in the predetermined direction by the industrial machinery based on the relative positional relationship. The vision sensor then captures an image of the visual pattern to determine the focus degree of the visual pattern, thereby moving the vision sensor to a focusing position that focuses on the visual pattern. The vision sensor has been calibrated using the vision pattern. To orient the vision sensor toward the predetermined direction based on the visual pattern, calibration data is used, which represents the position and orientation of the vision sensor relative to a predetermined reference position of the industrial machinery. The focus position is used to locate the shooting position of the vision sensor when performing a predetermined operation on a workpiece placed on the upper surface of the stage.

Images