Mark detection device and robot teaching system

Abstract

The present invention provides a sign detection device capable of detecting the position of a sign with high precision, and a robot teaching system utilizing the same. The sign detection device (220) includes: an image generation unit (221) that generates a composite image by overlaying image data obtained by a 2D camera with point group data obtained by a 3D camera taken from the same viewpoint as the 2D camera; a sign position calculation unit (222) that detects a sign and calculates its position based on the image data; a plane detection unit (223) that detects a plane based on the point group data; a sign plane determination unit (224) that determines a sign plane based on point group data in the area of ​​the sign based on the position of the sign calculated by the sign position calculation unit (222) and the plane detected by the plane detection unit (223); and a sign position correction unit (225) that corrects the position of the sign by projecting the position of the sign calculated by the sign position calculation unit (222) onto the sign plane.

Description

Technical Field

[0001] This invention relates to a sign detection device and a robot teaching system. Background Technology

[0002] In recent years, robots have become widely used in industry. These robots are used, for example, in the assembly, welding, and transportation of electronic and mechanical components, aiming to improve the efficiency and automation of factory production lines.

[0003] For example, in welding robots, a program needs to be created to perform the desired actions, and this program is pre-stored as so-called teach data. The operator uses a teach pendant to operate the actual robot in cooperation with the robot control unit, thereby recording the actions and generating the teach data.

[0004] In order to enable operators to actually make the robot move while generating teaching data using the teach pendant, the skill of the operator is highly dependent and sometimes takes a long time. Therefore, in order to reduce the burden on the operator, a robot system that uses AR devices and markers to recognize the robot's position has been disclosed (e.g., Patent Document 1).

[0005] In the technology disclosed in Patent Document 1, an AR device is used to simultaneously detect two reference markers, thereby recognizing their positional relationship with the markers used to determine the robot's coordinate system. This suppresses deviations in the display position of the AR graphic and allows for the recognition of the robot's position and orientation. In such a robot system, it is necessary to appropriately detect the reference markers in order to accurately recognize the position and orientation of the workpiece and the robot.

[0006] Existing technical documents

[0007] Patent documents

[0008] Patent Document 1: JP 2021-62463

[0009] However, in the technology disclosed in Patent Document 1, during the detection of the mark, for example when using a two-dimensional camera, the outline position of the mark may deviate during the contour determination in image processing, or the mark image may be distorted due to lens distortion or other reasons, corresponding to the field of view position, which may result in an error with the actual position of the mark. Summary of the Invention

[0010] Therefore, the present invention aims to provide a sign detection device capable of detecting the position of a sign with high precision, and a robot teaching system utilizing the same.

[0011] One aspect of the present invention relates to a sign detection apparatus comprising: an image generation unit that generates a composite image by overlaying image data acquired by a 2D camera with point group data acquired by a 3D camera from the same viewpoint as the 2D camera; a sign position calculation unit that detects a sign and calculates the position of the sign based on the image data in the composite image; a plane detection unit that detects a plane based on the point group data in the composite image; a sign plane determination unit that determines the sign plane in which the sign exists based on the point group data contained in the area of ​​the sign detected from the position of the sign calculated by the sign position calculation unit and the plane detected by the plane detection unit; and a sign position correction unit that corrects the position of the sign by projecting the position of the sign calculated by the sign position calculation unit onto the sign plane.

[0012] According to this method, the sign position calculation unit detects the sign and calculates its position based on image data from a composite image of image data and dot group data generated by the image generation unit. The plane detection unit detects the plane based on the dot group data in the composite image. Furthermore, the sign plane determination unit determines the sign plane in which the sign exists based on the dot group data contained in the area of ​​the sign detected from the position of the sign calculated by the sign position calculation unit and the plane detected by the plane detection unit. The sign position correction unit corrects the sign position by projecting the position of the sign calculated by the sign position calculation unit onto the sign plane. Thus, the sign position can be detected with high precision. While using only a distance camera or similar device to determine the sign position is also considered, the low resolution of the image formed from dot group data prevents the sign position from being recognized on the image, or even if it is recognized, the accuracy of the detected position is low. In contrast, in the sign detection apparatus according to one aspect of the present invention, since both image data and dot group data are used, the sign position can be detected with high precision.

[0013] In the above method, the plane may be determined as a sign plane if a given proportion or more of the point group data in the point group data of the region of the sign detected in the composite image is contained in the plane detected by the plane detection unit.

[0014] According to this method, since the sign plane determination unit uses point group data to determine the plane in which the sign exists, it can appropriately determine whether a sign exists in the plane detected by the plane detection unit. As a result, if a sign exists in the plane detected by the plane detection unit, the plane detected by the plane detection unit is determined to be the sign plane, and the sign position correction unit corrects the sign's position by projecting the sign's position calculated by the sign position calculation unit onto the sign plane. Therefore, the sign's position can be detected appropriately and with high accuracy.

[0015] In the above method, if a given proportion or more of the point group data in the point group data of the region of the sign detected in the composite image is not included in the plane detected by the plane detection unit, the virtual plane composed of the point group data in the region of the sign may be determined as the sign plane.

[0016] According to this method, the sign plane determination unit uses point group data to determine the plane in which the sign exists, thus enabling it to appropriately determine whether a sign exists in the plane detected by the plane detection unit. Consequently, if no sign is detected in the plane by the plane detection unit, a virtual plane composed of point group data within the area of ​​the sign is determined as the sign plane. The sign position correction unit then corrects the sign's position by projecting the position of the sign calculated by the sign position calculation unit onto this sign plane. Therefore, the position of the sign can be detected appropriately and with high accuracy.

[0017] In the above method, the mark plane determination unit may perform the following processing: when the mark is set on the workpiece, the plane detected by the plane detection unit is determined as the mark plane; when the mark is set on the robot, the virtual plane composed of point group data in the area of ​​the mark detected in the composite image is determined as the mark plane.

[0018] According to this method, when the mark is set on a workpiece, the mark plane determination unit determines the plane detected by the plane detection unit as the mark plane; when the mark is set on a robot, it determines the virtual plane composed of point group data within the area of ​​the mark as the mark plane. That is, the mark plane determination unit can appropriately determine the mark plane corresponding to the set position of the mark. As a result, the mark position correction unit can appropriately correct the position of the mark by projecting the position of the mark calculated by the mark position calculation unit onto the mark plane determined by the mark plane determination unit.

[0019] A robot teaching system according to one aspect of the present invention comprises: a camera unit having a 2D camera for capturing images including a workpiece and a mark, and a 3D camera for capturing images from the same viewpoint as the 2D camera; an image generation unit for generating a composite image by overlaying image data acquired by the 2D camera with point group data acquired by the 3D camera; a mark position calculation unit for detecting a mark and calculating the position of the mark based on image data in the composite image; a plane detection unit for detecting a plane based on point group data in the composite image; and a mark plane determination unit for determining the position of the mark based on the point group data in the composite image. The system calculates the location of the marker by using the point group data within the detected area of ​​the marker and the plane detected by the plane detection unit to determine the marker plane where the marker exists; the marker position correction unit corrects the position of the marker by projecting the position of the marker calculated by the marker position calculation unit onto the marker plane; the camera coordinate system setting unit sets the camera coordinate system based on the corrected position of the marker; and the program generation unit transforms the motion path of the robot in the camera coordinate system into the robot coordinate system set in the robot control device and generates a work program for making the robot move.

[0020] According to this method, the imaging unit includes a 2D camera and a 3D camera that captures images from the same viewpoint as the 2D camera. The marker position calculation unit detects a marker and calculates its position based on image data from a composite image of image data and point group data generated by the image generation unit. The plane detection unit detects a plane based on the point group data in the composite image. Furthermore, the marker plane determination unit determines the marker plane in which the marker exists based on the point group data contained within the region of the marker detected from the position calculated by the marker position calculation unit and the plane detected by the plane detection unit. The marker position correction unit corrects the marker position by projecting the position calculated by the marker position calculation unit onto the marker plane. The camera coordinate system setting unit sets the camera coordinate system based on the marker position corrected by the marker position correction unit. The program generation unit transforms the robot's motion path in the camera coordinate system into the robot coordinate system set in the robot control device and generates a work program for moving the robot. As a result, by setting the camera coordinate system based on the more accurately detected position of the markers and generating the work program, the robot arm can move with higher precision and appropriateness.

[0021] The effects of the invention

[0022] According to the present invention, a sign detection device capable of detecting the position of a sign with high precision and a robot teaching system utilizing the same are provided. Attached Figure Description

[0023] Figure 1This is a diagram illustrating the structure of a welding robot system 100 that includes the robot teaching system according to the first embodiment of the present invention.

[0024] Figure 2 This is a diagram illustrating the functional structure of the robot teaching system 200 according to the first embodiment of the present invention.

[0025] Figure 3 This diagram illustrates the situation where the marker M is set on the workpiece W, which is the object of welding, and the camera coordinate system is set with the position of the marker M as the origin O.

[0026] Figure 4 This diagram illustrates the specific processing in the mark detection device 220 that detects the position of the mark M set on the workpiece W.

[0027] Figure 5 This diagram illustrates the scenario where marker M is set on robot arm 3, and the camera coordinate system is established with the position of marker M as the origin O.

[0028] Figure 6 This diagram illustrates the specific processing in the marker detection device 220 that detects the position of the marker M set on the robot arm 3.

[0029] Figure 7 This is a flowchart illustrating the processing flow of the mark detection method M100 performed by the mark detection device 220 according to the first and second embodiments of the present invention.

[0030] Explanation of reference numerals in the attached figures

[0031] 1...Shooting terminal, 2...Robot control device, 3...Robot arm, 11...Control unit, 12...Shooting unit, 13...Communication unit, 14...Display unit, 21...Control unit, 22...Storage unit, 23...Communication unit, 24...Welding power supply unit, 31...Multi-joint arm, 32...Welding torch, 100...Welding robot system, 200...Robot teaching system, 211...Shooting unit, 212...Camera coordinate system setting unit, 213...Program generation unit, 220 ...marker detection device, 221...image generation unit, 222...marker position calculation unit, 223...plane detection unit, 224...marker plane determination unit, 225...marker position correction unit, C...communication cable, N...network, M...marker, W...workpiece, L1~L3...welding position, Dm...point group data, Wa, Wb...plane, BP...virtual plane, M100...marker detection method, S110~S170...steps of marker detection method M100 Detailed Implementation

[0032] The embodiments of the present invention will be specifically described below with reference to the accompanying drawings. Furthermore, the embodiments described below are merely specific examples for carrying out the present invention and are not intended to limit the scope of the invention. In addition, to facilitate understanding, the same reference numerals are used as much as possible to denote the same constituent elements in the drawings, and sometimes repeated descriptions are omitted.

[0033] <First Implementation>

[0034] [Basic Structure of Welding Robot System]

[0035] Figure 1 This is a diagram illustrating the structure of a welding robot system 100 that includes the robot teaching system according to the first embodiment of the present invention. Figure 1 As shown, the welding robot system 100 includes, for example, a camera terminal 1, a robot control device 2, and a robotic arm 3. The camera terminal 1 and the robot control device 2 are connected, for example, via a network N, and the robot control device 2 and the robotic arm 3 are connected, for example, via a communication cable C. The network N can be wired (including the communication cable) or wireless. Additionally, the welding robot system 100 may include a teach pendant. The teach pendant is an operating device used by the operator to teach the robotic arm 3 its movements.

[0036] The robotic arm 3 is a welding robot (industrial robot) that performs arc welding according to the construction conditions set in the robot control device 2. The robotic arm 3 has, for example, a multi-joint arm 31 mounted on a base member fixed to a factory floor or the like; and a welding torch 32 (end effector) connected to the front end of the multi-joint arm 31.

[0037] The robot control device 2 is a control component that controls the movement of the robotic arm 3, and includes, for example, a control unit 21, a storage unit 22, a communication unit 23, and a welding power supply unit 24.

[0038] The control unit 21 controls the robotic arm 3 and the welding power supply unit 24, for example, by executing the work program stored in the storage unit 22 by the processor.

[0039] The communication unit 23 controls communication with the shooting terminal 1 connected via network N, and controls communication with the robotic arm 3 connected via communication cable C.

[0040] The welding power supply unit 24, for example, supplies welding current and welding voltage to the robot arm 3 according to predetermined welding conditions in order to generate an electric arc between the tip of the welding wire and the workpiece. The welding conditions include data items such as welding conditions, welding start position, welding end position, arc discharge time, welding distance, torch posture, and torch movement speed. The welding power supply unit 24 can be separately mounted from the robot control device 2.

[0041] The shooting terminal 1 is, for example, a digital camera (2D camera), and can be a portable terminal with a built-in digital camera. Portable terminals include, for example, tablet computers, smartphones, portable information terminals (PDAs), and laptop PCs (personal computers) that can be carried and moved. The shooting terminal 1 includes, for example, a control unit 11, a shooting unit 12, a communication unit 13, and a display unit 14.

[0042] The control unit 11 controls each part of the shooting terminal 1 by executing a given program stored in the memory through the processor.

[0043] The imaging unit 12 includes, for example, a lens and an image sensor (image sensor), which converts the light from the subject that is illuminated by the lens into electrical signals (digital image data).

[0044] The communication unit 13 controls communication with the robot control device 2 connected via network N.

[0045] The display unit 14 is, for example, a display with a touch panel, which displays the image of the subject obtained by the shooting unit 12 and accepts input such as operation instructions from the operator. The display unit 14 can be, for example, a display device with a touch panel, and is separately provided from the shooting terminal 1.

[0046] Furthermore, the imaging terminal 1 also functions as a 3D camera, and may include distance measurement sensors such as LiDAR (Light Detection and Ranging) sensors, millimeter-wave sensors, and ultrasonic sensors. Typically, the distance measurement sensor illuminates the object being measured with a laser, and determines the shape of the object based on its reflected light, obtaining so-called 3D scan data as point group data. Additionally, the point group data may include, for example, position (distance) information shown in three-dimensional coordinates (X, Y, Z) and color information shown in (R, G, B).

[0047] In addition, the 2D camera that acquires the aforementioned image data and the 3D camera that acquires the point group data can be configured as a single shooting terminal, thus including two functions, or they can be configured as separate shooting terminals comprising each function.

[0048] [Structure of a Robot Teaching System]

[0049] Figure 2 This is a diagram illustrating the functional structure of the robot teaching system 200 according to the first embodiment of the present invention. Figure 2As shown, the robot teaching system 200, as a functional structure, includes, for example, a camera unit 211, a camera coordinate system setting unit 212, and a program generation unit 213. Furthermore, it includes a sign detection device 220 for detecting signs that serve as references for setting the camera coordinate system. The sign detection device 220 also includes an image generation unit 221, a sign position calculation unit 222, a plane detection unit 223, a sign plane determination unit 224, and a sign position correction unit 225.

[0050] The imaging unit 211, one of these functions, is a function possessed by the imaging terminal 1. On the other hand, each of the camera coordinate system setting unit 212, program generation unit 213, and marker detection device 220 can be fully implemented in either the imaging terminal 1 or the robot control device 2, or each function can be distributed between the imaging terminal 1 and the robot control device 2. Furthermore, other devices besides the imaging terminal 1 and the robot control device 2 may possess some or all of the aforementioned functions.

[0051] The imaging unit 211 is the same as the imaging unit 12 of the imaging terminal 1, and functions as a 2D camera to acquire image data. Furthermore, the imaging unit 211 functions as a 3D camera to acquire point group data. Additionally, the imaging unit 211 acquires an image that includes at least the mark and the workpiece to be welded.

[0052] The camera coordinate system setting unit 212 sets the camera coordinate system based on the markers contained in the image captured by the shooting unit 211. For example, the camera coordinate system setting unit 212 sets a three-dimensional orthogonal coordinate system composed of the X-axis, Y-axis and Z-axis that are orthogonal to each other at the origin in the image captured by the shooting unit 211, with the position of the marker as the origin, as the camera coordinate system.

[0053] Figure 3 This diagram illustrates a scenario where the marker M is set on the workpiece W, which is the object of welding, and the camera coordinate system is established with the position of marker M as the origin O. For example... Figure 3 As shown, the mark M is set on the base plate of the workpiece W. The camera coordinate system setting unit 212 sets a three-dimensional orthogonal coordinate system composed of the X-axis, Y-axis, and Z-axis, which are orthogonal to each other at the origin O, with the position of the mark M contained in the image captured by the shooting unit 211 as the origin O. Furthermore, the origin O can be any point among the marks M. For example, a predetermined point, a center point, or a selected point among the marks M can be set as the origin O.

[0054] Here, when detecting a marker M from an image captured by the imaging unit 211, the marker detection device 220 appropriately detects the position of the marker M based on image data obtained by the 2D camera and point group data obtained by the 3D camera.

[0055] The sign M can be any identifier that the imaging unit 211 can recognize as being placed in space. For example, an AR sign is preferred. By using an AR sign, when an AR sign is recognized in space, it is easy to make the user coordinate system, with the AR sign as its origin, coincide with the actual image. Details regarding the processing of the sign detection device 220 for detecting the sign M will be described later.

[0056] In the camera coordinate system set by the camera coordinate system setting unit 212, the motion path of the robot arm 3 is set based on the welding positions L1, L2, and L3 of the workpiece W contained in the image captured by the imaging unit 211. For example, the user can manually select the motion path of the robot arm 3 (welding torch 32), or calculate the distance between the welding positions L1, L2, and L3 and the welding torch 32 to automatically set the motion path of the robot arm 3 (welding torch 32), so that in the image captured by the imaging unit 211, the welding torch 32 mounted at the front end of the robot arm 3 moves along the welding positions L1, L2, and L3 of the workpiece W.

[0057] On the other hand, the robot control device 2 learns the position and orientation (robot coordinate system) of the robot arm 3 from information related to the angles of each axis controlled by the robot control device 2. Then, based on the setting position of the marker M and the position and orientation (setting position) of the robot arm 3 relative to the setting position of the marker M, calibration is performed to make the camera coordinate system and the robot coordinate system consistent. This calibration can be performed, for example, by aligning the tip of the robot arm 3 (welding torch 32) with the setting position of the marker M.

[0058] The program generation unit 213 transforms the motion path of the robot arm 3 set in the camera coordinate system into the robot coordinate system set in the robot control device 2, and generates a work program for making the robot arm 3 move.

[0059] Thus, in the robot teaching system 200, a camera coordinate system is set based on the image detection marker M captured by the imaging unit 211. The welding positions L1, L2, and L3 in this camera coordinate system and the motion path of the robot arm 3 are transformed into the robot coordinate system, and a work program is generated that allows the robot arm 3 to perform welding appropriately at the welding positions L1, L2, and L3. Then, the work program is stored in the storage unit of the robot control device 2, which controls the movement of the robot arm 3 based on the work program. As a result, as a welding robot system, welding can be performed appropriately at the welding positions L1, L2, and L3 of the workpiece W.

[0060] [Details of the processing in the sign detection device 220]

[0061] As described above, since the camera coordinate system is set with the position of the marker M as the origin O, it is important to properly detect the position of the marker M. The following details the processing of the marker detection device 220 for detecting the position of the marker M.

[0062] Figure 4 This diagram illustrates the specific processing steps in the mark detection device 220, which detects the position of the mark M set on the workpiece W. (See diagram for example.) Figure 4 As shown, the sign detection device 220 detects the position of the sign M based on image data obtained by the 2D camera and dot group data obtained by the 3D camera.

[0063] Image generation unit 221 generates a composite image by overlaying point group data acquired by the 3D camera onto image data acquired by the 2D camera. Figure 4 (a)). For example, an image including a workpiece W (the object of welding) and a mark M is captured by a digital camera, and this captured image is saved as image data acquired by a 2D camera. The position (distance) information corresponding to the workpiece W and the mark M is measured by a distance measurement sensor, and this position (distance) information is obtained as point group data shown in three-dimensional coordinate values ​​(X, Y, Z). Then, the image generation unit 221 generates a composite image by overlaying the point group data on the image data acquired by the 2D camera. Specifically, pixel (XY coordinate value) information that matches the image data acquired by the 2D camera is added to the three-dimensional coordinate values ​​(X, Y, Z) and RGB values ​​that correspond to each point of the point group data acquired by the 3D camera. In this way, a given drawing range based on the image data acquired by the 2D camera and the drawing positions of each point group data that constitute the point group data acquired by the 3D camera are recorded as corresponding.

[0064] Furthermore, in order to overlay point data acquired by a 3D camera onto image data acquired by a 2D camera, the image generation unit 221 preferably determines the relative position and orientation of the 2D camera and the 3D camera. The image data acquired by the 2D camera and the point data acquired by the 3D camera are typically captured from the same viewpoint.

[0065] The marker position calculation unit 222 detects the marker M based on image data in the composite image generated by the image generation unit 221, and calculates the position of the marker M. Figure 4(b)). For example, the marker position calculation unit 222 uses given image processing or the like to detect a pre-set image pattern of a marker M from the image data in the composite image generated by the image generation unit 221. Then, for the marker M detected from the image data, the marker position calculation unit 222 transforms the coordinates in the image data into coordinates in space to calculate the three-dimensional position. Here, the position of the marker M can be the outline range of the marker M or the position of the four corner points of the marker M.

[0066] The marker position calculation unit 222 can calculate the three-dimensional position of the marker M detected from the image data, for example, based on preset camera parameters including camera position, viewing direction, lens distortion, zoom, resolution, and other settings. Furthermore, the marker position calculation unit 222 can determine the point group data Dm contained within the area of ​​the marker M from the calculated three-dimensional position of the marker M.

[0067] Here, the three-dimensional position of the marker M calculated by the marker position calculation unit 222 is based on image data obtained from a 2D camera, and therefore may sometimes deviate from the actual position of the marker M. Figure 4 (c)).

[0068] The plane detection unit 223 detects planes based on point group data in the synthetic image generated by the image generation unit 221. Figure 4 (d)). For example, here, the plane detection unit 223 detects the bottom plate, i.e., plane Wa, and the back plate, i.e., plane Wb, of the workpiece W as planes with a given area.

[0069] The marker plane determination unit 224 determines the marker plane in which the marker M exists based on the point group data Dm contained in the region of the marker M detected in the synthetic image and the planes Wa and Wb detected by the plane detection unit 223. Figure 4 (e)). For example, here, the marker plane determination unit 224 determines whether the point group data Dm contained in the region of marker M is included in the point group data constituting plane Wa (Wb). Specifically, if a given proportion (e.g., 20% to 80%) or more of the point group data Dm contained in the region of marker M is included in the point group data constituting plane Wa, the marker plane determination unit 224 can consider that marker M exists on plane Wa and determine plane Wa as the marker plane.

[0070] On the other hand, such as Figure 4 As shown, since the point group data Dm contained in the region of the marker M is not included in the point group data constituting the plane Wb, the marker plane determination unit 224 does not determine the plane Wb as the marker plane.

[0071] The marker position correction unit 225 corrects the position of the marker M by projecting the position of the marker M calculated by the marker position calculation unit 222 onto the marker plane determined by the marker plane determination unit 224. Figure 4 (f) For example, the marker position correction unit 225 projects the position of the marker M calculated by the marker position calculation unit 222 onto the plane Wa, which is determined to be the marker plane by the marker plane determination unit 224, in the vertical direction. That is, it is determined that the position of the outline range of the marker M, detected from the position of the marker M calculated by the marker position calculation unit 222, is moved in the vertical direction based on the plane Wa, and the marker M is placed at a position intersecting the plane Wa. Then, the position of the marker M projected onto the plane Wa is set as the position where the marker M is set.

[0072] Thus, the sign detection device 220 can detect the position of the sign M with high accuracy by correcting the position of the sign M calculated based on the image data obtained by the 2D camera by projecting the point group data obtained by the 3D camera onto the sign plane (here, plane Wa) where the sign M exists.

[0073] As described above, in the sign detection apparatus 220 according to the first embodiment of the present invention, the sign position calculation unit 222 detects the sign M and calculates the position of the sign M based on the image data in the composite image of the image data and dot group data generated by the image generation unit 221. The plane detection unit 223 detects planes Wa and Wb based on the dot group data in the composite image. Then, the sign plane determination unit 224 determines the sign plane (plane Wa in this embodiment) in which the sign M exists based on the dot group data Dm contained in the region of the sign M detected by the sign position calculation unit 222 and the planes Wa and Wb detected by the plane detection unit 223. The sign position correction unit 225 corrects the position of the sign M by projecting the position of the sign M calculated by the sign position calculation unit 222 onto the sign plane. Thus, the position of the sign M can be detected with high accuracy.

[0074] Furthermore, according to the robot teaching system 200 utilizing the mark detection device 220 according to the first embodiment of the present invention, the camera coordinate system setting unit 212 sets the camera coordinate system with the position of the mark M as the origin O based on the position of the mark M detected with high precision by the mark detection device 220. The program generation unit 213 transforms the motion path of the robot arm 3 set in the camera coordinate system from the camera coordinate system to the robot coordinate system set in the robot control device 2 based on the setting position of the mark M, and generates a work program for moving the robot arm 3. As a result, welding can be performed with higher precision at appropriate welding positions.

[0075] Furthermore, in this embodiment, based on the point group data Dm contained within the region of the mark M and the point group data constituting the plane Wa, the mark M is considered to exist on the plane Wa, and the mark plane determination unit 224 determines the plane Wa as the mark plane. However, the method for determining the plane Wa as the mark plane is not limited to this. For example, the user can select the plane where the mark M exists, and the selected plane can be determined as the mark plane. When the mark M is set on the workpiece W, any plane detected by the plane detection unit 223 can be determined as the mark plane, or the user can select any plane from the plane to determine the mark plane.

[0076] Furthermore, in this embodiment, the marker position correction unit 225 corrects the position of the marker M by projecting the position of the marker M calculated by the marker position calculation unit 222 onto the marker plane determined by the marker plane determination unit 224. However, for example, if the positions of the marker M before and after correction are significantly different, the possibility that the marker M may not actually exist on the marker plane determined by the marker plane determination unit 224 can be considered. Therefore, if the difference between the positions of the marker M before and after correction exceeds a given threshold, the position of the marker M calculated by the marker position calculation unit 222 may not be projected onto the marker plane (plane Wa in this embodiment) determined by the marker plane determination unit 224, and the position of the marker M calculated by the marker position calculation unit 222 may be used as the position of the marker M.

[0077] <Second Implementation Method>

[0078] Next, in the second embodiment of the present invention, the case of appropriately detecting the position of the marker M when the marker M is set on the robot arm 3 will be described. Furthermore, the robot teaching system and the marker detection device are related to the use of... Figure 1 as well as Figure 2 The robot teaching system 200 and the mark detection device 220 described herein have the same structure. Therefore, detailed descriptions are sometimes omitted for the same processing and functions. In this embodiment, the differences from the first embodiment are mainly described in detail.

[0079] Figure 5 This diagram illustrates the scenario where marker M is set on robot arm 3, and the camera coordinate system is established with the position of marker M as the origin O. (Example) Figure 5As shown, the marker M is set on the robot arm 3. The camera coordinate system setting unit 212 uses the position of the marker M contained in the image captured by the imaging unit 211 as the origin O, and sets a three-dimensional orthogonal coordinate system composed of the X-axis, Y-axis and Z-axis that are orthogonal to each other at the origin O as the camera coordinate system. Furthermore, when detecting the marker M from the image captured by the imaging unit 211, the marker detection device 220 appropriately detects the position of the marker M based on the image data obtained by the 2D camera and the point group data obtained by the 3D camera.

[0080] [Details of the processing in the sign detection device 220]

[0081] The following details the processing of the sign detection device 220 for detecting the position of sign M in this embodiment.

[0082] Figure 6 This diagram illustrates the specific processing steps performed by the marker detection device 220, which detects the position of the marker M set on the robotic arm 3. (See diagram for example.) Figure 6 As shown, the sign detection device 220 detects the position of the sign M based on image data obtained by a 2D camera and point group data obtained by a 3D camera.

[0083] Image generation unit 221 generates a composite image by overlaying point group data acquired by a 3D camera onto image data acquired by a 2D camera. Figure 6 (a) The marker position calculation unit 222 detects the marker M based on image data in the composite image generated by the image generation unit 221, and calculates the position of the marker M. Figure 6 (b)). Furthermore, the marker position calculation unit 222 obtains the point group data Dm contained within the region of the marker M from the calculated three-dimensional position of the marker M. These are related to the method using the first embodiment. Figure 4 The treatment described in (a) and (b) is the same.

[0084] Furthermore, the marker position calculation unit 222 is consistent with the first embodiment. Figure 4 Similarly, (c) calculates the three-dimensional position of the marker M calculated by the marker position calculation unit 222 based on image data obtained from a 2D camera. Therefore, sometimes an error occurs between the actual position of the marker M and the position of the marker M. Figure 6 (c)).

[0085] The plane detection unit 223 detects planes based on point group data in the synthetic image generated by the image generation unit 221. Figure 6 (d)). For example, here, the plane detection unit 223 detects the bottom plate, i.e., plane Wa, and the back plate, i.e., plane Wb, of the workpiece W as planes with a given area.

[0086] The marker plane determination unit 224 determines the marker plane in which the marker M exists based on the point group data Dm contained in the region of the marker M detected in the synthetic image, and the planes Wa and Wb detected by the plane detection unit 223. Figure 6 (e)). In this embodiment, as Figure 5 As shown, the mark M is not set on either of the planes Wa and Wb constituting the workpiece W, but on the robot arm 3. Therefore, the mark plane determination unit 224 determines that a given proportion (e.g., 20% to 80%) or more of the point group data Dm contained in the region of mark M is not included in the point group data constituting planes Wa and Wb, and thus mark M does not exist on planes Wa and Wb. In this case, a virtual plane BP composed of the point group data Dm is generated based on the region of mark M, and this virtual plane BP is determined as the mark plane where mark M exists. Figure 6 (e)).

[0087] The marker position correction unit 225 corrects the position of the marker M by projecting the position of the marker M calculated by the marker position calculation unit 222 onto the marker plane determined by the marker plane determination unit 224. Figure 6 (f)). For example, the marker position correction unit 225 projects the position of the marker M calculated by the marker position calculation unit 222 onto the virtual plane BP, which is determined to be the marker plane by the marker plane determination unit 224, in the vertical direction. Then, the position of the marker M projected onto the virtual plane BP is set as the position for setting the marker M.

[0088] In this way, the sign detection device 220 generates a virtual plane BP in which the sign M virtually exists based on the point group data obtained by the 3D camera, uses the virtual plane BP as the sign plane, and corrects the position of the sign M calculated based on the image data obtained by the 2D camera by projecting it onto the sign plane (here, the virtual plane BP). Therefore, the position of the sign M is detected with high accuracy.

[0089] As described above, according to the second embodiment of the present invention, even if the mark M is not actually set on the plane constituting the workpiece W, the mark plane determination unit 224 determines the virtual plane BP as the mark plane where the mark M exists based on the point group data Dm contained in the area of ​​the mark M. Then, the mark position correction unit 225 corrects the position of the mark M calculated by the mark position calculation unit 222 by projecting it onto the mark plane. As a result, the position of the mark M can be detected with high accuracy.

[0090] In addition, in this embodiment, the point group data Dm contained in the area of ​​the marker M and the point group data constituting the planes Wa and Wb are used to determine that the marker M does not exist on the planes Wa and Wb. However, for example, it can also be determined that the marker M does not exist on the plane because the marker M is set on the robot arm 3. It can also be determined that it does not exist on the plane by the user's selection.

[0091] Furthermore, in this embodiment, the marker plane determination unit 224 generates a virtual plane BP and determines this virtual plane BP as the marker plane. However, in cases where the point group data Dm contained in the region of the marker M used to generate the virtual plane BP is extremely small or the deviation is large, the possibility that the virtual plane cannot be properly generated is considered. Therefore, if the difference between the positions of the marker M before and after correction exceeds a given threshold, the position of the marker M calculated by the marker position calculation unit 222 may not be projected onto the marker plane (virtual plane BP in this embodiment) determined by the marker plane determination unit 224, but instead the position of the marker M calculated by the marker position calculation unit 222 may be set as the position of the marker M.

[0092] [Mark Detection Method]

[0093] Next, the method for detecting marks by the mark detection device 220 according to the first and second embodiments of the present invention will be described in detail.

[0094] Figure 7 This is a flowchart illustrating the processing flow of the mark detection method M100 performed by the mark detection device 220 according to the first and second embodiments of the present invention. For example... Figure 7 As shown, the mark detection method M100 includes steps S110 to S170, each step being executed by the processor contained in the mark detection device 220.

[0095] In step S110, the mark detection device 220 generates a composite image by overlaying dot group data acquired by a 3D camera onto the image data acquired by a 2D camera (composite image generation step). As a specific example, the image generation unit 221 in the mark detection device 220 generates a composite image based on image data captured by a digital camera that simultaneously includes the workpiece W and the mark M, and dot group data measured by a distance measurement sensor.

[0096] In step S120, the sign detection device 220 detects a sign based on the image data in the composite image generated in step S110 and calculates the position of the sign (sign position calculation unit). As a specific example, the sign position calculation unit 222 in the sign detection device 220 detects a pre-set image pattern of the sign M from the image data and calculates the three-dimensional position of the sign M based on camera parameters, etc.

[0097] In step S130, the mark detection device 220 detects a plane (plane detection step) based on the point group data in the synthetic image generated in step S110. As a specific example, the plane detection unit 223 in the mark detection device 220 detects planes Wa and Wb, etc., that constitute the workpiece W, as planes with a given area or more.

[0098] In step S140, the sign detection device 220 determines the sign plane in which the sign exists based on the dot group data contained in the area of ​​the sign detected in step S120 and the plane detected in step S130 (sign plane determination step). As a specific example, the sign plane determination unit 224 in the sign detection device 220 determines whether a given proportion or more of the dot group data Dm contained in the area of ​​the sign M is included in the dot group data constituting the plane Wa (Wb) detected in step S130.

[0099] If a given proportion or more of the point group data in the point group data Dm is included in the point group data constituting the plane Wa (Wb) (if the flag M exists in the plane Wa), the flag plane determination unit 224 determines the plane Wa detected in step S130 as the flag plane (step S150).

[0100] On the other hand, if a given proportion or more of the point group data in the point group data Dm is not included in the point group data constituting the plane Wa (Wb) (if the flag M does not exist in the plane Wa (Wb), the marker plane determination unit 224 determines the virtual plane BP composed of the point group data Dm as the marker plane (step S160).

[0101] In step S170, the sign detection device 220 corrects the position of the sign by projecting the position calculated in step S120 onto the sign plane determined in step S150 or S160 (sign position correction step). As a specific example, the sign position correction unit 225 in the sign detection device 220 corrects the position of the sign M by projecting the position calculated in step S120 onto plane Wa or virtual plane BP.

[0102] As described above, according to the marker detection method M100, regardless of whether the marker M exists on a plane such as plane Wa, or is located on a robot arm 3 and does not exist on a plane, the state of the marker M is determined in step S140, and then plane Wa is determined as the marker plane in step S150, or the virtual plane BP is determined as the marker plane in step S160. Then, in step S170, the position of the marker M is corrected by projecting its position onto the marker plane (plane Wa or virtual plane BP). Therefore, the position of the marker M can be detected with high accuracy, regardless of whether the marker M exists on a plane such as plane Wa, or is located on a robot arm 3 and does not exist on a plane.

[0103] Furthermore, in the embodiments described above, the mark detection device 220 used in the robot teaching system 200 of the welding robot system 100 has been described as an example, but industrial robots using the mark detection device 220 are not limited to the welding robot system 100. For example, the mark detection device 220 can also be used in assembly robot systems, transport robot systems, etc., that assemble and transport electronic components and mechanical components.

[0104] The embodiments described above are intended to facilitate understanding of the present invention and are not intended to limit the scope of the invention. The elements, configurations, materials, conditions, shapes, and dimensions of the embodiments are not limited to the examples and can be appropriately modified. Furthermore, the structures shown in different embodiments can be partially substituted for or combined with each other.

Claims

1. A mark detection device, characterized in that, have: The image generation unit generates a composite image by overlaying image data acquired by a 2D camera with point group data acquired by a 3D camera from the same viewpoint as the 2D camera. The marker position calculation unit detects the marker and calculates its position based on image data in the synthesized image; The plane detection unit detects planes based on point group data in the synthesized image; The sign plane determination unit determines the sign plane in which the sign exists based on the point group data contained in the area of ​​the sign detected from the position of the sign calculated by the sign position calculation unit and the plane detected by the plane detection unit. and The sign position correction unit corrects the position of the sign by projecting the position of the sign calculated by the sign position calculation unit onto the sign plane. If a given proportion or more of the point cluster data within the region of the sign detected in the synthetic image is not included in the plane detected by the plane detection unit, the sign plane determination unit determines the virtual plane formed by the point cluster data within the region of the sign as the sign plane.

2. The mark detection device according to claim 1, characterized in that, If a given proportion or more of the point group data within the region of the marker detected in the composite image is contained in a plane detected by the plane detection unit, the marker plane determination unit determines that the plane is the marker plane.

3. The mark detection device according to claim 1, characterized in that, The mark plane determination unit performs the following processing: When the mark is set on the workpiece, the plane detected by the plane detection unit is determined to be the mark plane. When the marker is set on a robotic arm, the virtual plane formed by the point group data within the region of the marker detected in the synthetic image is determined as the marker plane.

4. A robot teaching system, characterized in that, have: The camera unit has a 2D camera that captures images of the workpiece and the mark, and a 3D camera that captures images from the same viewpoint as the 2D camera. An image generation unit overlays point group data acquired by a 3D camera onto image data acquired by the 2D camera to generate a composite image; The marker position calculation unit detects the marker and calculates its position based on image data in the synthesized image; The plane detection unit detects planes based on point group data in the synthesized image; The sign plane determination unit determines the sign plane in which the sign exists based on the point group data contained in the area of ​​the sign detected from the position of the sign calculated by the sign position calculation unit and the plane detected by the plane detection unit. The sign position correction unit corrects the position of the sign by projecting the position of the sign calculated by the sign position calculation unit onto the sign plane; The camera coordinate system setting unit sets the camera coordinate system based on the position of the corrected marker; and The program generation unit transforms the movement path of the robot arm in the camera coordinate system into the robot coordinate system set in the robot control device, and generates a work program to make the robot arm move. If a given proportion or more of the point cluster data within the region of the sign detected in the synthetic image is not included in the plane detected by the plane detection unit, the sign plane determination unit determines the virtual plane formed by the point cluster data within the region of the sign as the sign plane.

Images