Object detection system and robotic system

Abstract

This invention provides an object detection system and a robot system capable of detecting the position of feature line segments of objects within the image area of ​​a camera with greater accuracy. [Solution] The object detection system 30 includes a camera (22) having an array of image sensors arranged in a matrix along a first direction (U) and a second direction (V), a rotation actuator (38) that changes the roll angle of the camera (22), an image processing unit 50 that extracts feature line segments (90, 92) of objects (W1, W2) from within an image region (80) indicated by the image data Img, and a drive control unit 56 that controls the driving of the rotation actuator (38) so that the intersection angle of the extracted feature line segments (90, 92) with respect to the first direction (U) or the second direction (V) becomes larger before and after the rotation of the camera (22).

Description

[Technical Field]

[0001] The present invention relates to an object detection system for detecting objects, and a robot system comprising a robot for processing workpieces. [Background technology]

[0002] Object detection systems that use cameras to detect objects have been known for some time. For example, Patent Document 1 discloses a system that uses visual feedback to control the movement of a robot while aligning the terminals of a cable with a receptacle on a circuit board. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2024-066837 [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] By the way, in the system disclosed in Patent Document 1, depending on the position and orientation of the camera, the contour line of the terminal or receptacle may be substantially parallel to a first direction (e.g., horizontal direction) or a second direction (e.g., vertical direction) of the image area. Specifically, if the contour line is parallel to the first direction, detection accuracy for positions exceeding the image resolution (or pixel pitch) in the second direction cannot be obtained. Alternatively, if the contour line is parallel to the second direction, detection accuracy for positions exceeding the image resolution (or pixel pitch) in the first direction cannot be obtained.

[0005] This invention has been made in view of these problems, and its objective is to provide an object detection system and a robot system that can detect the position of a feature line segment of an object within the image area of ​​a camera with higher accuracy. [Means for solving the problem]

[0006] The object detection system of the present invention comprises: a camera having an array of image sensors arranged in a matrix along a first direction and a second direction, and generating image data by capturing an object in three-dimensional space; a rotation actuator that changes the roll angle of the camera by rotating the camera around an optical axis; an image processing unit that performs image processing on the image data generated by the camera and extracts feature line segments of the object from within the image region indicated by the image data; and a drive control unit that controls the driving of the rotation actuator so that the intersection angle between the feature line segments extracted by the image processing unit and the first direction or the second direction increases before and after the rotation of the camera.

[0007] Furthermore, the roll angle includes a first angle and a second angle, and the absolute value of the angle difference between the first angle and the second angle is greater than 0° and tangent. -1 (1 / 2)° or less is also acceptable.

[0008] Furthermore, the absolute value is tan -1 (1 / 10)° is also acceptable.

[0009] Furthermore, the first angle may be 0°, -45°, or 45°.

[0010] Furthermore, the drive control unit may perform drive control of the rotary actuator while tracking the object based on the image data sequentially generated by the camera.

[0011] The robot system of the present invention comprises a robot that processes a workpiece in three-dimensional space, a camera having an array of image sensors arranged in a matrix along a first direction and a second direction, and which captures the workpiece and generates image data, a rotation actuator that changes the roll angle of the camera by rotating the camera around an optical axis, and a control device to which the camera is connected and which controls the movement of the robot, wherein the control device performs image processing on the image data generated by the camera and extracts feature line segments of the workpiece from within the image region indicated by the image data, and performs control processing that controls the drive of the rotation actuator so that the intersection angle between the feature line segments extracted by the image processing and the first direction or the second direction becomes larger before and after the rotation of the camera.

[0012] Furthermore, the workpiece includes a cable and an electronic circuit board, the camera is mounted on the robot and attached to a tool that grips the cable, and the characteristic line segment may be a linear component of the contour line of the terminal of the cable or the receptacle of the electronic circuit board. [Effects of the Invention]

[0013] According to the present invention, the position of a feature line segment of an object within the image area of ​​a camera can be detected with higher accuracy. [Brief explanation of the drawing]

[0014] [Figure 1] This figure shows the overall configuration of a robot system incorporating an object detection system according to one embodiment of the present invention. [Figure 2] This figure shows an example of the device configuration of the tool shown in Figure 1. [Figure 3] Figure 1 is a block diagram illustrating the functions of the object detection system. [Figure 4] This flowchart shows an example of how to set the reference roll angle. [Figure 5]It is a diagram showing an example of an image obtained by the second camera in FIG. 1. [Figure 6] It is a diagram showing a change in spatial resolution accompanying switching of a roll angle. [Figure 7] It is a flowchart showing an example of robot motion control by the object detection system of FIGS. 1 and 3. [Figure 8] It is a diagram showing a change in a feature line segment accompanying a change in a roll angle. [Figure 9] It is a detailed flowchart relating to step SP26 in FIG. 7. [Figure 10] It is a diagram showing a change in a contour image in a comparative example. [Figure 11] It is a diagram showing a change in a contour image in an embodiment. DESCRIPTION OF EMBODIMENTS

[0015] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. To facilitate understanding of the description, the same constituent elements are denoted by the same reference numerals as much as possible in each drawing, and overlapping descriptions are omitted. Further, the term "part" may be replaced with other terms such as unit, module, device, or element, for example.

[0016] [Configuration of Object Detection System 30] <Overall Configuration of Robot System 10> FIG. 1 is a diagram showing an overall configuration of a robot system 10 incorporating an object detection system 30 according to an embodiment of the present invention. FIG. 2 is a diagram showing an example of a device configuration of a tool 18 in FIG. 1. Specifically, the main part of this robot system 10 is configured including a robot 12 and a control device 14.

[0017] Robot 12 is, for example, a vertical articulated robot having multiple joint axes. Robot 12 can perform various tasks, including gripping and moving workpieces, welding, and painting, by independently driving multiple joint axes in response to commands from the control device 14. In the example shown in Figures 1 and 2, robot 12 performs a connection operation in which the terminal 42 of cable W1 is inserted into a receptacle 44 on an electronic circuit board W2. In this case, cable W1 and electronic circuit board W2 correspond to the workpieces to be processed.

[0018] A tool 18 (or an end effector or manipulator) is attached to the end of the arm 16 of the robot 12. This allows the robot 12 to grasp the tip of the cable W1 and move the cable W1 to a desired position and orientation. A first camera 20 is attached to the arm 16, and a second camera 22 is attached to the tool 18. The first camera 20 and the second camera 22 are each connected to the control device 14.

[0019] The first camera 20 is an imaging device having an array of image sensors arranged in a matrix (hereinafter referred to as the image sensor array), which captures an object in three-dimensional space, generates an image signal for each frame, and outputs image data Img (Figure 3) that shows the time series of the image. The second camera 22 is an imaging device having an image sensor array similar to the configuration of the first camera 20, which captures an object in three-dimensional space, generates an image signal for each frame, and outputs image data Img (Figure 3) that shows the time series of the image. Hereinafter, when the plane formed by the image sensor array is defined as the UV plane, this two-dimensional coordinate space may be referred to as the "camera coordinate space". Furthermore, the camera coordinate space defined by the first camera 20 may be distinguished as the "first camera coordinate space", and the camera coordinate space defined by the second camera 22 may be distinguished as the "second camera coordinate space".

[0020] At least one of the first camera 20 and the second camera 22 is positioned so that the workpiece is within the field of view while the robot 12 is processing the workpiece. The first camera 20 and the second camera 22 consist of, for example, monocular cameras such as visible light cameras, infrared cameras, and TOF (Time of Flight) cameras, cameras of the same type, or a combination of cameras of different types.

[0021] The control device 14 is a computer that controls the motion of the robot 12. The control device 14 acquires image data Img (Figure 3) from the first camera 20 or the second camera 22 and performs visual feedback (VFB) control on the robot 12. Here, "visual feedback control" refers to a method of calculating the position, orientation, and movement of the controlled object through image processing and feeding this information back into the drive control of the actuator. Visual feedback control is a form of object tracking technology (or tracking technology) that tracks objects captured in an image. Specifically, the control device 14 is composed of a communication I / F 24, a processor 26, and a memory 28.

[0022] The communication interface 24 is an interface for communicating with external devices. This allows the control device 14 to exchange data with, for example, a programming pendant, a work terminal, or a higher-level device (none of which are shown).

[0023] The processor 26 may be a general-purpose processor including a CPU (Central Processing Unit), or it may be a dedicated processor including an FPGA (Field Programmable Gate Array) or a GPU (Graphics Processing Unit). The functions implemented by the processor 26 are explained in detail in Figure 3.

[0024] Memory 28 consists of a non-transient and computer-readable recording medium (or storage medium). Here, the computer-readable recording medium is a storage device including an HDD (Hard Disk Drive) or SSD (Solid State Drive), or a portable medium including a magneto-optical disk, ROM (Read Only Memory), CD (Compact Disc)-ROM, or flash memory.

[0025] Incidentally, the robot system 10 functions as an object detection system 30 that detects objects placed in real space. This object detection system 30 functions as a "multi-monocular camera system" that uses multiple cameras to detect objects without using triangulation. The number of cameras may be three or more, or it may be just one.

[0026] In the example shown in Figure 1, the plane forming the mounting surface of the electronic circuit board W2 is defined as the xy-plane, and the direction of the normal to the mounting surface is defined as the z-axis. In this case, the three-dimensional space is represented by a three-dimensional Cartesian coordinate system with x, y, and z as the three axes. The xyz coordinate system may be a "real coordinate system" representing real space, or it may be a "system coordinate system" (or world coordinate system) defined within the object detection system 30.

[0027] Furthermore, in order to control the movement of the robot 12, a robot coordinate space is defined that indicates the position and orientation of the robot 12. Here, the "robot coordinate space" is a three-dimensional Cartesian coordinate system with X, Y, and Z as the three axes. These state variables are formed by combining state values ​​such as, for example, [1] coordinate value on the X axis (X), [2] coordinate value on the Y axis (Y), [3] coordinate value on the Z axis (Z), [4] roll angle around the X axis (θ), [5] pitch angle around the Y axis (φ), and [6] yaw angle around the Z axis (ψ).

[0028] As shown in Figure 2, the tool 18 comprises a substantially flat base 32, a first support member 34, and a second support member 36. The first main surface of the base 32 is provided so as to be attachable to the tip of the arm portion 16. The first support member 34 and the second support member 36 are provided on the second main surface of the base 32 so as to extend in the normal direction.

[0029] The second camera 22 is rotatably mounted approximately in the center of the first support member 34. The second camera 22 is fixed so that the imaging surface at its tip is directed toward the second support member 36. A motor 38 (corresponding to a "rotation actuator") for rotating the second camera 22 around its optical axis is connected to the base end of the second camera 22. The roll angle A of the second camera 22 can be changed by driving the motor 38.

[0030] A clamper 40, 40 for gripping the cable W1 is provided approximately in the center of the second support member 36. As a result, the tip of the cable W1 is gripped from both sides at a position closer to the base end than the terminal 42. As shown in Figure 2, when the cable W1 is near the electronic circuit board W2, an image showing both the terminal 42 and the receptacle 44 can be obtained through imaging by the second camera 22.

[0031] <Functional Block Diagram> Figure 3 is a block diagram illustrating the functions of the object detection system 30 shown in Figure 1. Specifically, the main components of the object detection system 30 include a first camera 20, a second camera 22, a processor 26, a memory 28, and a motor 38 (Figure 2). The processor 26 functions as an image processing unit 50, an motion control unit 52, an angle setting unit 54, and a drive control unit 56 by reading and executing programs and data stored in the memory 28. The memory 28 also stores an image database (hereinafter referred to as image DB 66), layout information 68, and mapping information 70.

[0032] The image processing unit 50 performs image processing on the image data Img generated by the first camera 20 or the second camera 22, and extracts characteristic line segments (hereinafter referred to as "feature line segments") relating to an object from within the image region indicated by the image data Img. Specifically, the image processing unit 50 is composed of an image acquisition unit 60, a feature extraction unit 62, and a state conversion unit 64.

[0033] The image acquisition unit 60 sequentially acquires image data (Img) output from the first camera 20 or the second camera 22. Subsequently, the image acquisition unit 60 selects and outputs image data (Img) from among the multiple image data (Img) it has acquired that will be used for motion control of the robot 12 or drive control of the motor 38.

[0034] The feature extraction unit 62 performs extraction processing on the image data Img acquired by the image acquisition unit 60, and extracts feature line segments from within the image region. This extraction processing includes, for example, color conversion calculations, image binarization calculations, neighboring pixel concatenation, image matching calculations, or state quantity calculations. The matching calculation is, for example, template matching, which calculates the agreement rate between the image data Img and the basic image data ImgRef, and identifies an object if the agreement rate exceeds a threshold. The state quantities of the feature line segments include, for example, presence / absence, position, inclination, length, and distance.

[0035] The state conversion unit 64 converts the state quantities calculated by the feature extraction unit 62. For example, the state conversion unit 64 uses the layout information 68 and the mapping information 70 to convert the state quantities in the camera coordinate space into state quantities in the robot coordinate space. When the state conversion unit 64 uses image data Img from the first camera 20, it performs a conversion process from the first camera coordinate space to the robot coordinate space. When the state conversion unit 64 uses image data Img from the second camera 22, it performs a conversion process from the second camera coordinate space to the robot coordinate space.

[0036] The motion control unit 52 performs motion control of the robot 12 to track an object containing a feature line segment, based on the state quantities converted by the state conversion unit 64. Specifically, the motion control unit 52 determines the target position of the robot 12 from the state quantities in the robot coordinate space, and generates a movement path from the robot 12's current position to the target position using control information including the robot 12's encoder information. Then, the motion control unit 52 determines the control quantities necessary for the robot 12 to move along the movement path, and outputs a first control signal including these control quantities to the robot 12.

[0037] The angle setting unit 54 sets the angle around the optical axis of the second camera 22 (i.e., the roll angle A). For example, the angle setting unit 54 refers to information about the workpiece (hereinafter referred to as workpiece information), selects one of several complementary values, and supplies it to the drive control unit 56 as the reference roll angle Ab. The reference roll angle Ab is, for example, 0°, -45°, or 45°. Examples of workpiece information include the shape and type of the workpiece, the processing method of the workpiece, and the shape and orientation of the part of the workpiece of interest.

[0038] The drive control unit 56 performs drive control of the motor 38 using the reference roll angle Ab set by the angle setting unit 54. Specifically, the drive control unit 56 determines a control amount for the roll angle A of the second camera 22 and outputs a second control signal including this control amount to the motor 38. The drive control unit 56 also selects one of several detection modes and changes the roll angle A according to the update rules in that detection mode. Examples of detection modes include [1] a "fixed mode" in which the roll angle A is fixed regardless of the detection result of the camera image, or [2] a "switching mode" in which the roll angle A is dynamically switched according to the detection result of the camera image.

[0039] The drive control unit 56 executes the above-described switching mode while tracking an object is being performed, for example, based on image data Img sequentially generated by the first camera 20 or the second camera 22. Specifically, this switching mode is a detection mode for increasing the intersection angle with the two axes (U-axis or V-axis) that make up the camera coordinate space with respect to the feature line segment extracted by the image processing unit 50. The roll angle A can be switched in steps or continuously, and includes, for example, a first angle and a second angle. If the first angle is the reference roll angle Ab, then the second angle is (Ab + ΔA). |ΔA| is the absolute value of the angle difference between the first angle and the second angle. For example, |ΔA| = tan -1 (1 / n)° (where n is an integer greater than or equal to 2). That is, 0° < |ΔA| ≤ tan -1 The relationship (1 / 2)° is satisfied. Here, tan -1 (1 / 2)° ≈ 26.57°.

[0040] For example, if the feature line segment before switching the roll angle A extends in the U-axis direction, then with respect to the feature line segment after switching, the image resolution in the V-axis direction becomes effectively n times, and the pixel pitch becomes effectively (1 / n) times. In other words, from the viewpoint of position detection performance, a larger value of n is preferable. On the other hand, as the value of n increases, the length of the feature line segment required to detect the position error described above increases. From the viewpoint of feature line segment detection performance, the value of n is preferably in the range of 5 ≤ n ≤ 15. Specifically, when n = 10, tan -1 (1 / 10)° ≈ 5.71°.

[0041] If the drive control unit 56 defines the intersection angle as the smaller of the angle between the feature line segment and the U-axis, or the angle between the feature line segment and the V-axis, it may determine whether or not to switch the roll angle A depending on the relationship between the intersection angle and the threshold. Specifically, the drive control unit 56 determines that switching the roll angle A is necessary if the intersection angle is less than 5°, while determining that switching the roll angle A is unnecessary if the intersection angle is 5° or greater.

[0042] Image DB66 is a collection of reference images used in the matching calculation described above. The reference images correspond to camera images obtained sequentially when the robot 12 performs the task correctly. In the example of connection work, the reference images include [1] the gripping state of cable W1, [2] the movement state of cable W1, [3] the opposing state of cable W1 and electronic circuit board W2, and [4] a camera image showing the ideal connection state. To distinguish it from the image data Img that was actually obtained, the image data showing the reference images is called the reference image data ImgRef.

[0043] The layout information 68 includes information indicating the placement of the first camera 20 or the second camera 22. Examples of layout information 68 include camera identification information, coordinate values ​​indicating position or direction vectors indicating orientation, the type of tool 18, and the type of robot 12.

[0044] The mapping information 70 includes information for identifying coordinate transformation rules between different coordinate spaces. Examples of the mapping information 70 include matrix elements of a coordinate transformation matrix, or state variables for each feature point. Examples of coordinate transformations include [1] transformation from the first camera coordinate space of the first camera 20 to the second camera coordinate space of the second camera 22, [2] transformation from the first camera coordinate space to the robot coordinate space, or [3] transformation from the second coordinate space to the robot coordinate space.

[0045] [Operation of object detection system 30] The object detection system 30 in this embodiment is configured as described above. Next, the operation of changing the roll angle A by the object detection system 30, in particular the control device 14, will be explained with reference to Figures 4 to 11.

[0046] <1. Initial settings for the reference roll angle> First, the initial setting method for roll angle A will be explained with reference to Figures 4 and 5. Figure 4 is a flowchart showing an example of how to set the reference roll angle Ab. Steps SP11 to SP15 in Figure 4 correspond to step SP10 in Figure 7, which will be discussed later.

[0047] (Step SP11: Acquisition calculation) In step SP11 of Figure 4, the angle setting unit 54 of the processor 26 acquires information about the workpiece to be processed (i.e., workpiece information).

[0048] (Step SP12: Decision Calculation) In step SP12, the angle setting unit 54 determines the workpiece detection mode based on the workpiece information acquired in step S11.

[0049] (Step SP13: Verification calculation) In step SP13, the angle setting unit 54 proceeds to step SP14 if the detection mode determined in step SP12 is the first mode (step SP12: first mode), and to step SP15 if the detection mode is the second mode (step SP12: second mode).

[0050] (Steps SP14, SP15: Setting calculations) In step SP14, the angle setting unit 54 sets the reference roll angle corresponding to the first mode to Ab=0°. In step SP15, the angle setting unit 54 sets the reference roll angle corresponding to the second mode to Ab=45°. The angle setting unit 54 supplies the reference roll angle Ab set in steps SP14 and SP15 to the drive control unit 56.

[0051] Figure 5 shows an example of a reference image corresponding to the second camera 22 in Figure 1. More specifically, Figure 5 visualizes the rectangular image region 80 indicated by the reference image data ImgRef. The horizontal direction of the image region 80 represents the U-axis, and the vertical direction of the image region 80 represents the V-axis. Within this image region 80 are a cable region Rc indicating cable W1, a substrate region Rb indicating electronic circuit board W2, a terminal region Rt indicating terminal 42, and a receptacle region Rr indicating receptacle 44.

[0052] As the workpiece information described above, the contour lines of two regions of interest, Aoi1 and Aoi2, are used. Region of interest Aoi1 is the contour line of the receptacle region Rr. Region of interest Aoi2 is the contour line of the terminal region Rt. For example, if the contour lines of regions of interest Aoi1 and Aoi2 are approximately parallel to the U-axis (or V-axis), the "second detection mode" with a reference roll angle of Ab = 45° is selected.

[0053] Figure 6 shows the change in spatial resolution associated with the setting of the reference roll angle Ab. The left side of Figure 6 shows the image sensor array 82 with a reference roll angle of Ab = 0° when the first detection mode is selected. The right side of Figure 6 shows the image sensor array 82 with a reference roll angle of Ab = 45° when the second detection mode is selected. For the sake of explanation, we assume that the "x-axis" in real space and the "U-axis" in camera coordinate space coincide.

[0054] Each image sensor 84n, 84r has a square-shaped pixel area and is arranged at equal intervals with an array pitch ΔP. In the first detection mode, image sensor 84n is parallel to the x-axis, so the spatial resolution in the U-axis direction is the same as the array pitch, ΔP. In the second detection mode, image sensor 84r is rotated 45° with respect to the x-axis, so the spatial resolution in the U-axis direction is 1 / √2 of the array pitch (i.e., ΔP / √2). In other words, if the regions of interest Aoi1, Aoi2 are approximately parallel to the x-axis or y-axis, selecting the second detection mode can substantially improve the accuracy of position detection.

[0055] <2. Roll angle switching operation> Next, the operation of switching the roll angle will be explained with reference to Figures 7 to 11. Figure 7 is a flowchart showing an example of motion control of the robot 12 by the object detection system 30 shown in Figures 1 and 3.

[0056] (Step SP10: Setting calculation) In step SP10 of Figure 7, the drive control unit 56 of the processor 26 sets the reference roll angle Ab, which is the reference position of the second camera 22. This setting calculation is performed, for example, according to the flowchart shown in Figure 3.

[0057] (Step SP20: Control calculation) In step SP20, the operation control unit 52 of the processor 26 starts and continues to control the operation of the robot 12 according to visual feedback, for example.

[0058] (Step SP22: Verification calculation) In step SP22, the processor 26 checks whether terminal 42 of cable W1 has reached the target position (i.e., the mating position of receptacle 44). If terminal 42 of cable W1 has not yet reached the target position (step SP22: NO), the processor 26 proceeds to the next step SP24.

[0059] (Step SP24: Acquisition calculation) In step SP24, the image processing unit 50 (more specifically, the image acquisition unit 60) of the processor 26 acquires image data Img generated by the first camera 20 or the second camera 22.

[0060] (Step SP26: Extraction Operation) In step SP24, the image processing unit 50 (more specifically, the feature extraction unit 62) performs image processing on the image data Img acquired in step SP24 and extracts feature line segments 90 within the image region 80 corresponding to the image data Img. A specific example of the extraction method is explained in detail in Figure 9.

[0061] (Step SP28: Judgment calculation) In step SP28 of Figure 7, the drive control unit 56 determines whether or not a change in the roll angle A is necessary based on the state quantity of the feature line segment 90 extracted in step SP26. If a change in the roll angle A is necessary (step SP28: YES), the drive control unit 56 proceeds to the next step SP30. On the other hand, if a change in the roll angle A is not necessary (step SP28: NO), the drive control unit 56 skips the execution of step SP30 and returns to step SP20.

[0062] (Step SP30: Control calculation) In step SP30, the drive control unit 56 controls the motor 38 to rotate the second camera 22 to a desired roll angle A. The roll angle A is changed, for example, from [1] Ab to (Ab+ΔA) or from [2] (Ab+ΔA) to Ab. After executing step SP30, the processor 26 returns to step SP20.

[0063] Figure 8 shows the changes in feature lines 90 and 92 due to a change in the roll angle A. Figure 8 visualizes the rectangular image region 80 shown by the image data Img from the second camera 22. The horizontal direction of the image region 80 represents the U-axis, and the vertical direction of the image region 80 represents the V-axis. Point Ac is the centroid of the image region 80 and corresponds to the rotation center of the second camera 22.

[0064] Within this image region 80, a feature line segment 90 is displayed that represents the contours of the regions of interest Aoi1 and Aoi2 (Figure 5). When the roll angle of the second camera 22 is Ab=0°, the feature line segment 90 is positioned in the positional relationship shown by the dashed line. The feature line segment 90 is a line segment connecting two endpoints P1 and P2. In the example in Figure 8, since the feature line segment 90 is approximately parallel to the V-axis direction, the intersection angle with the V-axis direction becomes smaller than the threshold, and it is determined that a change in the roll angle A is necessary.

[0065] As the roll angle changes (Ab → Ab+ΔA), feature line segment 92 is displayed in place of feature line segment 90 within this image region 80. Feature line segment 92, shown as a solid line, connects two endpoints Q1 and Q2. Note that endpoints Q1 and Q2 correspond to points obtained by rotating endpoints P1 and P2 by ΔA around point Pc, respectively. In the example in Figure 8, since feature line segment 92 is inclined with respect to the V-axis direction, the intersection angle with the V-axis direction becomes larger than the threshold, and it is determined that changing the roll angle A is unnecessary.

[0066] Next, the processor 26 sequentially repeats steps SP20 to SP30 in Figure 7 until terminal 42 of cable W1 reaches the target position. When terminal 42 of cable W1 reaches the target position (step SP22: YES), the processor 26 terminates the execution of the flowchart in Figure 7.

[0067] <3. Roll angle switching operation> Next, the detection method for feature lines 90 and 92 (step SP26 in Figure 7) will be explained in detail with reference to the flowchart in Figure 9.

[0068] (Step SP40: Binarization operation) In step SP40, the image processing unit 50 (specifically, the feature extraction unit 62) performs a binarization process on the image data Img acquired by the image acquisition unit 60.

[0069] (Step SP42: Calculation) In step SP42, the feature extraction unit 62 performs detection processing on the image data Img that was binarized in step SP40 and calculates the state quantities (in this case, position and inclination) of the feature line segment 90 in the camera coordinate space.

[0070] (Step SP44: Acquisition Operation) In step SP44, the feature extraction unit 62 obtains the roll angle A that was set when generating the image data Img.

[0071] (Step SP46: Transformation Operation) In step SP46, the feature extraction unit 62 uses at least the roll angle A obtained in step SP44 to convert the state variables of the feature line segments 90 and 92 calculated in step SP42 into state variables in the robot coordinate space (in this case, position and inclination).

[0072] In this way, the image processing unit 50 completes the execution of the flowchart in Figure 9 (step SP26 in Figure 7). Subsequently, the image processing unit 50 supplies state values ​​in the robot coordinate space to the motion control unit 52 and state values ​​in the camera coordinate space to the drive control unit 56.

[0073] <4. Effects> Figure 10 shows the change in the contour image in the comparative example. Subregions 1n and 1s each contain the contour shape of feature line segment 90 (Figure 8) from the image captured using the second camera 22. Each cell constituting subregions 1n and 1s corresponds to a binarized pixel. A filled cell indicates a pixel with a value of 1, while an unfilled cell indicates a pixel with a value of 0. The "comparative example" shows the change in the contour image captured while fixing the roll angle A=0° and translating the position of tool 18 (Figures 1 and 2) in the U-axis direction. Specifically, subregion 1s corresponds to the case where subregion 1s is shifted 0.3 pixels to the positive side in the U-axis direction compared to subregion 1n. As can be seen from Figure 10, when the amount of translation is less than 1 pixel, there is no change in the image between subregions 1n and 1s. In other words, even though the position of feature line segment 90 is actually slightly different, it can be detected as being in the same position.

[0074] Figure 11 shows the change in the contour image in the embodiment. Subregions 94n and 94s each contain the contour shape of feature line segment 92 (Figure 8) from the image captured using the second camera 22. Each cell constituting subregions 94n and 94s corresponds to a binarized pixel. A filled cell indicates a pixel with a value of 1, while an unfilled cell indicates a pixel with a value of 0. In the "Embodiment," the roll angle is A = tan -1While fixing (1 / 10) ≒ 5.71°, this shows changes in contour images captured while translating the position of tool 18 (FIGS. 1 and 2) in the U-axis direction. Specifically, partial region 94s corresponds to a case shifted to the positive side in the U-axis direction by 0.3 pixels compared to partial region 94n. As understood from FIG. 11, even though the parallel translation amount is less than 1 pixel, image changes occur between partial regions 94n and 94s. That is, when the position of the feature line segment 92 changes slightly, it can be detected as a different position.

[0075] [Summary of Embodiment] As described above, according to the object detection system 30 and the robot system 10 in this embodiment, the motor 38 is driven and controlled such that the intersection angle between the feature line segment 90 extracted by the image processing unit 50 and the U-axis (first direction) or the V-axis (second direction) increases before and after the rotation of the second camera 22. Therefore, the position of the feature line segment 92 of the object (here, cable W1 or electronic circuit board W2) located within the image region 80 of the second camera 22 can be detected with higher accuracy.

[0076] Further, the roll angle includes the first angle and the second angle, and the absolute value of the angle difference between the first angle and the second angle is greater than 0° and tan -1 (1 / 2)° or less. When the feature line segment 90 before rotation extends in the U-axis direction, the image resolution in the V-axis direction for the feature line segment 92 after rotation becomes substantially twice or more, and the pixel pitch becomes substantially half or less. When the feature line segment 90 before rotation extends in the V-axis direction, the image resolution in the U-axis direction for the feature line segment 92 after rotation becomes substantially twice or more, and the pixel pitch becomes substantially half or less.

[0077] Further, the absolute value of the angle difference between the first angle and the second angle is tan -1 (1 / 10)°. This makes it possible, for example, to achieve both the detection accuracy of the position of the feature line segment 92 and the detection performance of the feature line segment 92 itself.

[0078] Furthermore, the first angle may be 0°, -45°, or 45°. This allows the image resolution to be increased by a factor of √2, if necessary.

[0079] Furthermore, if the workpiece is a cable W1 and an electronic circuit board W2, the second camera 22 is mounted on the robot 12 and attached to a tool 18 that grips the cable W1, and the characteristic line segments 90 and 92 may be the linear components of the contour lines of the terminal 42 of the cable W1 or the receptacle 44 of the electronic circuit board W2. By rotating the second camera 22 attached to the tool 18 mounted on the robot 12 in a timely manner, the alignment of the terminal 42 and the receptacle 44 can be performed with high precision.

[0080] [Differentiation] It should be noted that the present invention is not limited to the embodiments described above, and can be freely modified without departing from the spirit of the invention. Alternatively, the various components may be combined arbitrarily as long as no technical inconsistencies arise. Alternatively, the execution status or execution order of each step constituting the flowchart may be changed as long as no technical inconsistencies arise.

[0081] In the embodiments described above, the case in which the object detection system 30 functions as a "multi-monocular camera" system was used as an example, but the object detection system 30 can also be applied to a "stereo camera" system using triangulation. For example, the object detection system 30 normally selects stereo camera mode to detect objects, but may switch to multi-monocular camera mode to continue object detection while feature points within the field of view of one camera are temporarily lost.

[0082] In the embodiments described above, a vertical articulated robot was used as an example, but the type of industrial robot is not limited to this. For example, a horizontal articulated robot, a parallel link robot, or a Cartesian robot may be used. Furthermore, in the embodiments described above, the case in which the object detection system 30 is applied to the robot system 10 (Figure 1) was used as an example, but the object detection system 30 can be widely applied to other technical fields and applications. [Explanation of Symbols]

[0083] 10…Robot system, 12…Robot, 14…Control device, 16…Arm unit, 18…Tool, 20…First camera, 22…Second camera (camera), 26…Processor, 28…Memory, 30…Object detection system, 38…Motor (rotation actuator), 50…Image processing unit, 52…Motion control unit, 54…Angle setting unit, 56…Drive control unit, 80…Image area, 82…Image sensor array, 84n,84r…Image sensor, 90,92…Feature line segment, 94n,94r…Sub-region, Img…Image data, W1…Cable (object, workpiece), W2…Electronic circuit board (object, workpiece)

Claims

1. A camera having an array of image sensors arranged in a matrix along a first and a second direction, and which captures an object in three-dimensional space and generates image data, A rotation actuator that changes the roll angle of the camera by rotating the camera around its optical axis, An image processing unit performs image processing on the image data generated by the camera and extracts characteristic line segments of the object from within the image region indicated by the image data. A drive control unit controls the rotation of the rotation actuator so that the intersection angle between the feature line segment extracted by the image processing unit and the first or second direction increases over the period before and after the rotation of the camera, An object detection system characterized by comprising the following features.

2. The roll angle includes a first angle and a second angle, The absolute value of the angle difference between the first angle and the second angle is greater than 0° and tangent －１ The object detection system according to claim 1, characterized in that the temperature is (1 / 2)° or less.

3. The absolute value is tan －１ The object detection system according to claim 2, characterized in that the angle is (1 / 10)°.

4. The object detection system according to claim 2, characterized in that the first angle is 0°, -45°, or 45°.

5. The object detection system according to claim 1, characterized in that the drive control unit controls the drive of the rotating actuator while tracking the object is performed based on the image data sequentially generated by the camera.

6. A robot that processes a workpiece in three-dimensional space, A camera having an array of image sensors arranged in a matrix along a first direction and a second direction, and which captures the workpiece and generates image data, A rotation actuator that changes the roll angle of the camera by rotating the camera around its optical axis, The camera is connected to a control device that controls the movement of the robot, Equipped with, The control device is Image processing performed on the image data generated by the camera, extracting characteristic line segments of the workpiece from the image region indicated by the image data, A control process is performed to control the drive of the rotation actuator so that the intersection angle between the feature line segment extracted by the image processing and the first or second direction increases, both before and after the rotation of the camera. A robotic system characterized by performing the following actions.

7. The workpiece includes a cable and an electronic circuit board. The camera is attached to the robot and to a tool that grips the cable. The robot system according to claim 6, characterized in that the characteristic line segment is a linear component of the contour line of the terminal of the cable or the receptacle of the electronic circuit board.

Images