Object detection system and robot system

Abstract

To provide an object detection system capable of detecting at least the posture of an object in a simpler manner when detecting the object using two cameras, and a robot system.SOLUTION: An image analysis device (16) of an object detection system (10) is configured to: calculate a first angle (Am) indicative of a rotational position around a first optical axis (60a) of a first camera (14a) from first image data (Img1); calculate a second angle (Bm) indicative of a rotational position around a second optical axis (60b) of a second camera (14b) from second image data (Img2); and convert the first angle (Am) and the second angle (Bm) into angles (φm, θm, ψm) around three axes of an object (12) in three-dimensional space.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to an object detection system that detects an object, and a robot system that includes a robot that processes a workpiece. [Background technology]

[0002] Conventionally, object detection systems that use multiple cameras to detect objects have been known. For example, Patent Document 1 discloses a method for recognizing the three-dimensional position and orientation of an object based on the stereo measurement principle (so-called triangulation method). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 3525896 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the above-mentioned triangulation method has the problem that it takes time to perform stereo matching to calculate the common points between the two cameras, and also has the problem that the position and orientation of an object cannot be measured unless the measurement point is in the common field of view of the cameras.

[0005] The present invention has been made in consideration of these problems, and its purpose is to provide an object detection system and a robot system that can detect at least the orientation of an object using a simpler method when detecting an object using two cameras. [Means for solving the problem]

[0006] The object detection system of the present invention comprises a first camera that photographs an object in three-dimensional space and outputs first image data; a second camera that is positioned differently from the first camera and photographs the object and outputs second image data; and an image analysis device to which the first camera and the second camera are connected, wherein the image analysis device performs image processing that determines, from the first image data, a first angle that indicates the rotational position of the first camera about a first optical axis, and determines, from the second image data, a second angle that indicates the rotational position of the second camera about a second optical axis; and a conversion process that converts the first angle and the second angle determined by the image processing into angles about the three axes of the object in the three-dimensional space.

[0007] Furthermore, in the conversion process, when a straight line extending in an angular direction around the three axes converted from the first reference angle and the second reference angle is defined as a reference line, and a straight line extending in an angular direction around the three axes converted from the first measurement angle and the second measurement angle is defined as a measurement line, the first angle is the amount obtained by subtracting the first reference angle from the first measurement angle, and the second angle is the amount obtained by subtracting the second reference angle from the second measurement angle, the image analysis device may further perform a difference process to determine the amount of inclination of the measurement line with respect to the reference line as the amount of change in angle around the three axes.

[0008] Furthermore, the reference line may be a straight line intersecting a plane that contains the first optical axis and corresponds to the first reference angle with a plane that contains the second optical axis and corresponds to the second reference angle, and the measurement line may be a straight line intersecting a plane that contains the first optical axis and corresponds to the first measurement angle with a plane that contains the second optical axis and corresponds to the second measurement angle.

[0009] The robot system of the present invention comprises a robot that performs processing on a workpiece in three-dimensional space, a first camera that photographs the workpiece and outputs first image data, a second camera that is positioned differently from the first camera and photographs the workpiece and outputs second image data, and a control device to which the first camera and the second camera are connected and that controls the drive of the robot, wherein the control device performs image processing that determines a first angle that indicates the rotational position of the first camera about a first optical axis from the first image data and a second angle that indicates the rotational position of the second camera about a second optical axis from the second image data, a conversion process that converts the first angle and the second angle determined by the image processing into angles about three axes of the object in the three-dimensional space, and a determination process that determines a control amount related to the drive control based on the angles about the three axes converted by the conversion process. [Effects of the Invention]

[0010] According to the present invention, when an object is detected using two cameras, at least the orientation of the object can be detected by a simpler method. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a diagram showing the overall configuration of an object detection system according to an embodiment of the present invention. [Figure 2] 2 is a block diagram illustrating functions related to an object detection unit shown in FIG. 1. FIG. [Figure 3] 3 is a flowchart showing an example of an object detection method performed by the object detection unit of FIGS. 1 and 2. [Figure 4] FIG. 4 is a diagram showing an example of a method for calculating a measured roll angle in step SP14 of FIG. 3. [Figure 5] FIG. 4 is a first diagram showing a method of converting an angle in step SP16 of FIG. [Figure 6] FIG. 4 is a second diagram showing the angle conversion method in step SP16 of FIG. [Figure 7]1 is a diagram showing the overall configuration of a robot system according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings. To facilitate understanding of the description, the same components in each drawing will be denoted by the same reference numerals as much as possible, and duplicate explanations will be omitted. Furthermore, the term "part" may be replaced with other terms such as unit, module, device, or element.

[0013] [Configuration of object detection system 10] <Overall structure> FIG. 1 is a diagram showing the overall configuration of an object detection system 10 according to one embodiment of the present invention. The object detection system 10 functions as a "multi-monocular camera system" that uses multiple cameras to detect an object 12 located in real space without using triangulation. Specifically, the object detection system 10 includes three cameras 14a, 14b, and 14c and an image analysis device 16. The number of cameras may be two, four, or more.

[0014] The cameras 14a to 14c are imaging devices that generate signals for each frame by capturing images in a space and output them as image data that indicate a time series of images. The cameras 14a to 14c are configured, for example, as monocular cameras such as visible light cameras, infrared cameras, and TOF (Time Of Flight) cameras, or as a combination of cameras of the same type or different types. The cameras 14a to 14c are positioned so that the object 12 falls within the angle of view.

[0015] The image analysis device 16 performs an analysis process on the image data output from the cameras 14a to 14c to detect the state including the position and orientation of the object 12. The image analysis device 16 is configured to include a processor 18 and a memory 20.

[0016] The processor 18 may be a general-purpose processor including a CPU (Central Processing Unit), or may be a dedicated processor including an FPGA (Field Programmable Gate Array) or a GPU (Graphics Processing Unit). The processor 18 functions as an object detection unit 22 by reading and executing programs and data stored in the memory 20.

[0017] The memory 20 is a non-transitory computer-readable recording medium (or storage medium). Here, the computer-readable recording medium is a storage device such as a hard disk drive (HDD) or a solid state drive (SSD), or a portable medium such as a magneto-optical disk, a read-only memory (ROM), a compact disc (CD)-ROM, or a flash memory.

[0018] 1, the plane forming the placement surface of object 12 is defined as the XY plane, and the normal direction of the placement surface is defined as the Z axis. In this case, the three-dimensional space is expressed by a three-dimensional Cartesian coordinate system with X, Y, and Z as the three axes. The XYZ coordinate system may be a "real coordinate system" that indicates the real space, or may be a "system coordinate system" (or world coordinate system) defined within object detection system 10.

[0019] <Function block diagram> Fig. 2 is a block diagram illustrating functions related to the object detection unit 22 shown in Fig. 1. Specifically, the object detection unit 22 includes an image acquisition unit 30, an image processing unit 32, an angle conversion unit 34, a subtraction unit 36, and a position estimation unit 38. Furthermore, reference data 40 and layout information 42 are stored in the memory 20 (Fig. 1).

[0020] The reference data 40 is information indicating the reference position and orientation of the object 12, and includes an image set 44 and an angle set 46.

[0021] The image set 44 is a collection of reference images. The reference images correspond to images obtained by photographing the object 12 from various directions under a condition where the cameras 14a to 14c and the object 12 are in an ideal positional relationship. Hereinafter, image data representing the reference images will also be referred to as reference image data ImgRef.

[0022] Angle set 46 is a collection of angles indicating the position of the reference roll angle. The reference roll angle corresponds to the angle around the optical axis of cameras 14a to 14c (i.e., roll angle) when the above-mentioned reference image is captured.

[0023] The layout information 42 includes information indicating the arrangement of the cameras 14a to 14c. Examples of the layout information 42 include identification information of the cameras 14a to 14c, coordinate values indicating the positions of the cameras 14a to 14c, or direction vectors indicating the orientations of the cameras 14a to 14c.

[0024] The image acquisition unit 30 acquires image data output from the multiple cameras 14a to 14c and selects two pieces of image data from the multiple pieces of image data. For example, if image data from two cameras 14a and 14b are selected, the camera 14a corresponds to the "first camera" and the camera 14b corresponds to the "second camera." The image data from the cameras 14a and 14b are referred to as the first image data Img1 and the second image data Img2, respectively.

[0025] The image processing unit 32 performs image processing to determine the rotational position around the optical axis of the camera 14a (hereinafter referred to as the first optical axis 60a) from the first image data Img1 acquired by the image acquisition unit 30. This image processing includes, for example, template matching to compare the first image data Img1 with reference image data ImgRef corresponding to the camera 14a.

[0026] As with the first image data Img1, the image processing unit 32 performs image processing to determine the rotational position around the optical axis of the camera 14b (hereinafter, referred to as the second optical axis 60b) from the second image data Img2 acquired by the image acquisition unit 30. This image processing includes, for example, template matching to compare the second image data Img2 with reference image data ImgRef corresponding to the camera 14b.

[0027] Hereinafter, the rotational position around the first optical axis 60a will be defined as "roll angle A," and the rotational position around the second optical axis 60b will be defined as "roll angle B." The reference roll angles corresponding to the cameras 14a and 14b may be referred to as "Ar, Br" (corresponding to "first and second reference angles"), respectively. The roll angle calculated from the first image data Img1 may be referred to as "measured roll angle Am" (corresponding to "first measurement angle"), and the roll angle calculated from the second image data Img2 may be referred to as "measured roll angle Bm" (corresponding to "second measurement angle"). Furthermore, the combination of roll angles A and B may be collectively referred to as a pair of roll angles (A, B).

[0028] The angle conversion unit 34 converts the pair of roll angles (A, B) into triaxial angles (φ, θ, ψ), which are angles around the three axes of the object 12 on the XYZ axes, using the layout information 42. As a result, the pair of reference roll angles (Ar, Br) are converted into reference triaxial angles (φr, θr, ψr), and the pair of measurement roll angles (Am, Bm) are converted into measurement triaxial angles (φm, θm, ψm).

[0029] The angle conversion described above is performed through geometric calculations. Specifically, the angle conversion unit 34 obtains a linear equation for a line of intersection 70 (FIG. 5), which will be described later, and can obtain the triaxial angles from the obtained linear equation. This line of intersection 70 is a straight line where a plane that includes the first optical axis 60a and corresponds to roll angle A intersects with a plane that includes the second optical axis 60b and corresponds to roll angle B, or a straight line parallel to this line. Here, the line of intersection 70 corresponding to a pair of reference roll angles (Ar, Br) corresponds to a reference line 72 in FIG. 6, which will be described later. Furthermore, the line of intersection 70 corresponding to a pair of measured roll angles (Am, Bm) corresponds to a measurement line 74 in FIG. 6.

[0030] The subtraction unit 36 performs subtraction processing to subtract the reference three-axis angles (φr, θr, ψr) from the measured three-axis angles (φm, θm, ψm), thereby determining the change amounts (Δφ, Δθ, Δψ) in the three-axis angles.

[0031] [Operation of object detection system 10] The object detection system 10 in this embodiment is configured as described above. Next, the operation of this object detection system 10, particularly the operation of image analysis device 16, will be described with reference to Figures 3 to 6. Figure 3 is a flowchart showing an example of an object detection method performed by object detection unit 22 of Figures 1 and 2.

[0032] (Step SP10: Acquisition process) In step SP10 of FIG. 3, the image acquisition unit 30 acquires a plurality of image data output from the plurality of cameras 14a to 14c.

[0033] (Step SP12: Selection process) In step SP12, the image acquisition unit 30 selects two image data from the plurality of image data acquired in step S10, and outputs the selected first image data Img1 and second image data Img2.

[0034] (Step SP14: Image processing) In step SP14, the image processing unit 32 performs image processing on the first image data Img1 selected in step SP12 to calculate the angle around the first optical axis 60a (i.e., the measured roll angle Am). The image processing unit 32 also performs image processing on the second image data Img2 selected in step SP12 to calculate the angle around the second optical axis 60b (i.e., the measured roll angle Bm).

[0035] 4 is a diagram showing an example of a method for calculating the measurement roll angles Am and Bm in step SP14 of FIG. 3. The upper side of FIG. 4 corresponds to a rectangular image area 50r indicated by the reference image data ImgRef. The horizontal direction of the image area 50r indicates the U axis, and the vertical direction of the image area 50r indicates the V axis. An object area 52 viewed from the side of the object 12 exists approximately in the center of the image area 50r. An image reference point 54 is set at a position approximately in the center of the object area 52. Furthermore, an image reference line 56 is set within the object area 52, passing through the image reference point 54 and parallel to the V axis. The rotational position of this image reference line 56 corresponds to the reference roll angle Ar.

[0036] In contrast, the lower side of FIG. 4 corresponds to a rectangular image region 50m indicated by the first image data Img1. Similar to the image region 50r described above, an object region 52 exists approximately in the center of the image region 50m. However, the image reference point 55 is located at a different position from the image reference point 54, and the image reference line 57 extends in a direction tilted relative to the image reference line 56. The rotational position of this image reference line 57 corresponds to the measurement roll angle Am. Note that the reference roll angle Br and the measurement roll angle Bm can each be determined by performing pattern matching similar to that described above on the second image data Img2.

[0037] (Step SP16: Conversion process) 3, the angle conversion unit 34 converts the pair of roll angles (A, B) calculated in step SP14 into triaxial angles (φ, θ, ψ). As a result, the pair of reference roll angles (Ar, Br) are converted into reference triaxial angles (φr, θr, ψr). Also, the pair of measured roll angles (Am, Bm) are converted into measured triaxial angles (φm, θm, ψm).

[0038] 5 is a first diagram showing a method for converting angles in step SP16 of FIG. 3. The state of position and orientation in three-dimensional space is uniquely determined by, for example, six state quantities. These state quantities are, for example, a combination of [1] coordinate value (X) on the X axis, [2] coordinate value (Y) on the Y axis, [3] coordinate value (Z) on the Z axis, [4] angle around the X axis (φ), [5] angle around the Y axis (θ), and [6] angle around the Z axis (ψ).

[0039] The first optical axis 60a corresponds to the optical axis of the camera 14a, and the second optical axis 60b corresponds to the optical axis of the camera 14b. The plane 62a is located at a position rotated by a roll angle A around the first optical axis 60a. The plane 62b is located at a position rotated by a roll angle B around the second optical axis 60b. The intersection line 70 is a straight line where the two planes 62a and 62b intersect. From such geometric considerations, a pair of roll angles (A, B) can be uniquely associated with the triaxial angle (φ, θ, ψ) that indicates the orientation of the intersection line 70.

[0040] Fig. 6 is a second diagram showing the angle conversion method in step SP16 of Fig. 3. A reference line 72 is the intersection line 70 (Fig. 5) corresponding to a pair of reference roll angles (Ar, Br). A measurement line 74 is the intersection line 70 (Fig. 5) corresponding to a pair of measurement roll angles (Am, Bm).

[0041] (Step SP18: Subtraction process) 3, the subtraction unit 36 subtracts the reference three-axis angles (φr, θr, ψr) from the measured three-axis angles (φm, θm, ψm) converted in step SP16, thereby determining the change in the three-axis angles (Δφ, Δθ, Δψ).

[0042] In this way, the object detection unit 22 completes the detection operation shown in the flowchart of Fig. 3. As a result, the orientation of the object 12 in the XYZ coordinate space is calculated.

[0043] <Application example of object detection system 10> 7 is a diagram showing the overall configuration of a robot system 100 according to an embodiment of the present invention. Specifically, the robot system 100 includes a robot 102 and a control device 104.

[0044] The robot 102 is a vertically articulated robot having multiple joint axes. The robot 102 can perform various tasks, including gripping and moving a workpiece, welding, and painting, by independently driving the multiple joint axes in response to commands from a control device 104. In the example of FIG. 8, the robot 102 performs a connection task of inserting the tip end (e.g., a plug) of a cable 106 into a receptacle 108a on an electronic circuit board 108. In this case, the cable 106 or the electronic circuit board 108 corresponds to the workpiece to be processed.

[0045] An end effector 114 (or a tool or manipulator) is attached to the tip of an arm 112 of the robot 102. This allows the robot 102 to grip the tip of the cable 106 and move the cable 106 to a desired position and orientation. In addition, a pair of cameras 14a, 14b are attached to the tip of the arm 112. Each of the pair of cameras 14a, 14b is connected to a control device 104.

[0046] The control device 104 is a computer that controls the operation of the robot 102. The control device 104 acquires first image data Img1 and second image data Img2 (see FIG. 2) from the cameras 14a and 14b, respectively, and performs visual feedback (VFB) control on the robot 102. Specifically, the control device 104 includes a processor 120 and a memory 122.

[0047] The processor 120 may be a general-purpose processor including a CPU, similar to the processor 18 in Fig. 1, or may be a dedicated processor including an FPGA or GPU. The memory 122 is a non-transitory computer-readable recording medium, similar to the memory 20 in Fig. 1. The processor 120 functions as the object detection unit 22 and the drive control unit 124 by reading and executing the programs and data stored in the memory 122.

[0048] 2, and detects the positions and orientations of the cable 106 and the electronic circuit board 108. The configuration and operation of the object detection unit 22 are as described with reference to FIGS. 2 to 6.

[0049] The drive control unit 124 controls the drive of the robot 102 based on the detection result by the object detection unit 22. This drive control includes: [1] a determination process for determining a control amount based on the detection result by the object detection unit 22; and [2] a control process for controlling the drive of a servo motor (not shown) provided in the robot 102.

[0050] With this configuration, the robot 102 can perform the connection work of inserting the tip of the cable 106 into the receptacle 108a on the electronic circuit board 108 while sequentially correcting the position and posture of the end effector 114 through VFB control.

[0051] [Summary of the embodiment] As described above, according to the object detection system 10 and the robot system 100 in this embodiment, the image analysis device 16 or the control device 104 converts a pair of roll angles (Am, Bm) indicating the rotational position around the first optical axis 60a and the second optical axis 60b into three-axis angles (φm, θm, ψm) of the object 12 in the XYZ coordinate space. Therefore, when detecting the object 12 using two cameras 14a, 14b, at least the posture of the object 12 can be detected by a simpler method.

[0052] Furthermore, when a straight line extending in the direction of the triaxial angle (φr, θr, ψr) converted from a pair of reference roll angles (Ar, Br) is defined as the reference line 72, and a straight line extending in the direction of the triaxial angle (φm, θm, ψm) converted from a pair of measurement roll angles (Am, Bm) is defined as the measurement line 74, the image analysis device 16 or the control device 104 may calculate the amount of inclination of the measurement line 74 with respect to the reference line 72 as the amount of change (Δφ, Δθ, Δψ) in the triaxial angle. This makes it possible to easily calculate the amount of change (Δφ, Δθ, Δψ) through a geometric calculation method.

[0053] Furthermore, the reference line 72 is a straight line where a plane 62a that includes the first optical axis 60a and corresponds to the reference roll angle Ar intersects with a plane 62b that includes the second optical axis 60b and corresponds to the reference roll angle Br, and the measurement line 74 is a straight line where a plane 62a that includes the first optical axis 60a and corresponds to the measurement roll angle Am intersects with a plane 62b that includes the second optical axis 60b and corresponds to the measurement roll angle Bm. This makes it possible to easily determine the reference line 72 and the measurement line 74 through a geometric calculation method.

[0054] [Variations] The present invention is not limited to the above-described embodiment, and can be freely modified without departing from the spirit and scope of the present invention. Alternatively, the respective configurations may be arbitrarily combined within the scope of no technical contradiction. Alternatively, the execution or execution order of each step constituting the flowchart may be changed within the scope of no technical contradiction.

[0055] In the above embodiment, the position and orientation of the object 12 are detected using two cameras 14a and 14b, but the detection method is not limited to this. For example, two other cameras 14a and 14c (or cameras 14b and 14c) may be combined, and a composite value (e.g., a statistic including an average value) of the respective detection results may be obtained as the final detection result.

[0056] In the above embodiment, the object detection system 10 functions as a "multi-monocular camera" system, but the object detection system 10 can also be applied to a "stereo camera" system using triangulation. For example, the object detection system 10 normally selects the stereo camera mode to detect the object 12, but may switch to the multi-monocular camera mode to continue detecting the object 12 while a feature point within the angle of view of one of the cameras is temporarily lost.

[0057] In the above-described embodiment, a vertical articulated robot has been described as an example, but the type of industrial robot is not limited to this. For example, a horizontal articulated robot, a parallel link robot, or an orthogonal robot may be used. Furthermore, in the above-described embodiment, an example has been described in which the object detection system 10 is applied to the robot system 100 (FIG. 7), but the object detection system 10 can be widely applied to other technical fields and uses. [Explanation of symbols]

[0058] 10...object detection system, 12...object, 14a...camera (first camera), 14b...camera (second camera), 14c...camera, 16...image analysis device, 18,120...processor, 20,122...memory, 22...object detection unit, 30...image acquisition unit, 32...image processing unit, 34...angle conversion unit, 36...subtraction unit, 60a...first optical axis, 60b...second optical axis, 72...reference line, 74...measurement line, 100...robot system, 102...robot, 104...control device, 106...cable (work), 108...electronic circuit board (work), Img1...first image data, Img2...second image data

Claims

1. a first camera that captures an image of an object in a three-dimensional space and outputs first image data; a second camera that is disposed at a position different from the first camera and captures an image of the object and outputs second image data; an image analysis device to which the first camera and the second camera are connected, The image analysis device image processing to obtain a first angle indicating a rotation position of the first camera about a first optical axis from the first image data, and to obtain a second angle indicating a rotation position of the second camera about a second optical axis from the second image data; a conversion process for converting the first angle and the second angle obtained by the image processing into angles around three axes of the object in the three-dimensional space; An object detection system comprising:

2. In the conversion process, when a straight line extending in an angular direction around the three axes converted from the first reference angle and the second reference angle is defined as a reference line, and a straight line extending in an angular direction around the three axes converted from the first measurement angle and the second measurement angle is defined as a measurement line, the first angle is the first measurement angle minus the first reference angle, the second angle is the second measurement angle minus the second reference angle, The object detection system according to claim 1 , wherein the image analysis device further performs a differential process to determine an amount of tilt of the measurement line relative to the reference line as an amount of change in angle around the three axes.

3. the reference line is a straight line formed by an intersection of a plane that includes the first optical axis and corresponds to the first reference angle and a plane that includes the second optical axis and corresponds to the second reference angle, 3. The object detection system according to claim 2, wherein the measurement line is a straight line intersecting a plane that includes the first optical axis and corresponds to the first measurement angle and a plane that includes the second optical axis and corresponds to the second measurement angle.

4. A robot that processes workpieces in three-dimensional space, a first camera that photographs the workpiece and outputs first image data; A second camera that is arranged at a position different from the first camera and captures an image of the workpiece and outputs second image data; a control device to which the first camera and the second camera are connected and which controls the driving of the robot, The control device image processing to obtain a first angle indicating a rotation position of the first camera about a first optical axis from the first image data, and to obtain a second angle indicating a rotation position of the second camera about a second optical axis from the second image data; a conversion process for converting the first angle and the second angle obtained by the image processing into angles around three axes of the object in the three-dimensional space; a determination process for determining a control amount related to the drive control based on the angles around the three axes converted by the conversion process; A robot system characterized by executing the above.

Images