Three-dimensional measurement method for linear object

Abstract

To enable high-speed matching processing even in a case where there are a plurality of linear objects of the same color, and efficient determination on whether or not points of interest on two images correspond one-to-one, in a three-dimensional measurement method for linear objects using a stereoscopic camera.SOLUTION: A three-dimensional measurement method for linear objects has: an imaging step of imaging a plurality of linear objects 20 by a stereoscopic camera 10 having a first camera 11 and a second camera 12 to acquire a first image L and a second image R; and a determination step of selecting one of the linear objects as a target linear object on the first image and selecting an end point of the target linear object as a target end point, and determining that a re-imaging step is required to re-image the linear objects when an end point of another linear object exists on an epipolar line corresponding to the target end point on the first image, or when two or more end points of the linear objects exist on the epipolar line corresponding to the target end point on the second image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a three-dimensional measurement method for measuring the shape of a linear object such as a cable using a stereo system. [Background technology]

[0002] Stereo 3D measurement methods have traditionally been used to measure 3D positions using the parallax between two cameras. This method calculates the 3D position of a point of interest to be measured using the principles of triangulation from the relative positions of the two cameras and the point of interest captured in two images taken from different viewpoints. In this stereo method, the matching process of finding corresponding points on the two images places the heaviest burden on information processing and is the most costly process. For this reason, various methods have been proposed with the aim of improving the matching process.

[0003] Patent document 1 describes a method for achieving high-speed matching processing in three-dimensional measurement of linear objects using a stereo camera, in which the linear object is first extracted from the image using color differences, and then a matching process is performed by matching a point of interest on a first line image in the first image with the intersection of a second line image and an epipolar line in the second image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2019 / 017360 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the method described in Patent Document 1 requires a stereo camera capable of acquiring color images, and when there are multiple linear objects of the same color, the probability of not being able to uniquely determine corresponding points on the two images increases, so the effect of speeding up the matching process is limited.

[0006] The present invention has been made in consideration of the above, and aims to provide a 3D measurement method for linear objects that can perform high-speed matching processing even when a stereo camera captures monochrome images or when there are multiple linear objects of the same color. Another aim of the present invention is to provide a method for efficiently determining whether or not matching of points of interest on two images is possible, since it is necessary to capture images again if corresponding points cannot be uniquely determined. [Means for solving the problem]

[0007] The three-dimensional measurement method for linear objects of the present invention includes an imaging step of imaging a plurality of linear objects with a stereo camera having a first camera and a second camera to obtain a first image and a second image, and a determination step of selecting one of the linear objects on the first image as a linear object of interest and an endpoint of the linear object of interest as an endpoint of interest, and determining that a re-imaging step is necessary to image the linear objects again if an endpoint of another linear object exists on an epipolar line corresponding to the endpoint of interest on the first image, or if an endpoint of two or more linear objects exists on an epipolar line corresponding to the endpoint of interest on the second image.

[0008] This method allows for easy selection of endpoints of linear objects based solely on their shape in the first image by using the endpoints as focus points. Then, by determining the point of contact between the tip of the linear object and the epipolar line in the second image and setting that point as the corresponding point of the focus endpoint, high-speed matching processing becomes possible. Furthermore, by checking the number of endpoints on the epipolar line, it is easy to determine whether reimaging is necessary.

[0009] Preferably, the plurality of linear objects to be measured in the method for three-dimensionally measuring a linear object are of the same color. The method of the present invention is particularly effective when such linear objects are the target.

[0010] Preferably, in any of the above-described 3D measurement methods for a linear object, if it is determined in the determination step that the re-imaging step is necessary, the re-imaging step is performed after performing a rotation step of rotating the stereo cameras so that the baseline rotates within a plane perpendicular to a plane including the optical axis of the first camera and the optical axis of the second camera. Note that the plane including both optical axes always includes the baseline. With this method, even if the stereo cameras are installed at predetermined positions, the positional relationship between endpoints that were originally on the same epipolar line can be changed in the re-imaging image.

[0011] Preferably, in any of the above-described methods for 3D measurement of a linear object, the rotating step is a step of rotating the stereo cameras so that the baselines of the stereo cameras rotate around a point on the baseline. Alternatively, preferably, in any of the above-described methods for 3D measurement of a linear object, the rotating step is a step of rotating the stereo cameras so that the baselines of the stereo cameras rotate in an arc around a point not on the baseline. These rotation methods allow the viewpoint to be moved and re-imaged without significantly moving the field of view of the stereo cameras.

[0012] Preferably, in any of the above-mentioned three-dimensional measurement methods for linear objects, if it is determined in the judgment step that the re-imaging step is unnecessary, the method further includes a step of calculating the three-dimensional coordinates of the target endpoint and calculating the three-dimensional shape of the target linear object by sequentially calculating the three-dimensional coordinates of points on the target linear object starting from the target endpoint. [Effects of the Invention]

[0013] According to the 3D measurement method for linear objects of the present invention, by selecting an endpoint of the linear object as the point of interest whose position is to be measured, selection can be easily performed based only on the shape of the linear object on the first image. Then, by determining the point of contact between the tip of the linear object and the epipolar line on the second image and setting it as the corresponding point of the endpoint of interest, high-speed matching becomes possible. Furthermore, by checking the number of endpoints on the epipolar line, it is easy to determine whether the endpoint of interest can be matched on both images and whether re-imaging is required. [Brief explanation of the drawings]

[0014] [Figure 1] 1A and 1B are diagrams showing the arrangement of a linear object and a stereo camera when a method for three-dimensionally measuring a linear object according to an embodiment is carried out; FIG. 1B is a diagram showing two images; [Figure 2] 1 is a diagram illustrating an epipolar plane and an epipolar line, where A: the optical axes of the two cameras are parallel, and B: the optical axes of the two cameras intersect. [Figure 3] FIG. 1 is a process flow diagram of a method for three-dimensional measurement of a linear object according to an embodiment. [Figure 4] FIG. 10 is a diagram for explaining a matching process for a target endpoint. [Figure 5] 10A and 10B are diagrams showing an example in which the correspondence between the target endpoints in two images is not uniquely determined. [Figure 6] 10A and 10B are diagrams for explaining a case where the positions of the target endpoints are reversed between two images. [Figure 7] FIG. 10 is a diagram for explaining a method for calculating the shape of a linear object. [Figure 8] These are examples of how to rotate a stereo camera. A: When rotating around a point on the baseline, B: When rotating around a point not on the baseline. [Figure 9] These are examples of ineffective stereo camera rotation methods. A: Rotation around a point on the baseline, viewed from the V1 arrow in Figure 8A. B: Rotation around a point not on the baseline, viewed from the V2 arrow in Figure 8A. DETAILED DESCRIPTION OF THE INVENTION

[0015] An embodiment of the method for three-dimensional measurement of a linear object according to the present invention will be described with reference to the drawings. Note that, hereinafter, the method for three-dimensional measurement of a linear object may be simply referred to as the "measurement method."

[0016] Referring to FIG. 1, stereo camera 10 includes first camera 11 and second camera 12. Stereo camera 10 is placed at a predetermined position where tip (end point) 21 of linear object 20 is within its field of view. First camera 11 captures an image of linear object 20 to generate a first image L. Second camera 12 captures an image of linear object 20 to generate a second image R. In the following description, the camera on the left side when viewing the stereo camera from behind is referred to as the first camera, and the first camera is referred to as the "left camera," and the first image is referred to as the "left image." Similarly, the camera on the right side is referred to as the second camera, and the second camera is referred to as the "right camera," and the second image is referred to as the "right image."

[0017] The stereo camera 10 is connected to a control device 15, which is connected to a computing device 16. The stereo camera takes images in response to instructions from the control device and transmits the generated images to the control device. A programmable controller or the like can be used as the control device. The control device transfers the images received from the stereo camera to the computing device. The computing device performs required processing on the first image and the second image and performs various calculations such as calculating the three-dimensional shape of a linear object. A personal computer or the like can be used as the computing device.

[0018] There is no particular limitation on the type of linear object 20. Examples of linear objects include coated electric wires used in wiring of electrical appliances and wire harnesses, various wires, cables, optical fibers, and resin tubes used in medical devices such as catheters.

[0019] Referring to FIG. 2, the line segment connecting the camera centers C1 and C2 of the two cameras in the stereo camera is the baseline 13. The camera center is the center of the lens. For a point in the field of view of the stereo camera, the plane formed by that point and the camera centers C1 and C2 is called an epipolar plane, and the intersections between the epipolar plane and each image plane are epipolar lines. Therefore, when a point P1 is taken in space, an epipolar plane F1 containing that point P1 is defined, and epipolar lines EL1 and ER1 are defined on each of the left and right images L and R. If another point P2 is not on the epipolar plane F1, an epipolar plane F2 that includes point P2 and is separate from the epipolar plane F1 is defined, and epipolar lines EL2 and ER2 are defined on each of the left and right images L and R. Points P1 and P2 are always on corresponding epipolar lines on the left and right images. Although the actual images L and R are formed by flipping the scene upside down and left to right behind the center of the camera, in Fig. 2, the images L and R are drawn in front of the center of the camera for ease of explanation.

[0020] Figure 2A shows a case where the optical axes Z1 and Z2 of the left and right cameras are parallel. The epipolar lines EL1 and ER1 appear horizontally on the left and right images, and when the position of point P1 changes, the epipolar lines EL1 and ER1 move parallel on the images. Figure 2B shows a case where the optical axes Z1 and Z2 of the left and right cameras intersect. The epipolar lines EL1 and ER1 are tilted on the left and right images, and when the position of point P1 changes, the tilt of the epipolar lines EL1 and ER1 on the images changes. Even in this case, if the orientations of the camera optical axes Z1 and Z2 are known, the tilt of the epipolar lines on the images can be calculated. In the following explanation, it is assumed that the optical axes of the left and right cameras are parallel.

[0021] Next, the measurement method of this embodiment will be described based on the flow in Fig. 3. Here, the description will be given taking as an example a case where linear objects are measured in three dimensions and then grasped one by one by a robot hand based on the measurement results.

[0022] First, a linear object 20 is imaged by the stereo camera 10, including the tip (end point) 21 of the linear object, and a left image L and a right image R are generated.

[0023] Referring to FIG. 4, one of the endpoints 21 of the linear object on the left image L is selected as the target endpoint 21S. Specifically, for example, the left image L is binarized, noise such as isolated pixels is removed as necessary, the image of the linear object 20 is thinned, and one of the endpoints 21 is selected as the target endpoint 21S. Well-known methods can be used for the binarization process, noise removal process, and thinning process. The target endpoint 21S is an endpoint of the target linear object 20S.

[0024] A point corresponding to the target endpoint 21S on the left image L is searched for on the right image R. Since the corresponding point on the right image R is on the epipolar line ER corresponding to the target endpoint 21S, high-speed matching processing is possible by finding the point of contact between the tip of the linear object and the epipolar line on the right image and setting it as the corresponding point of the target endpoint.

[0025] A point corresponding to the target endpoint 21S on the left image L is searched for on the right image R, and if the corresponding point is uniquely determined on the right image, matching is successful. On the other hand, if the corresponding point is not uniquely determined on the right image, the position of the target endpoint 21S cannot be calculated, and the viewpoint of the stereo camera 10 must be changed and imaging must be repeated. Therefore, it is necessary to determine whether the corresponding point of the target endpoint 21S selected on the left image L can be uniquely determined on the right image R, and whether re-imaging is necessary.

[0026] The determination of whether re-imaging is necessary is made as follows. In the left image L, the target endpoint 21S is on the epipolar line EL corresponding to that target endpoint. In this case, in the right image R, the target endpoint 21S is also always on the epipolar line ER corresponding to that target endpoint. Therefore, if there is only one endpoint on each of the epipolar lines in the left and right images, the corresponding point is uniquely determined and matching is successful. In this case, re-imaging is not necessary.

[0027] On the other hand, as shown in Figure 5A, if there is an endpoint 21 of another linear object on the epipolar line EL corresponding to the endpoint 21S of interest in the left image L, the corresponding point in the right image cannot be uniquely determined. This is because if the two endpoints 21a and 21b are in the positional relationship shown in Figure 6, the left and right endpoints on the left and right images may be reversed. In this case, re-imaging is required.

[0028] Furthermore, even if there is only one endpoint on the epipolar line in the left image, if that endpoint on the epipolar line EL is an overlap of the endpoint of interest and another endpoint, as shown in Figure 5B, there will be two or more endpoints 21a and 21b on the epipolar line ER corresponding to the endpoint of interest 21S in the right image R, and the corresponding point on the right image cannot be uniquely determined. In this case, re-imaging is also required.

[0029] To summarize the above, if there are no other endpoints on the epipolar line corresponding to a target endpoint selected on the left image and only one endpoint exists on the epipolar line corresponding to the target endpoint on the right image, then the corresponding point of the target endpoint is uniquely determined on the right image, and it can be determined that re-imaging is not necessary.On the other hand, if there are other endpoints on the epipolar line corresponding to the target endpoint selected on the left image or two or more endpoints exist on the epipolar line corresponding to the target endpoint on the right image, then the corresponding point of the target endpoint is not uniquely determined on the right image, and it can be determined that re-imaging is necessary.

[0030] When it is determined that re-imaging is required for the selected target endpoint, if processing has not been completed for all linear objects for which re-imaging is not required, processing of the target endpoint is postponed, another endpoint is reselected as the target endpoint on the left image, and the steps after the imaging step are repeated.

[0031] If it is determined that reimaging of the selected target endpoint is unnecessary, the three-dimensional shape of the target linear object 20S is calculated. The position of the target endpoint in three-dimensional space is calculated, and the positions of points on the target linear object in three-dimensional space are sequentially calculated starting from the target endpoint, thereby calculating the three-dimensional shape of the target linear object. Specifically, for example, referring to FIG. 7, if a pixel adjacent to the target endpoint 21S on the image of the target linear object 20S in the left image L is selected as the target point 22S, the intersection of the epipolar line ER2 corresponding to the target point 22S and the image of the target linear object 20S in the right image R becomes a corresponding point. Based on this corresponding point, the position of the target point 22S in three-dimensional space is calculated. By sequentially selecting pixels to be used as the target point on the left image in a direction away from the target endpoint, and similarly finding corresponding points in the right image, the position of the target point is calculated. By repeating this process, the three-dimensional shape of the target linear object 20S can be calculated.

[0032] The target linear object 20S is grasped by the robot hand based on the calculated three-dimensional shape. If the processing has not been completed for all linear objects that do not require re-imaging, another endpoint on the left image is selected as the target endpoint, and the steps after the imaging step are repeated.

[0033] If the process is completed for all linear objects that do not require re-imaging as a result of the above, it is checked whether the process is completed for all linear objects, i.e., whether there are any unprocessed linear objects that are determined to require re-imaging. If the process is not completed for all linear objects, the viewpoint of the stereo camera 10 is changed and re-imaging is performed, and the above process is repeated.

[0034] As a method for changing the viewpoint of the stereo camera 10, if the stereo camera is movable, for example, if the stereo camera is attached to the tip of a robot arm, the stereo camera can be moved significantly and re-imaged. However, making the stereo camera movable requires a mechanism for this. Furthermore, to accurately measure a linear object, the linear object must be imaged at a certain magnification, so a control mechanism is also required to point the stereo camera at the tip of the linear object. For this reason, in actual manufacturing sites, the stereo camera is often fixed and installed at a predetermined position where it can capture a close-up of the tip of the linear object.

[0035] When the stereo camera 10 is installed at a predetermined position, the position of the stereo camera cannot be moved significantly in order to keep the tip of the linear object 20 from its field of view. In this embodiment, the stereo camera is rotated on the spot. At this time, the stereo camera is rotated so that the difference in distance from the center of the left and right cameras to the plane including the optical axis and baseline before rotation increases as the stereo camera rotates. Specifically, the stereo camera is rotated so that the baseline rotates within a plane perpendicular to the plane including the optical axis of the left camera and the optical axis of the right camera. As a result, the epipolar plane shifts due to the difference in distance from the baseline to the point of interest, and multiple endpoints that were on the same epipolar plane before rotation are separated into different epipolar lines in the re-captured image after rotation.

[0036] FIG. 8 shows an example of a method for rotating a stereo camera. FIG. 8A shows a method for rotating the stereo camera 10 around point X on the baseline 13. The stereo camera is rotated so that the baseline 13 rotates in a plane perpendicular to the plane including the optical axis Z1 of the left camera 11 and the optical axis Z2 of the right camera 12. Preferably, the stereo camera is rotated around the midpoint 14 of the baseline 13, thereby minimizing the movement of the field of view of the stereo camera. The rotation angle θ is preferably ±15 to 45 degrees. A larger rotation angle θ is preferable because it increases the deviation of the epipolar plane from the two endpoints. On the other hand, if the rotation angle θ is too large, the angle between a linear object on the image and the epipolar line becomes smaller, reducing the positional accuracy when determining the intersection of the two on the image.

[0037] FIG. 8B shows a method of rotating the stereo camera 10 around a point X that is not on the baseline. The stereo camera is rotated so that the baseline 13 rotates within a plane perpendicular to the plane containing the optical axis Z1 of the left camera 11 and the optical axis Z2 of the right camera 12. Preferably, the stereo camera is rotated around a point on the perpendicular bisector of the baseline 13, thereby minimizing the shift in the field of view of the stereo camera. The rotation angle θ is preferably ±15 to ±30 degrees. A larger rotation angle θ is preferable because it increases the deviation of the epipolar plane relative to the two endpoints due to the rotation. On the other hand, if the rotation angle θ is too large, the angle between a linear object on the image and the epipolar line becomes smaller, reducing the positional accuracy when determining the intersection of the two on the image. Furthermore, if the rotation angle is too large, the field of view of the stereo camera shifts too much. For the same reason, it is preferable that the radius of rotation be no more than twice the length of the baseline 13.

[0038] However, because actual stereo cameras have limitations in image resolution, if the deviation of the epipolar plane is too small, the epipolar lines on which the two endpoints lie cannot be distinguished in the image recaptured after rotation. Therefore, the inventors conducted a simulation to confirm the effect of rotation, taking into account the resolution of the stereo camera. The simulation was performed using a commercially available stereo camera (Kurabo Industries, Ltd., Classens C200) consisting of two cameras with a focal length of 12 mm connected with a baseline length of 150 mm, under the condition that a linear object with a diameter of 1 mm was imaged from a distance of approximately 500 mm.

[0039] In the case of Figure 5A, if the distance between the two endpoints in space was 3 mm or more, the two endpoints could be distinguished on the image recaptured after rotation by rotating the image by a rotation angle of ±20 degrees using either the rotation method in Figure 8A or Figure 8B. In the case of Figure 5B, if the distance between the two endpoints in space was 15 mm, the rotation method in Figure 8A could not detect the separation of the two epipolar lines even with a rotation angle of ±20 degrees, but the rotation method in Figure 8B could detect the separation of the two epipolar lines at rotation angles of ±20 degrees or more.

[0040] The same simulation results showed that the rotation methods shown in Figures 9A and 9B were ineffective. In Figure 9A, the stereo camera 10 is rotated around point X on the baseline 13, but the stereo camera is rotated so that the baseline 13 rotates within a plane including the optical axis Z1 of the left camera 11 and the optical axis Z2 of the right camera 12. In Figure 9B, the stereo camera is rotated so that the baseline 13 moves parallel to the side of the cylinder. Note that in Figure 9B, the right camera 12 is hidden behind the left camera 11. With either rotation method shown in Figures 9A and 9B, it is not expected that the epipolar plane on which the two endpoints lie will shift due to the rotation, and this was also confirmed by the simulation.

[0041] As a result of the above steps, if the processing for all linear objects is completed, the process ends.

[0042] The present invention is not limited to the above-described embodiment, and various modifications are possible within the scope of the technical concept thereof. [Explanation of symbols]

[0043] 10 Stereo Camera 11 Left camera (first camera) 12 Right camera (second camera) 13 Baseline 14 Midpoint of Baseline 15 Control device 16 Arithmetic unit 20 Linear objects 20S Linear object of interest 21 Endpoint 21S Target endpoint 22S Point of interest on a linear object C1, C2 camera center EL Epipolar line on the left image ER Epipolar line on the right image F1, F2 epipolar plane L Left image (first image) P1, P2 points in space R Right image (second image) X Stereo camera rotation center Optical axis of Z1 and Z2 cameras

Claims

1. an imaging step of capturing images of a plurality of linear objects using a stereo camera including a first camera and a second camera to obtain first and second images; a determination step of selecting one of the linear objects on the first image as a linear object of interest and an end point of the linear object of interest as an end point of interest, and determining that a re-imaging step of re-imaging the linear object is necessary when an end point of another linear object exists on an epipolar line corresponding to the end point of interest on the first image, or when an end point of two or more linear objects exists on an epipolar line corresponding to the end point of interest on the second image; A method for three-dimensional measurement of a linear object.

2. The plurality of linear objects are of the same color. The method for three-dimensional measurement of a linear object according to claim 1.

3. When it is determined in the determination step that the re-imaging step is necessary, a rotation step is performed in which the stereo camera is rotated so that a baseline rotates within a plane perpendicular to a plane including the optical axis of the first camera and the optical axis of the second camera, and then the re-imaging step is performed. The method for three-dimensional measurement of a linear object according to claim 1.

4. The rotating step is a step of rotating the stereo camera so that a baseline of the stereo camera rotates around a point on the baseline. The method for three-dimensional measurement of a linear object according to claim 3.

5. The rotating step is a step of rotating the stereo camera so that a baseline of the stereo camera rotates in an arc around a point that is not on the baseline. The method for three-dimensional measurement of a linear object according to claim 3.

6. The method further comprises a step of calculating the three-dimensional coordinates of the target end point when it is determined in the determination step that the re-imaging step is unnecessary, and calculating the three-dimensional shape of the target linear object by sequentially calculating the three-dimensional coordinates of points on the target linear object starting from the target end point. The method for three-dimensional measurement of a linear object according to claim 1.

Images