Positioning method and positioning system

Abstract

The invention is a positioning method for positioning a reference feature (10) with respect to a target feature (11) along a positioning plane within a workspace accessible by a robotic manipulator, wherein one of said reference and target features (10, 11) is on a first object attached to the robotic manipulator and the other one of said reference and target features (10, 11) is on a second object being arranged in the workspace and being detached from the robotic manipulator. First and second positioning steps are carried out and are repeated until at least one stop condition is achieved, wherein the at least one stop condition comprises the condition that the calculated respective physical distance is within a threshold limit. The invention also relates to a positioning system, a computer program product and a computer-readable medium for carrying out the method.

Description

[0001] POSITIONING METHOD AND POSITIONING SYSTEM

[0002] TECHNICAL FIELD

[0003] The subject matter disclosed herein relates to a positioning method, a positioning system, a computer program product and a computer readable medium for positioning a reference feature with respect to a target feature. More particularly, the disclosed subject matter relates to systems and methods for using visual servoing techniques in a unique way to achieve accurate positioning in case if exact calibrations, exact mutual positions / orientations, exact camera scale factors are not necessarily available. The subject matter disclosed herein is especially suitable for positioning a tool with respect to a workpiece to enable carrying out various operations in an accurately positioned way.

[0004] BACKGROUND ART

[0005] Automatic measurement and diagnosis of used printed circuit boards (PCBs) is a challenging task due to the need for precise fixturing of the boards in traditional measuring tools, as well as the PCB schematics, including the geometry, for robot programming. Hence, PCB repair shops usually employ human operators to find and identify the root cause of the malfunctions of the broken PCBs. In general, these shops face a large variety of products in small batch sizes but with many, frequently recurring product types. Even though measuring operations often contain repetitive steps - seemingly good candidates for automation - automated solutions cannot yet provide a flexibility for repair shops to be worth investing into.

[0006] There are known methods for automatic quality assurance and electrical testing of PCBs as a part of manufacturing processes. These electrical tests ensure that the bare or populated PCBs leaving the factory are tested thoroughly regarding their electrical parameters (voltages, currents, etc.). There are two main testing approaches: the in-circuit tester systems with bed-of-nails fixtures and the flying probe tests. The bed-of-nails approach is the fastest testing method due to its massive parallelization with the test-pins in the bed connecting simultaneously to the PCB. On the other hand, this method is the least flexible as it requires a pre- manufactured, accurate fixture, which can only be used for a single type of PCB. Accordingly, this method in general is only applicable in mass production of PCBs.

[0007] There is a need for eliminating the constraints of PCB-specific fixtures, and correspondingly, to flexibly enable tests without fixturing , especially with the flexible flying probe testers. However, prior art flying probe tests require accurately set up measurement point positions. These tests can be rapidly prepared from a digital geometric representation, such as a computer-aided design (CAD) model, due to tight manufacturing tolerances of PCBs. Therefore, flying probe tests are typically used for manufactured PCBs and not for repairing assembled PCB products, where geometric information may be missing. Even if the measurement point positions are defined offline, i.e. , with coordinates and not through manual teaching, the points might not be accessible from the usual vertical approach direction because of obstacles (cables, heatsinks, etc.) in the assembly.

[0008] Consequently, the diagnosis of used PCBs calls for even more flexible measurement methods. Beside managing a large product range, handling the variation between different products of the same type is also challenging. These variations commonly occur when the production is maintained over a longer time period, or when there are multiple manufacturers for some PCB components. Although connection point positions seldom change, having varying visual appearance and dimensions in minor features like heatsinks and mounted components is rather typical. Therefore, automated measurement point detection, together with automated program generation or adaptation would be highly beneficial, which calls for a flexible, robotized solution.

[0009] In classical robotic applications, such problems may be tackled using an online teaching method, where the tool center point (TCP) of the robot pose is set by an operator at a setup phase and the taught points are interpolated during the measuring cycle. This method can be performed with the precision determined by the repeatability of the robot, which is usually in the order of 0.1 -0.01 mm for industrial robot arms. However, taught points can only be used successfully if the robot repeatability, fixturing precision of the workpieces, and the manufacturing and assembly precision of the workpieces are such that the accumulating errors still allow the precision required for the desired subsequent action. Otherwise, the accumulating errors also need to be taken into account during the measurement.

[0010] Visual servoing, also known as vision-based robot control, is a known technique, which uses feedback information extracted from a vision sensor, i.e. visual feedback to control the motion of a robot. There are known solutions, which use visual servoing for probe positioning or for other types of PCB related positionings.

[0011] US 6,208,375 B1 discloses a test probe positioning method and system. A target feature is selected either by manual means such as a computer mouse, or by automatic means based on previously-stored information. The pixel address of the selected target feature within a displayed image is determined. A test probe equipped with probe-identifying markers is introduced into the field of view of the camera, and its pixel address is determined on the camera image. With target and probe addresses known, the probe is moved to bring it into contact with the target feature. Given that the camera provides real time images of the probe position, a closed loop system can be employed to target a feature to be probed and then to servo the probe to that feature. This prior art solution is based on point-to-point servoing, which has drawbacks as detailed later.

[0012] US 2015 / 0120047 A1 discloses a control device, which includes a reception unit that receives first operation information and second operation information different from the first operation information; and a process unit that instructs a robot to execute operations based on the first operation information and the second operation information using a plurality of captured images of an imaged target object, the images being captured multiple times while the robot moves from a first posture to a second posture different from the first posture. This prior art solution is also based on point-to-point servoing.

[0013] In point-to-point servoing it is typically required that the x-y plane of the imaging is in parallel with the manipulation plane. In this layout, it is possible to overlap a selected feature point in the image with a target feature point in the image. This approach is most effective in cases where the tool orientation is fixed, and the motion is 2-2.5 axis linear motion. Due to the constraints of the motion, this known solution is less flexible, cannot avoid obstacles, and is typically suitable for planar workpieces.

[0014] Point-to-point servoing with a two-camera arrangement also aims to bring a “tip” of the tool fixed to the manipulator into overlap with the target feature, and to bring the tip into contact with the target feature for the desired state before the action. In this case, the marker can be the tip of the tool, which is a point on the camera images, representing the straight-line projection of the lines determined by the focal point of the two cameras and the vertex corresponding to the tip (which is the same point for both camera images). These two lines intersect at the tip, and if the fine positioning is successful, the intersection will coincide with the target feature, establishing contact between the tip and the target feature. Fine positioning is not feasible for deviating lines. Alternatively, if an offset distance is desired between the tip of the tool and the target feature (instead of physical contact), then one marker point per camera can be set up further away on the tool instead of using the tip of the tool as a marker. In this case, calibration is required to ensure that the lines determined by these marker points intersect. As long as rigid body relations between the cameras and markers are maintained, marker points may only exist on the images as virtual marker points by drawing (marking) them on the camera images. In this case, their detection and localization are not necessary. For each different offset distance (i.e. the distance between the tool tip and the target feature perpendicular to the manipulation plane), the virtual marker point setup and calibration must be repeated. For this reason, the flexibility of the method is highly limited. This can be particularly problematic if the offset distance is not accurately achieved by moving the robot TCP to a preliminary, initial position. In this case, the intersection of the two virtual lines cannot be moved into the target feature by a planar (e.g. X-Y) movement, it is necessary to move in the offset direction (e.g. Z) in the fine positioning iteration, otherwise the overlapping of the markers and the target feature cannot be realized in both camera images at the same time, i.e. the system becomes overdetermined. The TCP frame directions for the offset movement can be calculated from a model or determined by measurement. The movement in this direction must be performed iteratively with the planar movement until the target feature point and the intersection point of the two virtual axes coincide.

[0015] Point-to-point servoing with a two-camera arrangement is inflexible also because the marker placement is constrained (the lines must intersect) and the fine positioning must either take place in a specific plane (corresponding to a fixed geometric distance) so that the intersection of the spatial lines coincides with the target feature, or along more than two axes, where it is not possible to separate the compensation vectors on the camera images to correspond to a single motion axis, due to the more than two axes of the movement. Furthermore, point-to-point overlapping can be more challenging to process, light conditions may vary, and it may not be possible to determine the reference feature, e.g. the tip of the measuring pin in the image due to light and visibility conditions e.g. due to glint of metallic parts, occlusions, or shadows. Thus, point-to-point servoing is sensitive to external light sources, and to visual or even geometric mismatches between subsequently processed workpieces.

[0016] The known positioning solutions do not provide a sufficiently flexible positioning. Thus, there is a need for a solution allowing an improvement over existing methods and systems. There is a need for a positioning method, a positioning system, a computer program product and a computer readable medium eliminating as much as possible the shortcomings of known techniques.

[0017] There is a need to provide such an arrangement and image processing algorithms that make the positioning insensitive to external light sources, visual or even geometric mismatches between subsequently processed workpieces. This kind of flexibility would allow the positioning to be applied not only to large workpiece volumes of mass productions, but also to smaller volumes, e.g. in small repair shops. Additional flexibility is desirable for enabling the transfer of defined target features between different robot cells, which may differ in manufacturing and assembly tolerance instances, so that a target feature does not need to be defined in each cell. In other words, there is a need for a solution allowing accurate positioning even in the case of uncalibrated surroundings, without knowing exact reference distances and / or scale factors of imaging cameras. There is a special need for a PCB testing solution that is suitable for, or can be effectively applied for robustly checking a variety of features on used PCBs arriving in small batches, especially in case the precise geometry of the product is not available.

[0018] It has been recognized that flexibility should also be preferably ensured in defining the target feature, so there is a need for a solution allowing model-based or model- free options. Model-free definition provides great flexibility, as visual and approximate geometric information can be used to store the target feature, e.g. capturing an image of the target feature and storing the robot TCP position when the robot is positioned close to the expected position of the action when defining the target feature.

[0019] DISCLOSURE OF INVENTION

[0020] It is an object of the invention to provide an efficient and flexible positioning solution, which can be easily implemented, necessitates low computational power, and which is tolerant to uncalibrated situations in which only estimates or assumptions are available on geometric relations, such as distances, angles and / or camera scale factors.

[0021] The above and other objects have been achieved by the positioning method according to claim 1 , by the positioning system according to claim 12, by the computer program product according to claim 13 and by the computer-readable medium according to claim 14. Preferred embodiments are defined in the dependent claims.

[0022] The invention is not limited to the measurement and testing of printed circuit boards using a robot or robotic arm equipped with a measurement tool. Other applications of the invention are also possible, therefore generic tasks / actions performed on generic workpieces, products or objects using generic manipulator(s) equipped with generic tool(s) or object(s) are within the scope of the present disclosure.

[0023] According to a preferred embodiment, a novel, robotic, visual servoing probe test method and corresponding measurement tool is proposed. An experimental PCB measurement cell is provided to automatically measure relevant measurement points of a PCB motherboard without precisely locating the motherboard assembly in the robot workspace. The preferred embodiment is suitable for repair shops to test PCBs using simple locating pins and basic test specifications, without the need for accurate geometric information about the product.

[0024] Assuming there are deviations caused by the inaccurate workpiece location, manufacturing and assembly tolerances of the PCB product, as well as the robot positioning repeatability, the positioning error can be in the range of a couple of millimeters. To be able to handle the position error of the measured features, a special visual servo technique is provided. The idea in this preferred embodiment is to observe the test-pin (connected to one of the terminals of the measuring instrument), and the measured feature simultaneously using a 2D camera. By identifying the axis of the test-pin together with the measured feature, a positioning error can be calculated in the image space, which determines the direction and an estimated magnitude of the required compensation in the physical space. To realize the compensation, motion commands are sent to the robot to reduce the error on the camera images. As planar compensation is required in a plane parallel to the top plane of the PCB, bi-directional observation is necessary. Therefore, a second camera is also introduced, which detects the position error perpendicular to the first one.

[0025] In this embodiment, the test-pin mounted on the robot converges to the measured feature through the servoing robot motion, wherein the planar motion is performed on a plane above the PCB (safety plane) using a board-specific safety distance to avoid collision with any mounted components. After the error is reduced below a predefined threshold in both directions, the test-pin’s axis becomes aligned with the measurement point and using a downward feeding motion along this axis, the electric measurement can be carried out by pushing the test-pin against the measured feature, which establishes the galvanic contact.

[0026] BRIEF DESCRIPTION OF DRAWINGS

[0027] Characteristics, objectives and advantages of embodiments of the subject matter will become apparent from the following description, which is given solely by way of illustration and is non-limiting, and is to be read with reference to the appended drawings in which

[0028] Fig. 1 schematically illustrates a part of a preferred embodiment of an inventive positioning system,

[0029] Fig. 2 schematically illustrates an enlarged portion of Fig. 1 ,

[0030] Fig. 3 schematically illustrates a view field of a first imaging unit of the system of Fig. 1 ,

[0031] Fig. 4 schematically illustrates a view field of a second imaging unit of the system of Fig. 1 ,

[0032] Figs. 5A and 5B schematically illustrate respective view fields of the first imaging unit in two phases of positioning in a first direction,

[0033] Figs. 6Aand 6B schematically illustrate respective view fields of the second imaging unit in two phases of positioning in a second direction,

[0034] Fig. 7 schematically illustrates a block diagram of visual servoing in a preferred embodiment of the invention,

[0035] Fig. 8 schematically illustrates a constellation-based matching usable for identifying a target feature,

[0036] Fig. 9 schematically illustrates a template matching usable for identifying a target feature,

[0037] Fig. 10 schematically illustrates a practical application of a preferred embodiment with a workpiece which is not aligned with respect to locating pins, and

[0038] Fig. 11 schematically illustrates an enlarged part of Fig. 10 showing the workpiece in a fixture-aligned position. MODES FOR CARRYING OUT THE INVENTION

[0039] Although some preferred embodiments are discussed below with respect to measurement and testing of printed circuit boards using a robot or robotic arm equipped with a measurement tool, the invention is not limited to this field. Other applications of the invention are also possible, therefore generic tasks / actions performed on generic workpieces, products or objects using generic manipulator(s) equipped with generic tool(s) or object(s) are within the scope of the present disclosure.

[0040] The description below mainly refers to an embodiment, where the tool is mounted on the robot’s flange and the workpiece is placed in the robot's workspace, in a robotic measurement cell. The aim is to adjust with high precision a reference feature (robot object feature) of a tool / workpiece / object mounted on a manipulator / robot arm to a defined geometric relationship with respect to a target feature (static or even moving object feature) of a workpiece / tool / object in a workspace of a manipulator / robot. In other words, the aim is to fit a reference feature and a target feature into a known relative geometric position with respect to each- other, in order to perform a certain subsequent action.

[0041] Fig. 1 schematically illustrates a part of a preferred embodiment of an inventive positioning system, while Fig. 2 schematically illustrates an enlarged portion of Fig. 1 . The subject matter disclosed herein is a positioning method and a positioning system for positioning a reference feature 10, such as a tip of a test pin as depicted in Fig. 2, with respect to a target feature 11 , such as contact point on a PCB as depicted in Fig. 2, along a positioning plane. The terms “target feature” and “reference feature” in the present context denote, each on a respective object, any specific point, location, area or portion, which are to be positioned with respect of each other. The action to be performed after positioning can be a galvanic contact formation and measurement task, with or without some kind of previous robotic motion.

[0042] Generally, one of said reference and target features 10, 11 is on a first object attached to a robotic manipulator 100 (see Figs. 7, 10 and 11 ) and the other one of said reference and target features 10, 11 is on a second object being arranged in a workspace accessible by the robotic manipulator 100 and being detached from, i.e. being not attached to the robotic manipulator 100. In the depicted example, the first object is a tool 12 comprising the test pin, while the second object is a workpiece 13, which is here a PCB to be tested. The positioning plane is preferably above the workpiece 13 and is practically parallel with a surface of the workpiece 13 carrying the target feature 11 , but the invention is also applicable in cases where the positioning plane is non-parallel with such a surface.

[0043] As depicted in Figs. 1 and 2, a first imaging unit 20 with an estimated first scale factor is used for imaging, from a first viewpoint 24, the target feature 11 and at least one marker 21 defining a virtual first line 22 in the image of the first imaging unit 20. Preferably there are two markers 21 formed as dots or crosshairs defining the first line 22. The first imaging unit 20 and the at least one marker 21 imaged by the first imaging unit 20 are fixed with respect to the reference feature 10.

[0044] A second imaging unit 30 with an estimated second scale factor is also used for imaging, from a second viewpoint 34 different from the first viewpoint 24, the target feature 11 and at least one marker 31 defining a virtual second line 32 in the image of the second imaging unit 30. Again, there are preferably two markers 31 formed as dots or crosshairs defining the second line 32. The second imaging unit 30 and the at least one marker 31 imaged by the second imaging unit 30 are also fixed with respect to the reference feature 10.

[0045] According to the invention, the viewpoints 24, 34 and the markers 21 , 31 are arranged so that a virtual first plane 23 defined by the first viewpoint 24 and the at least one marker 21 defining the first line 22, and a virtual second plane 33 defined by the second viewpoint 34 and the at least one marker 31 defining the second line 32 intersect each other along a virtual axis 40, which intersects the positioning plane. The detected marker(s) 21 , 31 and the respective viewpoints 24, 34, being preferably camera focal points or camera image center points, shall form the first and second planes 23, 33, respectively. The first line 22 and the second line 32 are projections of the spatial virtual first plane 23 and spatial virtual second plane 33 onto an image plane corresponding to the first imaging unit 20 and the second imaging unit 30, respectively. The intersection of the virtual planes 23, 33 defined by the two imaging units 20, 30 and detected markers 21 , 31 thus define the spatial virtual axis 40.

[0046] The term “viewpoint” in this context may denote the focal points or center points of the respective imaging units, practically corresponding to a center pixel or a center point of the imaging camera pixel matrix. Alternatively, other pixels or points of the imaging camera pixel matrix can also be assigned as a reference point used as a viewpoint for the positioning.

[0047] Ultimately, marker detection / localization has two purposes. Firstly, it needs to determine the image lines 22, 32, which can be done using any kind of line definition, such as using two points or a point and a direction vector. These can be represented by artificial or natural markers 21 , 31 , the markers being identifiable portions, areas or points of objects appearing in the images. Secondly, it needs to determine the virtual planes 23, 33 in order to form the virtual axis 40 corresponding to e.g. a tool. Marker detection can be carried out e.g. by conventional methods like blob or edge detection, or by machine learning-based methods. If the virtual axis 40 defined by the intersection of the planes does not intersect the reference feature 10, then the reference feature 10 needs to be calibrated to the virtual axis 40, in this way, after the positioning is successfully finished and the target feature is on the virtual axis, the reference feature gets to a known geometric relation relative to the target feature, which allows the performing of the desired action by considering the said relative geometric relation. Otherwise, if the virtual axis 40 does intersect the reference feature 10, it is sufficient for the intended operation if the virtual axis 40 is known in the robot TCP frame, relative to the TCP axes.

[0048] Positioning requires at least one, preferably two imaging units 20, 30 capable of providing different viewpoints, positioned so that the corresponding image planes are not parallel, i.e. the planes intersect. The imaging units 20, 30 can be realized by means of pinhole cameras or other cameras preferably with calibration and undistortion that match the perspective mapping. Preferred realizations can use - two cameras acting as two imaging units providing two different viewpoints, or

[0049] - two imaging units achieved by a single camera together with an optic system that results in a single camera image with multiple parts corresponding to different viewpoints, or

[0050] - a single imaging unit achieved by a single camera together with an optic system that results in a single camera image with a single viewpoint in one state, and a second camera image with a second viewpoint in a second state, with the actual state being controllable, practically an optic system alternatingly imaging different viewpoints into the single camera image, or

[0051] - any combination of the above options.

[0052] Fig. 3 schematically illustrates a view field of the first imaging unit 20 and Fig. 4 schematically illustrates a view field of the second imaging unit 30. The positioning method comprises first positioning steps in a first direction of the positioning plane, and second positioning steps in a second direction of the positioning plane. Each of the first positioning steps comprise:

[0053] - taking an image by the first imaging unit 20, the image containing a first imaging 111 of the target feature 11 ,

[0054] - determining a distance within the image between the first line 22 and the first imaging 111 of the target feature 11 ,

[0055] - calculating a respective physical distance based on the determined distance and the estimated first scale factor,

[0056] - checking, whether at least one stop condition is achieved, and if not, controlling the robotic manipulator 100 to a movement compensating the calculated respective physical distance.

[0057] The first positioning step is repeated until the at least one stop condition is achieved. The term “compensating” in the present context means a movement with the robotic manipulator 100 that aims to eliminate the physical distance present between an actual and a desired position. Distance determining within the image is preferably carried out by determining a distance between the first line 22 and a center point or center pixel of the first imaging 111 of the target feature 11 . Similarly, each of the second positioning steps comprise:

[0058] - taking an image by the second imaging unit 30, the image containing a second imaging 112 of the target feature 11 ,

[0059] - determining a distance within the image between the second line 32 and the second imaging 112 of the target feature 11 ,

[0060] - calculating a respective physical distance based on the determined distance and the estimated second scale factor,

[0061] - checking, whether the at least one stop condition is achieved, and if not, controlling the robotic manipulator 100 to a movement compensating the calculated respective physical distance.

[0062] The second positioning step is also repeated until the at least one stop condition is achieved.

[0063] The physical target feature 11 appears from a different viewpoint in each camera image, so the imagings 111 , 112 of the target feature vary from camera to camera in the image space. It is important that in each iteration both the target feature 11 to be detected and the virtual axis 40 to be detected remain the same.

[0064] As depicted in Figs. 3 and 4, the first and second lines 22, 32 are determined by means of respective imagings 211 , 311 of the markers 21 , 31 within the respective images.

[0065] If the rigid body relations are preserved with respect to the TCP frame, the imaging units 20, 30 and the markers 21 , 31 , then the markers 21 , 31 or even the first and second lines 22, 32 - i.e. the virtual image axes - can be drawn (marked) onto the image after calibration as virtual markers with known pixel addresses, and can be used instead of physical ones. In this case, until the TCP frame-camera-tool geometry does not change, there is no need to redetect the markers 21 , 31. The geometric calibration of the tool markers can be carried out at the time of tool manufacture. Geometric calibration of cameras, markers and of the reference feature 10 can also be performed during tool manufacture. When knowing the camera matrix of the calibrated camera, as well as the offset distance between the reference feature 10 and the target feature 11 , it is possible to use virtual markers instead of physical ones in such way that they are drawn (marked) on the camera image, as if they were on a parallel plane in a different height, e.g. by using perspective projection of the image points.

[0066] The at least one stop condition for the positioning method comprises the condition that the calculated respective physical distance is within a threshold limit, which corresponds to successful positioning, i.e. when the deviation between the desired and attained relative geometric relation between the reference feature 10 and the target feature 11 is sufficiently low. However, further conditions can also be applied as a stop condition. Preferably, the at least one stop condition also comprises one or more of the following further conditions:

[0067] - a number of repetitions of the first positioning step has reached an upper limit and / or a number of repetitions of the second positioning step has reached an upper limit and / or a time taken by the repeatedly performed first and second positioning steps has reached an upper limit;

[0068] - an upper distance limit has been achieved from an initial distance between the reference feature 10 and the target feature 11 ;

[0069] - a collision or close to collision state has been achieved by the first and / or second positioning steps, between any two components of the robotic cell.

[0070] A number of termination conditions can be thus defined for the servo method. For example, a maximum initial error radius can be assumed, i.e., if the robot TCP moves out of the circle defined by this radius around a starting point, the servo method can be terminated (displacement error). This phenomenon can occur when the detection and / or localization of the target feature is incorrect due to e.g., a significant visual deviation in the area of the target feature. For example, this occurs in case of template matching-based target feature detection, when the correlation between the template pattern and the camera image is weak and a false positive matching is found. Similarly, detecting a collision or close to collision state in the robotic cell may cause the method to stop. This can be detected e.g. by measuring the arising force between any two components in the robotic cell. In case of PCB measurement, arising force most probably originates from a collision between the measuring probe (tool 12) and the measured workpiece 13; if this reaches an upper force limit (force error), the servoing can be terminated. Furthermore, servoing can be terminated if the servo iteration does not converge quickly enough towards the acceptable alignment between the tool 12 and the measured point (control error). In all of the above error cases, the intended action, e.g. in case of PCB measurement, the forward feeding translational motion and measurement is omitted in order to avoid any collision, damage or potential shorting of components. Otherwise, if no error is detected after the servo target is reached, the desired action can be performed e.g. a forward translational motion, measurement and a backward translational motion back to a safety plane above the workpiece after the measured values are read from a measuring instrument.

[0071] Accordingly, the robot is positioned so that the virtual axis 40 and the target feature 11 become overlapped, or in other words become intersecting, or in still other words the target feature 11 gets to be on the virtual axis 40. If this does not happen within a given threshold (tolerance), the robot's position further iterates using the above detailed fine positioning. For this, the inventive method uses perspective projection in such a way that when an image line (image virtual axis) in an image of an imaging unit is overlapped by the imaging of the target feature, in reality, it is achieved that a virtual plane belonging to the imaging unit (whose projection is the image line) will be overlapped by the physical target feature. As this is achieved by fine positioning for two imaging units 20, 30 with different viewpoints 24, 34, the physical target feature will fall on the plane 23, 33 of both imaging units 20, 30, which means that the target feature will fall on the intersection, i.e. on the spatial virtual axis 40 defined by the planes 23, 33 of the two imaging units 20, 30. The spatial calibrated position of the virtual axis 40 is known in the robot's TCP frame, so that due to rigid body constraints, a motion command can be issued to the robot that can bring the physical reference feature 10 to the desired relative geometric position with respect to the physical target feature 11 .

[0072] Fine positioning is thus performed by performing compensatory movements identified in the camera images. If the exact value of the ratio between image space and physical space (pixel / mm), called as the scale factor of the imaging units, would be readily and steadily available, as well as an exact compensation motion could be carried out, while the physical position of the target feature remain the same, one compensatory motion would achieve an overlap of the virtual axis 40 with the target feature 11 within a predefined tolerance. In such cases servoing would not be necessarily as a single compensatory motion could achieve the fine positioned state. However, in real life processes no such conditions can be ensured or could only be ensured with high specific costs and with high computational demand.

[0073] The invention ensures that by iteratively performing the compensation process in a number of steps, fine positioning could be accomplished in a safe and simply implementable way. For the inventive accurate positioning, it is sufficient to know an assumed or estimated scale factor of the imaging units 20, 30. Experiments have shown that even up to a + / -50% deviance of the estimated scale factors from actual respective real scale factor values, the inventive visual servoing provides a converging and accurate positioning. Experiments have also proven that a deviation of up to + / -10% of the estimated scale factors results in a very fast converging visual servoing. In this case, even significantly less robustness is required for the image processing algorithms and for the robotic control,

[0074] In the case of a deviation between + / -10% and + / -50%, a higher noise tolerance may be required in the image processing algorithms, or for example, a higher depth of camera field may be needed, and the of the robot control may need to be adjusted more carefully.

[0075] Experiences have shown that in cases of up to + / -50% deviance of the estimated scale factors, proper robot control can be achieved by using a properly chosen proportional controller aligned with the system, i.e. using a P controller is sufficient, which reduces computational demand. It is noted that multiplying the distance determined in the image with the estimated scale factor can in itself be considered as a simple P controller. In case of higher deviations of the scale factor, using more complicated controllers, such as PI, PD or PID controllers may be more appropriate. Iterative compensation is more noise tolerant, allowing e.g. small displacements of the workpiece 13 during the process, e.g. due to vibration, and can handle inaccuracies caused by the robot repeatability.

[0076] The two calculated physical distances determined based on the image of the two imaging units 20, 30, i.e. the first and second compensation values - which may or may not correspond directly to the first and second different directions of motion - can be combined and output to the robot controller at the same time iteratively, until the distances between the virtual axis 40 and the target feature 11 in both directions fall below a threshold. Along an axis defined by a degree of freedom other than those of fine positioning, e.g. in the Z direction, a position and / or orientation defined by an initial position and / or orientation can be maintained throughout the process. However, there can be different strategies for setting the undefined degrees of freedom during the fine positioning process, e.g. by moving further away from the workpiece in the Z direction, the field of view of the camera can be increased, making the method more suitable for rough positioning with a larger initial error; then, when the error becomes lower, by moving towards the workpiece in the Z direction, the scale factor can be improved and more precise fine positioning can be achieved.

[0077] The threshold limit for the calculated respective physical distances can be adjusted according to the given application, in the light of the acceptable tolerances and expected accuracy. Individual steps of the first and second positioning steps can be merged, mixed and / or their order can be changed, as best suitable for the given application.

[0078] Essential prerequisites for the operation of the control underlying the visual servo method are that

[0079] - the distance between the virtual axis 40 and the target feature 11 during the application of the method converges to zero, and

[0080] - the specified accuracy (threshold limit) is physically achievable by the robotic system. In addition, a stop condition is required to avoid infinite iteration loops, e.g. by allowing a maximum number of iterations or a maximum positioning time.

[0081] The former condition means that the absolute distance between the virtual axis 40 and the target feature 11 must decrease both in the image and in physical space. It does not necessarily need to show a decrease in every iteration step, for example, it is possible that a false detection occurs in one iteration due to a flicker and therefore this distance temporarily increases, or it is possible that a vibration causes the target feature 11 to move further away from the virtual axis 40 between two iterations. Regardless, if the distance shows a decreasing trend over several iterations, the controller is considered adequate. This convergence can be more accurately captured in the image space: the robot's motion must be such that the distances between the first and second lines 22, 32 and the first and second imagings 111 , 112 of the target feature 11 in the image space is expected to decrease in each iteration.

[0082] The latter condition implies that the resolution of the system, i.e. the motion command that can be issued to the robot and the smallest step that the robot can take, as well as the scale factor must allow the specified accuracy to be achieved.

[0083] In a single iteration step of positioning, one of each of the first and second positioning steps can be carried out, or only one of those in an alternating manner. After controlling the robotic manipulator 100 to a movement compensating the calculated respective physical distance, a sufficient waiting time of e.g. 0.1 s can be introduced before taking the next image, so that the movement of the robotic manipulator 100 can settle.

[0084] One embodiment of the iterative solution of the visual servo method is a step-wise compensation. If the time of the operations performed in an iteration (imaging, image processing, compensation computation, command issuing, etc.) allows, i.e. the operations can be performed in near real time, continuous mode compensation can be performed instead of step-wise mode. In this case, for example, speed control may be used instead of position-based control, where imaging is performed in motion, e.g. by snapshot capture or by cropping a snapshot from a camera stream. In the visual servo method, a velocity can be determined from the calculated compensation distances in the iterations, taking into account the cycle time of the iteration, and compensation can be performed using the distance, computation time and velocity values. Continuous operation has the advantage of faster compensation, as it saves the acceleration and deceleration phases in each cycle of the robot's motion.

[0085] The target feature 11 can be a point or an area or a portion on an object and the reference feature 10 can be a point or an area or a portion of another object, where one of the two objects is attached to the robotic manipulator 100, while the other of the two objects is attached to the workspace, and not attached to the robotic manipulator 100. Also, the imaging units 20, 30 do not necessarily need to be fixed to the robot, for example, fixed external imaging units can also be used, with the markers 21 , 31 and the reference feature 10 fixed with respect to the workspace, while the target feature 11 is bound to the robot.

[0086] In addition, it is also possible that the reference feature 10 is not or not only bound to the tool 12 (e.g. attached to the robot’s flange), but e.g. if the tool is a gripper, the reference feature 10 can also be bound to the workpiece 13 attached to the gripper. For example, if a prismatic gripper is attached to the robot arm and a cylindrical workpiece is gripped in the gripper, the virtual axis 40 can be the axis of the cylindrical workpiece and the reference feature 10 can be on the cylindrical workpiece.

[0087] Target features 11 can be defined on the basis of a model of the workpiece 13, e.g. geometric information can be defined for a target feature 11 using the 3D model of the workpiece, and visual information can be prepared for the target feature 11 based on rendered images (using a virtual camera) of the area of the target feature 11 in the 3D model. In the absence of a precise model, e.g. CAD drawings of the workpiece 13, visual and geometric information is required for the target feature. For example, template patterns can be set on images captured of the target feature and the corresponding starting robot TCP pose (position and orientation) can preferably be manually taught for a PCB measurement point. A teaching operator can position the robot with the probe above the feature to be measured in a safety plane above the workpiece. At this point, images can be captured, possibly with additional, built in light source(s) (e.g. using LED panels) turned on, and the target feature can be selected for example on the zoomed camera images with pixel accuracy. This can be then stored together with the robot TCP coordinates by reading the values from the robot controller, and the operator can attach a label, a measurement type and reference electric parameters to the set measurement point. The TCP pose can serve as an initial pose for the positioning method, and this pose is preferably maintained during the positioning, except in the positioning directions determined by the positioning plane.

[0088] While in the safety plane, it is also possible to set a tilted robot approach direction instead of a vertical one by setting the probe in a tilt direction and angle, as well as the TCP orientation around the pin axis (or Z axis of the TCP frame). These parameters can be stored together with the measurement point. The initial TCP pose can be different for each different target feature, while the positioning directions can remain the same.

[0089] In case of a PCB measurement application, by organizing different measurement points in a sequence, a measurement profile can be formed, which can be assigned to a specific product type. During the testing of a PCB, using the product type identifier, the measurement profile can be automatically loaded and executed. In such an application, the fine positioning is carried out for the first measurement point (target feature), if the positioning is successful, the corresponding motions and measurement (desired action) are executed, next, the same operation is performed for the second measurement point, etc., until the operation is performed for every measurement point.

[0090] Manually set measurement points need to be robust enough to handle most visual differences. It is the responsibility of the teaching operator to consider lighting conditions and the object appearance and set the tilt and orientation of the tool in such a way that there are no significant noise and disturbance (e.g. shining spots or occlusions) in the visual information (e.g. template pattern) of the target feature.

[0091] For manual teaching it is important that the target features 11 are defined in a way that they can be consistently recognized throughout the fine positioning. An exact robot start pose (initial TCP pose) is not required; it is sufficient to define an approximate starting pose. In this case, no prior geometric information (e.g. digital model, shop drawing) is required.

[0092] Preferably, subsequent to achieving the stop condition that the calculated respective physical distance is within a threshold limit, a movement is carried out with the first object towards the second object, e.g. in form of a translational movement with the tool 12 in parallel with or along the virtual axis 40 until the tool 12 contacts the workpiece 13. In this way, an accurate positioning can be carried out with a distance from the workpiece 13, and thereafter an approaching movement can be carried out towards the workpiece 13 in order to start a respective action, or as a part of a respective action.

[0093] In an especially preferred embodiment, the viewpoints 24, 34 and the markers 21 , 31 are arranged so that the reference feature 10 is on the virtual axis 40. In such a case, the target feature 11 can always be touched by making an approximation in the direction of the virtual axis 40; the target feature 11 always remains on the virtual axis 40. In this case, the action with the tool 12 can be started as soon as positioning is complete and there is no need to further position the tool 12 to compensate any distance vector between the virtual axis 40 and the reference feature 10. In case the reference feature 10 is not on the virtual axis, an additional distance compensation is necessary. The reference feature 10 is not necessarily arranged on the virtual axis 40, and it can be an action point of a respective tool 12, e.g. the tip of a gauge tip, a bit head tip for screwing, or a welding tip for spot welding.

[0094] In the preferred case, if the same at least one marker 21 , 31 is imaged by both of the first imaging unit 20 and the second imaging unit 30, the system will be tolerant to changes in the imaging unit-spatial virtual axis relationship. Namely, in case of any imaging unit displacement, the displaced virtual planes still define the same intersection line, so the same spatial virtual axis 40 will remain, as it is defined by the stationary physical markers 21 , 31 .

[0095] Preferably, movements in the positioning plane in an X direction and a Y direction correspond to either

[0096] - the X and Y axes of the robot TCP frame, in case these axes are parallel to the positioning plane; or

[0097] - the projection of the X and Y axes of the robot TCP frame onto the positioning plane, in case the axes are not parallel to the positioning plane, as controlling the robot using the TCP frame is one of the typical methods for robot positioning.

[0098] In addition, preferably, for the sake of easy implementations and calculations, the imaging units 20, 30 are rigidly fixed to the robot’s flange and the viewpoints 24, 34 and the markers 21 , 31 are arranged so that the first plane 23 is perpendicular to the second plane 33, more preferably the first plane 23 is perpendicular to the X axis of the TCP frame and the second plane 33 is perpendicular to the Y axis of the TCP frame. This is related to the TCP-imaging unit-marker arrangement, which is considered to be fixed in a robotic system. In this case, each image has a preferred movement direction, and if the target feature 11 and the image virtual axis 40 overlap in one image, the fine positioning movement of the other imaging unit will not move the target feature 11 out of this already existing overlapping. It is noted that other TCP-imaging unit-marker arrangements are also possible, where the respective axes and planes are not aligned perpendicularly, in these cases the relative geometric relation between the planes and the axes need to be determined, and applied during the positioning process to be able issue motion commands suitable for the positioning directions in the positioning plane. This also means that the images do not have a preferred movement direction, and both images may influence the movement in both positioning directions.

[0099] Considering the above described, preferred arrangement, in case of the usual vertical approach, the controlled movements of the robot in the X and Y direction of the positioning plane are aligned with the X and Y axes of the TCP frame, which are also in parallel with the plane normal of the first and second virtual planes 23, 33, respectively. Thus, in this embodiment the viewpoints 24, 34 and the markers 21 , 31 are arranged so that the first plane 23 is perpendicular to a first controlled movement direction of the robotic manipulator 100, and the second plane 33 is perpendicular to a second controlled movement direction of the robotic manipulator 100. In this case, positioning is easy, as the compensation vector determined on the image of the first imaging unit 20 directly corresponds to the X directional motion in the positioning plane, and the image of the second imaging unit 30 directly corresponds to the Y directional motion in the positioning plane. Incidentally, in this scenario, the virtual axis 40 will be perpendicular to the positioning plane.

[0100] However, for the sake of higher flexibility, not using the usual vertical approach is also possible, i.e. it is allowed that the X and Y axes of the TCP frame are not in parallel with the positioning plane, resulting that the virtual axis 40 is also not perpendicular to the positioning plane. In this scenario, the projection of the X and Y axes of the robot TCP frame onto the positioning plane are used as X and Y positioning directions. Considering the above described preferred TCP-imaging unit-marker arrangement, positioning is still easy, as the compensation vector determined on the image of the first imaging unit 20 still directly corresponds to the X directional motion in the positioning plane, and the image of the second imaging unit 30 still directly corresponds to the Y directional motion in the positioning plane. An example for this scenario is when the TCP frame and attached tool 12 is tilted relative to the arrangement used in the usual vertical approach to avoid obstacles or improve visibility of the target feature. The TCP pose used for one positioning task is determined by the initial TCP pose assigned to the target feature, which can be set either using a model-based approach by the programming operator or using a model-free approach by the teaching operator.

[0101] The robot movement is preferably set up in such a way that each image has a preferred movement direction, and if the target feature 11 and the image virtual axis 40 overlap in one image, the fine positioning movement of the other imaging unit will not move the target feature 11 out of this already existing overlapping. The distance determining steps preferably comprise image processing for identifying respective positions of the first and second imagings 111 , 112 of the target feature 11 within the respective images and / or image processing for identifying respective positions of the imagings 211 , 311 of the markers 21 , 31 within the respective images.

[0102] In case if positioning in a Z direction is also desired, a positioning point can be defined on the virtual axis 40 by means of a third plane intersecting the virtual axis 40, wherein the third plane can be defined by a third viewpoint of a third imaging unit and / or further at least one marker defining a third line in an image of the first or second imaging units 20, 30.

[0103] Thus, for multi-degree of freedom fine positioning, including e.g. rotation around the Z-axis or Z-directional translation, can be implemented by defining multiple virtual planes. For example, the intersection of three virtual planes can result in a virtual point that can be overlapped with the target feature 11 . If two imaging units are used to position the target point using two virtual planes, then first the target point is positioned on the spatial virtual axis 40. For the actual positioning, the target point is then moved along the virtual axis 40. Without the inclusion of a third plane intersecting the virtual axis 40, reaching the desired position can be identified e.g. by means of metallic contact, force or other type of sensor signals. If a second plane is selected on one of the imaging units - i.e. three planes in total on the two imaging units - then the third plane can be used to overlap any point on the virtual axis 40 with the target point.

[0104] Fig. 7 schematically illustrates a block diagram of visual servoing in a preferred embodiment of the invention. Visual feedback for the visual servoing is provided by a compound image processing algorithm, e.g. by using standard or machine learning-based tools. The images are taken in step 50. The image processing is responsible for two main tasks, with the final goal being the calculation of the compensation vector for the servo technique. In a feature extraction step 51 , firstly, marker points are detected on the images of the imaging units 20, 30 to determine the virtual axis 40, which in the present embodiment ideally coincides with the testpin’s axis, from both viewpoints 24, 34 for the bi-directional error compensation. Secondly, the target feature is identified on the images of the imaging units 20, 30. For example, for each measurement point and for each camera, a template pattern can be selected during the target feature setup, which serves as a reference image. Continuing the above example, when testing a particular measurement point on the PCB, the corresponding pattern can be sought on each camera image in every servo iteration. The pixel position resulting in the highest correlation can be selected as the target point of the compensation vector. Finally, the distance within the image is computed between the imaging of the target feature 11 and the image virtual axis (in this case, representing the pin axis), perpendicular to the image virtual axis. This results in two separate compensation vectors in step 52 - utilized as a displacement command for the robot in step 53 - corresponding to two perpendicular directions in the plane parallel to the top PCB surface, the commands are preferably carried out by a robot controller in step 54.

[0105] The markers 21 , 31 can be identified and localized in the image space by visual imaging and image processing methods on the images of the imaging units 20, 30.

[0106] There are a number of possible options for managing the image processing for the target feature detection and localization for the visual servo method.

[0107] Image keypoints can be used to determine the target feature 11 on the images of the imaging units 20, 30, this could be invariant to image rotation and image scaling, and less sensitive to variation in the lighting conditions and colors.

[0108] Fig. 8 schematically illustrates a constellation-based matching usable for identifying a target feature, which is a variant of image keypoint-based matching. This can involve a machine learning-based or conventional detection algorithm for prominent objects on the images of the imaging units 20, 30. Whether or not an object is prominent is determined by the workpiece type. For example, many PCBs have via holes on the outer layer, which have a precise position on the board and are distinct enough to detect and localize. A set of via holes can be considered as a 2D point cloud on the image (“constellation”), which maps the image, and features can be localized relative to this constellation using e.g. homography. This can be performed when defining the target feature 11 on the reference image, and the resulting constellation can be compared with the constellation identified on the actual camera image during the visual servo process.

[0109] Fig. 9 schematically illustrates a template matching usable for identifying a target feature, e.g. by using normalized cross correlation.

[0110] It is also possible to use multiple reference images e.g. in different colors for defining the target feature, and these can be associated with a single target feature 11 , while an autonomous selection of the best fitting initial reference image can be provided for the visual servo operation. The reference images corresponding to target features 11 can also be masked to further improve robustness. Any kind of image processing / lighting-based preparation can be applied to improve robustness.

[0111] Fig. 10 schematically illustrates a practical application of a preferred embodiment with a workpiece 13, which is not aligned with respect to locating pins 60, and Fig. 11 schematically illustrates an enlarged part of Fig. 10 showing the workpiece 13 in a fixture-aligned position where the workpiece 13 is in abutment with the locating pins 60. Here the invention is used for screwing and unscrewing with a screwing tool 12. If the spatial virtual axis is constrained to the given screw, it may be useful to have two spatial virtual axes 40, one for the tool 12 and one for the screw. In this case, it is possible to calibrate the spatial virtual axis 40 associated with the current screw against the virtual axis of the tool 12 or even against the TCP frame each time the screw is gripped; or it is possible to use the image virtual axes associated with the screw for fine positioning. Even without calibrating the virtual axes 40 for the screws in every gripping case, it is possible to determine the reference feature 10 (the position of which varies relative to the TCP frame per screw grip), with a good approximation and thus to perform fine positioning. The action can be e.g. a twisting task, or some kind of translation and a twisting task. The invention does not necessarily require the workpiece 13 to be in an instrumented, exactly located state, (i.e. in known position and orientation). In case of physical contact actions, the workpiece 13 may be clamped without being precisely located in the positioning plane. In non-contact type action cases, neither clamping nor locating may be necessary. In case the workpiece is not in an exactly located state, a maximum deviation in the position of the workpiece 13 shall be such that the condition that the target feature 11 is in the field of view of the imaging units 20, 30 at the initial position of the fine positioning is still satisfied. This can be achieved, for example, by placing the workpiece 13 in the robot's workspace with a slight orientation error, e.g. by manually placing the workpiece 13 in a visually marked position without using locating pins. For larger position and / or orientation errors, an accuracy enhancement technique - e.g. homography-based measurement using an image of a larger portion of the workpiece, possibly containing identifiable portions of the workspace as well - can be used to determine the approximate in-plane position and / or orientation error of the workpiece 13. This further enhances the flexibility of the invention, as it does not necessarily require instrumentation, which can result in actions being performed on workpieces 13 arranged on a simple flat surface, such as a table.

[0112] The invention overcomes the disadvantages of prior art solutions of conventional positioning, which are due to accumulation of errors in robot, workpiece, fixturing and tool manufacturing, as well as errors in assembly and control (measurement, resolution) and errors in imaging and image processing. This often results in insufficient accuracy during positioning, which makes it impossible to successfully perform the desired action, or necessitate exceptional manufacturing and assembly precision, control and imaging resolution for the positioning to be feasible for the desired precision.

[0113] It is also an advantage of the invention that one of the reference and target features 10, 11 is bound to the robotic cell or even to the robot, and can be set when the robotic cell is installed, but the other one of the reference and target features 10, 11 is bound to the application (i.e. the workpiece or object on which the action is performed). This means that the definition of one of the said features can be flexible and simple so that it can be easily set from model, or manually by a teaching operator.

[0114] A low-cost, compact, robot-mountable measurement tool is provided by the invention, e.g. for the realization of flexible PCB measurement and visual servo function. The measuring tool may carry LED light sources together with their control modules to enable robust image processing with the imaging units 20, 30. Their role is to counteract any uncertainties of environmental lighting conditions. In a preferred embodiment, a test-pin holding probe has an elongated shape so that the measurement tool can reach measurement points even in the close vicinity of higher components on the PCB, such as heatsinks. The test-pin is fixed in the probe with a screw, which also provides the electric contact for the crimped positive terminal cable of the measuring instrument. A spring-loaded pin (pogo pin probe) is used for the measurement, which makes robot control simpler and more robust. The robot TCP is configured so that the origin of the TCP frame coincides with the tip of the pin and the Z axis of the TCP frame is aligned with the axis of the test-pin. The testpin can be easily replaced by unscrewing the fastener in case of damage or wear.

[0115] The imaging units 20, 30 may be fixed in such an angle that their axes intersect the axis of the test-pin under the tip of the pin. In this way, the images of the imaging units 20, 30 cover the measured feature as well as the probe. The probe preferably carries two marker points per imaging unit as shown in the Figs. 1 , 3 and 4, which preferably indicate the axis of the test-pin.

[0116] The invention is not only applicable to a single static target feature 11 , but it can also be extended to tracking, which can be conceived e.g. as a sequence of static target features 11 instead of a single static target feature 11 . This means that after a target feature 11 has finished its fine positioning, a new fine positioning starts at the next target feature 11 , while transitions can be handled e.g. by interpolation by inserting intermediate points. A dynamic target feature 11 may also be used, which target feature 11 is a point of interest in motion. Another possibility is to perform trajectory tracking in two steps, first, performing fine positioning for a target position sequence above the workpiece 13, e.g. in Z direction 10 mm above the surface, and saving the corresponding robot positions to perform the action, if contact is required, 10 mm below, at the surface of the workpiece. In this way, the range of applications can be significantly extended, e.g. to arc welding, laser welding or gluing, sealant dispensing.

[0117] Straight-line tracking can also be implemented by defining two target features 11 instead of one target feature 11 , and detecting and localizing both of them in both images of the imaging units 20, 30. The two target features 11 define a vector that makes an angle with the image virtual axes in both image of the imaging units 20, 30. By specifying a target angle value and varying the two rotations of the robot TCP frame, the angle between the image virtual axis and the target feature vector can be adjusted to the target angle value. Then, a linear trajectory can be traced between the two target features, or along the line defined by them, in the direction corresponding to the spatial virtual axis 40.

[0118] Instead of a single robot or manipulator, multiple robots can also be applied for example for PCB diagnostic tasks, for cases when different terminals need to be in contact with different target points, simultaneously. Depending on the given application, a manipulator with less degrees of freedom than a robotic arm can be applied.

[0119] In general, for successful visual servoing, first the spatial virtual axis 40 is preferably calibrated to the robot TCP frame, and thereafter the reference feature 10 is preferably calibrated to the virtual axis 40 or to the TCP frame. The cameras are assumed to be pin-hole cameras, or other cameras that are calibrated and have perspective, i.e. line preserving, undistorted images. Automated, semi-automated or manual geometric calibration can be performed using a calibration device and method that is able to calibrate the system in two levels:

[0120] - Geometric calibration of the spatial virtual axis 40 of e.g. a tool relative to the robot joint it is mounted to, for example to the Z axis of the joint; this involves the geometric relation of the imaging unit and the robot joint, as well as the geometric relation of the marker points relative to the robot joint. This can be realized by for example applying visual servoing to a target feature 11 using different robot TCP orientations. Theoretically, the spatial virtual axis 40 is going to intersect the same target feature 11 in different orientations. In each orientation, after the visual servoing was performed, using the robot’s internal measurement system, the TCP frame can be measured relative to the robot base. By formulating the rigid body constraint between the Z axis of the TCP frame and the spatial virtual axis 40, the orientation of the spatial virtual axis 40 can be determined e.g. using the least squares method.

[0121] - Geometric calibration of the position of the reference feature 10 relative to the identified spatial virtual axis 40; this can be determined for example manually, after visual servoing (with or without performing the action afterwards), the reference feature 10 can be moved by e.g. jogging the robot to meet the target feature with the desired geometric relation. The manually introduced displacement shows the spatial position of the reference feature 10 relative to the spatial virtual axis 40. An alternative, automated calibration method can be for example performing visual servoing to one or a set of artificial target features 11 with known positions relative to the robot base frame, and analyzing the image of the imaging units 20, 30. Using these images and the known and attained geometric positions, the reference feature 10 can be determined in the images of the imaging units 20, 30 relative to the target feature(s) 11 in the image space, and hence (e.g. by using homography) its physical position can also be determined in the robot base frame, in the robot workspace or in the robot TCP frame.

[0122] In principle, both calibrations can be avoided if the spatial virtual axis 40 coincides with one of the main axes of the TCP frame, and the reference feature 10 is coincident with the spatial virtual axis 40.

[0123] Industrial applicability of the invention follows from the above detailed description and advantages. Novel visual servoing method and system are provided e.g. to overcome the challenges of automated PCB measurement in repair shop environments. The method can handle inaccurately located workpieces and move the tool, e.g. measurement probe to a measurement point with the required accuracy. The invention provides

[0124] - precision enhancement over the robot repeatability, made possible by the resolution of the robot joint coordinate measuring instruments (e.g. encoders);

[0125] - enhanced flexibility as no precise geometric representation is necessary for point (target feature) teaching, only a basic measurement test specification is required;

[0126] - enhanced flexibility in the volume and variation of the workpieces to be processed;

[0127] - enhanced flexibility in the transferring of a defined measurement point between different robot cells, which may differ in manufacturing and assembly tolerance instances, without the need for manually re-defining the measurement point;

[0128] - the possibility of using different initial pose along / around the axes defined by the undefined degrees of freedom (that can be different for each measurement point), not only the conventional vertical approach is allowed, helping to avoid 3D obstacles and visibility issues;

[0129] - simple usage, the teaching process only requires pixel point selections on the camera images and an approximate positioning of the robot TCP;

[0130] - a representation in form of geometric and visual information for the measurement points;

[0131] - no precise fixturing is required, the visual servo method is capable of compensating a couple of millimeter errors in positioning, and the workspace of the robot arm allows large variation in PCB size;

[0132] - the method is less sensitive to deviations along / around the axes defined by the undefined degrees of freedom relative to the set initial pose;

[0133] - the technique is robust to deviations in the visual appearance and lighting conditions; and

[0134] - the measurement profile is easy to set up and modify, adding, removing and adjusting measurement points is simple. According to a further aspect, the invention is a computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out the steps of the inventive positioning method.

[0135] According to a still further aspect, the invention is a computer-readable medium comprising instructions which, when executed by a computer, cause the computer to carry out the steps of the inventive positioning method.

[0136] Embodiments of the invention can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment including both hardware and software elements. An embodiment may be implemented in software, which includes but is not limited to firmware, resident software, microcode, etc.

[0137] This written description uses examples to disclose the subject matter, and also to enable any person skilled in the art to practice the subject matter, including making and using any devices or systems and performing any incorporated methods. The patentable scope is defined by the claims, and may include other examples that occur to those skilled in the art. Such other examples are intended to be within the scope of the claims if they have structural elements that do not differ from the literal language of the claims, or if they include equivalent structural elements with insubstantial differences from the literal languages of the claims.

Claims

CLAIMS1. A positioning method for positioning a reference feature (10) with respect to a target feature (11 ) along a positioning plane within a workspace accessible by a robotic manipulator (100), wherein one of said reference and target features (10, 11 ) is on a first object attached to the robotic manipulator (100) and the other one of said reference and target features (10, 11 ) is on a second object being arranged in the workspace and being detached from the robotic manipulator (100), wherein a first imaging unit (20) with an estimated first scale factor is used for imaging, from a first viewpoint (24), the target feature (11 ) as well as at least one marker (21 ) defining a first line (22) in the image of the first imaging unit (20), the first imaging unit (20) and the at least one marker (21 ) imaged by the first imaging unit (20) being fixed with respect to the reference feature (10), and a second imaging unit (30) with an estimated second scale factor is used for imaging, from a second viewpoint (34) different from the first viewpoint (24), the target feature (11 ) as well as at least one marker (31 ) defining a second line (32) in the image of the second imaging unit (30), the second imaging unit (30) and the at least one marker (31 ) imaged by the second imaging unit (30) being fixed with respect to the reference feature (10), and wherein the viewpoints (24, 34) and the markers (21 , 31 ) are arranged so that a first plane (23) defined by the first viewpoint (24) and the at least one marker (21 ) defining the first line (22), and a second plane (33) defined by the second viewpoint (34) and the at least one marker (31 ) defining the second line (32) intersect each other along a virtual axis (40), and the virtual axis (40) intersects the positioning plane, the positioning method comprising: a first positioning step comprising:- taking an image by the first imaging unit (20), the image containing a first imaging (111 ) of the target feature (11 ),- determining a distance within the image between the first line (22) and the first imaging (111 ) of the target feature (11 ),- calculating a respective physical distance based on the determined distance and the estimated first scale factor,- checking, whether at least one stop condition is achieved, and if not, controlling the robotic manipulator (100) to a movement compensating the calculated respective physical distance, and repeating the first positioning step until at least one stop condition is achieved, the positioning method further comprising: a second positioning step comprising:- taking an image by the second imaging unit (30), the image containing a second imaging (112) of the target feature (11 ),- determining a distance within the image between the second line (32) and the second imaging (112) of the target feature (11 ),- calculating a respective physical distance based on the determined distance and the estimated second scale factor,- checking, whether the at least one stop condition is achieved, and if not, controlling the robotic manipulator (100) to a movement compensating the calculated respective physical distance, and repeating the second positioning step until at least one stop condition is achieved, wherein the at least one stop condition comprises the condition that the calculated respective physical distance is within a threshold limit.

2. The positioning method according to claim 1 , characterized in that the reference feature (10) is a point, a location, an area or a portion of the first object, and the target feature (11 ) is a point, a location, an area or a portion of the second object.

3. The positioning method according to claim 2, characterized in that subsequent to achieving the stop condition that the calculated respective physical distance is within a threshold limit, a movement is carried out with the first object towards the second object.

4. The positioning method according to any of claims 1 to 3, characterized in that the same at least one marker (21 , 31 ) is imaged by the first imaging unit (20) and the second imaging unit (30).

5. The positioning method according to any of claims 1 to 4, characterized in that the viewpoints (24, 34) and the markers (21 , 31 ) are arranged so that the reference feature (10) is on the virtual axis (40).

6. The positioning method according to any of claims 1 to 5, characterized in that the robotic manipulator (100) is controlled to movements in an X direction and a Y direction of the positioning plane, and the viewpoints (24, 34) and the markers (21 , 31 ) are arranged so that the first plane (23) is perpendicular to the second plane (33).

7. The positioning method according to any of claims 1 to 6, characterized in that the viewpoints (24, 34) and the markers (21 , 31 ) are arranged so that the first plane (23) is perpendicular to a first controlled movement direction of the robotic manipulator (100), and the second plane (33) is perpendicular to a second controlled movement direction of the robotic manipulator (100).

8. The positioning method according to any of claims 1 to 7, characterized in that the at least one stop condition also comprises one or more of the following further conditions:- a number of repetitions of the first positioning step has reached an upper limit and / or a number of repetitions of the second positioning step has reached an upper limit and / or a time taken by the repeatedly performed first and second positioning steps has reached an upper limit;- an upper distance limit has been achieved from an initial distance between the reference feature (10) and the target feature (11 );- a collision or a pre-collision state has been achieved by the first and / or second positioning steps.

9. The positioning method according to any of claims 1 to 8, characterized in that said distance determining steps comprise- image processing for identifying respective positions of the first and second imagings (111 , 112) of the target feature (11 ) within the respective images, and / orimage processing for identifying respective positions of imagings (211 , 311 ) of the markers (21 , 31 ) within the respective images.

10. The positioning method according to any of claims 1 to 9, characterized in that a positioning point is defined on the virtual axis (40) by means of a third plane intersecting the virtual axis (40), wherein the third plane is defined by a third viewpoint of a third imaging unit and / or further at least one marker defining a third line in a imaging unit image.11 . The positioning method according to any of claims 1 to 10, characterized in that the imaging units (20, 30) are- separate cameras,- a single camera combined with an optic system imaging different viewpoints into respective parts of the single camera image,- a single camera combined with an optic system alternatingly imaging different viewpoints into the single camera image, or- a combination of the above options.

12. A positioning system for positioning a reference feature (10) with respect to a target feature (11 ) along a positioning plane, the positioning system comprising a robotic manipulator (100) and a workspace accessible by the robotic manipulator (100), wherein one of said reference and target features (10, 11 ) is on a first object attached to the robotic manipulator (100) and the other one of said reference and target features (10, 11 ) is on a second object being arranged in the workspace and being detached from the robotic manipulator (100), the positioning system further comprising a first imaging unit (20) with an estimated first scale factor for imaging, from a first viewpoint (24), the target feature (11 ) as well as at least one marker (21 ) defining a first line (22) in the image of the first imaging unit (20), the first imaging unit (20) and the at least one marker (21 ) imaged by the first imaging unit (20) being fixed with respect to the reference feature (10), and a second imaging unit (30) with an estimated second scale factor for imaging, from a second viewpoint (34) different from the first viewpoint (24), the target feature(11 ) as well as at least one marker (31 ) defining a second line (32) in the image of the second imaging unit (30), the second imaging unit (30) and the at least one marker (31 ) imaged by the second imaging unit (30) being fixed with respect to the reference feature (10), wherein the viewpoints (24, 34) and the markers (21 , 31 ) are arranged so that a first plane (23) defined by the first viewpoint (24) and the at least one marker (21 ) defining the first line (22), and a second plane (33) defined by the second viewpoint (34) and the at least one marker (31 ) defining the second line (32) intersect each other along a virtual axis (40), and the virtual axis (40) intersects the positioning plane, the positioning system further comprising a processor adapted to perform the steps of the positioning method of claim 1 .

13. A computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out the steps of the positioning method of claim 1 .

14. A computer-readable medium comprising instructions which, when executed by a computer, cause the computer to carry out the steps of the positioning method of claim 1.

Images