Method for determining position of region of tool

Abstract

Method for determining the position of a region (101) of a tool (100) relative to an image acquisition device (200) located on the tool (100), the method comprising the steps of: placing the region (101) of the tool (100) at a fixed point, the position of which relative to a calibration device is known, the calibration device comprising at least one visual marking; determining, by the image acquisition device (200), the position of the visual marker relative to the image acquisition device (200); the position of the region (101) of the tool (100) relative to the image acquisition device (200) is determined on the basis of the position of the visual marker relative to the image acquisition device (200) and on the basis of the position of the fixed point relative to the calibration device.

Description

Technical Field

[0001] The present disclosure relates to a method for determining the position of a region of a tool relative to an image acquisition device located on the tool, and to a related device. Background Art

[0002] Determining the position of a region of a tool in a defined coordinate system, also referred to as "calibration of the region of the tool", is a fundamental element for ensuring the proper operation of a system for determining the position of the region of the tool.

[0003] In this regard, in the field of robotic arms, it is known to determine the position of a region of a tool (usually the tip of the robotic arm) located on the robotic arm by repeatedly pointing the region of the tool at a known position. Alternatively, to determine the position of the tip of the robotic arm, the intersection of a laser beam from a known position with the tip of the robotic arm is typically utilized.

[0004] However, existing solutions rely on knowledge of the encoder kinematics in the links that make up the robotic arm. Therefore, these solutions are not applicable to tools for which the tool kinematics are unknown. Summary of the Invention

[0005] The present disclosure improves this situation.

[0006] There is provided a method for determining the position of a region of a tool relative to an image acquisition device located on the tool, the method comprising the steps of:

[0007] - placing the region of the tool at a fixed point, the position of which relative to a calibration device is known,

[0008] wherein the calibration device comprises at least one visual marker,

[0009] - determining, by the image acquisition device, the position of the visual marker relative to the image acquisition device,

[0010] - determining the position of the region of the tool relative to the image acquisition device based on the position of the visual marker relative to the image acquisition device and based on the position of the fixed point relative to the calibration device.

[0011] Advantageously, the proposed method can determine the position of the region of the tool relative to an image acquisition device located on the tool without knowledge of the kinematic characteristics of the tool. Therefore, the proposed method is applicable to all types of tools. In particular, the proposed method is applicable to tools mounted on robotic arms as well as manual tools. Notably, compared to solutions in the prior art that require mechanical pointing or mechanical aiming, the proposed method has a lower complexity. The proposed method can complete the calibration of the region of the tool in approximately two minutes, while solutions in the prior art sometimes require approximately fifteen minutes to obtain a similar result.

[0012] Therefore, when the position of the image acquisition device is known, the position of the area of the tool can be inferred by the proposed method. Thus, the position of the area of the tool can be tracked based on the information of the position of the image acquisition device. In addition, according to one or more embodiments, the operations performed by the area of the tool (such as a fastening tool or a threading tool) can be controlled.

[0013] The features described in the following paragraphs can be implemented independently or in combination.

[0014] According to one or more embodiments, at least one visual marker is a two-dimensional visual marker. Advantageously, the position of at least one two-dimensional visual marker relative to the image acquisition device can be determined based on the two-dimensional image captured by the image acquisition device. According to one or more embodiments, determining the position of at least one two-dimensional visual marker relative to the image acquisition device can also be based on a three-dimensional image and / or at least one two-dimensional image capable of providing the spatial position and intensity of the visual marker pixels.

[0015] Alternatively, according to one or more embodiments, at least one visual marker is a three-dimensional visual marker. According to one or more embodiments, the at least one three-dimensional visual marker is asymmetric. Advantageously, the asymmetry can avoid ambiguity in position. Thus, according to one or more embodiments, at least one visual marker can be all or part of an asymmetric object. In particular, according to one or more embodiments, the asymmetric object can be obtained by 3D printing. Advantageously, the position of the at least one three-dimensional visual marker relative to the image acquisition device can be determined based on the three-dimensional image captured by the image acquisition device. According to one or more embodiments, the position of the at least one three-dimensional visual marker relative to the image acquisition device can also be determined based on the information of the computer-aided design (CAD) model of the three-dimensional visual marker.

[0016] According to one or more embodiments, the calibration device can include a plurality of visual markers, and the position of the image acquisition device can be determined relative to at least a part of the plurality of visual markers.

[0017] Advantageously, the calibration accuracy of the area of the tool can be improved by determining the position of the image acquisition device relative to at least a part of the plurality of visual markers. Indeed, the position of the image acquisition device can be obtained by using the information of the plurality of visual markers, thereby reducing the uncertainty of the relative position between the coordinate system of the image acquisition device and the coordinate system of the calibration device.

[0018] According to one or more embodiments, the method can include determining a plurality of positions of the at least one visual marker by the image acquisition device, and includes the following steps:

[0019] - For each of the determined multiple positions of the at least one visual marker relative to the image acquisition device, determine an initial position of the region of the tool relative to the image acquisition device based on the position of the visual marker relative to the image acquisition device and based on the position of the fixed point relative to the calibration device.

[0020] - Divide the multiple initial positions of the region of the tool into groups.

[0021] - Determine the size of the group that contains the largest number of initial positions.

[0022] - Based on the determined size of the group, determine the position of the region of the tool relative to the image acquisition device based on the initial positions of the group of the determined size.

[0023] "Dividing" is understood as data partitioning, or "data clustering". The goal of the partitioning is to divide the multiple initial positions of the region of the tool into different groups while minimizing the within-class inertia within the groups and maximizing the between-class inertia between the groups.

[0024] Advantageously, the proposed method can also determine the position of the region of the tool relative to the image acquisition device by utilizing the multiple initial positions, thereby improving the accuracy and precision of the calibration of the region of the tool. In particular, incorrect positions can be excluded from the calibration of the region of the tool, such as positions determined from images captured when the region of the tool is not placed at the fixed point.

[0025] According to one or more embodiments, the multiple positions of the at least one visual marker relative to the image acquisition device may include positions determined based on a portion of the multiple visual markers, the portion including a number of visual markers greater than a threshold.

[0026] Advantageously, the accuracy of the calibration of the region of the tool can be adjusted by changing the threshold.

[0027] According to one or more embodiments, the step of determining the multiple initial positions of the region of the tool relative to the image acquisition device is iteratively performed until the size of the group that contains the largest number of initial positions is greater than a threshold.

[0028] Advantageously, the accuracy of the calibration can be improved by changing the threshold. Thereby, intermediate positions of the region of the tool outside the fixed point can be excluded from the determination of the position of the region of the tool.

[0029] According to one or more embodiments, the calibration device includes at least a partial articulated connection or a spherical gimbal connection between a fixed point located at a known position at the fixed point and the region of the tool, the position of the fixed point relative to the calibration device being known.

[0030] Advantageously, the presence of a part of a hinge connection or a ball joint connection at a fixed point can increase at least one degree of freedom in the relative movement between the image acquisition device and the calibration device, while maintaining the position of the area of the tool relative to the fixed point. Thus, various positions between the image acquisition device and the calibration device can enrich the calibration data, thereby improving the accuracy and precision of the calibration. In addition, the presence of at least a part of the ball joint connection can make the data obtained in the three-dimensional space substantially uniform. In particular, the height of the image acquisition device relative to the calibration device can be changed so that the image acquisition device can acquire images of visual markers that were previously invisible.

[0031] According to one or more embodiments, the method will further include the following steps:

[0032] - Rotating the area of the tool about an axis of the area of the tool through a hinge connection,

[0033] - Determining, by the image acquisition device, a plurality of positions of at least one visual marker of the image acquisition device relative to the calibration device according to the rotation of the area of the tool about the axis,

[0034] - Determining the position of the axis of the area of the tool relative to the image acquisition device based on the plurality of positions of the image acquisition device relative to the visual marker that vary according to the rotation of the area of the tool about the axis.

[0035] The "axis of the area of the tool" is herein understood as an axis along which an attachment attached to the area of the tool can extend. One end of the attachment attached to the area of the tool can thus be located on the axis of the area of the tool. The position where one end of the attachment is fixed to the area of the tool can vary according to the size of the attachment.

[0036] Advantageously, knowing the axis of the area of the tool can limit the space of the allowed positions of the area of the tool (or one end of the attachment attached to the area of the tool) relative to the image acquisition device.

[0037] Alternatively, according to one or more embodiments, determining the axis of the area of the tool can include the following steps:

[0038] - Adding an attachment to the area of the tool,

[0039] - Placing the area of the attachment at a fixed point at a known position,

[0040] - Determining, by the image acquisition device, the position of the visual marker relative to the image acquisition device,

[0041] - Determining the position of the area of the attachment relative to the image acquisition device based on the position of the visual marker relative to the image acquisition device and based on the position of the fixed point relative to the calibration device,

[0042] - Determine the position of the axis of the region of the tool relative to the image acquisition device based on the position of the tool-based region relative to the image acquisition device and the position of the attachment-based region relative to the image acquisition device.

[0043] According to one or more embodiments, to detect a change in a second attachment attached to the region of the tool, the method will further include the following steps:

[0044] - Determine the position of one end of the second attachment relative to the image acquisition device based on the axis of the region of the tool by the image acquisition device.

[0045] Advantageously, automatic detection of changes in the second attachment attached to the region of the tool can thus be achieved. Thus, calibration of the end of the second attachment relative to the image acquisition device can be achieved without repeating all the steps of the method proposed. Thus, calibration of the region of the tool can be extended to all types of attachments attached to the region of the tool.

[0046] According to another aspect, there is provided an image acquisition device for implementing the method according to this description.

[0047] According to another aspect, there is provided a device for calibrating the position of a region of a tool relative to an image acquisition device located on the tool, the device comprising:

[0048] - A fixed point of known position relative to the device,

[0049] - At least one visual marker of known position relative to the device,

[0050] - Optionally, at least a partially articulated connection or a ball and socket joint at the fixed point of known position for receiving the region of the tool.

[0051] According to another aspect, there is provided a computer program comprising instructions which, when executed by a processor, implement all or part of the method as defined herein. According to another aspect, there is provided a non-volatile computer-readable storage medium having such a program stored thereon. Description of the Drawings

[0052] Other features, details and advantages will become apparent from the following detailed description and the analysis of the accompanying drawings, in which:

[0053] Figure 1

[0054] Figure 1 An example of a robotic arm including a tool equipped with an image acquisition device according to one or more embodiments is shown.

[0055] Figure 2

[0056] Figure 2 Shows an example of a hand tool equipped with an image acquisition device according to one or more embodiments.

[0057] Figure 3

[0058] Figure 3 Shows a flowchart of an exemplary implementation of a method according to one or more embodiments.

[0059] Figure 4

[0060] Figure 4 Shows a flowchart of an exemplary implementation of a method according to one or more embodiments.

[0061] Figure 5

[0062] Figure 5 Shows a block diagram of a tool equipped with an image acquisition device according to one or more embodiments.

[0063] Figure 6

[0064] Figure 6 Shows a flowchart of an exemplary implementation of a method according to one or more embodiments.

[0065] Figure 7

[0066] Figure 7 Shows a top view of a calibration device according to one or more embodiments.

[0067] Figure 8

[0068] Figure 8 Shows a calibration device and a tool equipped with an image acquisition device according to one or more embodiments. DETAILED DESCRIPTION

[0069] In the following detailed description of embodiments, many specific details are set forth in order to provide a more thorough understanding. However, those skilled in the art will understand that the embodiments may be practiced without these specific details. In other instances, well-known features have not been described in order to avoid unnecessarily complicating the description.

[0070] This description refers to the flowcharts and block diagrams of methods and apparatuses according to one or more embodiments. Each described flowchart can be implemented by hardware, software (including embedded software (firmware), or middleware), microcode, or any combination thereof. In the case of a software implementation, the flowchart can be implemented by computer program instructions or software code, which can be stored or transmitted on a computer-readable medium, including a non-volatile medium, or a medium loaded into the memory of a general-purpose or special-purpose computer, or any other programmable data processing device or apparatus to process data, thereby creating a machine such that the computer program instructions or software code executed on the computer or programmable data processing device or apparatus constitute means for implementing these functions.

[0071] Examples of computer-readable media include, but are not limited to, computer storage media and communication media, including any medium that facilitates the transfer of a computer program from one location to another. "Computer storage medium / media" refers to a physical medium that can be accessed by a computer. Examples of computer storage media include, but are not limited to, flash memory components or disks or other flash devices (e.g., USB drives, memory sticks, floppy disks), CD-ROMs or other optical data storage devices, DVDs, magnetic data storage devices or other magnetic data storage devices, data storage components, RAM, ROM, EEPROM, smart cards, solid-state drives (SSDs), and any other medium that can be used to transport or store or hold data or data structures readable by a computer processor.

[0072] In addition, various forms of computer-readable media can transmit or carry instructions to a computer via a router, gateway, server, or any data transmission device, whether wired (via coaxial cable, fiber optic, telephone line, DSL cable, or Ethernet cable), wireless (via infrared, radio, cellular, microwave), or virtualized transmission devices (virtual routers, virtual gateways, virtual tunnel endpoints, virtual firewalls). According to an embodiment, the instructions can include the code of any computer programming language or computer program element, such as, but not limited to, assembly language, C, C++, Visual Basic, Hypertext Markup Language (HTML), Extensible Markup Language (XML), Hypertext Transfer Protocol (HTTP), Hypertext Preprocessor (PHP), Structured Query Language (SQL), MySQL, Java, JavaScript, JavaScript Object Notation (JSON), Python, and bash scripts.

[0073] In addition, the terms "in particular", "for example", and "exemplary" are used in this description to specify non-limiting examples or illustrations that do not necessarily correspond to embodiments that are more preferred or more advantageous relative to other possible embodiments or aspects.

[0074] As used in this description, the terms "located on", "mounted on", "equipped with" and their various variants and forms refer to couplings, connections, assemblies that can be direct or indirect. Thus, an image acquisition device mounted on a tool can be located directly on the tool or on an intermediate element between the image acquisition device and the tool.

[0075] Figure 1 As a non-limiting example of tool 100, according to one or more embodiments, a fastening tool or threading tool mounted on a robotic arm RBT is shown. Alternatively, tool 100 can be a machining tool, such as a drilling tool, deburring tool, chamfering tool, polishing tool or any other tool suitable for implementing the proposed method, such as a welding tool. In Figure 1 the example of, the area 101 of the tool to be calibrated corresponds to the head of tool 100, i.e., the point at which tool 100 performs the operation to be carried out. Thus, in Figure 1 the case of the fastening tool shown in, the head of tool 100 can correspond to the end of the sleeve. In the case of a drilling tool, the head of the tool can particularly correspond to one end of the drill bit.

[0076] Furthermore, tool 100 is equipped with an image acquisition device 200. The image acquisition device 200 is configured to acquire images. For example, the image acquisition device 200 includes a two-dimensional (2D) camera or a three-dimensional (3D) camera. According to one or more embodiments, the image acquisition device 200 includes a 2D camera and a 3D camera, so that the information obtained by the two cameras can be utilized. According to one or more embodiments, the 2D camera includes two three-color lenses (red, green, blue, RGB) and is connected to an inertial measurement unit. According to one or more embodiments, the 3D camera is an active stereo camera. In addition, the image acquisition device 200 is operatively coupled to a processor capable of implementing the method described in this description.

[0077] As Figure 1 shown, the area 101 of the tool can be separate from the image acquisition device 200. More specifically, in Figure 1 it can be observed that there is an offset between the center 201 of the image acquisition device 200 and the area 101 of the tool, denoted as T_tcp.

[0078] Alternatively, Figure 2 a portable manual tool is shown as a non-limiting example of tool 100'. More specifically, Figure 2 the example of represents a manual fastening tool, more specifically a ratchet wrench. Alternatively, tool 100' can be a manual tool, such as a manual threading tool, manual drilling tool, manual deburring tool, manual chamfering tool, manual polishing tool, manual welding tool or any other manual tool suitable for implementing the proposed method.

[0079] InFigure 1 Similar to the tool 100 in [reference], the image acquisition device 200' is located on the tool 100'. The offset between the center 201' of the image acquisition device 200' and the region of interest 101' of the tool is denoted as T_tcp'. The region of interest of the tool is, for example, the end of the socket in the case of a ratchet wrench.

[0080] In each of the above cases, it is necessary to know the offsets T_tcp, T_tcp' in order to infer the position of the regions of interest 101, 101' of the tool relative to the image acquisition device 200 from the images acquired by the image acquisition device.

[0081] For this purpose, Figure 3 is a flowchart of an exemplary method 300 for calibrating the region of a tool according to one or more embodiments.

[0082] In a first step PLACE_TOOL 301, the regions of interest 101, 101' of the tool are placed at a fixed point. The fixed point is a point whose position is known relative to the position of the calibration device. Thus, the positions of the fixed point in the calibration device coordinate system, denoted as coordinates T_offset and T_offset' respectively in Figure 6 and Figure 7 are known.

[0083] In addition, the calibration device includes at least one visual marker. The position of the visual marker relative to the calibration device is known. Thus, the position of the fixed point relative to the at least one visual marker is also known. For example, as shown in Figure 7 and 8 and as discussed hereinafter, according to one or more embodiments, the at least one visual marker is a two-dimensional visual marker.

[0084] In a second step DET_POS_MARK / CAM 302, the position of the visual marker relative to the image acquisition devices 200, 200' is determined by the image acquisition devices 200, 200'.

[0085] According to one or more embodiments, the determination 302 of the position of the visual marker relative to the image acquisition devices 200, 200' includes, in a first sub-step, detecting at least one visual marker by the image acquisition devices 200, 200'. In particular, the at least one visual marker can be detected based on the two-dimensional image captured by the image acquisition devices 200, 200'. The two-dimensional image can be a grayscale image. Alternatively, the two-dimensional image can be a color image. According to one or more embodiments, the image acquisition devices 200, 200' detect at least one visual marker based on the two-dimensional image by using a computer vision library (such as the OpenCV library). According to one or more embodiments, the ArUco library (S. Garrido-Jurado, R.) can be used. F.J. Madrid-Cuevas and M.J. Marín-Jiménez. 2014. "Automatic Generation and Detection of Highly Reliable Benchmark Markers under Occlusion". Pattern Recogn. 47, 6 (June 2014), 2280 - 2292. DOI = 10.1016 / j.patcog.2014.01.005).

[0086] According to one or more embodiments, in a second sub-step, the three-dimensional position of the at least one visual marker relative to the image acquisition devices 200, 200' is determined. According to one or more embodiments, the three-dimensional position of the at least one visual marker relative to the image acquisition devices 200, 200' is obtained based on the two-dimensional image. More specifically, according to one or more embodiments, the three-dimensional position of the at least one visual marker relative to the image acquisition devices 200, 200' is obtained by a computer vision library that takes the two-dimensional image as input. According to one or more embodiments, the computer vision library can also take one or more intrinsic parameters of the image acquisition devices 200, 200' as input. For example, the computer vision library is the OpenCV library. More specifically, the ArUco library can be used. Alternatively, according to one or more embodiments, when the image acquisition devices 200, 200' include a three-dimensional camera, the three-dimensional position of the at least one visual marker relative to the image acquisition devices 200, 200' can be directly obtained by the three-dimensional camera.

[0087] According to one or more embodiments, in the third sub-step, a three-dimensional transformation between the coordinate system of the reference frame of the image acquisition devices 200, 200 and the coordinate system of the reference frame of the calibration device is determined. According to one or more embodiments, the three-dimensional transformation between the coordinate system of the reference frame of the image acquisition devices 200, 200 and the coordinate system of the reference frame of the calibration device is determined based on the three-dimensional position of the at least one visual marker relative to the image acquisition devices 200, 200' and based on the three-dimensional position of the at least one visual marker relative to the calibration device. According to one or more embodiments, the three-dimensional position of the at least one visual marker relative to the calibration device is generally known and recorded. According to one or more embodiments, the three-dimensional transformation between the coordinate system of the reference frame of the image acquisition devices 200, 200 and the coordinate system of the reference frame of the calibration device is represented by a matrix of size 3×3, denoted as M T_CAL , such that the three-dimensional position of the at least one visual marker relative to the image acquisition devices 200, 200' and the three-dimensional position of the at least one visual marker relative to the calibration device, denoted as X CAM and X MARK , respectively, can be represented by the following relation [Mathematical formula 1]:

[0088] [Mathematical formula 1]

[0089] X CAM = M T_CAL * X MARK

[0090] Therefore, the three-dimensional transformation between the coordinate system of the calibration reference frame and the coordinate system of the reference frame of the image acquisition devices 200, 200 is also known. In fact, the three-dimensional transformation matrix between the coordinate system of the calibration reference frame and the coordinate system of the reference frame of the image acquisition devices 200, 200 can be represented as the inverse matrix of matrix M T_CAL .

[0091] In the third step DET_POS_TOOL / CAM 303, the positions of the regions 101, 101' of the tool relative to the image acquisition devices 200, 200' are determined, denoted as T_tcp and T_tcp' in Figure 6 and 7 respectively. The positions of the regions 101, 101' of the tool relative to the image acquisition devices 200, 200' (T_tcp, T_tcp') are obtained based on the position of the visual marker relative to the image acquisition devices 200, 200' and the positions of the fixed points relative to the calibration device (T_offset, T_offset'). According to one or more embodiments, the positions of the regions 101, 101' of the tool relative to the image acquisition devices 200, 200' (T_tcp, T_tcp') are obtained according to the following relations [Mathematical formula 2] and [Mathematical formula 2']:

[0092] [Mathematical formula 2]

[0093] T_tcp = M T_CAL *T_offset

[0094] [Mathematical formula 2']

[0095] T_tcp' = M T_CAL *T_offset'

[0096] The above reference Figure 3 The method 300 described above can be iterated. According to one or more embodiments, the image acquisition devices 200, 200' can capture multiple images. Capturing multiple images can be performed at regular time intervals. For example, the capture frequency can be between 10 Hz and 100 Hz, more preferably between 10 Hz and 2 Hz, and even more preferably about 15 Hz.

[0097] According to one or more embodiments, in the case where the positions of the at least one visual marker relative to the image acquisition devices 200, 200' are captured, the Figure 4 calibration method 400 shown can be implemented.

[0098] According to one or more embodiments, in the first step PLACE_TOOL 301', the region of interest 101, 101' of the tool is placed at a fixed point, similar to the Figure 3 step PLACE_TOOL 301 described.

[0099] For one or more of the multiple captured images, and even for each of the multiple captured images, the detection of at least one visual marker can be achieved, similar to the detection of at least one visual marker referred to Figure 3 in the description.

[0100] According to one or more embodiments, in the case where the calibration device includes multiple visual markers, the method 400 can include an optional second step DETECT_MARK 401. The step DETECT_MARK 401 can include: for one or more of the multiple captured images, and even for each of the multiple captured images, the number of visual markers detected in the image can be determined.

[0101] According to one or more embodiments, in the case of implementing the optional step DETECT_MARK 401, the method may include a likewise optional third step MIN_MARK 402. Step MIN_MARK 402 may include: for one or more of the plurality of captured images, and even for each of the plurality of captured images, comparing the number of visual markers determined in step DETECT_MARK 401 with a first threshold σ. The first threshold σ may advantageously be selected between 1 and 20, preferably between 2 and 15, and more preferably approximately 4. According to one or more embodiments, if for an image, the number of visual markers determined in step DETECT_MARK 401 is greater than the first threshold σ, as Figure 4 shown by the OK arrow exiting the diamond 402 in Figure 4 then step 302' may be implemented. Alternatively, if for an image, the number of visual markers determined in step DETECT_MARK 401 is less than the first threshold σ, as shown by the NOK arrow exiting the diamond 402 in

[0102] Figure 3 then step 401 may be implemented on other images of the plurality of captured images. According to one or more embodiments, for one or more of the plurality of captured images, and even for each of the plurality of captured images, the determination 302' of the position of at least one visual marker by the image acquisition devices 200, 200' may be performed as described above for the determination 302 of the position of at least one visual marker by the image acquisition devices 200, 200' in

[0103] For the sake of brevity, it will not be repeated here. In particular, according to one or more embodiments, the determination 302' of the position of at least one visual marker by the image acquisition devices 200, 200' is performed only for images for which the number of visual markers determined in step DETECT_MARK 401 is greater than the above-mentioned first threshold σ. Figure 3

[0104] According to one or more embodiments, for one or more of the plurality of captured images, and even for each of the plurality of captured images, the determination 303' of the position of the region of interest 101, 101' of the tool relative to the image acquisition devices 200, 200' may be performed as described above for the determination 303 of the position of the region of interest 101, 101' of the tool relative to the image acquisition devices 200, 200' inAccording to one or more embodiments, in step CLUST_POS_TOOL / CAM 403, the regions of interest 101, 101' of the tool determined in step 303 can be clustered relative to the position of the image acquisition device. According to one or more embodiments, the k-nearest neighbor (k-NN) algorithm can be used. According to one or more embodiments, the k-nearest neighbor (k-NN) algorithm can use two classes. Alternatively, according to one or more embodiments, the mean shift algorithm can be used. According to one or more embodiments, the clustering can be implemented through a library for machine learning, such as the Scikit-Learn library (Pedregosa et al., 2011, "Machine Learning in Python: Scikit-Learn", JMLR 12, pp. 2825-2830). In particular, according to one or more embodiments, the DBSCAN algorithm can be used. According to one or more embodiments, the maximum distance between two points can be adjusted so that they are considered members of the same group. The selection of the maximum distance between two points can advantageously be between 1 mm and 3 mm, preferably between 1.5 mm and 2.5 mm, and more preferably about 2 mm.

[0105] According to one or more embodiments, an optional step MIN_MAX_CLUST 404 can be implemented. Step MIN_MAX_CLUST 404 can include determining the size of a group selected from the multiple groups determined in step CLUST_POS_TOOL / CAM 403. More specifically, the size of the group containing the regions of interest 101, 101' of the tool with the most relative to the image acquisition device can be determined. The determined size of the group can be compared with a second threshold α. The second threshold α can advantageously be selected between 50 and 250, preferably between 100 and 200, and more preferably about 150. According to one or more embodiments, if the determined size of the group is greater than the second threshold α, as Figure 4 shown by the OK arrow exiting the diamond 404, then step 405 can be implemented. Alternatively, if the determined size of the group is less than the second threshold α, as Figure 4 shown by the NOK arrow exiting the diamond 404, then step 401 can be implemented on the images among the multiple captured images on which steps 401, 402, 302', 303' and 403 were not implemented.

[0106] According to one or more embodiments, in step DET_FINAL_POS_TOOL / CAM 405, based on the positions of the groups of the determined sizes, the "final" positions of regions 101, 101', 101" of the tool are determined. In particular, according to one or more embodiments, the "final" positions of regions 101, 101', 101" of the tool are the average of the positions of the groups of the determined sizes. Alternatively, the "final" positions of regions 101, 101', 101" of the tool can be the median of the positions of the groups of the determined sizes.

[0107] As Figure 5 shown in the schematic diagram of, according to one or more embodiments, an attachment (denoted as ACC) can be attached to region 101" of the tool.

[0108] According to one or more embodiments, the position of the end (denoted as 502) of attachment ACC can be determined according to one of the methods described with reference to Figure 3 and 4 the description.

[0109] Alternatively, according to one or more embodiments, the three-dimensional position of end 502 of attachment ACC can be inferred by image acquisition device 200" relative to image acquisition device 200" based on the knowledge of the axis (denoted as A500) of the region of the tool.

[0110] More specifically, in the two-dimensional image captured by image acquisition device 200" that includes end 502, the two-dimensional position of end 502 relative to image acquisition device 200" can be obtained. Based on this two-dimensional position and based on the knowledge of axis A500 of the region of the tool, the three-dimensional position of end 502 relative to image acquisition device 200" can be obtained.

[0111] For example, according to one or more embodiments, based on the knowledge of axis A500 of the region of the tool, the distance (denoted as d) between the center (denoted as 201") of image acquisition device 200" and the point (denoted as 501) that is coplanar with the plane of the lens of image acquisition device 200", and the plane passes through center 201", can be obtained. The distance d can be referred to as the baseline.

[0112] Point 501 can define the center of a virtual image acquisition device along the direction of axis A500 of the region of the tool. The two-dimensional position of end 502 relative to the virtual image acquisition device can be inferred. In particular, end 502 can be located at the center of the virtual image that the virtual image acquisition device would capture.

[0113] Therefore, based on the distance d between the center 201″ of the image acquisition device 200″ and the center 501 of the virtual image acquisition device, and based on the position of the end 502 relative to the image acquisition device 200″ and the position of the end 502 relative to the virtual image acquisition device, the three-dimensional position of the end 502 relative to the image acquisition device 200″ can be obtained by triangulation. In particular, the distance between the position of the end 502 relative to the image acquisition device 200″ and the position of the end 502 relative to the virtual image acquisition device can be determined, which is called disparity (denoted as δ). According to one or more embodiments, the disparity δ can be measured in pixels between the position of the end 502 relative to the image acquisition device 200″ and the position of the end 502 relative to the virtual image acquisition device. A third coordinate or depth (denoted as z) can be calculated from the disparity δ and the focal length (denoted as f) of the image acquisition device 200″. The depth z can be calculated according to the following formula [Mathematical Formula 3]:

[0114] [Mathematical formula 3]

[0115] z=(d*f) / δ

[0116] Alternatively, according to one or more embodiments, when the image acquisition device 200 ″ includes a 3D camera, the 3D position of the end 502 relative to the image acquisition device 200 ″ can be directly obtained from a 3D image containing the end 502 captured by the image acquisition device 200 ″.

[0117] therefore, Figure 6 is a flow chart of a method 600 for detecting an end of an accessory attached to a region 101 ″ of a tool, the location of which has been determined by one of the methods described above.

[0118] In a first step DETECT_AXIS 601 , the axis A500 of the area 101 ″ of the tool is determined.

[0119] According to one or more embodiments, when the position of the area 101″ of the tool relative to the image acquisition device 200″ is known, for example, determined by one of the methods described above, and when the position of the end of the accessory fixed to the area 101″ of the tool relative to the image acquisition device 200″ is also known, for example, determined by one of the methods described above, the axis A500 of the area 101″ of the tool can be determined as a unique axis passing through these two positions.

[0120] Alternatively, according to one or more embodiments, the calibration device includes a hinge connection between a fixed point P of known position at the fixed point P and a region of the tool, the position of the fixed point P relative to the calibration device being known. According to one or more embodiments, the axis A500 of the region 101” of the tool is determined by rotating the region 101” of the tool about the axis A500 of the region 101” of the tool through the hinge connection. More specifically, a plurality of positions of at least one visual marker of the image acquisition device 200” relative to the calibration device can be acquired, and these positions vary according to the rotation about the axis A500 of the region of the tool. Then, the equation of the axis A500 of the region of the tool relative to the image acquisition device 200” can be determined from the plurality of positions of the image acquisition device 200” relative to at least one visual marker. Indeed, the plurality of positions of the image acquisition device 200” relative to at least one visual marker form an arc in a plane perpendicular to the axis A500. The direction of the axis A500 can be inferred from a known plane perpendicular to the axis A500. The direction of the axis A500 is the same as the normal direction of the plane perpendicular to the axis A500. Therefore, combining a point on the axis A500 (i.e., the fixed point P of known position) and the direction of the axis A500, the axis A500 can be uniquely determined.

[0121] In the second step DET_COORD_PICT 602, the two-dimensional position of the end 502 relative to the image acquisition device 200” is acquired in the two-dimensional image captured by the image acquisition device 200” and containing the end 502.

[0122] According to one or more embodiments, the two-dimensional position of the end 502 relative to the image acquisition device 200” is obtained by using a deep learning algorithm. For example, a convolutional neural network (CNN) can be used. In particular, a region-based convolutional neural network (R-CNN) can be used. In particular, a masked region-based convolutional neural network (Mask R-CNN) can be used. The deep learning algorithm can be trained with training images to detect the end 502. The training images can include real images and / or simulated images. The extended training image set can be obtained by data augmentation from an initial training image set.

[0123] Alternatively, according to one or more embodiments, the two-dimensional position of the end 502 relative to the image acquisition device 200” is obtained by analyzing the light intensity of the pixels along the axis A500 to detect the end 502. For example, edge detection can be used to extract the end 502. In particular, a Canny edge detector can be used to extract the end 502.

[0124] Alternatively, according to one or more embodiments, the two-dimensional position of the end 502 relative to the image acquisition device 200” is obtained by analyzing a plurality of images captured by the image acquisition device 200”. In fact, since the end 502 is fixed relative to the image acquisition device 200”, the position of the end 502 relative to the image acquisition device 200” can be obtained by determining the position of the fixed point in the plurality of images. In particular, according to one or more embodiments, the plurality of images are a video stream captured by the image acquisition device 200”.

[0125] In the third step DET_POS_TOOL / CAM 603, by utilizing the knowledge of the axis A500 of the area of the tool explained in Figure 5 the two-dimensional position of the end 502 relative to the image acquisition device 200” is converted into the three-dimensional position of the end 502 relative to the image acquisition device 200”.

[0126] Figure 7 Fig. shows a top view of a calibration plate as a non-limiting example of the calibration device 700. Alternatively, according to one or more embodiments (not shown), the calibration device is a non-planar three-dimensional object.

[0127] The calibration device 700 includes a fixed point P, the position of which relative to the calibration device is known.

[0128] The depicted calibration device 700 includes a plurality of visual markers, including visual markers 701 and 702, the positions of which relative to the device 700 are known. As Figure 7 shown, the position of the fixed point P relative to the corner of the visual marker 702 is T_offset_702. For clarity, Figure 7 the position of the fixed point P in the coordinate system of each visual marker is not shown. Figure 8 The visual markers shown in

[0129] are the corners of the markers in the OpenCV ArUco library. According to one or more embodiments (not shown), the visual markers can be the corners of the April Tags markers that can be found at the following address: https: / / april.eecs.umich.edu / software / apriltag. Alternatively, according to one or more embodiments (not shown), the calibration device 700 includes a single visual marker the position of which relative to the calibration device 700 is known.

[0130] As Figure 8As shown, the spherical gimbal connection component 800 located at the fixed point P can be a hollow hemisphere. The region 101 of the tool can be equipped with a spherical tip configured to be inserted into the hemisphere during calibration. In this configuration, the region 101 of the tool can coincide with the fixed point P. Alternatively, according to one or more embodiments (not shown), the articulated connection component located at the fixed point P can include two nested cylinders.

[0131] As described above, the region 101 of the tool is manually placed, for example by an operator, at the fixed point P whose position relative to the calibration device 700 is known. The position T_cal_702 of the visual marker 702 (the corner of the marker) relative to the image acquisition device 200 is determined by the image acquisition device 200. The robot RBT can move the region 101 of the tool over the fixed point P via the connection 800. During these movements, the captured visual marker may change, but this does not adversely affect the implementation of the proposed method. Then, the position T_tcp of the region 101 of the tool relative to the image acquisition device 200 can be determined from the position T_cal_702 of the visual marker 702 relative to the image acquisition device 200 and the position T_offset_702 of the fixed point P relative to the visual marker 702.

[0132] According to the selected embodiments, certain actions, operations, events, or functions in each method described herein can be performed in a different order than described herein, or can be added, combined, or can be not performed or not occur. Additionally, according to one or more embodiments, certain actions, operations, or events can occur in parallel rather than sequentially.

[0133] In particular, according to one or more embodiments, the tool can be equipped with multiple image acquisition devices. The number n of the image acquisition devices among the multiple image acquisition devices can advantageously be selected to be between 2 and 4 (including 2 and 4). Then, the above-described calibration method can be implemented for the n image acquisition devices. According to one or more embodiments, the above-described calibration method can be implemented in parallel n times using the same calibration device.

[0134] Although described through a certain number of detailed exemplary embodiments, the proposed calibration method and the apparatus for implementing the method include different variations, modifications, and improvements that are obvious to those skilled in the art, and it is understood that these different variations, modifications, and improvements fall within the scope of the present disclosure defined by the claims. Additionally, the various aspects and features described above can be implemented together, or separately, or replaced with each other, and all combinations and sub - combinations of all aspects and features are within the scope of the present disclosure. Moreover, some of the above - mentioned systems and devices may not include all the modules and functions described in the preferred embodiments.

Claims

1. A method (300) for determining the position of a region (101, 101', 101") of a tool relative to an image acquisition device (200, 200', 200") located on the tool (100, 100', 100"), the method comprising the steps of: - placing (301) the region (101, 101', 101") of the tool at a fixed point (P), the position of which relative to a calibration device (700) is known, the calibration device (700) comprising at least one visual marker (701, 731), - determining (302), by means of the image acquisition device (200, 200', 200"), the position of the visual marker (701, 731) relative to the image acquisition device (200, 200', 200"), - determining (303) the position of the region (101, 101', 101") of the tool relative to the image acquisition device (200, 200', 200") based on the position of the visual marker (701, 731) relative to the image acquisition device (200, 200', 200") and based on the position of the fixed point (P) relative to the calibration device (700).

2. The method according to claim 1, characterized in that the calibration device (700) comprises a plurality of visual markers (701, 731), and wherein the position of the image acquisition device is determined relative to at least a part of the plurality of visual markers (701, 731).

3. The method according to any one of the preceding claims, characterized in that a plurality of positions of the at least one visual marker (701, 731) relative to the image acquisition device (200, 200', 200") are determined by means of the image acquisition device (200, 200', 200"), the method further comprising the steps of: - for each of the determined plurality of positions of the at least one visual marker (701, 731) relative to the image acquisition device (200, 200', 200"), determining an initial position of the region (101, 101', 101") of the tool relative to the image acquisition device (200, 200', 200") based on the position of the visual marker (701, 731) relative to the image acquisition device (200, 200', 200") and based on the position of the fixed point (P) relative to the calibration device (700), - dividing the plurality of initial positions of the region (101, 101', 101") of the tool relative to the image acquisition device (200, 200', 200") into groups, - determining the size of the group containing the largest number of initial positions, - determining the position of the region (101, 101', 101") of the tool relative to the image acquisition device (200, 200', 200") based on the determined size of the group and based on the initial positions of the group of the determined size.

4. The method according to claims 2 and 3, characterized in that The plurality of positions of the at least one visual marker (701, 731) relative to the image acquisition device (200, 200', 200") includes positions determined based on a portion of the plurality of visual markers (701, 731), the portion including a number of visual markers (701, 731) greater than a threshold value.

5. The method according to claim 4, wherein, the step of determining the regions (101, 101', 101") of the tool relative to the plurality of initial positions of the image acquisition device (200, 200', 200") is iteratively performed until the size of the group containing the largest number of initial positions is greater than a threshold value.

6. The method according to any one of the preceding claims, wherein, the calibration device (700) includes at least a partial articulated connection or a spherical gimbal connection (800) between a fixed point (P) located at a known position and the regions (101, 101', 101") of the tool at the fixed point (P), and the position of the fixed point relative to the calibration device (700) is known.

7. The method according to the previous claim, further comprising the following steps: - rotating the regions (101, 101', 101") of the tool about an axis (A500) of the regions (101, 101', 101") by means of the articulated connection, - determining, by means of the image acquisition device (200, 200', 200"), a plurality of positions of the at least one visual marker (701, 731) of the image acquisition device (200, 200', 200") relative to the calibration device (700) according to the rotation of the regions (101, 101', 101") of the tool about the axis (A500), - determining the position of the axis (A500) of the regions (101, 101', 101") of the tool relative to the image acquisition device (200, 200', 200") based on the plurality of positions of the image acquisition device (200, 200', 200") relative to the visual marker (701, 731) that vary according to the rotation of the regions (101, 101', 101") of the tool about the axis (A500).

8. The method according to any one of the preceding claims, further comprising the following steps: - adding an attachment (ACC) to the regions (101, 101', 101") of the tool, - placing the region (502) of the attachment on a fixed point (P) at a known position, - determining, by means of the image acquisition device (200, 200', 200"), the position of the visual marker (701, 731) relative to the image acquisition device (200, 200', 200"), - determining the position of the region (502) of the attachment relative to the image acquisition device (200, 200', 200") based on the position of the visual marker (701, 731) relative to the image acquisition device (200, 200', 200") and based on the position of the fixed point (P) relative to the calibration device (700). - Determine the position of the axis (A500) of the tool-based region (101, 101', 101") relative to the image acquisition device (200, 200', 200") based on the position of the tool-based region (101, 101', 101") relative to the image acquisition device (200, 200', 200") and the position of the attachment-based region (502) relative to the image acquisition device (200, 200', 200").

9. The method according to claim 8, further comprising the following steps to detect a change in a second attachment attached to the tool-based region (101, 101', 101"): - Determine the position of one end of the second attachment relative to the image acquisition device (200, 200', 200") through the image acquisition device (200, 200', 200") based on the axis (A500) of the tool-based region.

10. A computer program, comprising instructions that, when executed by a processor, implement the method according to any one of claims 1 to 9.

11. A non-volatile computer-readable storage medium, having stored thereon a program that, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images