Method for determining the location of a tool's zone - Patent Application 20070122997

Abstract

A method for determining the position of a zone (101) of a tool (100) relative to an image capture device (200) arranged on the tool (100), the method comprising the steps of: - positioning a zone (101) of the tool (100) at a fixed point, the position of which is known relative to a calibration device including at least one visual marker; - determining (302) using the image capture device the position of the visual marker relative to the image capture device; - determining (303) the position of the zone of the tool relative to the image capture device based on the position of the visual marker relative to the image capture device and based on the position of the fixed point relative to the calibration device.

Description

[Technical Field]

[0001] FIELD OF THE DISCLOSURE The present disclosure relates to the field of methods for determining the location of a zone of a tool relative to an image-capturing device and associated equipment located on the tool. [Background technology]

[0002] Determining the position of the tool zone within the defined reference frame can also be referred to as "calibrating the tool zone" and is an essential element in ensuring proper functioning of the system for determining the position of the tool zone.

[0003] In this regard, it is known in the field of robotic arms to determine the position of a zone of a tool located on a robot arm, most often the tip of the robot arm, by repeatedly pointing the zone of the tool at a known position. Alternatively, it is common to use the intersection between a laser beam from a known position and the tip of the robot arm to determine the position of the tip of the robot arm.

[0004] However, existing solutions rely on knowledge of the kinematics of encoders in the links forming the robot arm, and therefore these solutions are not suitable for tools whose kinematics are unknown. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] French Patent Application Publication No. 3069691 [Patent Document 2] US Patent Application Publication No. 2013 / 010081 [Non-patent literature]

[0006] [Non-Patent Document 1] S. Garrido-Jurado, R. Munoz-Salinas, FJ Madrid-Cuevas, and MJ Marin-Jimenez. 2014. "Automatic generation and detection of highly reliable fiducial markers under occlusion". Pattern Recogn. 47, 6 (June 2014), 2280-2292. DOI=10.1016 / j.patcog.2014.01.005 [Non-patent document 2] Pedregosa et al. 2011. “Scikit-learn: Machine Learning in Python,” JMLR 12, pp. 2825-2830 Summary of the Invention [Problem to be solved by the invention]

[0007] The present disclosure aims to improve that situation. [Means for solving the problem]

[0008] A method is provided for determining the location of a zone of a tool relative to an image capture device disposed on the tool, the method comprising the steps of: - positioning a zone of the tool at a fixed point whose position is known relative to a calibration device comprising at least one visual marker; - using an image capture device to determine the position of the visual marker relative to the image capture device; - determining the position of said zone of the tool relative to the image capture device based on the position of the visual marker relative to the image capture device and based on the position of the fixed point relative to the calibration device.

[0009] Advantageously, the proposed method allows for determining the position of the tool zone relative to an image capture device located on the tool without requiring knowledge of the tool kinematics. Therefore, the proposed method is suitable for all types of tools. In particular, the proposed method is suitable for manual tools as well as tools attached to robotic arms. It is noteworthy that the proposed method is less complex than prior art solutions that require mechanical pointing or mechanical sights. Therefore, the proposed method allows for calibrating the tool zone within approximately two minutes, while prior art solutions can take approximately 15 minutes to achieve similar results.

[0010] The proposed method therefore allows for estimating the position of the tool zone when the position of the image capture device is known, thereby allowing for tracking the position of the tool zone based on the position information of the image capture device, and according to one or more embodiments, for controlling the actions performed by the tool zone, e.g., tightening or screwing.

[0011] The features described in the following paragraphs may optionally be implemented independently of each other or in combination with each other.

[0012] According to one or more embodiments, the at least one visual marker is a 2D visual marker. Advantageously, determining the position of the at least one 2D visual marker relative to the image capture device may be based on a 2D image captured by the image capture device. According to one or more embodiments, determining the position of the at least one 2D visual marker relative to the image capture device may further be based on a 3D image and / or at least one 2D image that may provide spatial positions and intensities of pixels of the visual marker.

[0013] Alternatively, according to one or more embodiments, the at least one visual marker is a three-dimensional (3D) visual marker. According to one or more embodiments, the at least one three-dimensional visual marker is asymmetric. Advantageously, the asymmetry allows for avoiding positioning ambiguities. Thus, according to one or more embodiments, the at least one visual marker may be all or part of an asymmetric object. In particular, according to one or more embodiments, the asymmetric object may be obtained by 3D printing. Advantageously, determining the position of the at least one 3D visual marker relative to the image capture device may be based on a 3D image captured by the image capture device. According to one or more embodiments, determining the position of the at least one 3D visual marker relative to the image capture device may further be based on information originating from a computer-aided design (CAD) model of the 3D visual marker.

[0014] According to one or more embodiments, the calibration device may include a plurality of visual markers, and the position of the image capture device may be determined relative to at least a portion of the plurality of visual markers.

[0015] Advantageously, determining the position of the image capture device relative to at least some of the plurality of visual markers allows for increased accuracy in the calibration of the zones of the tool. Indeed, the position of the image capture device can be obtained by utilizing information from several visual markers, while reducing the uncertainty in the relative position between the reference frame of the image capture device and the reference frame of the calibration device.

[0016] According to one or more embodiments, the method includes determining, by an image capture device, a plurality of positions of at least one visual marker relative to the image capture device, and the steps of: - for each of the plurality of determined positions of the at least one visual marker relative to the image capture device, determining a primary position of the zone of the tool relative to the image capture device based on the position of the visual marker relative to the image capture device and the position of the fixed point relative to the calibration device; - partitioning into groups a plurality of primary positions of said zone of the tool relative to the image capture device; - determining the size of the group containing the most prominent position; - determining, as a function of the determined size of the group, the position of said zone of the tool relative to the image capture device based on the main position of the group of determined size.

[0017] "Partitioning" is understood to mean dividing data or "data clustering." Partitioning aims to divide multiple key locations into groups of different key locations while minimizing intra-class inertia within the groups and maximizing inter-class inertia between the groups.

[0018] Advantageously, the proposed method also makes it possible to increase precision and accuracy in calibrating the tool zone by utilizing multiple key positions to determine the position of the tool zone relative to the image capture device, in particular, erroneous positions, e.g., determined from images captured while the tool zone is not positioned on a fixed point, can be excluded from the calibration of the tool zone.

[0019] According to one or more embodiments, the multiple positions of at least one visual marker relative to the image capture device may include positions determined based on a portion of the multiple visual markers, the portion including a number of visual markers greater than a predetermined threshold.

[0020] Advantageously, the calibration accuracy of the tool's zones can be adjusted by varying the threshold value.

[0021] According to one or more embodiments, the step of determining a plurality of key positions of the zone of the tool relative to the image capture device is repeated until the size of the group containing the most key positions is greater than a predetermined threshold.

[0022] Advantageously, by varying the threshold, the accuracy of the calibration can be improved, so that intermediate positions of the tool's zone outside the fixed point can be excluded from determining the position of the tool's zone.

[0023] According to one or more embodiments, the calibration device includes at least a portion of a pivot or ball joint connection at a fixed point having a known position relative to the calibration device between the fixed point having a known position and a zone of the tool.

[0024] Advantageously, the presence of a pivot or ball-joint connection at the fixed point adds at least one degree of freedom to the relative motion between the image capture device and the calibration device while maintaining the position of the tool zone relative to the fixed point. This allows for enriched calibration data at various positions between the image capture device and the calibration device, improving calibration accuracy. Additionally, the presence of at least a portion of the ball-joint connection allows for obtaining substantially uniform data in three-dimensional space. In particular, the height of the image capture device relative to the calibration device can be changed to allow the image capture device to capture images of visual markers that would otherwise be invisible.

[0025] According to one or more embodiments, the method further comprises the steps of: - rotating the tool zone about its axis by means of a pivot connection; - determining, by the image capture device, a plurality of positions of the image capture device relative to at least one visual marker of the calibration device as a function of rotation about an axis of the zone of the tool; - determining an axis of the tool zone relative to the image capture device based on a plurality of positions of the image capture device relative to the visual marker as a function of rotation about the axis of the tool zone.

[0026] "Axis of the zone of the tool" is understood here to mean the axis along which an accessory attached to the zone of the tool can extend. One end of an accessory attached to the zone of the tool can therefore be located on the axis of the zone of the tool. The position of the end of an accessory fixed to the zone of the tool on the axis of the zone of the tool may vary depending on the size of the accessory.

[0027] Advantageously, knowing the axis of the tool zone therefore makes it possible to constrain the space of allowable positions of the tool zone, or one end of an accessory attached to the tool zone, relative to the image capture device.

[0028] Alternatively, according to one or more embodiments, determining the axis of the zone of the tool may include the following steps: - adding accessories to the tool's zone, - placing a zone of the accessory at a fixed point whose position is known; - using an image capture device to determine the position of the visual marker relative to the image capture device; - determining a position of the zone of the accessory relative to the image capture device based on a position of the visual marker relative to the image capture device and based on a position of a fixed point relative to the calibration device; - determining an axis of the tool zone relative to the image capture device based on the position of the tool zone relative to the image capture device and based on the position of the accessory zone relative to the image capture device;

[0029] According to one or more embodiments, to detect a change in a second accessory attached to a zone of the tool, the method further includes the steps of: - determining, by the image capture device, a position of one end of the second accessory relative to the image capture device based on an axis of the zone of the tool.

[0030] Advantageously, automatic detection of changes in the second accessory attached to the tool zone can be achieved in this manner. As a result, calibration of the end of the second accessory relative to the image capture device can be achieved without having to reproduce all steps of the proposed method. In this way, calibration of the tool zone can be generalized to all types of accessories attached to the tool zone.

[0031] According to another aspect, there is provided an image capture device for performing the methods described herein.

[0032] According to another aspect, there is provided an apparatus for calibrating the position of a zone of a tool relative to an image capture device located on the tool, the apparatus including: - a fixed point whose position relative to the device is known, - at least one visual marker whose position relative to the device is known; - Optionally, at least part of a pivot or ball joint connection located at a fixed point for receiving a zone of the tool.

[0033] According to another aspect, there is provided a computer program comprising instructions for performing all or part of the methods defined herein when the program is executed by a processor. According to another aspect, there is provided a non-transitory computer-readable storage medium having stored thereon such a program. [Brief explanation of the drawings]

[0034] [Figure 1] 1 illustrates a robotic arm with a tool equipped with an image capture device according to one or more embodiments. [Figure 2] 1 illustrates a hand tool with an image capture device according to one or more embodiments. [Figure 3] 1 is a flowchart illustrating an exemplary embodiment of a method according to one or more embodiments. [Figure 4] 1 is a flowchart illustrating an exemplary embodiment of a method according to one or more embodiments. [Figure 5] FIG. 1 is a diagram of a tool with an image capture device according to one or more embodiments. [Figure 6] 1 is a flowchart illustrating an exemplary embodiment of a method according to one or more embodiments. [Figure 7] FIG. 1 is a top view of a calibration device according to one or more embodiments. [Figure 8] 1 illustrates a tool with a calibration device and an image capture device according to one or more embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0035] Other features, details and advantages will become apparent from reading the following detailed description and examining the accompanying drawings.

[0036] In the following detailed description of embodiments of the present invention, numerous specific details are presented to provide a more thorough understanding. However, those skilled in the art will recognize that the embodiments may be practiced without these specific details. In other instances, well-known features are not described to avoid unnecessarily complicating the description.

[0037] This description refers to illustrations of flowcharts and diagrams of methods and apparatus according to one or more embodiments. Each of the described flowcharts can be implemented by hardware, software (including embedded software (firmware) or middleware), microcode, or any combination thereof. In the case of a software implementation, the flowchart diagrams can be implemented by computer program instructions or software code, which can be stored or transmitted on a computer-readable medium, including a non-transitory medium, or a medium loaded into the memory of a general-purpose or specific computer, or any other programmable data processing apparatus or device for processing data to create a machine, such that the computer program instructions or software code executed on a computer or programmable data processing apparatus or device constitutes means for performing these functions.

[0038] Embodiments of computer-readable media include, but are not limited to, computer storage media and communication media, including any medium that facilitates transfer of a computer program from one place to another. A "computer storage medium / media" refers to any physical medium that can be accessed by a computer. Examples of computer storage media include, but are not limited to, flash memory components or disks, or any other flash memory devices (e.g., USB keys, memory keys, memory sticks, key disks), CD-ROMs or other optical data storage devices, DVDs, magnetic disk data storage devices or other magnetic data storage devices, data memory components, RAM, ROM, EEPROM, smart cards, SSD ("solid state drive") memory, and any other form of medium that can be used to carry, store, or preserve data or data structures that can be read by a computer processor.

[0039] Additionally, various forms of computer-readable media may transmit or carry instructions to a computer, such as a router, gateway, server, or any data transmission device, whether wired (via coaxial cable, fiber optic, phone line, DSL cable, or Ethernet cable), wireless (via infrared, radio, cellular, microwave), or virtualized transmission device (virtual router, virtual gateway, virtual tunnel endpoint, virtual firewall). The instructions may, depending on the embodiment, include code in any computer programming language or computer program element, such as, but not limited to, assembly language, C, C++, Visual Basic, Hypertext Markup Language (HTML), Extensible Markup Language (XML), Hypertext Transfer Protocol (HTTP), Hypertext Preprocessor (PHP), SQL, MySQL®, Java, JavaScript, Javascript Object Notation (JSON), Python, and bash script.

[0040] Furthermore, the terms "particularly," "for example," and "exemplary" are used herein to indicate non-limiting examples or descriptions of embodiments that do not necessarily correspond to embodiments that are preferred or advantageous over other possible embodiments or aspects.

[0041] As used herein, the terms "disposed," "attached," "comprising," and various variations and forms thereof refer to coupling, connection, or assembly, which may be direct or indirect. Thus, an image capture device disposed / mounted on a tool may be disposed directly on the tool or may be disposed on an intermediate element located between the image capture device and the tool.

[0042] FIG. 1 illustrates, as a non-limiting example of a tool 100, a tightening or screwing tool attached to a robot arm RBT according to one or more embodiments. Alternatively, the tool 100 can be one of a machining tool, such as a drilling tool, a deburring tool, a chamfering tool, a grinding tool, or any other tool (e.g., a welding tool) suitable for implementing the proposed method. In the example of FIG. 1, the zone 101 of the tool to be calibrated corresponds to the head of the tool 100 and refers to the point on the tool 100 where the operation to be performed is performed. Thus, in the case of a tightening tool shown in FIG. 1, the head of the tool 100 can correspond to the end of a socket. In the case of a drilling tool, the head of the tool can correspond, in particular, to one end of a drill bit.

[0043] The tool 100 also includes an image capture device 200. The image capture device 200 is configured to acquire images. The image capture device 200 may, for example, be a two-dimensional (2D) camera or a three-dimensional (3D) camera. According to one or more embodiments, the image capture device 200 includes a 2D camera and a 3D camera, allowing for utilizing information obtained by the two cameras. According to one or more embodiments, the 2D camera includes two three-color lenses (red, green, blue, RGB) coupled to an inertial measurement unit. According to one or more embodiments, the 3D camera is an active stereo camera. Furthermore, the image capture device 200 is operably coupled to a processor that enables the implementation of the methods described herein.

[0044] As shown in Figure 1, the tool zone 101 may be spaced apart from the image capture device 200. More specifically, in Figure 1, an offset T_tcp between the center 201 of the image capture device 200 and the tool zone 101 can be observed.

[0045] Alternatively, Fig. 2 shows a handheld manual tool as a non-limiting example of tool 100'. More specifically, the example of Fig. 2 represents a manual tightening tool, more specifically a ratchet. Alternatively, tool 100' may be a manual tool such as a manual threading tool, a manual drilling tool, a manual deburring tool, a manual chamfering tool, a manual polishing tool, a manual welding tool, or any other manual tool suitable for carrying out the proposed method.

[0046] Similar to tool 100 of Figure 1, image capture device 200' is positioned on tool 100'. The offset between center 201' of image capture device 200' and zone of interest 101' of the tool is denoted as T_tcp'. The zone of interest of the tool is, for example, the end of the socket in the case of a ratchet.

[0047] In each of the above cases, knowing the offsets T_tcp, T_tcp', it is required to infer the location of the zones of interest 101, 101' relative to the image capture device 200 from the images acquired by the image capture device.

[0048] To this end, FIG. 3 is a flowchart illustrating a proposed method 300 for calibrating zones of a tool, according to one or more embodiments.

[0049] In the first step 301 (PLACE_TOOL), the tool's zone of interest 101, 101' is placed at a fixed point. A fixed point is a point whose position relative to the calibration device is known. Therefore, the coordinates of the fixed point in the reference frame of the calibration device are known, denoted T_offset and T_offset' in Figures 6 and 7, respectively.

[0050] The calibration device further includes at least one visual marker. The position of the visual marker relative to the calibration device is known. Therefore, the position of the fixation point relative to the at least one visual marker is also known. According to one or more embodiments, the at least one visual marker is a two-dimensional (2D) visual marker, as shown in, for example, Figures 7 and 8 and as described below.

[0051] In a second step 302 (DET_POS_MARK / CAM), the position of the visual marker relative to the image capture device 200, 200' is determined by the image capture device 200, 200'.

[0052] According to one or more embodiments, determining 302 the position of the visual marker relative to the image capture device 200, 200′ comprises, in a first sub-step, detecting at least one visual marker by the image capture device 200, 200′. In particular, detecting at least one visual marker by the image capture device 200, 200′ may be performed based on a 2D image acquired by the image capture device 200, 200′. The 2D image may be a grayscale image. Alternatively, the 2D image may be a color image. According to one or more embodiments, detecting at least one visual marker by the image capture device 200, 200′ based on the 2D image is performed by a computer vision library, for example the OpenCV library. According to one or more embodiments, the ArUco library can be used (Non-Patent Document 1).

[0053] According to one or more embodiments, in a second substep, a 3D position of the at least one visual marker relative to the image capture device 200, 200′ is determined. According to one or more embodiments, the 3D position of the at least one visual marker relative to the image capture device 200, 200′ is obtained based on a 2D image. More particularly, according to one or more embodiments, the 3D position of the at least one visual marker relative to the image capture device 200, 200′ is obtained by a computer vision library that receives the 2D image as input. According to one or more embodiments, the computer vision library may also take one or more parameters specific to the image capture device 200, 200′ as input. For example, the computer vision library is the OpenCV library. More specifically, the ArUco library may be used. Alternatively, according to one or more embodiments, if the image capture device 200, 200′ is equipped with a 3D camera, the 3D position of the at least one visual marker relative to the image capture device 200, 200′ may be obtained directly by the 3D camera.

[0054] According to one or more embodiments, in a third sub-step, a 3D transformation between the coordinate system of the reference frame of the image capture device 200, 200′ and the coordinate system of the reference frame of the calibration device is determined. According to one or more embodiments, the 3D transformation between the coordinate system of the reference frame of the image capture device 200, 200′ and the coordinate system of the reference frame of the calibration device is determined based on the 3D position of the at least one visual marker relative to the image capture device 200, 200′ and the 3D position of the at least one visual marker relative to the calibration device. According to one or more embodiments, the 3D position of the at least one visual marker relative to the calibration device is in principle known and recorded. According to one or more embodiments, the 3D transformation between the coordinate system of the reference frame of the image capture device 200, 200′ and the coordinate system of the reference frame of the calibration device is determined based on a matrix (M T_CAL) and the 3D position of the at least one visual marker relative to the image capture device 200, 200′ and the 3D position of the at least one visual marker relative to the calibration device (denoted by X and X, respectively). CAM and X MARK ) can be expressed by the following relation [Equation 1]:

[0055] [Number 1] X CAM =M T_CAL *X MARK

[0056] As a result, the 3D transformation between the coordinate system of the reference frame of the calibration device and the coordinate system of the reference frame of the image capture device 200, 200 is also known. In fact, the matrix of the 3D transformation between the coordinate system of the reference frame of the calibration device and the coordinate system of the reference frame of the image capture device 200, 200 can be expressed as the inverse matrix of the matrix MT_CAL.

[0057] In a third step 303 (DET_POS_TOOL / CAM), the position of the zone of interest of the tool 100, 100' relative to the image capture device 200, 200' is determined, denoted by T_tcp, T_tcp' in Figures 6 and 7, respectively. The position T_tcp, T_tcp' of the tool's zone 101, 101' relative to the image capture device 200, 200' is obtained based on the position of the visual marker relative to the image capture device 200, 200' and the position T_offset, T_offset' of the fixed point relative to the calibration device. According to one or more embodiments, the position T_tcp, T_tcp' of the tool's zone 101, 101' relative to the image capture device 200, 200' is obtained according to the following respective relationships [Equation 2], [Equation 2']:

[0058] [Number 2] T_tcp = MT_CAL * T_offset.

[0059] [Number 2'] T_tcp'=MT_CAL*T_offset'

[0060] The method 300 described above with reference to FIG. 3 can be repeated. According to one or more embodiments, the image capture device 200, 200′ acquires multiple images. The acquisition of the multiple images may occur at regular time intervals. For example, the acquisition frequency can be between 10 Hz and 100 Hz, more preferably between 10 Hz and 2 Hz, and even more preferably about 15 Hz.

[0061] According to one or more embodiments, when multiple positions of at least one visual marker relative to the image capture device 200, 200' are obtained, a calibration method 400 shown in FIG. 4 may be performed.

[0062] According to one or more embodiments, in a first step 301′ (PLACE_TOOL), the zone of interest 101, 101′ of the tool is placed at a fixed point, similar to step 301 (PLACE_TOOL) described with reference to FIG.

[0063] Detection of at least one visual marker similar to the detection of at least one visual marker described with reference to FIG. 3 may be performed for one or more of the acquired plurality of images, or even for each image of the acquired plurality of images.

[0064] According to one or more embodiments, if the calibration device includes multiple visual markers, method 400 may include an optional second step 401 (DETECT_MARK), which may include, for one or more images in the multiple acquired images, or even for each image in the multiple acquired images, determining the number of visual markers detected in the image.

[0065] According to one or more embodiments, if optional step 401 (DETECT_MARK) is performed, the method may include a third step 402 (MIN_MARK), which is also optional. Step 402 (MIN_MARK) may include comparing the number of visual markers determined in step 401 (DETECT_MARK) for one or more of the plurality of acquired images, or even for each image of the plurality of acquired images, with a first threshold value σ. The first threshold value σ may advantageously be selected to be between 1 and 20, preferably between 2 and 15, and more preferably about 4. According to one or more embodiments, if the number of visual markers determined in step 401 (DETECT_MARK) for an image is greater than the first threshold value σ, as represented by an OK arrow exiting diamond 402 in FIG. 4, step 302′ may be performed. Alternatively, if the number of visual markers determined in step 401 (DETECT_MARK) for an image is less than a first threshold σ, as represented by the NOK arrow exiting diamond 402 in FIG. 4, step 401 may be performed for another image of the plurality of acquired images.

[0066] According to one or more embodiments, the determination 302' of the position of the at least one visual marker by the image capture device 200, 200' for one or more images of the plurality of acquired images, or even for each image of the plurality of acquired images, may be performed according to one or more embodiments described above in relation to Figure 3 for the determination 302 of the position of the at least one visual marker by the image capture device 200, 200', the description of which will not be repeated here for the sake of brevity in this description. In particular, according to one or more embodiments, the determination 302' of the position of the at least one visual marker by the image capture device 200, 200' is performed only for images in which the number of visual markers determined in step 401 (DETECT_MARK) is greater than the first threshold σ, as described above.

[0067] According to one or more embodiments, for one or more images of the plurality of acquired images, or even for each image of the plurality of acquired images, the determination 303' of the position of the zone of interest of the tool 100, 100' relative to the image capture device 200, 200' may be performed according to one or more embodiments described above in relation to FIG. 3 for the determination 303 of the position of the zone of interest of the tool 100, 100' relative to the image capture device 200, 200', the description of which will not be repeated here for the sake of brevity in this description.

[0068] According to one or more embodiments, in step 403 (CLUST_POS_TOOL / CAM), the positions of the zones of interest of the tool 100, 100' relative to the image capture device 200, 200' determined in step 303 may be divided into groups ("clustering"). According to one or more embodiments, a k-nearest neighbors (k-NN) algorithm may be used. According to one or more embodiments, a k-nearest neighbors algorithm may be used with two classes. Alternatively, according to one or more embodiments, a mean shift algorithm may be used. According to one or more embodiments, the division may be achieved by a library intended for machine learning, for example the Scikit-Learn library (Non-Patent Document 2). In particular, according to one or more embodiments, a DBSCAN algorithm may be used. According to one or more embodiments, the maximum distance between two points that can be considered to be in the same group may be adjusted. The maximum distance between two points that allows them to be considered to be in the same group can advantageously be chosen to be between 1 mm and 3 mm, preferably between 1.5 mm and 2.5 mm, more preferably about 2 mm.

[0069] According to one or more embodiments, optional step 404 (MIN_MAX_CLUST) can be performed. Step 404 (MIN_MAX_CLUST) can include determining a group size from among the groups determined in step 403 (CLUST_POS_TOOL / CAM). More specifically, a group size can be determined that includes most positions of the zone of interest of the tool 100, 100′ relative to the image capture device 200, 200′. The determined group size can be compared with a second threshold α. The second threshold α can advantageously be selected to be between 50 and 250, preferably between 100 and 200, and more preferably approximately 150. According to one or more embodiments, if the determined size of the group is greater than the second threshold α, as represented by an OK arrow exiting diamond 404 in FIG. 4, step 405 can be performed. Alternatively, as represented by the NOK arrow exiting diamond 404 in FIG. 4, if the determined group size is less than the second threshold α, step 401 may be performed for those images of the plurality of acquired images for which none of steps 401, 402, 302′, 303′ and 403 have been performed.

[0070] According to one or more embodiments, in step 405 (DET_FINAL_POS_TOOL / CAM), a so-called "final" position of the zone of the tool 100, 100', 100" is determined based on the positions of the group of determined size. In particular, according to one or more embodiments, the "final" position of the zone of the tool 100, 100', 100" is the average of the positions of the group of determined size. Alternatively, the "final" position of the zone of the tool 100, 100', 100" may be the median of the positions of the group of determined size.

[0071] As shown schematically in FIG. 5, according to one or more embodiments, accessories, designated ACC, may be attached to a zone of the tool 100″.

[0072] According to one or more embodiments, the position of the end 502 of the accessory ACC may be determined according to one of the methods described with reference to FIGS.

[0073] Alternatively, according to one or more embodiments, the 3D position of the end 502 of the accessory ACC relative to the image capture device 200″ may be estimated by the image capture device 200″ based on knowledge of the axes of the tool zone indicated by A500.

[0074] More specifically, in a 2D image captured by image capture device 200" and comprising end portion 502, the 2D position of end portion 502 relative to image capture device 200" can be obtained. The 3D position of end portion 502 relative to image capture device 200" can be obtained based on the 2D position and based on knowing the axis A 500 of the tool's zone.

[0075] For example, according to one or more embodiments, based on knowing the axis A 500 of the tool's zone, it is possible to obtain the distance d between the center 201" of the image capture device 200" and a point 501 that is coplanar with the plane of the lens of the image capture device 200" and that passes through the center 201". The distance d can be expressed in terms of a baseline.

[0076] Point 501 can define the center of a virtual image capture device oriented along tool zone axis A 500. The 2D position of edge 502 relative to the virtual image capture device can then be estimated. In particular, edge 502 can be located at the center of a virtual image captured by the virtual image capture device.

[0077] Therefore, based on the distance d between the center 201" of the image capture device 200" and the center 501 of the virtual image capture device, and the position of the edge 502 relative to the image capture device 200" and the virtual image capture device, the 3D position of the edge 502 relative to the image capture device 200" can be obtained by triangulation. In particular, a distance designated in terms of disparity, denoted δ, can be determined between the position of the edge 502 relative to the image capture device 200" and the position of the edge 502 relative to the virtual image capture device. According to one or more embodiments, the disparity δ may be measured in pixels between the position of the edge 502 relative to the image capture device 200" and the position of the edge 502 relative to the virtual image capture device. A third coordinate, i.e., the depth (denoted z), can be calculated from the disparity δ and the focal length (denoted f) of the image capture device 200". The depth z can specifically be calculated by the following relationship [Equation 3]:

[0078] [Number 3] z=(d*f) / δ

[0079] Alternatively, according to one or more embodiments, if the image capture device 200″ comprises a 3D camera, the 3D position of the end 502 relative to the image capture device 200″ may be obtained directly from a 3D image captured by the image capture device 200″ and comprising the end 502.

[0080] Accordingly, FIG. 6 is a flowchart illustrating a method 600 for detecting the end of an accessory attached to a zone of a tool 100″, the location of which has been determined by one of the methods presented above.

[0081] In a first step 601 (DETECT_AXIS), the axis A 500 of the zone of the tool 100'' is determined.

[0082] According to one or more embodiments, if the position of a zone of tool 100" relative to image capture device 200" is known, for example, as determined by one of the methods presented above, and if the position of the end of an accessory fixed to the zone of tool 100" relative to image capture device 200" is also known, for example, as determined by one of the methods presented above, axis A500 of the zone of tool 100" may be determined as a single axis passing through these two positions, for example, as determined by one of the methods presented above.

[0083] Alternatively, according to one or more embodiments, the calibration device includes a pivot connection between a fixed point P having a known position relative to the calibration device and the tool zone at a fixed point P having a known position relative to the calibration device. According to one or more embodiments, the axis A500 of the tool 100″ zone is determined by performing a rotation about the axis A500 of the tool 100″ zone by the pivot connection. More specifically, multiple positions of the image capture device 200″ relative to the at least one visual marker of the calibration device can be obtained as a function of rotation about the tool zone axis A500. Then, from the multiple positions of the image capture device 200″ relative to the at least one visual marker, an equation of the tool zone axis A500 relative to the image capture device 200″ can be determined. In effect, the multiple positions of the image capture device 200″ relative to the at least one visual marker form an arc of a circle in a plane perpendicular to the axis A500. Thus, the direction of the axis A500 can be inferred from knowing the plane perpendicular to the axis A500. The direction of axis A500 is perpendicular to a plane perpendicular to axis A500. Therefore, axis A500 can be uniquely determined by knowing the point on axis A500 located at fixed point P, the position of which is known, and the direction of axis A500.

[0084] In a second step 602 (DET_COORD_PICT), in a 2D image taken by the image capture device 200'' and including the edge 502, the 2D position of the edge 502 relative to the image capture device 200'' is obtained.

[0085] According to one or more embodiments, the 2D position of the edge 502 relative to the image capture device 200" is obtained by a deep learning algorithm. For example, a convolutional neural network (CNN) can be used. In particular, a regional convolutional neural network (R-CNN) can be used. In particular, a mask regional convolutional neural network (Mask R-CNN) can be used. The deep learning algorithm may be trained using training images to detect the edge 502. The training images may include real images and / or simulated images. Data augmentation can be performed from an initial set of training images to obtain an augmented set of training images.

[0086] Alternatively, according to one or more embodiments, to detect the edge 502, the 2D position of the edge 502 relative to the image capture device 200″ is obtained by analyzing the light intensity of the pixels along the axis A 500. For example, edge detection can be used to extract the edge 502. In particular, a Canny edge detector can be used to extract the edge 502.

[0087] Alternatively, according to one or more embodiments, the 2D position of the end 502 relative to the image capture device 200" is obtained by analyzing multiple images captured by the image capture device 200". Indeed, since the end 502 is fixed relative to the image capture device 200", the position of the end 502 relative to the image capture device 200" can be obtained by determining the position of a fixed point within the multiple images. In particular, according to one or more embodiments, the multiple images are obtained from a video stream captured by the image capture device 200".

[0088] In a third step 603 (DET_POS_TOOL / CAM), the 2D position of the end 502 relative to the image capture device 200'' is transformed into a 3D position of the end 502 relative to the image capture device 200'' by utilizing knowledge of the tool's zone axis A 500, as described with reference to FIG.

[0089] 7 shows a top view of a calibration board as a non-limiting example of a calibration device 700. Alternatively, according to one or more embodiments (not shown), the calibration device is a non-planar 3D object.

[0090] Calibration device 700 includes a fixed point P whose position relative to the calibration device is known.

[0091] The illustrated calibration device 700 includes multiple visual markers, including visual markers designated 701 and 702, whose positions relative to the device 700 are known. As shown in FIG. 7 , a fixed point P has a position T_offset_ 702 relative to a corner of the visual marker 702. For clarity, the location of the fixed point P within each visual marker's frame of reference is not shown in FIG. 7 . The visual markers shown in FIG. 8 are corners of markers from the OpenCV ArUco library. According to one or more embodiments (not shown), the visual markers may be corners of April Tags markers, which can be found at the address "https: / / april.eecs.umich.edu / software / apriltag." Alternatively, according to one or more embodiments (not shown), the calibration device 700 includes a single visual marker whose position relative to the calibration device 700 is known.

[0092] Optionally, according to one or more embodiments, the calibration device 700 includes a portion of a pivot or ball joint connection located at a fixed point P to receive a zone of the tool 100, 100', 100".

[0093] 8, the portion of the ball joint connection 800 located at the fixed point P may be a hollow hemisphere. The zone of the tool 100 may comprise a spherical tip configured to fit into the hemisphere during calibration. In this configuration, the zone of the tool 100 may therefore coincide with the fixed point P. Alternatively, according to one or more embodiments (not shown), the portion of the pivot connection located at the fixed point P may comprise two nested cylinders.

[0094] As described above, the zone of the tool 100 is manually positioned, for example by an operator, on a fixed point P whose position relative to the calibration device 700 is known. The position T_cal_702 of the visual marker 702 relative to the image capture device 200, which is the corner of the marker, is determined by the image capture device 200. The robot RBT can generate movements of the zone of the tool 100 on the fixed point P by means of the connection 800. During these movements, the acquired visual markers may change without adversely affecting the execution of the proposed method. The position T_tcp of the zone of the tool 100 relative to the image capture device 702 may then be determined from the position T_cal_702 of the visual marker 702 relative to the image capture device 200 and from the position T_offset_702 of the fixed point P relative to the visual marker 200.

[0095] Depending on the embodiment selected, certain operations, actions, events, or functions of each method described herein may be performed or occur in an order different from the order in which they are described, or may be added, merged, not performed, or not occur, as the case may be. Furthermore, in certain embodiments, certain operations, actions, or events may be performed or occur simultaneously rather than sequentially.

[0096] In particular, according to one or more embodiments, the tool may comprise a plurality of image capture devices. The number n of image capture devices in the plurality of image capture devices may advantageously be selected to be greater than or equal to 2 and less than or equal to 4. The calibration method described above may then be performed for the n image capture devices. Advantageously, according to one or more embodiments, the calibration method described above may be performed n times in parallel using the same calibration device.

[0097] Although described through several detailed exemplary embodiments, it is understood that the proposed calibration method and apparatus for implementing embodiments of the method include various variations, modifications, and improvements that will be apparent to those skilled in the art, and that these variations, modifications, and improvements are within the scope of the present disclosure, as defined by the following claims. Furthermore, the various aspects and features described above may be implemented together or separately, or substituted for one another, and all various combinations and subcombinations of these aspects and features are within the scope of the present disclosure. Furthermore, some of the systems and apparatus described above may not incorporate all of the modules and functionality described with respect to the preferred embodiments.

Claims

1. 1. A method (300) for determining the position of a zone (101, 101', 101") of a tool (100, 100', 100") relative to an image capture device (200, 200', 200") disposed on the tool, comprising: a step (301) of positioning said zone (101, 101', 101") of said tool at a fixed point (P) whose position is known relative to a calibration device (700) comprising at least one visual marker (701, 731); - determining (302) using said image capture device (200, 200', 200") the position of said visual marker (701, 731) relative to said image capture device (200, 200', 200"); - determining (303) the position of the zone (101, 101', 101") of the tool relative to the image capture device (200, 200', 200") based on the position of the visual marker (701, 731) relative to the image capture device (200, 200', 200") and based on the position of the fixed point (P) relative to the calibration device (700); A method comprising:

2. The method of claim 1 , wherein the calibration device (700) includes a plurality of visual markers (701, 731), and the position of the image capture device is determined relative to at least a portion of the plurality of visual markers (701, 731).

3. a plurality of positions of the at least one visual marker (701, 731) relative to the image capture device (200, 200', 200") are determined by the image capture device (200, 200', 200"); - for each of the determined positions of the at least one visual marker (701, 731) relative to the image capture device (200, 200', 200"), determining a primary position of the zone (101, 101', 101") of the tool relative to the image capture device (200, 200', 200") based on the position of the visual marker (701, 731) relative to the image capture device (200, 200', 200") and the position of the fixed point (P) relative to the calibration device (700); - dividing into groups a number of primary positions of said zones (101, 101', 101") of said tool relative to said image capture device (200, 200', 200"); - determining the size of the group containing the most dominant positions; - determining, as a function of the determined size of the group, the position of the zone (101, 101', 101") of the tool relative to the image capture device (200, 200', 200") based on the main position of the group of the determined size; The method of claim 1 or 2, further comprising:

4. 4. The method of claim 2 or 3, wherein the plurality of positions of the at least one visual marker (701, 731) relative to the image capture device (200, 200′, 200″) includes the positions determined based on a portion of the plurality of visual markers (701, 731), the portion including a number of visual markers (701, 731) greater than a predetermined threshold.

5. 5. The method of claim 4, wherein the step of determining the plurality of primary positions of the zone (101, 101', 101") of the tool relative to the image capture device (200, 200', 200") is repeated until the size of the group containing most primary positions is greater than a predetermined threshold.

6. 6. The method according to claim 1, wherein the calibration device (700) comprises, at the fixed point (P) whose position is known relative to the calibration device (700), at least a part of a pivot or ball joint connection (800) between the fixed point (P) whose position is known and the zone (101, 101', 101") of the tool.

7. - rotating said zone (101, 101', 101") of said tool by said pivot connection around the axis (A500) of said zone (101, 101', 101") of said tool; - determining, by said image capture device (200, 200', 200"), a plurality of positions of said image capture device (200, 200', 200") relative to said at least one visual marker (701, 731) of said calibration device (700) as a function of rotation of said zone (101, 101', 101") of said tool about said axis (A500); determining the axis (A500) of the zone (101, 101', 101") of the tool relative to the image capture device (200, 200', 200") based on the multiple positions of the image capture device (200, 200', 200") relative to the visual marker (701, 731) as a function of rotation about the axis (A500) of the zone (101, 101', 101") of the tool; The method of any one of claims 1 to 6, further comprising:

8. - adding an accessory (ACC) to said zone (101, 101', 101") of said tool; - placing a zone (502) of said accessory at said fixed point (P) whose position is known; - using said image capture device (200, 200', 200"), determining the position of said visual marker (701, 731) relative to said image capture device (200, 200', 200"); - determining the position of the zone (502) of the accessory relative to the image capture device (200, 200', 200") based on the position of the visual marker (701, 731) relative to the image capture device (200, 200', 200") and based on the position of the fixed point (P) relative to the calibration device (700); - determining an axis (A500) of the zone (101, 101', 101") of the tool relative to the image capture device (200, 200', 200") based on the position of the zone (101, 101', 101") of the tool relative to the image capture device (200, 200', 200") and based on the position of the zone (502) of the accessory relative to the image capture device (200, 200', 200"); The method of any one of claims 1 to 7, further comprising:

9. To detect a change in a second accessory attached to said zone (101, 101', 101") of said tool, - determining, by said image capture device (200, 200', 200"), the position of one end of said second accessory relative to said image capture device (200, 200', 200"), based on the axis (A500) of said zone of said tool; The method of claim 8 further comprising:

10. A computer program comprising instructions for carrying out the method according to any one of claims 1 to 9.

11. A computer-readable storage medium having stored thereon a program for performing the method of any one of claims 1 to 9 when executed by a processor.

Images