Method for determining a position of a zone of a tool

Abstract

A method for determining a position of a zone (101) of a tool (100) with respect to an image-capturing device (200) located on the tool (100), the method comprising steps of: placing the zone (101) of the tool (100) on a fixed point the location of which with respect to a calibration device is known, said calibration device comprising at least one visual reference; determining, via the image-capturing device (200), the position of the visual reference with respect to the image-capturing device (200); and determining the position of said zone (101) of the tool (100) with respect to the image-capturing device (200), based on the position of the visual reference with respect to the image-capturing device (200) and on the location of the fixed point with respect to the calibration device.

Description

Description Title: Method for determining the position of a tool zone Technical field [1] The present disclosure relates to the field of methods for determining a position of an area of ​​a tool relative to an image-taking device located on the tool and associated devices. Prior art [2] Determining the position of a tool area, which can also be referred to as "tool area calibration", in a defined reference frame is an essential element in ensuring the proper functioning of a tool area location system. [3] In this regard, in the field of robotic arms, it is known to determine the position of the area of ​​a tool located on the robotic arm, most often the tip of the robotic arm, by pointing the area of ​​the tool repeatedly towards a known location. Alternatively, in order to determine the position of the tip of the robotic arm, it is common to exploit the crossing between laser beams from known positions and the tip of the robotic arm. [4] However, pre-existing solutions rely on knowledge about the kinematics of encoders in the links forming the robotic arm. These solutions are therefore not suitable for tools for which the kinematics of the tool are not known. Summary [5] This disclosure improves the situation. [6] A method is provided for determining a position of an area of ​​a tool relative to an image-taking device located on the tool, the method comprising the steps of: placing the area of ​​the tool on a fixed point of known location relative to a calibration device, said calibration device comprising at least one visual reference mark, determining, via the image-taking device, the position of the visual reference mark relative to the image-taking device, determining the position of said area of ​​the tool relative to the image-taking device, from the position of the visual reference mark relative to the image-taking device and the location of the fixed point relative to the calibration device. [7] Advantageously, the proposed method makes it possible to determine the position of the tool area relative to the image-taking device located on the tool without requiring knowledge of the kinematics of the tool. Therefore, the proposed method is suitable for all types of tools. In particular, the proposed method is suitable for tools mounted on robotic arms as well as for manual tools. Notably, the proposed method proves to be less complex than the solutions of the prior art requiring mechanical pointing or a mechanical sight. The proposed method thus allows calibration of the tool area in a time of the order of two minutes, while the solutions of the prior art sometimes reach durations of the order of fifteen minutes to obtain a similar result. [8] Thus, thanks to the proposed method, it is made possible, when the position of the image-taking device is known, to deduce therefrom the position of the tool area. As a result, tracking of the position of the tool area can be implemented from information concerning the position of the image-taking device. Also, according to one or more embodiments, operations carried out by the tool area, for example tightening or screwing, can be controlled. [9] The features set out in the following paragraphs may, optionally, be implemented independently of each other or in combination with each other.

[0010] According to one or more embodiments, the at least one visual reference mark is a 2D visual reference mark. Advantageously, the determination of the position of the at least one 2D visual reference mark relative to the image-taking device can be done from a 2D image taken by the image-taking device. According to one or more embodiments, the determination of the position of the at least one 2D visual reference mark relative to the image-taking device can further be done from a 3D image and / or at least one 2D image capable of providing positions in space and an intensity of the pixels of the visual reference mark.

[0011] Alternatively, according to one or more embodiments, the at least one visual marker is a three-dimensional, 3D, visual marker. According to one or more embodiments, the at least one three-dimensional visual marker is non-symmetrical. Advantageously, an asymmetry makes it possible to avoid positioning ambiguities. Thus, according to one or more embodiments, the at least one visual marker may be all or part of an asymmetrical object. In particular, according to one or more embodiments, the asymmetrical object may be obtained by 3D printing. Advantageously, the determination of the position of the at least one 3D visual marker relative to the image-taking device may be done from a 3D image taken by the image-taking device.According to one or more embodiments, the determination of the position of the at least one 3D visual reference point relative to the image-taking device can also be done from information from a computer-aided design, CAD, model of the 3D visual reference point.

[0012] According to one or more embodiments, the calibration device may comprise a plurality of visual markers and the position of the image-taking device may be determined relative to at least a portion of the plurality of visual markers.

[0013] Advantageously, determining the position of the image-taking device relative to at least a portion of the plurality of visual references makes it possible to increase the accuracy of the calibration of the area of ​​the tool. Indeed, the position of the image-taking device can thus be obtained by using information from several visual references, reducing uncertainty as to a relative position between a reference frame of the image-taking device and a reference frame of the calibration device.

[0014] According to one or more embodiments, the method may comprise a determination of a plurality of positions of the at least one visual reference mark relative to the image-taking device by means of the image-taking device and the steps consisting of: determining, for each of the positions of the plurality of determined positions of the at least one visual reference mark relative to the image-taking device, a primary position of said area of ​​the tool relative to the image-taking device, from the position of the visual reference mark relative to the image-taking device and the location of the fixed point relative to the calibration device, partitioning the plurality of primary positions of said area of ​​the tool relative to the image-taking device into groups, determining a size of a group containing the most primary positions, determining, as a function of the determined group size,the position of said tool area relative to the image-taking device, from the primary positions of the group of the determined size.,

[0015] By "partitioning" we mean data partitioning, or "data clustering" in English. Partitioning aims to divide the plurality of primary positions into different groups of primary positions, minimizing intra-class inertia of the groups, while maximizing inter-class inertia between the groups.

[0016] Advantageously, the proposed method also makes it possible to increase the precision and accuracy of the calibration of the tool area by determining the position of the tool area relative to the image-taking device by exploiting a plurality of primary positions. In particular, erroneous positions, for example determined from images acquired while the tool area was not placed on the fixed point, can thus be excluded from the calibration of the tool area.

[0017] According to one or more embodiments, the plurality of positions of the at least one visual marker relative to the image-taking device may comprise the positions which have been determined from a portion of the plurality of visual markers comprising a number of visual markers greater than a threshold.

[0018] Advantageously, the accuracy of the calibration of the tool area can thus be modulated by varying the threshold value.

[0019] According to one or more embodiments, the step of determining the plurality of primary positions of the tool area relative to the image-taking device will be iterated until the size of the group containing the most primary positions is greater than a threshold value.

[0020] Advantageously, the accuracy of the calibration can be increased by varying the threshold value. An intermediate position of the tool area outside the fixed point can thus be excluded from the determination of the position of the tool area.

[0021] According to one or more embodiments, the calibration device will comprise at the fixed point of known location relative to the calibration device, at least part of a pivot connection or a ball joint connection between the fixed point of known location and the area of ​​the tool.

[0022] Advantageously, the presence of a portion of a pivot connection or a ball joint connection at the fixed point adds at least one degree of freedom in the relative movements between the image-taking device and the calibration device, while maintaining the position of the tool area relative to the fixed point. Thus, a variety of positions between the image-taking device and the calibration device can enrich the calibration data, making it possible to increase the precision and accuracy of the calibration. Additionally, the presence of at least a portion of a ball joint connection makes it possible to obtain substantially uniform data in a three-dimensional space. In particular, a height of the image-taking device relative to the calibration device can be modified, allowing the image-taking device to acquire images of visual landmarks that would not have been visible otherwise.

[0023] According to one or more embodiments, the method will further comprise the steps of: rotating the tool area around an axis of the tool area, via the pivot connection, determining, via the image capture device, a plurality of positions of the image capture device relative to the at least one visual reference mark of the calibration device as a function of the rotation around the axis of the tool area, determining an axis of the tool area relative to the image capture device, from the plurality of positions of the image capture device relative to the visual reference mark as a function of the rotation around the axis of the tool area.

[0024] By "tool area axis" is meant here an axis along which an accessory attached to the tool area can extend. One end of an accessory attached to the tool area can thus be located on the tool area axis. The position of the end of the accessory attached to the tool area on the tool area axis can vary depending on a size of the accessory.

[0025] Advantageously, knowledge of the axis of the tool zone thus makes it possible to constrain the space of admissible positions of the tool zone, or of one end of an accessory fixed to the tool zone, relative to the image-taking device.

[0026] Alternatively, according to one or more embodiments, the determination of the axis of the tool zone may comprise the steps of: adding an accessory to the tool zone, placing an accessory zone on the fixed point of known location, determining, via the image-taking device, the position of the visual reference mark relative to the image-taking device, determining the position of said area of ​​the accessory relative to the image-taking device, from the position of the visual reference mark relative to the image-taking device and the location of the fixed point relative to the calibration device, determining an axis of the area of ​​the tool relative to the image-taking device, from the position of said area of ​​the tool relative to the image-taking device and the position of said area of ​​the accessory relative to the image-taking device.

[0027] According to one or more embodiments, in order to detect a change of a second accessory attached to the area of ​​the tool, the method will further comprise the steps of: determining, via the image-taking device, the position of an end of the second accessory relative to the image-taking device, from the axis of the area of ​​the tool.

[0028] Advantageously, automatic detection of a change of a second accessory attached to the tool area can thus be achieved. Consequently, calibration of the end of the second accessory relative to the image-taking device can be achieved without requiring reproduction of all the steps of the proposed method. A generalization of the calibration of the tool area can thus be achieved for all types of accessories attached to the tool area.

[0029] According to another aspect, an image capturing device is provided for implementing a method according to the present description.

[0030] According to another aspect, there is provided a device for calibrating a position of an area of ​​a tool relative to an image-taking device located on the tool, the device comprising: a fixed point of known location relative to the device, at least one visual reference of known location relative to the device, optionally, at least part of a pivot or ball joint located at the fixed point to receive the area of ​​the tool.

[0031] According to another aspect, there is provided a computer program comprising instructions for implementing all or part of a method as defined herein when this program is executed by a processor. According to another aspect, there is provided a non-transitory, computer-readable recording medium on which such a program is recorded. Brief description of the drawings

[0032] Other features, details and advantages will become apparent upon reading the detailed description below, and upon analyzing the attached drawings, in which: Fig. 1

[0033] [Fig. 1] shows a robotic arm comprising a tool provided with an image-taking device according to one or more embodiments. Fig. 2

[0034] [Fig. 2] shows a hand tool equipped with an image-taking device according to one or more embodiments. Fig. 3

[0035] [Fig. 3] shows a diagram illustrating an example of implementation of the method according to one or more embodiments. Fig. 4

[0036] [Fig. 4] shows a diagram illustrating an example of implementation of the method according to one or more embodiments. Fig. 5

[0037] [Fig. 5] shows a diagram of a tool provided with an image-taking device according to one or more embodiments. Fig. 6

[0038] [Fig. 6] shows a diagram illustrating an example of implementation of the method according to one or more embodiments. Fig. 7

[0039] [Fig. 7] shows a top view of a calibration device according to one or more embodiments. Fig. 8

[0040] [Fig. 8] shows a calibration device and a tool equipped with an image-taking device according to one or more embodiments. Description of the embodiments

[0041] In the following detailed description of embodiments of the invention, numerous specific details are presented to provide a more complete understanding. However, those skilled in the art may realize that embodiments can be practiced without these specific details. In other instances, well-known features are not described to avoid unnecessarily complicating the description.

[0042] The present description refers to diagrammatic illustrations of the methods and devices according to one or more embodiments. Each of the described diagrams may be implemented in hardware, software (including embedded software ("firmware"), or "middleware"), microcode, or any combination thereof. In the case of a software implementation, the diagrammatic illustrations may be implemented by computer program instructions or software code, which may be stored or transmitted on a computer-readable medium, including a non-transitory medium, or a medium loaded into the memory of a generic, specific computer, or any other apparatus or device programmable data processing apparatus to produce a machine, such that the computer program instructions or software code executed on the computer or programmable data processing apparatus or device constitute means of implementing these functions.

[0043] Embodiments of a computer-readable medium include, but are not limited to, computer storage media and communication media, including any medium that facilitates the transfer of a computer program from one location to another. "Computer storage medium(s)" means any physical medium that can be accessed by a computer.Examples of computer storage media include, but are not limited to, flash memory disks or components or any other flash memory devices (e.g., USB keys, memory keys, memory sticks, key disks), CD-ROMs or other optical data storage devices, DVDs, magnetic disk data storage devices or other magnetic data storage devices, data memory components, RAM, ROM, EEPROM memories, memory cards ("smart cards"), SSD ("Solid State Drive") memories, and any other form of media usable for transporting or storing or memorizing data or data structures that can be read by a computer processor.

[0044] In addition, various forms of computer-readable media may transmit or carry instructions to a computer, such as a router, gateway, server, or any data transmission equipment, whether wired (via coaxial cable, fiber optics, telephone wires, DSL cable, or Ethernet cable), wireless (via infrared, radio, cellular, microwave), or virtualized transmission equipment (virtual router, virtual gateway, virtual tunnel endpoint, virtual firewall).The instructions may, depending on the embodiments, include code of any computer programming language or computer program element, such as, without limitation, assembly languages, C, C++, Visual Basic, HyperText Markup Language (HTML), Extensible Markup Language (XML), HyperText Transfer Protocol (HTTP), Hypertext Preprocessor (PHP), SQL, MySQL, Java, JavaScript, JavaScript Object Notation (JSON), Python, and bash scripting.

[0045] Furthermore, the terms "notably", "for example", "example" are used in the present description to designate examples or illustrations of non-limiting embodiments, which do not necessarily correspond to preferred or advantageous embodiments compared to other aspects or possible embodiments.

[0046] The terms "located", "mounted", "equipped" and their various variants and forms used in this description refer to couplings, connections, assemblies which may be direct or indirect. Thus, an image-taking device located / mounted on a tool may be directly located on the tool or on an intermediate element positioned between the image-taking device and the tool.

[0047] Figure 1 shows, as a non-limiting example of tool 100, a tightening or screwing tool mounted on a robotic arm RBT according to one or more embodiments. Alternatively, the tool 100 may be a machining tool, for example one of a drilling tool, a deburring tool, a chamfering tool, a polishing tool, or any other tool suitable for implementing the proposed method such as a welding tool. In the example of Figure 1, the area 101 of the tool for which the calibration is sought corresponds to the head of the tool 100, that is to say the point of the tool 100 where the action to be performed takes place. Thus, in the case of the clamping tool shown in Figure 1, the head of the tool 100 may correspond to one end of a socket. In the case of a drilling tool, the head of the tool may in particular correspond to one end of a drill bit.

[0048] Furthermore, the tool 100 is provided with an image-taking device 200. The image-taking device 200 is configured to acquire images. The image-taking device 200 comprises, for example, a two-dimensional, 2D, camera or a three-dimensional, 3D, camera. According to one or more embodiments, the image-taking device 200 comprises a 2D camera and a 3D camera, making it possible to take advantage of the information obtained by the two cameras. According to one or more embodiments, the 2D camera comprises two three-color lenses (red, green, blue, RGB) coupled to an inertial unit. According to one or more embodiments, the 3D camera is an Active Stereo type camera. Furthermore, the image-taking device 200 is functionally coupled to a processor enabling the implementation of the methods described in the present description.

[0049] As shown in Figure 1, the area 101 of the tool may be distinct from the image-taking device 200. More particularly, an offset, referenced T_tcp, between the center 201 of the image-taking device 200 and the area 101 of the tool can be observed in Figure 1.

[0050] Alternatively, Figure 2 shows, as a non-limiting example of tool 100', a portable hand tool. More particularly, the example of Figure 2 represents a hand tightening tool, more specifically a ratchet. Alternatively, the tool 100' may be a hand tool among a hand screwing tool, a hand drilling tool, a hand deburring tool, a hand chamfering tool, a hand polishing tool, a hand welding tool, or any other hand tool suitable for implementing the proposed method.

[0051] As for the tool 100 of Figure 1, an image-taking device 200' is located on the tool 100'. The offset between the center 201' of the image-taking device 200' and the area of ​​interest 101' of the tool is referenced T_tcp'. The area of ​​interest of the tool is for example the end of a socket in the case of a ratchet.

[0052] In each of the cases set out above, knowledge of the offset T_tcp, T_tcp' is sought in order to deduce from an image acquired by the image-capturing device the position of the area of ​​interest 101, 101' relative to the image-capturing device 200.

[0053] For this purpose, Figure 3 is a diagram illustrating the method 300 for calibrating the area of ​​the tool proposed according to one or more embodiments.

[0054] In a first step PLACE_TOOL 301, the area of ​​interest 101, 101' of the tool is placed on a fixed point. The fixed point is a point of known location relative to a calibration device. Thus, the coordinates, referenced T_offset, T_offset' in Figures 6 and 7 respectively, of the fixed point in a frame of reference of the calibration device are known.

[0055] In addition, the calibration device comprises at least one visual reference mark. A location of the visual reference mark relative to the calibration device is known. Therefore, the location of the fixed point relative to the at least one visual reference mark is also known. As for example shown in Figures 7 and 8, and discussed below, according to one or more embodiments, the at least one visual reference mark is a two-dimensional, 2D, visual reference mark.

[0056] In a second step DET_POS_MARK / CAM ​​302, the position of the visual marker relative to the image-capturing device 200, 200' is determined via the image-capturing device 200, 200'.

[0057] According to one or more embodiments, the determination 302 of the position of the visual reference mark relative to the image-taking device 200, 200' comprises, in a first sub-step, a detection of the at least one visual reference mark by the image-taking device 200, 200'. In particular, the detection of the at least one visual reference mark by the image-taking device 200, 200' can be done from a 2D image taken by the image-taking device 200, 200'. The 2D image can be a grayscale image. Alternatively, the 2D image can be a color image. According to one or more embodiments, the detection of the at least one visual reference mark by the image-taking device 200, 200' from the 2D image is performed via a computer-aided vision library, for example the OpenCV library. According to one or more embodiments, the ArUco library can be used (S. Garrido-Jurado, R. Munoz-Salinas, FJMadrid-Cuevas, and M. J. Marin-Jiménez. 2014. "Automatic generation and detection of highly reliable fiducial markers under occlusion". Pattern Recogn. 47, 6 (June 2014), 2280-2292. D0l=10.1016 / j. patcog.2014.01 .005).

[0058] According to one or more embodiments, in a second sub-step, a 3D position of the at least one visual reference point relative to the image-taking device 200, 200' is determined. According to one or more embodiments, the 3D position of the at least one visual reference point relative to the image-taking device 200, 200' is obtained from the 2D image. More particularly, according to one or more embodiments, the 3D position of the at least one visual reference point relative to the image-taking device 200, 200' is obtained via a computer vision library taking the 2D image as input. According to one or more embodiments, the computer vision library may also take one or more parameters intrinsic to the image-taking device 200, 200' as input. For example, the computer vision library is the OpenCV library.More particularly, the ArUco library can be used. Alternatively, according to one or more embodiments, when the image-taking device 200, 200' comprises a 3D camera, the 3D position of the at least one visual marker relative to the image-taking device 200, 200' can be directly acquired by the 3D camera.

[0059] According to one or more embodiments, in a third substep, a 3D transformation between the coordinate system of the reference frame of the image-taking device 200, 200 and the coordinate system of the reference frame of the calibration device is determined. According to one or more embodiments, the 3D transformation between the coordinate system of the reference frame of the image-taking device 200, 200 and the coordinate system of the reference frame of the calibration device is determined. calibration is determined from the 3D position of the at least one visual reference point relative to the image-taking device 200, 200' and a 3D position of the at least one visual reference point relative to the calibration device. According to one or more embodiments, the 3D position of the at least one visual reference point relative to the calibration device is known a priori and recorded. According to one or more embodiments, the 3D transformation between the coordinate system of the reference frame of the image-taking device 200, 200 and the coordinate system of the reference frame of the calibration device is represented by a matrix, denoted MT_CAL, of size 3 x 3 such that the 3D position of the at least one visual reference point relative to the image-taking device 200, 200' and the 3D position of the at least one visual reference point relative to the calibration device, respectively denoted XCAM and XMARK can be expressed by the relation [Math. 1] below:

[0060] [Math. 1] XCAM = MT_CAL * XMARK

[0061] Consequently, the 3D transformation between the coordinate system of the calibration reference frame and the reference frame of the image-taking device 200, 200 is also known. Indeed, a matrix of the 3D transformation between the coordinate system of the calibration reference frame and the reference frame of the image-taking device 200, 200 can be expressed as the inverse of the matrix MT_CAL.

[0062] In a third step DET_POS_TOOL / CAM 303, the position of the area of ​​interest of the tool 101, 101' relative to the image-taking device 200, 200', referenced T_tcp, T_Tcp' in FIGS. 6 and 7 respectively, is determined. The position T_tcp, T_Tcp' of the area 101, 101' of the tool relative to the image-taking device 200, 200' is obtained from the position of the visual reference relative to the image-taking device 200, 200' and the location T_offset, T_offset' of the fixed point relative to the calibration device. According to one or more embodiments, the position T_tcp, T_Tcp' of the zone 101, 101' of the tool relative to the image-taking device 200, 200', is obtained according to the relation [Math. 2], respectively [Math. 2'], below:

[0063] [Math. 2] T_tcp = MT_CAL * T_offset

[0064] [Math. 2'] T_tcp' = MT_CAL * T_offset'

[0065] The method 300 described above with reference to FIG. 3 may be iterated. According to one or more embodiments, the image-taking device 200, 200' acquires a plurality of images. The acquisition of the plurality of images may be done at regular time intervals. For example, an acquisition frequency may be between 10 Hz and 100 Hz, more preferably between 10 Hz and 2 Hz, more preferably of the order of 15 Hz.

[0066] According to one or more embodiments, in cases where a plurality of positions of the at least one visual reference mark relative to the image-taking device 200, 200' is acquired, the calibration method 400 illustrated in FIG. 4 can be implemented.

[0067] According to one or more embodiments, in a first step PLACE_TOOL 301 ', the area of ​​interest 101 , 101 ' of the tool is placed on a fixed point similarly to the step PLACE_TOOL 301 described with reference to FIG. 3.

[0068] For one or more of the images of the plurality of acquired images, or even for each image of the plurality of acquired images, a detection of the at least one visual cue similar to the detection of the at least one visual cue described with reference to FIG. 3 can be implemented.

[0069] According to one or more embodiments, in cases where the calibration device comprises a plurality of visual markers, the method 400 may comprise a second optional DETECT_MARK 401 step. The DETECT_MARK 401 step may comprise, for one or more of the images of the plurality of acquired images, or even for each image of the plurality of acquired images, a determination of a number of visual markers detected in the image.

[0070] According to one or more embodiments, in cases where the optional step DETECT_MARK 401 is implemented, the method may comprise a third step MIN_MARK 402 which is also optional. The step MIN_MARK 402 may comprise, for one or more of the images of the plurality of acquired images, or even for each image of the plurality of acquired images, a comparison of the number of visual markers determined in the step DETECT_MARK 401 with a first threshold value a. The first threshold value a may advantageously be chosen to be between 1 and 20, preferably between 2 and 15, more preferably of the order of 4. According to one or more embodiments, if, for an image, the number of visual markers determined in the step DETECT_MARK 401 is greater than the first threshold value a, as represented by the OK arrow coming from the block 402 in Figure 4, step 302' may be implemented. Alternatively, if, for an image, the number of visual markers determined in step DETECT_MARK 401 is less than the first threshold value a, as represented by the arrow NOK starting from block 402 in Figure 4, step 401 may be implemented on another image of the plurality of acquired images.

[0071] According to one or more embodiments, for one or more of the images of the plurality of acquired images, or even for each image of the plurality of acquired images, the determination 302' of the position of the at least one visual marker by the image-taking device 200, 200' can be carried out according to one or more embodiments described above in connection with FIG. 3 for the determination 302 of the position of the at least one visual marker by the image-taking device 200, 200', the description of which is not repeated here for the purposes of conciseness of the present description. In particular, according to one or more embodiments, the determination 302' of the position of the at least one visual marker by the image-taking device 200, 200' is carried out only for the images for which the number of visual markers determined in the DETECT_MARK step 401 is greater than the first threshold value a as indicated previously.

[0072] According to one or more embodiments, for one or more of the images of the plurality of acquired images, or even for each image of the plurality of acquired images, the determination 303' of the position of the area of ​​interest of the tool 101, 101' relative to the image-taking device 200, 200' can be carried out according to one or more embodiments described above in connection with FIG. 3 for the determination 303 of the position of the area of ​​interest of the tool 101, 101' relative to the image capture device 200, 200', the description of which is not repeated here for the purposes of conciseness of the present description.

[0073] According to one or more embodiments, in a step CLUST_POS_TOOL / CAM 403 the positions of the area of ​​interest of the tool 101, 101' relative to the image-taking device 200, 200' determined in step 303 may be partitioned into groups. According to one or more embodiments, a k-nearest neighbors (k-NN) algorithm may be used. According to one or more embodiments, the k-nearest neighbors (k-NN) algorithm may be used with two classes. Alternatively, according to one or more embodiments, a meanshift algorithm may be used. According to one or more embodiments, the partitioning can be obtained via a library intended for machine learning, for example the Scikit-Learn library (Pedregosa et al. 2011. “Scikit-learn: Machine Learning in Python, JMLR 12, pp. 2825-2830.).In particular, according to one or more embodiments, the DBSCAN algorithm may be used. According to one or more embodiments, a maximum distance between two points for them to be considered as in the same group may be adjusted. The maximum distance between two points for them to be considered as in the same group may advantageously be chosen to be between 1 mm and 3 mm, preferably between 1.5 mm and 2.5 mm, more preferably of the order of 2 mm.

[0074] According to one or more embodiments, an optional MIN_MAX_CLUST step 404 may be implemented. The MIN_MAX_CLUST step 404 may comprise a determination of a size of a group among the groups determined in the CLUST_POS_TOOL / CAM step 403. More particularly, the size of the group containing the most positions of the area of ​​interest of the tool 101, 101' relative to the image-taking device 200, 200' may be determined. The determined group size may be compared to a second threshold value a. The second threshold value a can advantageously be chosen to be between 50 and 250, preferably between 100 and 200, more preferably of the order of 150. According to one or more embodiments, if the size of the determined group is greater than the second threshold value a, as represented by the OK arrow starting from block 404 in FIG. 4, step 405 can be implemented.Alternatively, if the determined group size is less than the second threshold value a, as represented by the arrow NOK starting from block 404 in FIG. 4, step 401 may be implemented on an image of the plurality of acquired images for which none of steps 401, 402, 302', 303' and 403 has been implemented.

[0075] According to one or more embodiments, in a step DET_FINAL_POS_TOOL / CAM 405, a so-called “final” position of the tool area 101, 101', 101” is determined from the positions of the group of the determined size. In particular, according to one or more embodiments, the “final” position of the tool area 101, 101', 101” is an average of the positions of the group of the determined size. Alternatively, the “final” position of the tool area 101, 101', 101” may be a median of the positions of the group of the determined size.

[0076] As shown schematically in Figure 5, according to one or more embodiments, an accessory, referenced ACC, can be attached to the area of ​​the tool 101”.

[0077] According to one or more embodiments, a position of an end, referenced 502, of the ACC accessory can be determined according to one of the methods set forth with reference to Figures 3 and 4.

[0078] Alternatively, according to one or more embodiments, a 3D position of the end 502 of the accessory ACC relative to the image-taking device 200” can be deduced, via the image-taking device 200”, from knowledge of an axis of the tool zone, referenced A500.

[0079] More particularly, a 2D position of the end 502 relative to the image-taking device 200” can be obtained in a 2D image taken by the image-taking device 200” and including the end 502. A 3D position of the end 502 relative to the image-taking device 200” can be obtained from the 2D position and knowledge of the axis A500 of the tool area.

[0080] For example, according to one or more embodiments, from the knowledge of the axis of the area of ​​the tool A500, a distance, noted d, between a center, noted 201”, of the image-taking device 200” and a point, noted 501, coplanar with a plane of an objective of the image-taking device 200” passing through the center 201” can be obtained. The distance d can be designated by the term baseline.

[0081] The point 501 may define the center of a virtual image-taking device directed along the axis of the tool A500. A 2D position of the end 502 relative to the virtual image-taking device may then be deduced. In particular, the end 502 may be located at the center of a virtual image that would be taken by the virtual image-taking device.

[0082] Thus, from the distance d between the center 201” of the image-taking device 200” and the center 501 of the virtual image-taking device, the positions of the end 502 relative to the image-taking device 200” and the virtual image-taking device, the 3D position of the end 502 relative to the image-taking device 200” can be obtained by triangulation. In particular, a gap, designated by the term disparity, noted 5, can be determined between the position of the end 502 relative to the image-taking device 200” and the position of the end 502 relative to the virtual image-taking device. According to one or more embodiments, the gap 5 can be measured in pixels between the position of the end 502 relative to the image-taking device 200” and the position of the end 502 relative to the virtual image-taking device.A third coordinate, or depth, denoted z, can be calculated from the disparity 5 and a focal length, denoted f, of the image-taking device 200”. The depth z can notably be calculated according to the formula [Math. 3] below:.

[0083] [Math. 3] z = (d*f) / 5

[0084] Alternatively, according to one or more embodiments, when the image capture device 200” comprises a 3D camera, the 3D position of the end 502 relative to the device image capture device 200” can be directly obtained from a 3D image taken by the image capture device 200” and including the end 502.

[0085] Thus, Figure 6 is a diagram illustrating the method 600 of detecting an end of an accessory attached to the area of ​​the tool 101”, the position of the area of ​​the tool 101” having been determined by one of the methods presented above.

[0086] In a first step DETECT_AXIS 601, the axis A500 of the tool area 101” is determined.

[0087] According to one or more embodiments, when the position of the area of ​​the tool 101” relative to the image-taking device 200” is known, for example determined by one of the methods presented above, and when a position of the end of an accessory fixed to the area of ​​the tool 101” relative to the image-taking device 200” is also known, for example determined by one of the methods presented above, the axis A500 of the area of ​​the tool 101” can be determined as a single axis passing through these two positions.

[0088] Alternatively, according to one or more embodiments, the calibration device comprises at the fixed point P of known location relative to the calibration device a pivot connection between the fixed point P of known location and the area of ​​the tool. According to one or more embodiments, the axis A500 of the area of ​​the tool 101” is determined by performing a rotation around the axis A500 of the area of ​​the tool 101” via the pivot connection. More particularly, a plurality of positions of the image-taking device 200” relative to the at least one visual reference mark of the calibration device can be acquired as a function of the rotation around the axis A500 of the area of ​​the tool. An equation of the axis A500 of the area of ​​the tool relative to the image-taking device 200” can then be determined from the plurality of positions of the image-taking device 200” relative to the at least one visual reference mark.Indeed, the plurality of positions of the image-taking device 200” relative to the at least one visual reference mark forms an arc of a circle in a plane orthogonal to the axis A500. A direction of the axis A500 can thus be deduced from the knowledge of the plane orthogonal to the axis A500. The direction of the axis A500 is the normal to the plane orthogonal to the axis A500. Thus, knowing a point of the axis A500, which is located at the fixed point P of known location, and a direction of the axis A500, the axis A500 can be uniquely determined.

[0089] In a second step DET_COORD_PICT 602, the 2D position of the end 502 relative to the image-taking device 200” is obtained in a 2D image taken by the image-taking device 200” and comprising the end 502.

[0090] According to one or more embodiments, the 2D position of the end 502 relative to the image capture device 200” is obtained via a deep learning algorithm. For example, a convolutional neural network, or “Convolutional Neural Network (CNN)” may be used. A regional convolutional neural network, or “Regional Convolutional Neural Network (R-CNN)” may in particular be used. In particular, a mask regional convolutional neural network, or “Mask Regional Convolutional Neural Network (Mask R-CNN)” may be used. The deep learning algorithm may be trained with training images for detecting the end 502. The training images may include real images and / or simulated images. Data augmentation may be performed from an initial set of training images to obtain an augmented set of training images.

[0091] Alternatively, according to one or more embodiments, the 2D position of the end 502 relative to the image capture device 200” is obtained by analyzing a light intensity of pixels along the axis A500 to detect the end 502. For example, edge detection can make it possible to extract the end 502. In particular, a Canny edge detector can be used to extract the end 502.

[0092] Alternatively, according to one or more embodiments, the 2D position of the end 502 relative to the image-taking device 200” is obtained by analyzing a plurality of images taken by the image-taking device 200”. Indeed, since the end 502 is fixed relative to the image-taking device 200”, the position of the end 502 relative to the image-taking device 200” can be obtained by determining a position of a fixed point on the plurality of images. In particular, according to one or more embodiments, the plurality of images comes from a video stream taken by the image-taking device 200”.

[0093] In a third step DET_POS_TOOL / CAM 603, the 2D position of the end 502 relative to the image-taking device 200” is converted into the 3D position of the end 502 relative to the image-taking device 200” by exploiting the knowledge of the axis A500 of the tool area as explained with reference to FIG. 5.

[0094] Figure 7 shows, as a non-limiting example of a calibration device 700, a top view of a calibration board. Alternatively, according to one or more embodiments not shown, the calibration device is a non-planar 3D object.

[0095] The calibration device 700 comprises a fixed point P of known location relative to the calibration device.

[0096] The calibration device 700 shown comprises a plurality of visual markers, including visual markers referenced 701 and 702, of known locations relative to the device 700. As shown in Figure 7, the fixed point P has a location T_offset_702 relative to a corner of the visual marker 702. The location of the fixed point P in the reference frame of each of the visual markers is not shown in Figure 7 for the sake of clarity. The visual markers shown in Figure 8 are marker corners from the OpenCV ArUco library. According to one or more embodiments not shown, the visual markers may be marker corners of the April Tags type which can be found at the following address: https: / / april.eecs.umich.edu / software / apriltag.Alternatively, according to one or more embodiments not shown, the calibration device 700 comprises a single visual marker of known location relative to the calibration device 700.

[0097] Optionally, according to one or more embodiments, the calibration device 700 comprises a part of a pivot or ball joint located at the fixed point P to receive the area of ​​the tool 101, 101', 101”.

[0098] As seen in Figure 8, the portion of the ball joint 800 located at the fixed point P may be a hollow half-sphere. The area of ​​the tool 101 may be provided with a spherical tip configured to fit into the half-sphere during calibration. In this configuration, the area of ​​the tool 101 may thus be merged with the fixed point P. Alternatively, according to one or more embodiments not shown, the portion of the pivot joint located at the fixed point P may comprise two nested cylinders.

[0099] As previously described, the area of ​​the tool 101 is placed, for example manually by an operator, on the fixed point P of known location relative to the calibration device 700. The position T_cal_702 of the visual reference 702 relative to the image-taking device 200 which is a corner of a marker is determined by means of the image-taking device 200. The robot RBT can generate movements of the area of ​​the tool 101 on the fixed point P by means of the link 800. During these movements, the acquired visual references can vary without harming the implementation of the proposed method. The position T_tcp of the area of ​​the tool 101 relative to the image-taking device 200 can then be determined from the position T_cal_702 of the visual reference point 702 relative to the image-taking device 200 and the location T_offset_702 of the fixed point P relative to the visual reference point 702.

[0100] Depending on the embodiment selected, certain acts, actions, events, or functions of each of the methods described herein may be performed or occur in a different order than they were described, or may be added, merged, or may not be performed or occur, as the case may be. In addition, in some embodiments, certain acts, actions, or events are performed or occur concurrently and not successively.

[0101] In particular, according to one or more embodiments, the tool may be provided with a plurality of image-taking devices. A number n of image-taking devices of the plurality of image-taking devices may advantageously be chosen between 2 and 4. The calibration method described above may then be implemented for the n image-taking devices. Advantageously, according to one or more embodiments, the calibration method described above may be implemented n times in parallel, using the same calibration device.

[0102] Although described through a number of detailed exemplary embodiments, the proposed calibration method and the device for implementing an embodiment of the method include various variations, modifications and improvements which will be obvious to those skilled in the art, it being understood that these various variations, modifications and improvements are part of the scope of the present disclosure, as defined by the claims which follow. In addition, different aspects and characteristics described above can be implemented together, or separately, or substituted for each other, and all various combinations and subcombinations of the aspects and features are within the scope of this disclosure. Furthermore, some systems and equipment described above may not incorporate all of the modules and functions described for the preferred embodiments.

Claims

Claims

1. Method (300) for determining a position of an area (101, 101', 101”) of a tool relative to an image-taking device (200, 200', 200”) located on the tool (100, 100', 100”), the method comprising the steps of: placing (301) the area (101, 101', 101”) of the tool on a fixed point (P) of known location relative to a calibration device (700), said calibration device (700) comprising at least one visual marker (701, 731), determining (302), via the image-taking device (200, 200', 200”), the position of the visual marker (701, 731) relative to the image-taking device (200, 200', 200”), determining (303) the position of said area (101, 101', 101”) of the tool relative to the image-taking device (200, 200', 200”), from the position of the visual reference mark (701, 731) relative to the image-taking device (200, 200',200”) and the location of the fixed point (P) relative to the calibration device (700).,

2. The method of claim 1, wherein the calibration device (700) comprises a plurality of visual markers (701, 731) and wherein the position of the image-taking device is determined relative to at least a portion of the plurality of visual markers (701, 731).

3. Method according to one of the preceding claims in which a plurality of positions of the at least one visual marker (701, 731) relative to the image-taking device (200, 200', 200”) is determined by means of the image-taking device (200, 200', 200”), the method further comprising the steps of: determining, for each of the positions of the plurality of determined positions of the at least one visual marker (701, 731) relative to the image-taking device (200, 200', 200”), a primary position of said zone (101, 101', 101”) of the tool relative to the image-taking device (200, 200', 200”), from the position of the visual marker (701, 731) relative to the image capture device (200, 200', 200”) and the location of the fixed point (P) relative to the calibration device (700), partition the plurality of primary positions of said zone (101, 101',101 ”) of the tool relative to the image-taking device (200, 200', 200”) in groups, determining a size of a group containing the most primary positions, determining, based on the determined group size, the position of said area (101, 101 ', 101 ”) of the tool relative to the image-taking device (200, 200', 200”), from the primary positions of the group of the determined size.,

4. A method according to claims 2 and 3, wherein the plurality of positions of the at least one visual marker (701, 731) relative to the image-taking device (200, 200', 200”) includes positions that were determined from a portion of the plurality of visual cues (701, 731) comprising a number of visual cues (701, 731) greater than a threshold.

5. A method according to claim 4, wherein the step of determining the plurality of primary positions of said area (101, 101', 101”) of the tool relative to the image-taking device (200, 200', 200”) is iterated until the size of the group containing the most primary positions is greater than a threshold value.

6. Method according to any one of the preceding claims, in which the calibration device (700) comprises, at the fixed point (P) of known location relative to the calibration device (700), at least part of a pivot connection or a ball joint connection (800) between the fixed point (P) of known location and the zone (101, 101', 101”) of the tool.

7. Method according to the preceding claim, further comprising the steps of: rotating the area (101, 101', 101”) of the tool about an axis (A500) of the area (101, 101', 101”) of the tool, via the pivot connection, determining, via the image-taking device (200, 200', 200”), a plurality of positions of the image-taking device (200, 200', 200”) relative to the at least one visual reference mark (701, 731) of the calibration device (700) as a function of the rotation about the axis (A500) of the area (101, 101', 101”) of the tool, determining an axis (A500) of the area (101, 101', 101") of the tool relative to the image-taking device (200, 200', 200"), from the plurality of positions of the image-taking device (200, 200', 200") relative to the visual reference mark (701, 731) as a function of the rotation about the axis (A500) of the area (101, 101', 101") of the tool.

8. Method according to any one of the preceding claims, further comprising the steps of: adding an accessory (ACC) to the area (101, 101', 101") of the tool, placing an area (502) of the accessory on the fixed point (P) of known location, determining, via the image-taking device (200, 200', 200"), the position of the visual marker (701, 731) relative to the image-taking device (200, 200', 200"), determining the position of said area (502) of the accessory relative to the image-taking device (200, 200', 200"), from the position of the visual marker (701, 731) relative to the image-taking device (200, 200', 200") and from the location of the fixed point (P) relative to the calibration device (700), determining an axis (A500) of the zone (101, 101', 101”) of the tool relative to the image-taking device (200, 200', 200”), from the position of said zone (101, 101',101”) of the tool relative to the image-taking device (200, 200', 200”) and the position of said area (502) of the accessory relative to the image-taking device (200, 200', 200”).,

9. A method according to claim 8, further comprising, in order to detect a change of a second accessory attached to the area (101, 101', 101") of the tool, the steps of: determining, via the image-taking device (200, 200', 200"), the position of one end of the second accessory relative to the image-taking device (200, 200', 200”), from the axis (A500) of the tool area.

10. Computer program comprising instructions for implementing the method according to one of claims 1 to 9 when this program is executed by a processor.

11. Non-transitory recording medium readable by a computer on which is recorded a program for implementing the method according to one of claims 1 to 9 when this program is executed by a processor.

Images