Control of manual operations using portable hand tools in the production of component assemblies

Abstract

The present invention relates to a system for controlling manual operations during the production of an assembly of parts, the system comprising a portable hand-held tool allowing an assembly technician to perform manual operations on a plurality of elements of the assembly of parts and a positioning device at least partially integrated into an electronic module attached to the portable hand-held tool, the positioning device being able to determine the position of at least one manual operation associated with an element by the position of the portable hand-held tool in a three-dimensional coordinate system associated with a set of modeling data representing a three-dimensional modeling image of the assembly of parts. According to the invention, the positioning device comprises a depth camera belonging to the electronic module and a processing module capable of estimating the position of the portable hand-held tool based on the image captured by the depth camera.

Description

Technical Field

[0001] The present invention generally relates to the operation of assemblies of production and / or maintenance components using portable handheld tools to allow assembly technicians to perform manual operations on components of the assembly.

[0002] The portable hand-held tool may be a tightening tool (wrench or screwdriver) designed to perform tightening operations on tightening elements (such as bolts and nuts forming a bolt assembly). The portable hand-held tool may also be a tool designed to fit elements such as rivets, or a tool designed to pierce, weld or cut at various locations on an assembly of components. Background Art

[0003] In many industries, such as those related to transportation and in particular the aerospace industry where assembly processes can be highly complex, it is essential to control the manual operations performed by assembly technicians using hand-held tools during the assembly of components.

[0004] In particular, it is necessary to control the tightening operation, since the complexity of the assembly and the generally large number of tightening elements on which the assembly technician must act are sources of error that are difficult to track. Among the requirements determined to ensure assembly reliability, in particular there is a need to ensure that all bolt assemblies involved in the assembly of the predetermined components are actually tightened. It is also necessary to ensure that each bolt assembly that is actually tightened is properly tightened in terms of tightening torque. In particular, an over-tightened bolt assembly may irreversibly damage the assembled component. Conversely, an insufficiently tightened bolt assembly may loosen under the effect of vibrations.

[0005] Ultimately, it would be desirable to allow assembly technicians to exercise real-time control over the manual operations they perform with handheld tools during the production of an assembly of components, and to optimize the time they spend on that production.

[0006] Document US 8311658 relates to a system for controlling tightening operations during the production process of an assembly of a component, and for locating each tightening operation performed by an assembly technician manipulating the portable tightening tool by locating the portable tightening tool when the portable tightening tool moves at the position of each tightening operation. To this end, the system uses a positioning device, which includes a radio frequency signal transmitter equipped on the portable tightening tool on the one hand, and a plurality of radio frequency beacons placed in the production space of the assembly on the other hand. The position of the tightening tool in a three-dimensional coordinate system associated with the production space and the position of the tightening element tightened by the tightening tool are traditionally obtained, for example, using a time-of-flight and true-range multilateration technique based on a radio frequency signal emitted from the position of the tightening tool and received by a radio frequency beacon. The radio frequency signal emitted from the position of the tightening tool may further include information related to the state of each tightening operation (such as an indication that the tightening operation has been started, is in progress and / or has ended). The various positions occupied by the tightening tool determined by the positioning device are transmitted to a reporting and display system via a communication network (WAN, LAN or Internet), which particularly includes a portable device screen (phone, tablet) and a fixed computer screen. The reporting and display system converts each position received by it in a three-dimensional coordinate system associated with the production space into a position in a three-dimensional coordinate system associated with a set of modeling data (e.g., CAD files) representing a three-dimensional modeling image of an assembly of components. The reporting and display system can then control the display of images of the position of each tightening element on the three-dimensional modeling image and the state of the tightening operation associated with each tightening element on various screens in real time. The system described in document US 8311658 therefore makes it possible to remotely locate, monitor and save the states of various tightening operations. In addition, the technician responsible for the tightening operation can also control the progress state of the tightening operation.

[0007] However, this system has many disadvantages. First, the system is expensive because it requires the use of a large number of receiving beacons, which may be significant depending on the complexity and size of the components to be produced. In addition, the positioning technology based on the use of waves may be biased by the working environment in which the assembly technician moves. For example, the metal frames and skins of aircraft and spacecraft form a Faraday cage through which waves cannot pass. In addition, it is difficult to find the correct position of the beacon that optimizes the communication between the tightening tool and the beacon. Therefore, it is usually necessary to optimize the transmission protocol to overcome the problem of signal reflections. Finally, such a system cannot obtain the level of accuracy required in some cases for positioning (usually on the order of millimeters or less). Summary of the invention

[0008] In order to overcome the above mentioned drawbacks, the present invention provides a low cost solution for locating and identifying with high accuracy manual operations performed by hand tools on elements of an assembly of components.

[0009] More specifically, the present invention relates to a system for controlling manual operations during the production process of an assembly of components, the system comprising a portable hand-held tool allowing an assembly technician to perform manual operations on a plurality of components of the assembly of components and a positioning device at least partially integrated into an electronic module attached to the portable hand-held tool, the positioning device being capable of determining the position of at least one manual operation associated with a component by positioning the portable hand-held tool in a three-dimensional coordinate system, the system being characterized in that,

[0010] The three-dimensional coordinate system is associated with a three-dimensional model of the assembly of components,

[0011] The control system comprises a memory storing a plurality of CAD files allowing the construction of a three-dimensional modeled image of an assembly of components at various stages of production,

[0012] And the positioning device includes a depth camera belonging to the electronic module and a processing module capable of estimating and saving the current position of the portable handheld tool, wherein the current position is estimated by aligning a point cloud extracted from at least one current image captured by the depth camera with a grid or point cloud extracted from a three-dimensional modeling image representing the current state of the component, and the three-dimensional modeling image representing the current state is constructed based on a CAD file from a plurality of CAD files stored in a memory.

[0013] Other advantageous and non-limiting features of the system according to the invention:

[0014] - the positioning device may also comprise an inertial measurement unit integrated into the electronic module, the processing module being capable of estimating, based on the measurements delivered by the inertial measurement unit, a movement of the portable hand-held tool from a last estimated and saved current position, and being capable of estimating, based on the last current position and said movement, at least one updated position of the portable hand-held tool;

[0015] - the processing module may be integrated into the electronic module or into the remote device; in the latter case, the electronic module comprises communication means capable of transmitting the images captured by the depth camera and the measurements from the inertial measurement unit to the remote device for processing by the processing module;

[0016] - the system may further comprise at least one portable user interface capable of automatically indicating to said assembly technician the set point parameters to be applied to said component as a function of the position of the associated manual operation determined by said positioning device;

[0017] - said at least one portable user interface is integrated into said electronic module and / or mobile phone and / or portable tablet computer;

[0018] - the user interface is preferably a display screen;

[0019] - the set point parameters are pre-stored in a storage module of the system, the storage module storing a set of set point parameters to be applied to each of the plurality of elements of the assembly of the component;

[0020] - the electronic module may comprise a confirmation device capable of triggering the automatic saving of information related to the manual operation;

[0021] - the confirmation means may be a manual operation button, actuation of which triggers said automatic saving of information relating to the manual operation;

[0022] - said information comprises the confirmation time and a reference mark allowing unique identification of the element associated with the manual operation;

[0023] The hand tool is, for example, a nutrunner or a screwdriver, a riveter, a stapler, a drill, a cutter or a welder.

[0024] Another subject of the invention is a method for controlling manual operations during the production of an assembly of components, the method comprising the following steps:

[0025] - an assembly technician using a portable hand-held tool to perform manual operations on multiple elements of the component's assembly; and

[0026] - determining the position of a manual operation associated with an element by locating the position of said portable hand-held tool in a three-dimensional coordinate system,

[0027] The method is characterized in that

[0028] A three-dimensional coordinate system is associated with the three-dimensional model of the assembly of the component,

[0029] The method comprises the steps of storing in a memory a plurality of CAD files allowing the construction of a three-dimensional modeled image of an assembly of components at various stages of production,

[0030] And the determining step includes image capture by a depth camera belonging to an electronic module attached to the portable handheld tool, and a step of estimating and saving the current position of the portable handheld tool by aligning a point cloud extracted from at least one current image captured by the depth camera with a mesh or point cloud extracted from a three-dimensional modeling image representing the current state of the assembly of the component, wherein the three-dimensional modeling image representing the current state is constructed based on a CAD file from among multiple CAD files contained in a memory.

[0031] According to some possible implementations of the method:

[0032] - The method may also comprise the steps of: estimating the movement of the portable hand-held tool from the last estimated and saved current position based on the measurements delivered by the inertial measurement unit belonging to said electronic module, and estimating at least one updated position of said portable hand-held tool based on the last current position and said movement;

[0033] - the method may further comprise the step of automatically transmitting to a portable user interface the set point parameters to be applied to said element as a function of the position of the associated manual operation from the step of determining;

[0034] - said transmitting step preferably comprises displaying said set point parameter on a display screen, said display screen being integrated into said electronic module, mobile phone and / or portable tablet. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The following description, provided with reference to the accompanying drawings given by way of non-limiting example, will make it easy to understand what the invention consists of and how it can be implemented. In the drawings:

[0036] Figure 1 A system for controlling a manual operation, in particular a tightening operation, according to a first possible embodiment of the invention is shown in simplified form;

[0037] Figure 2 The block diagram shows Figure 1 An electronic module for controlling a manually operated system;

[0038] Figure 3 A system for controlling manual operation according to another possible embodiment of the present invention is shown in simplified form;

[0039] Figure 4 Possible steps to be implemented in a method for estimating the position of a movable object based on images captured by a depth camera and data measured by an inertial measurement unit of an electronic module attached to or integrated into the movable object are shown, for example in Figures 1 to 3The invention is implemented in a system for controlling manual operation, in particular tightening operation. DETAILED DESCRIPTION

[0040] In the drawings, the same or equivalent elements will have the same reference numerals.

[0041] In the following, the invention will be described in the non-limiting context of controlling a manual operation of the tightening type using a portable hand-held tool (such as a screwdriver or a wrench, and hereinafter referred to as a "portable tightening tool"). However, the invention is also applicable to controlling other types of manual operations involving hand-held tools. Thus, the portable hand-held tool may also be a tool designed to fit elements such as rivets or staples, or a tool designed to perforate holes at various locations on an assembly, weld various parts of an assembly, or cut parts of an assembly.

[0042] Several systems for controlling a tightening operation using a portable tightening tool will now be described with reference to the accompanying drawings. All of these systems use a positioning device that is completely ( Figure 1 and Figure 2 ) or at least partially ( Figure 3 ) are integrated into an electronic module attached to the portable tightening tool, these positioning devices are able to determine the position of at least one tightening operation associated with the tightening element by positioning the portable tightening tool (when facing the tightening element) in a three-dimensional coordinate system associated with a set of modeling data representing a three-dimensional modeling image of an assembly of the component. In the following, the portable tightening tool is described as a tightening wrench for nuts. However, the present invention is applicable to other types of tightening tools designed for other types of tightening elements, such as screwdrivers for tightening screws, or, as mentioned above, to other types of hand-held tools that allow other types of manual operations, such as staplers, nailing machines, riveting machines, drills, welders or cutting machines. The hand-held tool can be any pre-existing hand-held tool (e.g., a conventional manual wrench or torque wrench) to which the electronic module is subsequently attached. As a variant, the portable hand-held tool is designed to contain the electronic module from the beginning.

[0043] Figure 1 By way of non-limiting example, a portable tightening tool according to a first possible embodiment of the invention is schematically shown, here a tightening wrench 1 for nuts, which belongs to a system for controlling a given type of manual operation (here tightening) when producing an assembly of parts. Figure 1, reference numeral 2 denotes an example of a component that an assembly technician has to assemble in an assembly of components (not shown) and two nuts 20, 21 that the technician has to tighten using a portable tightening tool 1 in order to assemble the component. The portable tightening tool 1 comprises an electronic module 3 which, in this first embodiment, contains all the functions that allow the technician to have real-time control of the tightening operations they perform.

[0044] Figure 2 A block diagram of the electronic module 3 according to this first embodiment is shown. As described above, the electronic module 3 here includes all positioning devices capable of locating the position of the tightening wrench 1. These positioning devices first include a depth camera 30, which captures a three-dimensional image of the environment of the tightening wrench when the assembly technician moves the wrench to continuously perform various tightening operations in the assembly production process. The depth camera 30 can be a stereo camera, or a camera operating according to the time of flight principle (TOF camera), or a camera called a structured light camera. In an embodiment that is particularly advantageous for the reasons given in the rest of this disclosure, the positioning device of the electronic module 3 can also include an inertial measurement unit 31, which can deliver the measurement results of the linear acceleration and the measurement results of the angular velocity along the three orthogonal axes associated with the electronic module (and therefore associated with the tightening wrench). The positioning device also includes a processing module 32, which receives the three-dimensional image captured by the depth camera 30, and, where appropriate, receives the measurement results performed by the inertial measurement unit 31. According to the invention, the processing module 32 comprises a calculation algorithm that makes it possible to estimate the position of the electronic module 3, and therefore of the tightening wrench 1, based on at least a three-dimensional image captured by the depth camera 30. This position is estimated in a three-dimensional coordinate system associated with a 3D model of the assembly, obtained by means of three-dimensional modeling data contained in a CAD file stored in the memory 33a, as will be explained below. The database 33b of the electronic module 3 makes it possible to store data related to tightening operations, in particular identifiers of tightening elements on which the technician acts during the assembly process.

[0045] like Figure 2As shown, in this example, the electronic module 3 also includes various user interfaces, in particular a display screen 34. The user interface may also include one or more manual operation buttons 35, and one or more visual indicators 36 (such as light emitting diodes, etc.). The electronic module 3 also includes one or more communication modules 37, allowing the electronic module to exchange information with external devices such as a portable tablet, a PC or a server. If the handheld tool is a connected tool (for example, a smart torque wrench), a communication module can be set to allow the electronic module 3 to exchange data with the connected tool. Any wired (Ethernet, etc.) or wireless (Wi-Fi, Bluetooth, etc.) communication protocol can be used. Finally, Figure 2 A microprocessor 38 a and its associated memory 38 b are shown, making it possible to control the operation of the various components of the electronic module 3 and of the power battery 39 .

[0046] Figure 1 and Figure 2 The system can be used as follows: the assembly technician picks up the nutrunner 1 and moves it until the nutrunner faces one of the bolt assemblies to be tightened, e.g. Figure 1 The bolt assembly 20 shown. Based on the images captured by the depth camera 30, the electronic module 3 will be able to calculate and save the position of the electronic module 3 relative to the three-dimensional model of the assembly. More specifically, as will be explained in more detail below, the current position of the electronic module 3 can be estimated by aligning a point cloud extracted from at least one current 3D image delivered by the depth camera 30 with a point cloud extracted from a three-dimensional image (e.g., a CAD file) that models the current state of the assembly of the component, wherein the three-dimensional image is obtained by the processing module 32 from the memory 33a. As a variant, the point cloud extracted from the image delivered by the depth camera 30 can be aligned directly on the three-dimensional grid that models the current state of the assembly of the component. The three-dimensional grid can directly correspond to the CAD file that models the assembly. In any case, the electronic module 3 is able to know its relative position relative to the current state of the modeled assembly at any time or at least regularly. The estimation based on the 3D image makes it possible to obtain an estimated position with an accuracy level of less than a millimeter. Therefore, when the tightening tool 1 faces the bolt assembly on which the assembly technician is about to act, the system makes it possible to locate and thus identify the bolt assembly with high precision. As will be explained below, when the electronic module 3 is further equipped with an inertial measurement unit 31, the processing module 32 can advantageously be configured to estimate the movement of the tightening tool 1 from its last estimated and saved current position based on the image captured by the depth camera 30, and to update the position of the tightening tool 1 according to the movement and the last estimated current position.

[0047] The set point parameters for each manual operation that the technician must perform can be advantageously saved in advance, for example, in a CAD file for modeling the assembly saved in the storage module 33a of the electronic module 3. In the non-limiting example of the tightening tool 1, the parameter is a tightening set point to be applied to each tightening element. For a riveting handheld tool, the set point parameter relates to the force to be applied. For a drill-type handheld tool, the set point parameter includes, for example, the diameter and / or rotation speed of the drill bit. For a welder-type handheld tool, the set point parameter includes, for example, the diameter and speed of the welding wire, the pulse frequency and / or intensity of the welding arc. Therefore, in a possible embodiment, according to the position of the tightening element in the assembly estimated by the electronic module 3 based on the image captured when the tightening tool faces a specific tightening element, the electronic module 3 can retrieve the set point (here is the tightening set point (typically the tightening torque)) suitable for the tightening element that the assembly technician is acting on from the storage module 33a based on the estimated position, and automatically control the display of the tightening set point on the screen 34 of the electronic module 3. It should be noted that other user interfaces may be used in addition to or in place of the screen 34, for example, an audio indicator may be used that provides an audio signal to the assembly technician to indicate the tightening set point parameters that they must apply to the tightening element according to the position of the tightening operation associated with that tightening element, or a display screen on a remote device, such as a portable tablet computer ( Figure 1 (not shown), the electronic module 3 can exchange information with the tablet computer via the wireless communication module using, for example, Wi-Fi or Bluetooth protocols.

[0048] In any case, there is an undoubted saving of time for the assembly technician relative to all prior art solutions in which he must consult documentation in parallel in order to know the set point parameters he must apply for each element of the assembly.

[0049] Visual indicator 36 (e.g., diode) can emit light of a predetermined color (e.g., green light) to indicate to the assembly technician that they are indeed acting on the correct tightening element. Once tightening is performed, the technician can confirm the tightening operation by, for example, manually operating button 35. Then, the actuation of the button triggers the automatic preservation of information related to the tightening operation just performed by the assembly technician, preferably stored in database 33b. The information advantageously includes an accurate timestamp of the end of the tightening operation, such as corresponding to the time when the assembly technician confirms the tightening operation by button 35 (the time when the button is triggered), and includes a reference mark (reference) for uniquely identifying the tightening element associated with the tightening operation. As a variation relative to the actuation of the manually operated button, in some embodiments, the confirmation of the tightening operation and the preservation of information related to the tightening operation can be automatic: for example, in the case where the handheld tool is a mechanical torque wrench, when there is an inertial measurement unit 31, the measurement results performed by the inertial measurement unit 31 can be used to identify the sudden acceleration of the wrench around the bolt assembly, corresponding to the click of the wrench when the tightening torque is reached. In another embodiment in which the handheld tool is a connected torque wrench, that is to say a torque wrench which is in particular capable of transmitting information, the wrench can be arranged to transmit an indication to the electronic module 3 via a communication module 37 configured with a communication protocol matching the communication protocol used by the wrench that the tightening torque input as a set point has been reached, which corresponds to a confirmation of the operation.

[0050] The measurements made by the inertial measurement unit 31 can be used to estimate the tightening torque actually applied. As a variant, in the case where the portable tightening tool 1 is a smart wrench or a connection wrench capable of measuring the tightening torque actually applied, the electronic module 3 can also be arranged to retrieve and save the tightening torque measured by the wrench.

[0051] Actuation of the button 35 may also trigger the automatic display on the screen 34 of the next tightening operation to be performed by the assembly technician, e.g. Figure 1 The assembly technician repeats all the above operations for each tightening operation they have to perform in the assembly. On completion of these operations, the assembly technician can be prompted to take a photo of the produced assembly, for example by a specific message displayed on the screen 34. The manually operated button 35 can serve as a trigger for taking a photo with the aid of the camera 30. The photos are advantageously saved, for example in a database 33b. Thus, it is possible to have a visual proof of the quality of the assembly process.

[0052] In the above described embodiments, the assembly technician has a fully autonomous handheld tool that allows for control, inspection and monitoring of manual operations of assembly of components.

[0053] Figure 3Another embodiment of the system for controlling a tightening operation according to the present invention is shown. Figure 1 and Figure 2 The system described is similar in that it comprises an electronic module 3 attached to the portable tightening tool 1, which module comprises at least a depth camera 30 and preferably also an inertial measurement unit 31. However, the actual processing of the 3D images captured by the depth camera and, where appropriate, the measurements from the inertial measurement unit is not performed locally in the electronic module 3 but in a remote device 4. The remote device is preferably a portable tablet intended to be used by an assembly technician. In other words, the above reference to the 3D image captured by the depth camera is not performed locally in the electronic module 3 but in a remote device 4. Figure 2 The processing module 32, storage module 33a and database 33b described above are no longer integrated into the electronic module 3, but are integrated into the remote device 4 ( Figure 3 All the data required to estimate the position of the tool relative to the assembly model are transmitted by the communication module 37 of the electronic module 3 to a corresponding communication module (not shown) of the portable tablet 4 via a communication link 5 (for example a wireless link of the Wi-Fi or Bluetooth type). Figure 3 The screen 40 of the remote device is connected to the screen 34 of the module 3 in the processing module, thereby facilitating the control and monitoring operation. For example, the current position of the tightening tool estimated by the processing module can be viewed on the screen 40 superimposed with the three-dimensional modeled image of the component. Here again, it is advantageously possible to retrieve the tightening set point suitable for the tightening element to be acted on by the assembly technician from the CAD model file stored in the database based on the estimated position of the tightening tool that enables the identification of the tightening element, and indicate it to the assembly technician by displaying it on the screen 40.

[0054] With reference above Figure 1 and Figure 2 As in the system described, once the tightening has been performed, the tightening operation is confirmed manually or automatically by the technician by means of the manual operation button 35. In any case, the confirmation triggers the automatic saving of information related to the tightening operation just performed by the assembly technician in the database of the electronic module 3 and / or in a remote database of a remote device.

[0055] The control system according to the invention therefore allows assembly technicians to check in real time that they have indeed tightened all tightening elements and, in a variant in which the adapted tightening set points are additionally automatically supplied to the technician, to ensure that each tightening element has been tightened with the correct tightening parameters.

[0056] All information stored by the electronic module 3 and / or by the remote device 4 can be retrieved and displayed at any time. Table 1 below gives an exemplary representation of the status of a tightening operation related to a given tightening element.

[0057] [Table 1]

[0058] time Reference markings for tightening elements Tightening torque Tightening action status 11:57 11532 120N.m OK

[0059] In the above example, the assembly technician or any other person can verify that the tightening element carrying the reference mark 11532 was tightened at 11:57 (corresponding to the confirmation time of the tightening operation) with a set point or estimated tightening torque equal to 120 N.m. Other information can be added, such as the date of the tightening action or an identifier uniquely associated with the assembly technician.

[0060] Now refer to Figure 4 Various possible steps to be implemented in a method for locating a movable object in a given environment based on images captured by a depth camera and data measured by an inertial measurement unit of an electronic module attached to or integrated into the movable object are described in detail. The movable object to be located may be the portable handheld tool 1 described above in the non-limiting context of controlling a manual tightening operation, in which case the location method may be implemented by a computational algorithm of a processing module 32 of the electronic module 3 or a remote processing module in a remote device 4. More generally, the object to be located may be any object capable of moving or being moved within an environment, such as the end of a robotic arm.

[0061] In the following it is considered in a non-limiting manner that the environment in which the movable object is to be positioned corresponds to a production space in which an assembly of components is produced.

[0062] Recall that the data that can be used to estimate the position of an electronic module and therefore the position of a movable object to which the module is fixed or into which it is integrated are:

[0063] - CAD files containing data allowing three-dimensional modeling of the components of the assembly;

[0064] - a three-dimensional image captured by a depth camera; and

[0065] - Measurements delivered by the inertial measurement unit.

[0066] In the context of application to control manual operations such as tightening, the CAD file must make it possible to model the assembly of components at various stages in the assembly process. In other words, modeling data is available that allows the construction of various digital models representing the assembly at various stages in the production of this assembly.

[0067] The purpose of the algorithm described below is to identify the relative position of the electronic module integrating a depth camera and an inertial measurement unit in a three-dimensional coordinate system associated with a digital model corresponding to the current state of the assembly of the component.

[0068] 1. Determine the digital model that corresponds to the modeled current state of the component ( Figure 4 Step 100) :

[0069] As mentioned above, the assembly can be in a variety of states depending on the parts that have been assembled or not yet assembled. In order to obtain the position, the algorithm needs to know the current state of the digital model, because the algorithm compares the information observed by the depth camera with the information observed digitally, that is, with the three-dimensional modeling image of the assembly representing the current state of the component. The current digital model can be directly input by a third party (operator, production monitoring software) and selected from a CAD file representing the current state of the digital model. As a variant, the current digital model is determined based on the selection of points of interest extracted from the current three-dimensional image provided by the depth camera, which are associated with local parameters calculated according to the techniques conventionally used in 2D / 3D object recognition. In order to determine the modeled current state of the component, the calculated local parameters are compared with the parameters obtained in the same way as seen by the digital simulation camera based on the 3D model of the component at various stages.

[0070] exist Figure 4 On the right, an exemplary three-dimensional current image Im of a real component captured by a depth camera is shown. C , and a modeled three-dimensional image Im corresponding to the modeled current state of the component M In this example, the component in question corresponds to Figure 1 and Figure 3 Part 2 and nuts 20, 21 in it.

[0071] 2. Generate a point cloud and determine the first position / current position of the module ( Figure 4 Steps 110 and 120 in :

[0072] The general principle for determining the relative position of a module with respect to a digital model is to search for the captured three-dimensional image Im C 3D image with modeling M The transformation required for alignment. In practice, the calculation algorithm is configured to extract in step 110:

[0073] - on the one hand, the point cloud of the current 3D image captured by the depth camera (this point cloud is called the “source” point cloud); and

[0074] - On the other hand, a point cloud of the modeled three-dimensional image (this point cloud is referred to as the "target" point cloud).

[0075] Figure 4 The right hand part shows the current three-dimensional image Im C Extracted "source" point cloud N C And from the modeled 3D image Im M Extracted “target” point cloud N M .

[0076] In order to obtain the most accurate representation of the possible "source" point cloud and thus facilitate the alignment of the two point clouds, step 110 will preferably use a plurality of images captured continuously by the depth camera over a given period of time, each captured image being converted into a point cloud. The capture time of these consecutive images is preferably close enough (e.g., corresponding to a capture frequency of 30 images per second) so that the individual shots are similar. The point clouds obtained by these consecutive shots are then aligned so as to form only one "source" point cloud. In order to align these point clouds and form the source point cloud, step 110 will calculate the relative movement between each shot based on the measurements delivered by the inertial measurement unit. By proceeding in this way, a very high density of points is advantageously obtained while reducing the digital noise potentially generated by the depth camera.

[0077] Preferably, the process of aligning the "source" point cloud with the "target" point cloud is performed in two stages, first a coarse alignment (sub-step 121) and then a fine alignment (sub-step 122).

[0078] For the implementation of a rough alignment, sub-step 121 comprises, for example, randomly selecting a plurality of points in the "source" point cloud. Each selected point is paired with a point in the "target" point cloud having local parameters closest to the local parameters associated with the selected point, and the transformation is sought to be optimized to minimize the distance between the same pair of points.

[0079] Figure 4 The right hand portion of the sub-step 121 shows the current three-dimensional image Im C Extracted "source" point cloud N C With the 3D image from modeling Im M Extracted “target” point cloud N M The rough alignment result.

[0080] Starting from the alignment found in sub-step 121, sub-step 122 implements an algorithm of ICP (Iterative Closest Point) type, which consists in associating each point of the "source" point cloud with the closest point in terms of Euclidean distance of the "target" point cloud, and then iteratively searching for the transformation required to minimize the distance between each paired point. For each iteration of the optimization, the points are reassociated.

[0081] At the completion of sub-step 122 , the electronic module knows the transformations necessary to obtain the alignment of the points in the cloud and can therefore infer therefrom its position in the three-dimensional coordinate system associated with the modeled image.

[0082] As mentioned above, in a variant implementation of the method, the point cloud extracted from the image delivered by the depth camera 30 may be aligned not on a point cloud modeling the assembly, but directly on a three-dimensional mesh modeling the current state of the assembly of parts.

[0083] 3. Update the position by measuring the acceleration ( Figure 4 Step 130) :

[0084] In order not to have to repeat steps 110 to 120 too often, which is expensive in terms of computing time, the calculation algorithm will advantageously update the position by calculating the movement of the module relative to the previously saved position based on the measurements delivered by the inertial measurement unit in a step 130. Thus, the position can be updated in a conventional manner using, for example, Kalman filtering.

[0085] However, for each update of the position, a new three-dimensional image is captured by the depth camera. This image can be advantageously transformed into a point cloud and aligned with the "target" point cloud based on the previously calculated movement, and then added to the "source" point cloud. In order to prevent the "source" point cloud from containing too many points (which could cause memory problems), the algorithm can advantageously filter the "source" cloud by randomly removing points when areas are too dense. More intelligent ways of selecting excess points can also be envisioned, by preferentially removing relatively inconsistent points and / or by replacing small dense point sets with the centroid of the small dense point set.

[0086] Step 130 may be repeated a certain number of times, for example a predetermined number of times N. Figure 4 As shown. However, the movement calculation loses significant accuracy over time, especially because the acceleration measurements are integrated twice to calculate the movement. To combat this phenomenon, after N iterations, the calculation algorithm can advantageously use, for example, an ICP type algorithm to perform a realignment of all the point clouds acquired by the depth camera to the "target" point cloud in step 130.

[0087] 4. Reliability of estimated position ( Figure 4 Step 140) :

[0088] As long as the estimated position is considered reliable, the update of the position estimate of the electronic module using the measurements delivered by the inertial measurement unit can be performed. Therefore, it is desirable to check the reliability of the estimate periodically. To this end, in step 140, the calculation algorithm can be advantageously configured to align the point cloud extracted from the image captured by the camera with the "source" point cloud based on the estimated position of the "source" point cloud. The reliability of the position is then defined as the sum of the Euclidean distances of each point in the "source" cloud relative to the "target" cloud. If the reliability is above a certain predetermined threshold, the position estimate using the measurements from the inertial measurement unit must be considered unreliable. The calculation algorithm is then configured to restart the steps from step 110.

[0089] In addition to the savings in computing costs, associating the measurements from the inertial measurement unit with the processing of the images captured by the depth camera can also have many advantages. Thus, for example, in the context of tightening operations in an assembly process in which an electronic module is applied to control components, a depth camera may not be able to give a sufficiently clear image for objects that are too close. Therefore, even when the portable tightening tool is as close as possible to the tightening element, the measurements delivered by the inertial measurement unit can fill in the periods where the camera may be insufficient by making it possible to estimate the position of the portable tightening tool. In addition, although the depth camera already makes it possible to obtain millimeter-level accuracy levels for position estimates, the gain in accuracy can still be improved by associating acceleration measurements to update the estimated position (for example by Kalman filtering).

Claims

1. A system for controlling manual operations during the production of an assembly of components, the system comprising a portable hand-held tool (1) allowing an assembly technician to perform manual operations on a plurality of elements (20, 21) of the assembly of components, and a positioning device at least partially integrated into an electronic module (3) attached to the portable hand-held tool (1), the positioning device being capable of determining the position of at least one manual operation associated with an element (20, 21) by positioning the portable hand-held tool (1) in a three-dimensional coordinate system, characterized in that The three-dimensional coordinate system is associated with a three-dimensional model of the assembly of components, The system comprises a memory (33a) storing a plurality of CAD files allowing the construction of a three-dimensional modeled image of the assembly of the component at various stages of production, The positioning device comprises a depth camera (30) belonging to the electronic module (3) and a processing module (32) capable of estimating and saving the current position of the portable handheld tool (1), wherein the current position is estimated by aligning a point cloud extracted from at least one current image captured by the depth camera (30) with a grid or point cloud extracted from a three-dimensional modeling image representing the current state of the component, wherein the three-dimensional modeling image representing the current state is constructed based on a CAD file from among the multiple CAD files stored in the memory (33a).

2. The system according to claim 1, characterized in that The positioning device also includes an inertial measurement unit (31) integrated into the electronic module (3), and the processing module (32) is capable of estimating the movement of the portable handheld tool (1) from the last estimated and saved current position based on the measurement results delivered by the inertial measurement unit (31), and is capable of estimating at least one updated position of the portable handheld tool (1) based on the last current position and the movement.

3. A system according to any one of the preceding claims, characterised in that The processing module (32) is integrated into the electronic module (3).

4. The system according to claim 1, characterized in that The processing module (32) is integrated into the remote device (4).

5. The system according to claim 2, characterized in that The processing module (32) is integrated into the remote device (4).

6. The system according to claim 5, characterized in that The electronic module (3) comprises communication means (37) capable of transmitting images captured by the depth camera (30) and measurements from the inertial measurement unit (31) to the remote device (4) for processing by the processing module (32).

7. The system according to any one of claims 1, 2, 4 to 6, characterized in that: The system also includes at least one portable user interface (34, 40) capable of automatically indicating to the assembly technician set point parameters to be applied to the component (20, 21) based on the position of the associated manual operation determined by the positioning device.

8. The system according to claim 7, characterized in that The at least one portable user interface (34, 40) is integrated into the electronic module (3) and / or the mobile phone and / or the portable tablet computer.

9. The system according to claim 7, characterized in that The user interface is a display screen.

10. The system according to claim 8, characterized in that The user interface is a display screen.

11. The system according to any one of claims 8 to 10, characterized in that The set point parameters are pre-stored in the memory (33a) of the system, the memory storing a set of set point parameters to be applied to each of the plurality of elements (20, 21) of the assembly of the component.

12. The system according to any one of claims 1, 2, 4 to 6, 8 to 10, characterized in that The electronic module (3) comprises a confirmation device capable of triggering the automatic saving of information related to the manual operation.

13. The system according to claim 12, characterized in that The confirmation device is a manual operation button (35), the actuation of which triggers the automatic saving of information related to the manual operation.

14. The system according to claim 12, characterized in that Said information comprises the time of confirmation and a reference mark allowing unique identification of the element associated with said manual operation.

15. The system according to claim 13, characterized in that Said information comprises the time of confirmation and a reference mark allowing unique identification of the element associated with said manual operation.

16. The system according to any one of claims 1, 2, 4 to 6, 8 to 10, 13 to 15, characterized in that The handheld tool (1) is a tightening wrench or a screwdriver, a riveting machine, a stapler, a drill, a cutting machine or a welding machine.

17. A method for controlling manual operations in a production process of an assembly of components, the method comprising the steps of: - an assembly technician uses a portable hand tool (1) to perform manual operations on a plurality of elements (20, 21) of the assembly of the component; as well as - determining the position of a manual operation associated with an element (20, 21) by locating the position of the portable hand tool (1) in a three-dimensional coordinate system, The method is characterized in that The three-dimensional coordinate system is associated with a three-dimensional model of the assembly of components, The method comprises the steps of storing in a memory a plurality of CAD files allowing the construction of a three-dimensional modeled image of the assembly of the component at various stages of production, And the determining step includes image capture by a depth camera (30) belonging to an electronic module (3) attached to the portable handheld tool (1), and a step (110-120) of estimating and saving the current position of the portable handheld tool (1) by aligning a point cloud extracted from at least one current image captured by the depth camera (30) with a grid or point cloud extracted from a three-dimensional modeling image representing the current state of the component, wherein the three-dimensional modeling image representing the current state is constructed based on one CAD file from among the multiple CAD files stored in the memory (33a).

18. The method according to claim 17, characterized in that The method further comprises the following steps (130): estimating the movement of the portable handheld tool (1) from a last estimated and saved current position based on measurement results delivered by an inertial measurement unit (31) belonging to the electronic module (3), and estimating at least one updated position of the portable handheld tool (1) based on the last current position and the movement.

19. The method according to claim 17 or 18, characterized in that The method further includes the step of automatically transmitting to a portable user interface (34, 40) set point parameters to be applied to the element based on the location of the associated manual operation from the determining step.

20. The method according to claim 19, characterized in that The transmission step comprises displaying said set point parameters on a display screen which is integrated into said electronic module (3), mobile phone and / or portable tablet computer.

Images