Interactive 3D method and system for assisting during manual assembly or disassembly operations

Abstract

The invention relates to a method for assisting a user in performing an assembly sequence comprising a series of part assembly operations. For each operation in the series, the method comprises the following steps: displaying a 3D image of a real part to be manipulated within the user's field of vision using a display (2) held in front of the user's eyes, while leaving the user's surroundings visible in the field of vision; capturing, by means of a capture device (SNS) attached to the user's hand, translational and / or rotational movements of the user; moving the displayed image of the part by applying the captured movements in an active state; detecting, by the capture device, a command applied by the user in the active state; and upon detection of the command in the active state, activating the next operation in the series. Figure 1

Description

Title of the invention: 3D interactive method and system for assisting during manual assembly or disassembly operations. Technical field

[0001] The present invention relates to the assembly of mechanical parts, particularly those supplied in a kit for assembly. The present invention also relates to maintenance operations requiring the partial or complete disassembly of a system, particularly for replacing a faulty part. State of the art

[0002] Many objects, such as furniture or more complex systems, are sold in kit form, containing parts to be assembled. Generally, printed assembly instructions are provided with the kit. Such instructions are also used to describe assembly procedures in the manufacturing industry and for training purposes. Furthermore, printed maintenance manuals are produced to describe maintenance procedures for complex systems. These manuals typically describe sequences for assembling and / or disassembling and reassembling the components of a system.

[0003] However, it is often difficult to read and understand diagrams presented in such assembly or maintenance instructions, especially since creating sufficiently clear diagrams can be quite challenging. Furthermore, some people have difficulty visualizing parts in three dimensions from two-dimensional diagrams. Thus, it can be difficult to identify a part from its representation in a two-dimensional diagram and to determine the correct position for assembly. In addition, creating sufficiently clear and comprehensive assembly or maintenance instructions requires considerable expertise and time, and can therefore be relatively expensive. As a result, many manufacturers neglect this essential task.

[0004] Furthermore, assembly and maintenance operations are manual tasks. It can therefore be difficult to consult printed instructions while simultaneously handling parts or tools. In addition, some maintenance operations must be carried out under harsh conditions that make consulting printed instructions difficult. Such harsh conditions can be encountered, for example, during work at great heights, in confined spaces, or when handling fluids such as mechanical oils, outdoors in adverse climatic conditions, underwater or even in space.

[0005] It is therefore desirable to be able to provide access to assembly and disassembly information for parts, sufficiently explicit to be useful in any situation or environment. It is also desirable to be able to leave the operator's hands free while allowing access to this information at any time. It may also be desirable to reduce the duration of assembly and disassembly operations, reduce the risk of errors, and allow results with the required level of quality to be achieved without depending on the operator's experience and training. Summary

[0006] Embodiments relate to a method for assisting a user in performing an assembly sequence comprising a series of operations for assembling mechanical parts, the method comprising, for each operation in the series of operations, steps consisting of: displaying a three-dimensional image of a real part to be manipulated during the operation, in a part of the user's field of vision using a display held on the user's head and in front of the user's eyes, while leaving the user's environment visible in another part of the user's field of vision; capturing, using a capture device associated with a hand of the user, rotational movements of the user; moving the displayed image of the part by applying the captured movements, in an active state of the capture device; detecting the end of an operation in the active state;and upon detection of the end of an operation in the active state, activate the next operation in the series of operations.

[0007] Thanks to these arrangements, the user can perform a sequence of manual operations, with the ability to view an image of the part to be manipulated from all angles at each operation. Furthermore, viewing the image of the part leaves the user's hands free to perform the operation using the part and does not require space for a printed diagram or display screen. The user thus remains completely free to move around and can do so without having to use their hands to carry equipment other than that which may be necessary to perform the sequence of operations. The method can therefore be implemented in any environment, and particularly in harsh or confined environments.

[0008] According to one embodiment, the end of the operation is triggered by: a first command applied by the user in the active state, and / or by a pattern recognition application during recognition in an image of the field of user vision, provided by an image sensor, that the displayed part P2 is correctly mounted.

[0009] Thus, thanks to the implementation of a shape recognition application, a subsequent operation in the sequence of assembly operations can be activated automatically. This feature informs the user that they have correctly assembled the part corresponding to the detected assembly operation.

[0010] According to one embodiment, the method comprises steps consisting of: detecting with the capture device a second command applied in the active state by the user; upon detection of the second command in the active state, switching to an inactive state in which the displayed image of the part is frozen; detecting with the capture device a third command applied in the inactive state by the user; and upon detection of the third command in the inactive state, switching to the active state in which the displayed image of the part follows the hand movements captured by the capture device.

[0011] Thus, the user can use both hands without the image of the displayed part moving.

[0012] According to one embodiment, the method comprises steps consisting of: detecting, using the capture device, a fourth command applied in the active state by the user; and upon detection of the fourth command in the active state, activating a previous operation in the series of operations.

[0013] Thanks to the fourth command combined with the command enabling the activation of a subsequent operation, the user can navigate through the sequence of operations to be executed.

[0014] According to one embodiment, the method comprises steps consisting of: detecting, using the capture device, a fifth manual command applied in the active state by the user; upon detection of the fifth command, switching to a first state in which the displayed image of the part is frozen; detecting, using the capture device, the sixth, seventh, and eighth manual commands applied in the first state by the user; upon detection of the sixth command in the first state, enlarging the displayed image of the part; upon detection of the seventh command in the first state, shrinking the displayed image of the part; and upon detection of the eighth command in the first state, switching to the active state.

[0015] Through these commands, the user can access a state where he can enlarge or shrink the image of the displayed part, and return to the active state in which he can manipulate the image of the displayed part.

[0016] According to one embodiment, the commands are provided by the user: by pressing a push button, and / or by applying shocks to the capture device, and / or by hand gestures recognized by a recognition application shape in images of the user's field of vision provided by an image sensor, and / or by voice commands recognized by a voice command application coupled to a microphone positioned in the capture device or on the display.

[0017] According to one embodiment, the method includes a step of animating the image of the part to move it from a current position to a final position in which the corresponding real part must be positioned to perform the operation.

[0018] Embodiments may also relate to a system for assisting a user in carrying out a sequence of mechanical parts assembly, the assembly sequence comprising a series of operations, the system comprising: a motion capture device to be associated with a hand of the user to capture translational and / or rotational movements of the user; a display to be placed on the head and in front of the eyes of the user, while leaving the environment of the user visible in a part of the user's field of vision; and a processing unit connected to the capture device and the display, and configured to acquire motion data provided by the set of sensors and control the display, the system being configured to implement the method defined above.

[0019] According to one embodiment, the system includes augmented reality glasses, or a head-up display associated with a visor.

[0020] Thus, the user does not have to carry equipment when he has to move around, and does not need a nearby location to place a display screen, or a control device such as a mouse, or even printed documents.

[0021] According to one embodiment, the capture device includes translation and / or rotation motion sensors housed on the back of a glove, or in a ring to be put on a finger of the user, or in a bracelet to be placed around the wrist of the user.

[0022] According to one embodiment, the capture device includes a push button that can be actuated by a finger of the hand, to manually control changes in the state of the system.

[0023] Being associated with motion sensors, such a push button can be operated with the same hand supporting the sensor assembly.

[0024] According to one embodiment, the push button is positioned on the index finger of the hand, so that it can be operated by the thumb of the hand in the closed position.

[0025] According to one embodiment, the capture device includes sensors configured to detect translational and rotational movements along three perpendicular directions.

[0026] According to one embodiment, the sensors are associated with the user's hand such that a first direction of the three directions is oriented from the wrist to the fingertips or vice versa, a second direction of the three directions is oriented from an inner lateral edge to an outer lateral edge of the hand, or vice versa, and a third direction of the three directions is oriented in a direction perpendicular to the palm of the hand.

[0027] According to one embodiment, the system includes an image sensor associated with a pattern recognition application, and / or a microphone associated with a speech recognition application. Brief description of the figures

[0028] The present invention will be better understood with the aid of the following description of exemplary embodiments with reference to the accompanying figures, in which identical reference signs correspond to structurally and / or functionally identical or similar elements.

[0029] [Fig-1] Figure 1 schematically represents an execution assistance system of mechanical part assembly operations, according to a method of embodiment,

[0030] [Fig.2] Figure 2 schematically represents a motion capture device used in the assistance system, according to one embodiment,

[0031] [Fig. 3] Figure 3 schematically represents a state and transition diagram implemented in the assistance system, according to one embodiment,

[0032] [Fig. 4] Figures 4A, 4B and 4C represent an example of a sequence of images displayed by the assistance system, according to one embodiment,

[0033] [Fig. 5] Figures 5A and 5B respectively represent the dorsal surface and the palm of a hand on which is arranged a system for assisting in the execution of mechanical part assembly operations, according to another embodiment. Detailed description

[0034] Figure 1 shows a system for assisting in the execution of mechanical part assembly operations, according to one embodiment. The assistance system comprises a motion capture device SNS installed, for example, in or on a glove 1, for example in a part corresponding to the back of the hand, a processing unit UC, an interface and power supply circuit INT, and augmented reality glasses 2. The interface circuit INT is connected by wired links L1, L2 to the sensor SNS and includes a power supply circuit VS for the SNS capture device and a wireless communication circuit CM3 for communicating with the processing unit UC via a wireless communication interface CM1 associated with the processing unit. The processing unit UC also communicates with a control unit PU installed in the glasses 2. Communication is achieved via the CM1 communication interface and a CM2 wired or wireless communication interface, also installed in the glasses 2 and connected to the PU control unit. The glasses allow the wearer to see both their surroundings and information displayed superimposed on them. Thus, part of the user's field of vision is obscured by the displayed information.

[0035] Figure 1 shows a dorsal view of glove 1. The SNS motion capture device associated with glove 1 is configured to capture rotational and optionally translational movements of the hand along orthogonal X, Y, and Z directions, with the X direction oriented from the wrist to the fingertips, the Y direction oriented from an inner to an outer lateral edge of the glove, and the Z direction extending perpendicularly to the back of the glove. The signals provided by the SNS capture device may include linear displacement amplitudes or velocities along each of the X, Y, and Z axes, and rotational angles or velocities about the X, Y, and Z axes.

[0036] The INT interface circuit can be separate or integrated into the same housing as the SNS capture device. The SNS capture device can be connected via a wired connection L3 to a push button 5, allowing the user to input commands other than by movements of the hand wearing the glove 1. In one embodiment, the button 5 is positioned so that it can be operated using the thumb of the same hand, for example by making a fist.

[0037] The processing unit (PU) is a fixed or mobile terminal connected to the Internet, for example, a personal or laptop computer, tablet, or mobile phone. The PU includes an application for displaying 3D mechanical parts, with the glasses 2 being coupled to the PU to provide the display monitor function. The PU can access a library of 3D parts, for example, accessible via the Internet. The 3D parts can be presented in assembly or maintenance instructions.

[0038] The wireless connection between the processing unit UC and the interface circuit INT (communication circuit CM3) can be of the Bluetooth, BLE ("Bluetooth Low Energy") or WiFi™ type. The connection between the processing unit UC and the communication interface CM2 can be wired or wireless, for example of the Bluetooth or WiFi™ type.

[0039] Figure 2 shows circuits of the SNS capture device, according to one embodiment. The SNS capture device comprises an IPS sensor array integrating accelerometric and gyroscopic sensors along three perpendicular detection axes, a processing unit MC, such as, for example, a microcontroller or microprocessor, and an input / output circuit IO connected to the wired links L1, L2. The SNS capture device is associated with the glove 1 in such a way to ensure that the detection axes of the IPS sensors coincide with the X, Y, Z axes described previously.

[0040] According to one embodiment, the MC processing unit of the SNS capture device is configured to operate according to the state and transition diagram shown in Figure 3. This diagram includes the states PF off, PN on, ACT active, STB inactive, FF forward, RW backward, and ZM zoom, and the transitions T1 to T13. The T1 transition changes the SNS sensor from the PF off state to the PN on state. The T1 transition can be triggered by switching an on / off button on the INT interface or on the SNS capture device to the on state. In the PN on state, the MC processing unit attempts to establish a wireless communication link with the UC processing unit. The T2 transition from the PN on state to the ACT active state is triggered by the establishment of the wireless link between the INT interface circuit and the UC processing unit.The transition T3 from any state to the off state PF is triggered by switching the on / off button to the off state. In the active state ACT, the processing unit UC commands the three-dimensional (3D) display of a user-selected mechanical part through the glasses 2. Simultaneously, the processing unit MC collects measurement signals from the IPS sensors, converts them into digital measurements, and transmits them to the processing unit UC via the previously established wireless connection. In response to receiving these measurements, the processing unit UC commands rotational and, optionally, translational movements of the mechanical part displayed by the glasses 2.

[0041] The T4 transition from the active state ACT to the inactive state STB is activated by a brief press of button 5. In the STB state, the processing unit MC does not send any measurements wirelessly to the processing unit UC. The user can thus use both hands to perform the current operation without altering the display of the part on the goggles 2. The T5 transition from the inactive state STB to the active state ACT is activated by another brief press of button 5. Accessing the feed state FF allows the processing unit UC to display the next mechanical part on the goggles 2, in a sequence of assembly operations currently being executed and previously selected by the user. The feed state FF is reached from the active state ACT by the T6 transition, triggered, for example, by two successive brief presses of button 5.The return to the ACT state via transition T7 occurs as soon as the command to display the next part has been transmitted to the processing unit CU or when the next part is displayed by the glasses 2. Accessing the rewind state RW allows the processing unit CU to display a previous mechanical part in the currently running assembly sequence. The rewind state RW is reached from the active state ACT via transition T8, triggered, for example, by three successive short presses of button 5. The return... to the ACT state by the T9 transition is carried out as soon as the display command for the previous part has been transmitted to the processing unit UC or when the previous part is displayed by glasses 2.

[0042] The zoom state ZM is reached from the active state ACT by transition T10 and from the inactive state STB by transition T12, these transitions being triggered by a long press (e.g., greater than 2 s) on button 5. In this state, the processing unit MC transmits only the rotation measurements of the glove around an axis, for example, the X-axis, via the wired connection to the processing unit UC. In response to these measurements, the processing unit UC commands the enlargement or reduction of the mechanical part displayed by the glasses 2. For example, a rotation around the X-axis or a translation along the X-axis of the hand (forward) triggers an enlargement of the displayed part, and a contrary movement triggers a reduction of this part. The magnitude of the enlargement or reduction is defined according to the magnitude of the movement measured by the capture device SNS.The transition T11 from the zoom state ZM to the active state ACT is triggered, for example, by a short press on button 5. The transition T13 from the zoom state ZM to the inactive state STB is triggered, for example, by another long press on button 5.

[0043] Thanks to these arrangements, the user can virtually view and manipulate the displayed mechanical part in active mode (ACT) or zoom mode (ZM) with the hand wearing glove 1, while simultaneously viewing the actual part through glasses 2 and having their other hand free to manipulate the actual part or a tool. If the user wishes to use both hands to perform this operation without modifying the part display parameters via the control unit (CU), they can switch to inactive mode (STB). The user thus has both hands available to perform complex operations while having a detailed image in their field of vision showing the operation to be performed and the actual system to which this operation is to be applied.

[0044] According to one embodiment, the transitions to the zoom states ZM, FF and RW can be triggered by shocks or impacts applied to the glove, for example with the other hand, and detected from the motion measurements provided by the SNS capture device.

[0045] According to another embodiment, the active state ACT is activated as long as button 5 is pressed, while the inactive state STB is activated when button 5 is not pressed. In this way, the user can easily perform several repeated hand movements from an initial position to control a large-amplitude movement of the image of the part in the same direction without the part recoiling when the user returns their hand to the initial position to repeat the same hand movement. Furthermore, from an ergonomic point of view, pressing button 5 can This is similar to grasping the part to be manipulated, until the button is released. In this case, the ACT and STB states can be swapped in the diagram in Figure 3, and the transitions between the active state and the zoom state are no longer necessary.

[0046] Figures 4A, 4B and 4C represent examples of images that can be displayed by the glasses 2 controlled by the processing unit UC. Figures 4A-4C show images of two parts PI, P2 to be assembled, the image of part PI being fixed and the image of part P2 being movable using glove 1. In Figure 4A, the image of part P2 does not have a correct assembly orientation with the image of part PL. An arrow Fl can be displayed to show what rotational movement can be applied to the image of part P2 so that it is presented in a correct assembly orientation relative to the image of part PL. In the example in Figure 4A, a rotation around the X-axis can correctly orient the image of part P2 relative to the image of part PL. An animation, for example highlighting the image of part P2, can indicate to the user when they have correctly oriented the image of part P2.In Figure 4B, the image of part P2 is displayed with its correct orientation relative to the image of part PL. An arrow F2 can be displayed to indicate to the user a translational movement to apply to the image of part P2 to assemble it with part PL. This translational movement is, for example, to be performed in a direction perpendicular to the Z direction. Figure 4C shows the images of the two parts PI, P2 correctly assembled together.

[0047] Thus, the user can virtually manipulate the part P2 to be assembled and visualize how it should be assembled, while having the possibility of manipulating the real parts with his hands at the same time.

[0048] Instead of the arrows Fl, F2, the correct movements to be applied to the image of part P2 can be indicated by animations controlled by the processing unit CU, slowly moving the image of part P2 between the initial position and the final position to be reached. This animation can be displayed when the first or subsequent operation of a sequence of operations to be executed is activated, and / or following the activation of a command provided for this purpose using glove 1.

[0049] According to one embodiment, the glasses 2 are equipped with one or two image sensors 6 (Figure 1) transmitting images of the user's field of vision to the control unit CU. Furthermore, the control unit CU includes a pattern recognition application that the user can activate using a command applied to the glove 1 or the button 5. The pattern recognition application is configured to analyze the images of the user's field of vision, transmitted by the camera 6, and identify the actual part held by the user in The transmitted images are then displayed, and a highlighting element is shown indicating whether the image of the displayed part P2 corresponds to the part held by the user. The displayed highlighting element could be, for example, a color change in the displayed image P2 of the part.

[0050] The pattern recognition application can also be used to recognize hand gestures of the user captured by the image sensor 6 in the user's field of vision, and trigger the state changes described above according to the gestures detected.

[0051] The pattern recognition application can also be used to recognize in the user's field of vision that the displayed part P2 is correctly mounted and automatically proceed to the next operation in the sequence of operations. This recognition can simply consist of comparing the image of the part provided by the image sensor 6 in the user's field of vision with an image of the part correctly mounted on the system being assembled, stored by the processing unit CU.

[0052] According to one embodiment, the assistance system may include a microphone 7 installed in the SNS capture device or on the glasses 2 and coupled to the processing unit CU. Furthermore, the processing unit CU includes a speech recognition application and translates the words spoken by the user and recognized by the application into system commands to trigger the state changes described above. Thus, the button 5 can be omitted by implementing a gesture recognition application or a speech recognition application.

[0053] It will be evident to those skilled in the art that the present invention is susceptible to various embodiments and applications. In particular, the invention is not limited to the implementation of augmented reality glasses. Indeed, such glasses can be replaced by any display device attached to the user's head and configured to display, within the user's field of vision, or project into the user's eyes, images partially covering the user's field of vision. Thus, the glasses 2 can be replaced, for example, by a visor attached to the user's head and associated with a head-up display.

[0054] The invention is also not limited to the implementation of motion sensors disposed in a glove. Indeed, such sensors can be housed in a ring to be placed on a finger of the user or a bracelet to be placed around the wrist.

[0055] According to an embodiment illustrated by Figures 5A and 5B, the SNS capture device is disposed in a fabric pocket 11 attached to the back of the hand 10 by means of a wristband or strap 12; the interface and power supply circuit INT can be attached to the wrist by means of a wristband or strap or housed in the pocket 11. The strap 12 can be held around the hand 10 by a fastening device 15, for example of the hook and loop type. The push button 5 is mounted on a ring 13, for example formed by a strap to be placed on the index finger of the hand, the pocket 11 and the ring 13 being linked to each other by a sleeve 14 housing the wire connection L3.

[0056] In certain applications, it may not be necessary to capture hand movements along three perpendicular directions. Thus, the SNS capture device may only detect rotational and possibly translational movements, along a single direction or two perpendicular directions. Nor is it necessary for the detection directions of the SNS capture device to be perpendicular. Indeed, it is sufficient that these directions be different and that the possible third direction is not located in the plane defined by the other two directions.

Claims

Demands

1. 1. A method for assisting a user in performing an assembly sequence comprising a series of operations for assembling mechanical parts, the method comprising, for each operation in the series of operations, the steps of: displaying a three-dimensional image (P2) of an actual part to be manipulated during the operation, in a portion of the user's field of vision using a display (2) held on the user's head, while leaving the user's environment visible in another portion of the user's field of vision; capturing, using a capture device (SNS, 6), rotational movements of the user's hand; moving the displayed image of the part by applying the captured movements, in an active state of the capture device; detecting the end of an operation in the active state; and upon detection of the end of an operation in the active state, activating a subsequent operation in the series of operations.

2. 2. Method according to claim 1, wherein the end of operation is triggered by: a first command applied by the user in the active state, and / or by a pattern recognition application during recognition in an image of the user's field of vision, provided by an image sensor (6), that the displayed part P2 is correctly mounted.

3. 3. A method according to claim 1 or 2, comprising steps of: detecting a second command applied in the active state by the user; upon detection of the second command in the active state, transitioning to an inactive state in which the displayed image (P2) of the part is frozen; detecting, using the capture device, a third command applied in the inactive state by the user; and upon detection of the third command in the inactive state, transitioning to the active state in which the displayed image of the part follows hand movements captured by the capture device (SNS,

4. oj.

4. A method according to any one of claims 1 to 3, comprising steps of: detecting a fourth command applied in the active state by the user; and upon detection of the fourth command in the active state, activating a previous operation in the series of operations.

5. 5. A method according to any one of claims 1 to 4, comprising steps of: detecting a fifth command applied in the active state by the user; upon detection of the fifth command, transitioning to a first state in which the displayed image of the part is frozen; detecting sixth, seventh and eighth commands applied in the first state by the user; upon detection of the sixth command in the first state, enlarging the displayed image (P2) of the part; upon detection of the seventh command in the first state, shrinking the displayed image of the part; and upon detection of the eighth command in the first state, transitioning to the active state.

6. 6. Method according to claim 5, wherein the commands are provided by the user: by pressing a push button (5), and / or by applying shaking to the capture device (SNS), the capture device being associated with the hand, and / or by hand gestures recognized by a pattern recognition application in images of the user's field of vision provided by an image sensor (6), and / or by voice commands recognized by a voice control application coupled to a microphone positioned in the capture device (SNS) or on the display (2).

7. 7. A method according to any one of claims 1 to 6, comprising a step of animating the image of the part to move it from a current position to a final position in which the corresponding real part must be positioned to perform the operation.

8. 8. A system to assist a user in performing a sequence of mechanical parts assembly, the assembly sequence comprising a series of operations, the system comprising: a motion capture device to be associated with a hand of the user to capture translational and / or rotational movements of the user; a display to be placed on the head, while leaving the user's environment visible in a part of the user's field of vision; and a processing unit connected to the capture device and the display, and configured to acquire motion data provided by the sensor set and control the display, the system being configured to implement the method according to any one of claims 1 to 7.

9. 9. System according to claim 8, wherein the display comprises augmented reality glasses (2), or a head-up display associated with a visor.

10. 10. System according to claim 8 or 9, wherein the capture device (SNS) comprises translational and / or rotational motion sensors (IPS) housed on the back of a glove (1), or in a ring to be put on a finger of the user, or in a bracelet to be placed around the wrist of the user.

11. 11. System according to any one of claims 8 to 10, wherein the capture device (SNS) includes a push button (5) which can be actuated by a finger of the hand, to manually control changes of state of the system.

12. 12. System according to claim 11, wherein the push button (5) is positioned on the index finger of the hand, so that it can be operated by the thumb of the hand in the closed position.

13. 13. System according to any one of claims 8 to 12, wherein the capture device (SNS) comprises sensors (IPS) configured to detect translational and rotational movements along three perpendicular directions (X, Y, Z).

14. 14. A system according to claim 13, wherein the sensors (IPS) are associated with the user's hand such that a first direction (X) of the three directions is oriented from the wrist to the fingertips or vice versa, a second direction (Y) of the three directions is oriented from an inner lateral edge to an outer lateral edge of the hand, or vice versa, and a third direction (Z) of the three directions is oriented in a direction perpendicular to the palm of the hand.

15. 15. System according to claim 14, comprising: an image sensor (6) associated with a pattern recognition application, and / or a microphone (7) associated with a speech recognition application.

Images