Augmented Management of Operator Training for Industrial Processes
An augmented reality system with real-time gesture analysis and feedback improves industrial training efficiency by ensuring correct gesture execution and reducing risks, enabling operators to learn autonomously.
Patent Information
- Application Number
- FR2022006122
- Authority / Receiving Office
- FR · FR
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2022-06-22
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2042-06-22
AI Technical Summary
Existing training methods for industrial processes are inefficient, requiring long training times, posing risks to operators, and failing to provide effective feedback on hand gestures, leading to incomplete understanding of real-world tasks.
An augmented reality-based system that captures and analyzes an operator's hand gestures in real-time, using machine learning algorithms to compare them with predetermined gestures, providing immediate feedback and correction guidance.
Enhances training efficiency by ensuring correct execution of gestures, reducing training time, and minimizing risks, while allowing operators to learn and validate their actions autonomously.
Smart Images

Figure 00000023_0000 
Figure 00000024_0000 
Figure 00000024_0001
Abstract
Description
Title of the invention: Augmented Training Management for Operators Performing Industrial Processes Technical field
[0001] The invention relates to instructions provided to an operator performing industrial processes. More particularly, the invention relates to an augmented management system based on a coupling between augmented reality tools and the measurement of finger movements of an operator during training for tasks associated with one or more industrial processes. Context
[0002] In the field of industrial processes, assembly, disassembly, operation, and maintenance work involves complex actions. Each action requires specific knowledge and techniques for each machine and piece of equipment, depending on the associated tasks. Due to the variety of machines and equipment, industrial companies are obligated to provide regular training for operators, users, workers, and technicians (collectively, "operators"). Therefore, improving operators' autonomy in managing the skills necessary for their versatility through a new training method would be advantageous for industry.
[0003] Among known training methods, on-the-job training takes place directly in the work environment. It is the most effective because the operator learns through experience by directly following the instructions provided during the training. However, this type of training can require lengthy training periods. Some machines and services may therefore need to be shut down due to the operators' absence. Also, if equipment is damaged during training, additional repair costs will increase. Furthermore, industrial processes often include hazardous tasks that pose risks to operators.
[0004] In industrial processes where manual intervention is anticipated, a hand gesture is a non-verbal form of communication between a person and a machine. Hand gesture recognition is therefore considered an important aspect of human-computer interaction (HCI), which enables computers to capture and understand hand gestures and then execute actions. Standardized hand gesture recognition systems are vision-based, incorporating preprocessing (including hand area detection), feature extraction, and Labeling. A commonly used training technique is that of surface electromyography (sEMG) sensors for integrated hand gestures. Machine learning (ML) is used to solve the problem of recognizing hand gestures from sEMG signals. Classifiers for hand gesture recognition include, but are not limited to, support vector machines (SVMs), k-nearest neighbors (kNNs), decision trees, random forests, linear discriminant analysis, artificial neural networks (ANNs), convolutional neural networks (CNNs), and their equivalents.
[0005] Hand gestures are frequently used in HCI and augmented reality systems due to their interpretability and speed of communication. The variety and diversity of gestures have a significant effect on the identification rate and the effectiveness of the hand gesture identification method. Machine learning methods can be used to improve the rate at which hand gestures are recognized.
[0006] Various technologies can enable an operator to perform training in a mixture of real and virtual worlds (integrating, for example, elements of both). For example, hand gesture recognition could be obtained from a depth sensor (for example, a sensor of the type called "Kinect") to measure the dissimilarity between hand shapes (see Robust Part-Based Hand Gesture Recognition Using Kinect Sensor, Zhou Ren, Junsong Yuan, Jingjing Meng, Zhengyou Zhang, IEEE TRANSACTIONS ON MULTIMEDIA, Vol. 15, No. 5 (August 2013) ("the Zhou reference") (KINECT® is a registered trademark owned by Microsoft Corporation). Instead of wearing a data glove, the depth sensor can robustly detect and segment hands to build a valid basis for hand gesture recognition.
[0007] Furthermore, real-time hand gesture recognition could be achieved from sEMG signals (see Real-Time Surface EMG Pattern Recognition for Hand Gestures Based on an Artificial Neural Network, Zhen Zhang, Kuo Yang, Jinwu Qian and Lunwei Zhang, www.mdpi.com / journal / sensors (July 18, 2019)). A wristband acquires the sEMG signals, and a sliding window approach is applied to segment the corresponding data in feature extraction. An artificial neural network (ANN) is trained on the dataset, and a testing method is used in which the gesture will be recognized when the recognized labeling times reach the activation threshold of the ANN classifier. Therefore, the recognition system could be capable of re- to know a gesture before it is finished.
[0008] In another example, certain display devices, such as various head-mounted displays, may have transparent displays that allow the superimposition of displayed images onto a real-world background environment. The images may appear in front of the real-world background environment when viewed through the transparent display. In particular, the images may be displayed on the transparent screen in such a way that they appear blended with elements of the real-world background environment in what may be called augmented reality.
[0009] In an augmented environment, augmented reality technology can improve and increase the efficiency of training for tasks associated with industrial processes, enabling training based on a learning-by-doing approach. For example, US patent 9,643,314 discloses a method for controlling a robot to manipulate objects in a virtual world by acquiring the operator's movements. In this case, the operator's hand gesture is translated to a robot and not compared to an expected result.
[0010] In another example, US patent 10,952,488 discloses a welding interface device that uses an optical sensor and a time-of-flight sensor to collect an image of a welding environment. The device also includes an augmented reality controller to determine a simulated object to be presented in a field of view, a position of the simulated object in the field of view, and a perspective of the simulated object in the field of view. A semi-transparent display of the device presents the rendered simulated object in the field of view according to the position determined by the augmented reality controller, at least a portion of the welding environment being observable through the semi-transparent display when the display presents the rendered simulated object.
[0011] It is therefore understood that training could be conducted virtually to avoid an operator being injured during training, which does not give operators an understanding of the actual work environment and the tools they must use, including a lack of understanding of the actual work process. Consequently, operators cannot fully grasp the appropriate technique.
[0012] Thus, the disclosed invention relates to an automatic recognition and evaluation system for training tasks associated with industrial processes. The disclosed invention proposes a solution for preparing augmented reality-based training sequences, focusing on the acquisition, evaluation, and comparison of different hand gestures. The problem addressed by the invention is that of automatic task recognition to identify tasks associated with a predetermined process and the order in which they are performed. Thus, The invention differs from the prior art by offering real-time analysis of the operator's work, allowing the operator to correct and validate their work. Summary of the invention
[0013] The invention relates to a method for training operators on a predetermined process having a sequence of characteristic gestures, the method implemented by a computer system comprising one or more processors which are capable of executing programmed instructions stored in a memory operationally connected to one or more processors to carry out the process, characterized in that the method comprises the following steps: - an acquisition step carried out by a receiving module of the computer system to receive data corresponding to the operator's first-person view, during which the operator's hand gestures are captured in real time in the first-person view during a training session conducted by the operator, and during which the computer system obtains data from a hand gesture associated with a predetermined process in which the operator participates, this step comprising: - a hand tracking step from a computer system display device incorporating an augmented reality type tracking component, whereby the operator's hand tracking allows detection of a position that corresponds to the beginning or end of a sequence of gestures characteristic of a predetermined ongoing process; - a hand motion tracking step performed by an extraction module of the computer system to extract a set of characteristic gestures sought in the predetermined process in progress, the motion tracking of which from a display device of the computer system makes it possible to control the trajectory and movement of the operator's fingers during a sequence of characteristic gestures, and during which the display device detects all the gestures in the sequence associated with the predetermined process; - a step of identifying the operator's hand position relative to the characteristic gestures of the predetermined process, during which parameters of the movement performed by the operator and recorded during the motion tracking step are used to identify a characteristic gesture, this step being carried out by an extraction analysis module of the computer system to identify the characteristic gesture associated with the predetermined process; and - a comparison step during which the identified characteristic gesture and the characteristic gestures of the predetermined process are compared to determine a variance between them during the predetermined process; so that the actions of an operator performing the predetermined process are recorded for use in the current training session.
[0014] In embodiments of the method of the invention, the acquisition step includes a step of capturing the gestures made by the operator in a field of vision of the display device of the computer system.
[0015] In embodiments of the process of the invention, the comparison step comprises: - the generation of a quality check of the action to indicate where the execution of the identified action is correct; and - the generation of an alert message to indicate where the execution of the identified action is incorrect; - so that the verification and the alert message are generated in real time.
[0016] In embodiments of the method of the invention, the verification of the quality of the gesture and the verification of the quality of the alert message include visual instructions in a field of vision of the display device worn by the operator during the training session.
[0017] In embodiments of the method of the invention, the method further includes a step of providing one or more visual instructions in a text field and / or in an image field in a field of vision of the display device of the computer system.
[0018] In embodiments of the method of the invention, the generation of the alert message includes the generation of one or more tips to prompt the operator to correct incorrect gestures before performing the downstream gestures.
[0019] In embodiments of the method of the invention, the hand movement tracking step includes a step for detecting the end of the sequence of gestures.
[0020] In embodiments of the method of the invention, during the comparison step, one or more gesture recognition models use: - machine learning algorithms; or - deep learning algorithms.
[0021] In embodiments of the method of the invention, the method further includes a step of obtaining temporal synchronization between gestures made by the operator and the characteristic gestures expected during the training session.
[0022] In embodiments of the method of the invention, during the comparison step, the time synchronization obtained is used in sending checks and / or alert messages; so that the operator receives in real time an indication of success or failure of the identified action.
[0023] Other aspects of the invention will become evident from the following detailed description. Brief description of the drawings
[0024] The nature and various advantages of the invention will become more evident upon reading the following detailed description, together with the accompanying drawings, in which the same reference numbers designate identical parts throughout, and in which: [Fig.1] Fig.1 represents a flow diagram of an embodiment of a method of the invention for training operators based on finger movements. [Fig.2] Fig.2 represents examples of grasping gestures including static hand postures. [Fig.3] Fig.3 represents a capture from a video incorporating hand gestures made by the operator and instructions received while carrying out a predetermined process. [Fig. 4] [Fig. 4] represents an operator's field of vision during a training session of an exemplary flat node process [Fig.5] Fig.5 represents a decomposition of steps in an exemplary flat knot process. Detailed description
[0025] Referring now to the figures, in which the same numbers identify identical elements, [Fig. 1] represents a flow diagram of an embodiment of Method 100 of the invention. Method 100 relates to a method for training operators based on finger movements. Method 100 makes it possible to detect the correct performance of one or more gestures associated with one or more specific processes based on the detection of the operator's hand gestures. As used here, any reference to a "hand" (singular or plural) may include one or more fingers (including thumbs) and / or the corresponding wrist.
[0026] As used herein, “operator” refers to a single user comprising an individual participant in a predetermined process. The operator may be an individual member of a team or group participating in a predetermined process. The operator may include a machine (e.g., a robot or a cobot) with built-in privileges that replicate movements in an anthropomorphic manner. A human (or non-human) operator could manage such a machine by remotely managing a live, predetermined process. It is understood that an operator (whether human or machine) may employ one or more electronic systems and / or devices configured to receive command input and configured to automatically send data to at least one other operator (e.g., a human operator managing a process upstream or downstream of the predetermined process currently being carried out).
[0027] As used herein, the term “method” or “process” may include one or more steps performed by at least one computer system comprising one or more processors to execute instructions that identify the activities. Any sequence of tasks is given by way of example and does not limit the described methods to any particular sequence.
[0028] The method 100 of the invention is implemented by a computer system (or "system") to instantly project, based on a first-person view of an operator, gestures of the operator performing a predetermined process. In the context of operator training, the method of the invention allows the operator's gestures to be compared to the gestures associated with one or more predetermined processes to ensure the correct performance of the processes. As used here, a set of gestures constitutes an established way of performing a process in a context that requires the correct performance of the gestures. It is understood that a "characteristic gesture" includes a gesture being associated with a predetermined process of which the gesture is a part.
[0029] Method 100 of the invention uses gesture decomposition methods to associate visual gestures with processes performed based on the first-person view of the operator being trained. The algorithm used in implementing Method 100 allows for continuous improvement throughout all training sessions, ensuring that the system performing the process improves based on the experience it acquires through correction or validation by the operator, particularly regarding the choice of the order of gestures to detect and / or perform during a predetermined process. Thus, the gesture processing performed by the disclosed invention makes it possible to differentiate between the recognition of gestures associated with each process, gestures associated with a particular process, and the recognition of a process in its entirety.It is understood that tools based on artificial intelligence (or "AI") could also be used in the implementation of process 100.
[0030] The operator's gesture data input, including at least one video incorporating one or more first-person views of the operator, can facilitate gesture identification in the operator's first-person view. A gesture analysis process is performed manually during Method 100, and the identification parameters are found by trial and error. In embodiments of Method 100, gestures performed by the operator and identified in their views can be associated with one or more predetermined processes. The operator's gesture data input, captured during one or more operator training sessions, can include a starting position of the operator (or "neutral position") derived from the operator's first-person view and referring to a particular hand (and / or a particular finger) of the operator. This captured gesture data input can also include an operator finger position associated with at least some of the predetermined processes in which the operator intends to train.
[0031] The method 100 therefore comprises extracting features from the operator's first-person view. In one embodiment, these features are based, at least in part, on characteristic gestures associated with predetermined processes recognized during a training session in which the operator participates. The disclosed method uses machine learning methods (e.g., neural network training) to recognize characteristic gestures associated with the processes performed, and to correct gestures that prevent the correct execution of one or more associated processes.
[0032] Referring again to [Fig. 1], a computer system (or "system") that performs process 100 records the operator's hand gestures during a training session conducted by the operator. The system obtains data on a hand gesture possibly associated with a particular process in which the operator participates (or "predetermined process"). It is understood that the gesture data obtained can be associated with a characteristic gesture (or a sequence of characteristic gestures) of a predetermined process in which the operator is being trained.
[0033] The computer system (or "system") that performs process 100 includes a communication network incorporating one or more communication servers (or "servers") that manage data entering the system from various sources. The communication network may include wired or wireless connections and may implement any data transfer protocol known to a person skilled in the art. Examples of wireless connections may include, but are not limited to, radio frequency (RF), satellites, mobile phones (analog or digital), Bluetooth®, Wi-Fi, infrared, ZigBee, local area network (LAN), wireless local area network (WLAN), wide area network (WAN), NFC (near field communication), other wireless communication configurations and standards, their equivalents, and combinations thereof.
[0034] The term "processor" (or, alternatively, the term "programmable logic circuit") refers to one or more devices capable of processing and analyzing data and comprising one or more software programs for processing such data (for example, one or more integrated circuits known to those skilled in the art as being included in a computer, one or more controllers, one or more microcontrollers, one or more microcomputers, one or more programmable logic controllers (or "PLCs"), one or more application-specific integrated circuits, one or more neural networks, and / or one or more other known equivalent programmable circuits). The processor includes software for processing the data captured by the elements associated with the computer system (and the corresponding data obtained) as well as software for identifying and locating variances and identifying their sources in order to correct them.
[0035] The computer system includes at least one display device worn on the operator's head during a training session. The display device incorporates an optical system giving the operator a real, direct first-person view of the space in front of him / her. As used here, "real, direct view" refers to the ability to directly see the operator's hands (including wrists) in a physical environment with the human eye and not created image representations (for example, watching a video of a finger on a screen is not a real, direct view of the finger).It is understood that the display device may include a mobile network device in communication with the computer system's communication network (including devices of the type "augmented reality" and / or "virtual reality", and / or any combinations and / or any equivalents), (see, for example, the head-mounted display device of US patent 11,017,231).
[0036] The display device may include at least one video camera (or "camera") facing the physical environment in which the operator is conducting the training session. The camera may capture video and still images of the actions performed by the operator during training. As used herein, "image," in the singular or plural, refers respectively to still images and images captured in video sequences. The camera may be selected from commercially available cameras, including, without limitation, depth cameras and visible light cameras (RGB cameras). In the following description, the terms "video camera," "camera," "camera," "sensor," and "optical sensor" may be used interchangeably and may refer to one or more devices configured to capture video and still images.
[0037] The display device used records the first-person view incorporating the hand gestures made by the operator during one or more training sessions. The recorded gestures include the operator's behavioral actions during training in a predetermined process. The display device may include various electronic components (including, for example, loudspeakers, inertial sensors, a GPS transceiver, and / or a temperature sensor). However, it is not necessary to map the operator's hands in the physical environment or to identify them relative to the geographic locations of other objects in the operator's view during the training session.
[0038] The display device includes an integrated tracking component to monitor the hand gestures made by the operator during a training session. The component The tracking system can take into account the perception of hand movement (including individual finger and / or wrist movements), which is primarily defined by the movement control from the operator's perspective. The tracking component can be a commercially available type.
[0039] In one embodiment, the display device includes an augmented reality tracking component (for example, of the HoloLens® type offered by Microsoft). In this type of device, sensors are embedded within the device, eliminating the need for a fixed camera that the operator must position. Furthermore, a display device incorporating an augmented reality tracking component provides a first-person view for instantaneous projection of the operator's actions onto the display. The display integrated into such a display device enables dynamic interaction, with rapid alerts (allowing immediate correction of the action before other actions in a predetermined process) and / or immediate validations (allowing reinforcement of correct actions and continuous execution of the predetermined process).
[0040] With the use of augmented reality technology, the specification of correct grips and hand gestures for a predetermined process is taken into account. For example, referring to [Fig. 2], a grasping gesture can be seen as encompassing any static hand posture with which an object can be held securely regardless of hand orientation ([Fig. 2] corresponds to Figure 16 in the reference An Approach to Preparing Hand Gesture Maintenance Training Based on Virtual Reality, Manoch Numfu, Université Grenoble Alpes, https: / / tel.archives-ouvertes.fr / tel-03158358, (3 Tuesday 2021)), (“the Numfu reference”) (citing the reference Huagen, Shuming, & Qunsheng, 2004). Grasping gestures can be classified according to a basic shape, such as the cuboid, cylinder, sphere, triangle, etc. (citing Liu X. Cui, Song, & Xu, 2014).Thus, in the real world, hand orientation is achieved in a real space without movement constraints.
[0041] The predetermined process chosen for training consists of a sequence of steps marked by the specific hand gestures necessary to correctly perform the predetermined process. Instantiations of a generic process model can be used to specify training sequences that can be followed by the operator, provided that the corresponding gesture sequences have been recorded beforehand (for example, by filming an expert performing the sequence). Within the sequence of each process step, it is necessary to specify the appropriate gesture (and, if necessary, the tool(s) to correctly perform the process, based on the characteristics of the chosen process. The gestures will interact with the operator's hand to constitute the model which describes the hand gestures used for each step of a predetermined process.
[0042] The display device can communicate by wire or wirelessly (for example, via WiFi, Bluetooth®, infrared, RFID transmission, Universal Serial Bus (USB), cellular, and / or other wireless means of communication) on the computer system's communication network.
[0043] The communication network of the computer system may include one or more communication devices (or "devices") that capture and transmit data collected from the physical environment in which the operator conducts a training session. The communication device(s) may include one or more portable devices such as a mobile network device belonging to an identified operator or an operator trainer (for example, a mobile phone, a laptop computer, one or more portable devices connected to the network, including "augmented reality" and / or "virtual reality" devices, and / or any combination thereof and / or any equivalent). The communication device(s) may also include one or more remote computers capable of transferring data over the communication network.
[0044] The computer system comprises one or more modules that are implemented by the computer system and executable by one or more processors. The modules include a receiving module configured to receive data corresponding to the operator's first-person view, captured by the display device. The modules also include an extraction module configured to extract a set of characteristic gestures sought in a predetermined process. This set of characteristic gestures may include characteristic gestures that appear in the operator's view.The modules also include an analysis module configured to identify, based at least in part on the set of characteristic gestures associated with the predetermined process being carried out during a training session, the particular characteristic gesture associated with the predetermined process (or with a task of the predetermined process) in order to recognize the correct execution of the characteristic gesture in a predetermined combination and / or order.
[0045] The display device mounted on the operator's head can display virtual content to the operator. With reference to Figures 3 and 4, the head-mounted display device can have a field of view that indicates the hand gestures made by the operator during a training session (including, without limitation, the position of the tip of a right index finger Dd, the position of the joint of a right phalanx Dm, and the orientation of the respective left and right wrists PG, Pd as represented by the respective axes) (see [Fig. 3]). The field of view of the display device defines a volume of space in which the operator can see the view at first person with hands performing the gesture correctly.
[0046] With further reference to [Fig. 3] and [Fig. 4], other indices can be used to label the pixels corresponding to different parts of the hands (including the wrists). For example, the pixels representing the left hand (LM) can be labeled with a different index than the pixels representing the right hand (MD). In another example, the pixels representing the right index finger (D) can be labeled with a different index than the pixels representing the thumb (T).
[0047] Gesture recognition can also be achieved by labeling pixels in any appropriate way. For example, an artificial neural network can be trained to classify each pixel with the appropriate indices / labels. In this way, the different characteristics of one hand or another can be identified computationally, thus making it possible to distinguish different gestures. For example, the three-dimensional position of the fingers can be tracked from one training session to another, making it possible to distinguish parameters such as finger position, finger angle, finger speed, finger acceleration, finger proximity, etc.
[0048] This information is continuously processed to detect specific positions or postures at various stages of the action and to track movements in order to verify whether the sequence of actions conforms to the objective of the action to be performed. These stages and the associated teaching methods are defined by experts in the field and trainers. In the example of the reef knot provided below, the transfer of the thread from the right hand (RH) to the left hand (LH) involves criteria for this stage of the action to be performed correctly. The end result (holding both threads in the left hand) can be achieved differently, but this will not be a "correct action" because it is not suitable for all situations (for example, thin thread, thick cable, presence of tension, etc.). The tolerances are adapted to different human morphologies and must therefore allow for some flexibility in execution without compromising the objective of the action.Therefore, this point is specific to each application.
[0049] Example of a combination of gestures, postures and controls on the flat knot
[0050] Referring again to Figures 3 and 4, and further to [Fig. 5], by way of example, we consider the combination of gestures, postures, and controls associated with a process on the flat node. Each gesture in the process comprises a sequence that is broken down into several steps as shown in [Fig. 5]. [Fig. 5] represents a detection zone with respect to a monitored hand, some of whose joints are positioned within a volume close to certain reference points (for example, the intersection of the hand plane and the finger axis or the wrist orientation).
[0051] Referring to [Fig.5], in the context of a "proof-of-concept" of a flat knot process (performed here by right-handed people), there is a combination of four (4) essential steps for creating a knot: - The process starts from a starting position where one end of a thread is held in each hand and the hands are separated by a predetermined minimum distance. Referring to [Fig. 5], the defined steps are as follows: The right hand performs an initial crossing step (“Step 1”) during which the thread held in the right hand is brought to the left hand by crossing another thread over it. Upon completing this step, the system recognizes the correct movement. This recognition is communicated to the operator as a reinforcement and / or encouragement message displayed in the field of vision of the display device (see, for example, text field 50 shown in [Fig. 4]). This message could include an instruction to proceed to the next step. If this step is performed incorrectly (for example, the thread held in the right hand is brought to the left hand by crossing another thread underneath it), the display device incorporates an instructional message to identify the error and encourage its correction (for example, a message in text field 50 that reads “Incorrect movement, pass the right hand over the thread”). The right hand performs a second crochet step ("Step 2") during which the yarn is held between the right thumb and index finger. Simultaneously, the left hand brings the yarn over to form a first knotless crochet. If this step is performed correctly, the display displays a message to acknowledge it and proceed to the next step. If this step is performed incorrectly (for example, the left hand brings the yarn underneath instead of over), the display displays an instructional message to identify the error and encourage its correction (for example, a message in text field 50 that says "No knot, try again" or a message providing further instructions). Thus, the system focuses the operator on the correct movements that ensure the desired knot is formed, and prevents the operator from learning movements that hinder this process. - The right hand performs a third crossing step ("Step 3") during which the thread held in the right hand is brought to the left hand by crossing a thread underneath. If this step is performed correctly, the display displays a message to acknowledge it and proceed to the next step. If this step is performed incorrectly (for example, the left hand brings the thread over instead of underneath), the display displays an instructional message to identify the error and encourage its correction (for example, a message in text field 50 that ensures quality control by the operator). - The right hand performs a fourth and final crochet step ("Step 4") during which the yarn crossing is held between the right thumb and right index finger of the right hand. Simultaneously, the left hand brings the yarn underneath to form a second hook (and therefore the knot). If this step is performed correctly, the display device incorporates a message to acknowledge it and to ensure quality control by the operator. In the event that this step is performed incorrectly (for example, the left hand brings the thread over instead of under), the display device incorporates an instructional message to alert the operator and to encourage correction (for example, a message in text field 50 that says "Over or under?").
[0052] Referring again to the example of combining gestures on the flat node, the operator could have follow-up instructions incorporating guidance for successfully performing the expected characteristic gestures. In the example shown in [Fig. 4], the instructions that appear in the field of vision of the display device could include, for example: - an identification of the gesture "loop 1 detected" which triggers a sequence of characteristic gestures (signaled by the instruction "loop 2 expected"); - a quality check of identified gesture (“Check the quality of the knot”); and - an image of the correct knot for comparison with the knot, made by the operator during the training session.
[0053] It is understood that the display device could include a reference image (see, for example, [Fig. 4], in which the field of vision of the display device incorporates an image field 52 that represents the correct node and the crossings above and below). The system can provide instructions at each step of the process to ensure that the operator takes them into account in conjunction with the action performed (the instructions being readable in the text field 50 and / or visual instructions in the image 52). These instructions could be combined with one or more alerts that indicate a need to modify the action (including, for example, audible alerts).
[0054] It is understood that each step can be performed correctly or incorrectly, but a "bad" combination of correctly performed steps can lead to a bad knot or the absence of a knot. Therefore, the combination itself must also be checked. The movements are thus presented with possible combinations, "forbidden" combinations that introduce a quality problem, the correct combination, and combinations that do not form knots (in this last case, there is no risk because the operator realizes their error). This decomposition is valid for right-handed operators and is symmetrical (right-left) for left-handed operators.
[0055] By using the display device incorporating an augmented reality tracking component, the operator's hand tracking makes it possible to monitor the execution of a movement, verify checkpoints, control the orientations of the wrist and fingers, and validate gesture combinations to verify whether the gesture performed conforms to the predetermined process. The display means of the device The display allows errors and / or risks to be indicated to the operator in real time, to give advice, to emphasize educational messages, to learn how to do quality checks with a reminder of the standard, and to monitor performance (for example, by giving a "challenge" on the number of correct actions in a row).
[0056] With further reference to Figures 1 to 5, a detailed description is given by way of example of a method 100 of the invention (“method”) for training operators based on finger movements. The method 100 is implemented by the computer system described above (or by a site and / or facility incorporating the computer system). Each gesture is broken down into several steps, and each step is controlled according to the flow diagram shown in [Fig. 1]. In the method 100, the operation of the gesture control incorporates the following options: - Posture control; - Gesture tracking; - Display of instructions; and - Alert display.
[0057] The instruction(s) and alert(s) displays are made in the field of view of the display device worn by the operator during a training session of a predetermined process.
[0058] By starting the method 100 of the invention, the method includes the acquisition step 102 during which the operator's hand gestures are captured in the first-person view during a training session conducted by the operator. More specifically, the operator's hand gestures are captured in the field of view of the real-time display device corresponding to the execution of the gesture. During this step, the system obtains data from a hand gesture possibly associated with a predetermined process in which the operator participates.
[0059] Acquisition step 102 includes a hand-tracking step, using a display device incorporating an augmented reality tracking component. The hand tracking detects a position corresponding to the beginning or end of a characteristic gesture sequence. During this step, the position of one or both of the operator's hands is tracked to determine if this position is within a predetermined tolerance for starting the ongoing predetermined process. This step denotes the start of a gesture sequence in an expected order from an initial hand position.
[0060] The method 100 further includes a hand motion tracking step 104 during which the display device detects all gestures in the sequence associated with the predetermined process chosen for the current training. The motion tracking makes it possible to control the trajectory and movement of the operator's fingers during a sequence of characteristic gestures. This step includes a End-of-sequence detection step. Due to hand movement tracking during this step, it is not necessary for the operator to assume a particular initial position when starting the predetermined process gesture sequence during step 102.
[0061] It is understood that the operator could use one or both hands, depending on the training and the predetermined process chosen.
[0062] Hand gestures and the sequences they form part of are detected within the field of view of the display device. The detected gestures can be recorded for use in ongoing training and for reviewing the operator's performance at the end of that training. For robust hand gesture recognition, one or more gesture recognition models primarily use machine learning algorithms (such as K-Nearest Neighbor (KNN) and Support Vector Machine (SVM)) and deep learning algorithms (such as Recurrent Neural Network (RNN) and Deep Convolutional Neural Network (CNN)). These models are used to detect gestures characteristic of the predetermined process from one or more gestures made by the operator during a training session.
[0063] It is understood that a graph convolutional neural network (or “Graph CNN”) can be employed during process 100 to reconstruct a complete 3D mesh of the hand surface that contains richer information about the shape and pose of the hand. To train the networks with full supervision, a synthetic dataset can be created containing both the shape and pose of the hand. In this way, deep neural networks that are trained end-to-end can be used to retrieve the 3D mesh of the hand directly from a single RGB image (e.g., an image of one or more hands of an expert who regularly performs the correct sequence, including the correct gestures, associated with the predetermined process) (see “3D Hand Shape and Pose Estimation from a Single RGB Image”, Liuhao Ge et al., Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (2019) (the “Ge reference”).It is understood that other methods can be employed, including learning about non-linear variations in hand shape.
[0064] The process 100 further includes a step 106 for identifying the position of the operator's hands relative to the characteristic gestures of the predetermined process. During this step, the key parameters of the movement performed by the operator (recorded during the motion tracking step 104) are used to identify the hand position relative to the characteristic gestures of the predetermined process.
[0065] The method 100 further includes a comparison step 108 during which the characteristics of the gesture identified during step 106 (for example, the position of the tip of the index finger Dd, the position of the thumb joint T and the orientation Wrist movements (PG, PD) are taken into account to validate the movement according to the predetermined process. During this step, the identified movement and the characteristic movements of the predetermined process are compared to determine the variance between the identified movement and the corresponding correct movement of the predetermined process. This comparison triggers either a quality check 108a or an alert message 108b to warn the operator. Both check 108a and alert message 108b are performed in real time so that the validation / correction aspect of the movement is linked to the sequence of characteristic movements of the predetermined process to be performed during training. Both check 108a and alert message 108b include written messages within the field of vision of the display device (as represented by instructions 110 in [Fig. 5]).It is understood that these written messages could be accompanied by audio messages and / or haptic signals.
[0066] During the comparison step 108, a check 108a of the gesture triggers an additional check 108a' to validate that the identified gesture conforms to the movement tolerances established for the predetermined process. This additional validation can be performed before the next gesture in the sequence of characteristic gestures, so that the operator learns the behavior of the current gesture.
[0067] If the additional validation 108a' indicates that the identified gesture has been performed incorrectly, an alert message 108b' appears in the field of vision of the display device. If an alert message 108b, 108b' indicates that the identified gesture has been performed incorrectly, the operator is warned of this fact before performing a subsequent gesture. Sending an alert message indicates a risk of failure to perform the predetermined process so that the operator can correct their gesture in real time during the ongoing training. An alert message 108b, 108b' may include one or more prompts to encourage the operator to correct the incorrect gestures before performing the subsequent gestures (for example, one or more readable instructions in a text field of type 50 shown in [Fig. 4] and / or one or more visual instructions in an image field of type 52 shown in [Fig. 4]).The sending of alert messages 108b, 108b' can be repeated until the operator correctly performs the identified action (including performing the action in the correct order of a sequence of actions characteristic of the predetermined process).
[0068] Thus, the system implementing method 100 of the invention monitors in real time the suitability of the operator's work and uses a sequence of steps to validate the quality of the action (and therefore its potential to achieve the correct result). This sequence of steps can be used to avoid cases where a good result is obtained through inappropriate actions (for example, for reasons of ergonomics, quality, repeatability, and / or transferability to other processes). The system will instruct the operator to compare their work with the expected result. By these In these aspects, the result obtained is controllable by the operator and stems from validated procedures that they have assimilated. Consequently, the operator is encouraged to engage in self-criticism and validate their ability to act autonomously on the actual process, since they have acquired the capacity for self-control.
[0069] In all embodiments of the method 100, temporal synchronization is achieved between gestures performed by the operator and a training session conducted by the operator (for example, to verify that the gestures maintain the correct pace of the predetermined process). Temporal synchronization is achieved by using time synchronization points recorded by the display device that correspond in time to a predetermined process in the training session. The predetermined characteristic gestures of the process and the time synchronization points are used to align the gestures performed by the operator with the times of the gestures in the training session on the same clock. In this way, the operator's gestures are synchronized in time with the training session so that each gesture will correspond to a characteristic gesture at the appropriate time on the clock.Since a predetermined process involves a temporal sequence, time can be used as an explicit signal to identify the relationship between the gestures within the display device's field of vision and the expected characteristic gestures. During step 108, the comparison of process 100, this synchronization could be used to send the checks 108a, 108a' and / or the alert messages 108b, 108b', so that the operator receives real-time feedback on the success or failure of the gesture. The operator can then react appropriately to ensure the correct execution of the ongoing predetermined process.
[0070] In all embodiments of method 100, a neural network can be trained to recognize characteristic gestures during training sessions and to create bounding boxes (or "framed regions") around the recognized characteristic gestures. The framed regions can represent gestures performed correctly to reinforce good habits by carrying out the predetermined process that is the subject of the current training session (for example, a framed region around the poorly articulated thumb T). During this training, the coordinates of the bounding box of the characteristic gesture are correlated with the coordinates of the hand positions. These coordinates are used to calculate characteristics of the gesture, including, without limitation, the completion time of the gesture and the completion time of the sequence of which the gesture is a part.The reframed regions and characteristic gestures are fed into a neural network (e.g., one or more CNNs) to jointly learn the representation of the characteristic gesture by different operators during training.
[0071] In all embodiments of process 100 of the invention, one or more steps can be carried out iteratively.
[0072] The computer system may include pre-programmed management information. For example, a setting of process 100 may be associated with the parameters of typical physical environments in which the system operates. In embodiments of process 100, the computer system (and / or an installation incorporating the computer system) may receive voice commands or other audio data representing, for example, a step or a stop to video capture. The command may include a request for the current state of the predetermined process. A generated response may be represented audibly, visually, tactilely (for example, using a haptic interface), and / or virtually and / or augmented. This response, along with the corresponding data, may be recorded in the neural network.
[0073] The disclosed invention aims to minimize disruption to the trainee (operator). Therefore, the movement must be guided to ensure it is fluid and "automatic," while providing feedback on quality, precision, and checks to be performed, or reinforcing the pedagogical messages from the initial training (see the example of the flat knot above). An alert message for a detected error emphasizes the self-checking that the operator must systematically perform. By focusing on the characteristics of the movements in real time, the operator's training remains autonomous, regardless of their individual characteristics (e.g., hand size, skin color, presence of a hand disease or disability, gender, etc.).
[0074] The terms "at least one" and "one or more" are used interchangeably. The ranges presented as being "between a and b" encompass the values "a" and "b".
[0075] Although particular embodiments of the disclosed apparatus have been illustrated and described, it will be understood that various changes, additions, and modifications can be made without departing from the spirit or scope of this disclosure. Therefore, no limitations should be imposed on the scope of the invention described except those set forth in the appended claims.
Claims
1. Claims A method (100) for training operators on a predetermined process having a sequence of characteristic gestures, the method implemented by a computer system comprising one or more processors which are capable of executing programmed instructions stored in a memory operatively connected to one or more processors to carry out the method, characterized in that the method comprises the following steps: - an acquisition step (102) performed by a receiving module of the computer system to receive the data corresponding to the first-person view of the operator, during which the hand gestures of an operator are captured in real time in the first-person view during a training session performed by the operator, and during which the computer system obtains data of a hand gesture associated with a predetermined process in which the operator participates, this step comprising: - a step of tracking the hands from a display device of the computer system incorporating an augmented reality type tracking component, the tracking of the operator's hands of which makes it possible to detect a position which corresponds to a start or an end of a sequence of gestures characteristic of a predetermined process in progress; - a hand movement tracking step (104) performed by an extraction module of the computer system to extract a set of characteristic gestures sought in the predetermined process in progress, the movement tracking of which from a display device of the computer system makes it possible to control the trajectory and the movement of the fingers of the operator during a sequence of characteristic gestures, and during which the display device detects all the gestures in the sequence associated with the predetermined process; - a step of identifying (106) a position of the operator's hands in relation to the characteristic gestures of the predetermined process, during which parameters of the movement carried out by the operator and recorded during the movement tracking step (104) are used to identify a characteristic gesture, this step carried out by an extraction analysis module of the computer system to identify the characteristic gesture associated with the predetermined process; and - a comparison step (108) during which the characteristic gesture- The identified characteristic gestures and the characteristic gestures of the predetermined process are compared to determine a variance between them during the predetermined process; so that the gestures of an operator doing the predetermined process are recorded for use in the current training session.
2. The method (100) of claim 1, wherein, during the acquisition step (102), the capture of the gestures made by the operator is carried out in a field of vision of the display device of the computer system.
3. The method (100) of claim 2, wherein the comparing step (108) comprises: - generating a gesture quality check (108a, 108a') to indicate where the performance of the identified gesture is correct; and - generating an alert message (108b, 108b') to indicate where the performance of the identified gesture is incorrect; such that the check (108a) and the alert message (108b) are generated in real time.
4. The method (100) of claim 3, wherein the verification (108a, 108a') of the quality of the gesture and the alert message (108b, 108b') generated during the comparison step comprise visual instructions in a field of vision of the display device of the computer system comprising a display device which is worn by the operator during the training session.
5. The method (100) of claim 4, further comprising a step of providing one or more visual instructions in a text field (50) and / or in an image field (52) in a field of view of the display device of the computer system.
6. The method (100) of any one of claims 3 to 5, wherein generating the alert message (108b, 108b') includes generating one or more tips to cause the operator to correct the incorrect gestures before performing the downstream gestures.
7. The method (100) of any preceding claim, wherein the step of tracking hand movement (104) comprises a step of detecting the end of a gesture sequence.
8. The method (100) of any preceding claim, wherein, during the comparing step (108), one or more gesture recognition models use: - machine learning algorithms; or - deep learning algorithms.
9. The method (100) of any preceding claim, further comprising a step of obtaining time synchronization between gestures made by the operator and the characteristic gestures expected during the training session.
10. Method (100) of claim 9, wherein, during the comparison step (108), the obtained time synchronization is used in sending the verifications (108a, 108a') and / or the alert messages (108b, 108b'); so that the operator receives in real time an indication of success or failure of the identified gesture.
