Tool operator training system and method

Abstract

A tool operator training system is provided, comprising first operator sensors (102, 104) configured to be placed on an expert operator and a first tool (103) with first tool sensor (106). A first controller (105) is configured to measure (S301) expert operator movements and expert operator tool movements, and generate an expert operator avatar. Second operator sensors (112, 114) are configured to be placed on a trainee operator and a second tool (113) with second tool sensors (116). A second controller (115) is configured to generate (S302) a trainee avatar for a trainee operator, and measure (S303) trainee operator movements and trainee operator tool movements. A difference between the expert avatar and the trainee avatar performing an action is calculated, when the trainee operator is performing the action. Guidance is displayed at an augmented reality headset to guide the trainee operator to adapt movements to minimize the calculated difference when the trainee operator is performing the action.

Description

[0001] Tool Operator Training System and Method

[0002] Field of Invention

[0003] The present invention relates to a training system and method for training an operator of a tool.

[0004] Background

[0005] Traditionally, learning a sport that uses a club, a bat, or a racket involves a coach that guides the athlete step-by-step on the right movements to perform. The coach also shows the correct handling of the club (or similar tool) to the trainee or new learner, so that their movement can be copied by the trainee. However, while performing the movement, the trainee athlete should look at the tool or the ball instead of looking at the coach, making memorizing the correct movement separate from the act of performing it. Furthermore, a coach is usually expensive and the time with the coach limited, thereby slowing down the potential learning rate of the trainee player.

[0006] An objective of the present invention is to overcome the aforementioned challenges, amongst others.

[0007] Summary of Invention

[0008] In a first aspect, there is provided a tool operator training system, the system comprising: one or more first operator sensors configured to be placed on an expert operator of a tool, wherein the one or more first operator sensors comprise a first augmented reality headset; a first tool with one or more first tool sensors; a first controller configured to measure one or more expert operator movements with the one or more first operator sensors, and one or more expert operator tool movements with the one or more first tool sensors, for the expert operator performing an action with the first tool, and generate an expert operator avatar based upon the measured expert operator movements and expert operator tool movements; one or more second operator sensors configured to be placed on a trainee operator of the tool, wherein the one or more second operator sensors comprise a second augmented reality headset; a second tool with one or more second tool sensors; a second controller configured to: generate a trainee avatar for the trainee operator based upon preliminary trainee operator characteristics; measure one or more trainee operator movements with the one or more second operator sensors, and one or more trainee operator tool movements with the one or more second tool sensors when the trainee operator is performing the action with the second tool and represent these as the trainee avatar performing the action; calculate a difference between the expert operator avatar performing the action and the trainee avatar performing the action when the trainee operator is performing the action; and display guidance at a display of the second augmented reality headset, wherein the guidance comprises instructions to guide the trainee operator to adapt movements to minimize the calculated difference when the trainee operator is performing the action.

[0009] Optionally, the first tool and the second tool are a same type of tool.

[0010] Optionally, the first tool and the second tool are sports equipment.

[0011] Optionally, the first tool and the second tool are a golf club, and the action comprises a golf swing.

[0012] Optionally, the first operator sensors comprise one or more of an accelerometer, a gyroscope, and a magnetometer, and the second operator sensors comprise one or more of an accelerometer, a gyroscope, and a magnetometer.

[0013] Optionally, the second operator sensors comprise a camera of the second augmented reality headset configured to capture video.

[0014] Optionally, the first controller is configured to measure the one or more expert operator movements with the one or more first operator sensors, and the one or more expert operator tool movements with the one or more first tool sensors, for the expert operator performing an action with the first tool, until a predetermined measurement ending condition is met. Optionally, the measuring one or more expert operator movements with the one or more first operator sensors, and the one or more expert operator tool movements with the one or more first tool sensors, for the expert operator performing the action with the first tool, and the generating the expert operator avatar based upon the measured expert operator movements and expert operator tool movements comprises: measuring one or more expert operator movements with the one or more first sensors, and one or more expert operator tool movements of the first tool, respectively for each of a plurality of expert operators when each of the plurality of expert operators performs the action with the first tool; and generating an expert operator avatar for each of the plurality of expert operators based upon the measured respective expert operator movements and expert operator tool movements.

[0015] Optionally, the second controller is configured to determine a closest matching expert operator avatar of the plurality of expert operator avatars to the trainee avatar, such that the calculated difference is calculated between the closest matching expert operator avatar and the trainee avatar.

[0016] Optionally, the first controller is configured to receive a first activation signal, wherein the first activation signal triggers the first controller to measure the expert operator movements and the expert operator tool movements.

[0017] Optionally, the first controller is configured to store the generated expert operator avatar in an avatar datastore.

[0018] Optionally, the second controller is configured to receive a second activation signal, wherein the second activation signal triggers the second controller to measure the preliminary trainee operator characteristics.

[0019] Optionally, the second controller is configured to measure the preliminary trainee operator characteristics by measuring one or more preliminary trainee operator movements with the second operator sensors and / or measuring one or more preliminary trainee operator tool movements with the second tool sensors.

[0020] Optionally, the second controller is configured to, until a predetermined ending condition is met, repeatedly: measure the one or more trainee operator movements with the second operator sensors, and the one or more trainee operator tool movements with the second tool sensors when the trainee operator is performing the action and represent these as the trainee avatar performing the action; calculate the difference between the expert operator avatar performing the action and the trainee avatar performing the action when the trainee operator is performing the action; and display the guidance at the display of the second augmented reality headset when the trainee operator is performing the action.

[0021] In a second aspect, there is provided a tool operator training method, the method comprising: measuring one or more expert operator movements with one or more first operator sensors, wherein the one or more first operator sensors are configured to be placed on an expert operator of a tool and comprise a first augmented reality headset, and measuring one or more expert operator tool movements with one or more first tool sensors of a first tool, for an expert operator performing an action with the first tool, and generating an expert operator avatar based upon the measured expert operator movements and expert operator tool movements; generating a trainee avatar for a trainee operator based upon preliminary trainee operator characteristics; measuring one or more trainee operator movements with one or more second operator sensors wherein the second operator sensors are configured to be placed on a trainee operator of the tool and the one or more second operator sensors comprise a second augmented reality headset, and one or more trainee operator tool movements with one or more second tool sensors of a second tool when a trainee operator is performing the action with the second tool, and representing these as the trainee avatar performing the action; calculating a difference between the expert operator avatar performing the action and the trainee avatar performing the action when the trainee operator is performing the action; displaying guidance at a display of the second augmented reality headset, wherein the guidance comprises instructions to guide the trainee operator to adapt movements to minimize the calculated difference when the trainee operator is performing the action.

[0022] Optionally, the second aspect can include the optional features of the first aspect, as appropriate. In a third aspect there is provided a first non-transitory computer readable medium storing instructions thereon that when executed by one or more processors cause the one or more processors to: measure one or more expert operator movements with one or more first operator sensors, wherein the one or more first operator sensors are configured to be placed on an expert operator of a tool and comprise a first augmented reality headset, and measure one or more expert operator tool movements with one or more first tool sensors of a first tool, for an expert operator performing an action with the first tool; and generate an expert operator avatar based upon the measured expert operator movements and expert operator tool movements; a second non-transitory computer readable medium storing instructions thereon that when executed by one or more processors cause the one or more processors to: generate a trainee avatar for a trainee operator based upon preliminary trainee operator characteristics; measure one or more trainee operator movements with one or more second operator sensors wherein the second operator sensors are configured to be placed on a trainee operator of the tool and the one or more second operator sensors comprise a second augmented reality headset, and one or more trainee operator tool movements with one or more second tool sensors of a second tool when a trainee operator is performing the action with the second tool, and represent these as the trainee avatar performing the action; calculate a difference between the expert operator avatar performing the action and the trainee avatar performing the action when the trainee operator is performing the action; display guidance at a display of the second augmented reality headset, wherein the guidance comprises instructions to guide the trainee operator to adapt movements to minimize the calculated difference when the trainee operator is performing the action.

[0023] Optionally, the third aspect can include the optional features of the first aspect, as appropriate.

[0024] In a fourth aspect, there is provided an expert operator system of a tool operator training system, the expert operator system comprising: one or more first operator sensors configured to be placed on an expert operator of a tool, wherein the one or more first operator sensors comprise a first augmented reality headset; a first tool with one or more first tool sensors; and a first controller configured to measure one or more expert operator movements with the one or more first operator sensors, and one or more expert operator tool movements with the one or more first tool sensors, for the expert operator performing an action with the first tool, and generate an expert operator avatar based upon the measured expert operator movements and expert operator tool movements.

[0025] Optionally, the expert operator system of the tool operator training system of the fourth aspect can also include the corresponding optional features of the first aspect, as appropriate.

[0026] In a fifth aspect, there is provided a trainee operator system of a tool operator training system, the trainee operator system comprising: one or more second operator sensors configured to be placed on a trainee operator of a tool, wherein the one or more second operator sensors comprise a second augmented reality headset; a second tool with one or more second tool sensors; a second controller configured to: generate a trainee avatar for the trainee operator based upon preliminary trainee operator characteristics; measure one or more trainee operator movements with the one or more second operator sensors, and one or more trainee operator tool movements with the one or more second tool sensors when the trainee operator is performing an action with the second tool and represent these as the trainee avatar performing the action; calculate a difference between an expert operator avatar performing the action and the trainee avatar performing the action when the trainee operator is performing the action; and display guidance at a display of the second augmented reality headset, wherein the guidance comprises instructions to guide the trainee operator to adapt movements to minimize the calculated difference when the trainee operator is performing the action.

[0027] Optionally, the trainee operator system of the tool operator training system of the fifth aspect can also include the corresponding optional features of the first aspect, as appropriate.

[0028] In a sixth aspect, there is provided an expert operator method of a tool operator training method, the method comprising: measuring one or more expert operator movements with one or more first operator sensors, wherein the one or more first operator sensors are configured to be placed on an expert operator of a tool and comprise a first augmented reality headset, and measuring one or more expert operator tool movements with one or more first tool sensors of a first tool, for an expert operator performing an action with the first tool and generating an expert operator avatar based upon the measured expert operator movements and expert operator tool movements.

[0029] Optionally, the expert operator method of the tool operator training method of the sixth aspect can also include the corresponding optional features of the first aspect, as appropriate.

[0030] In a seventh aspect, there is provided a trainee operator method of a tool operator training method, the method comprising: generating a trainee avatar for a trainee operator based upon preliminary trainee operator characteristics; measuring one or more trainee operator movements with one or more second operator sensors wherein the second operator sensors are configured to be placed on a trainee operator of the tool and the one or more second operator sensors comprise a second augmented reality headset, and one or more trainee operator tool movements with one or more second tool sensors of a second tool when a trainee operator is performing the action with the second tool, and representing these as the trainee avatar performing the action; calculating a difference between an expert operator avatar performing the action and the trainee avatar performing the action when the trainee operator is performing the action; displaying guidance at a display of the second augmented reality headset, wherein the guidance comprises instructions to guide the trainee operator to adapt movements to minimize the calculated difference when the trainee operator is performing the action.

[0031] Optionally, the expert operator method of the tool operator training method of the sixth aspect can also include the corresponding optional features of the first aspect, as appropriate. In an eighth aspect there is provided a first non-transitory computer readable medium storing instructions thereon that when executed by one or more processors cause the one or more processors to: measure one or more expert operator movements with one or more first operator sensors, wherein the one or more first operator sensors are configured to be placed on an expert operator of a tool and comprise a first augmented reality headset, and measure one or more expert operator tool movements with one or more first tool sensors of a first tool, for an expert operator performing an action with the first tool; and generate an expert operator avatar based upon the measured expert operator movements and expert operator tool movements.

[0032] Optionally, the eighth aspect can include the optional features of the first aspect, as appropriate.

[0033] In a ninth aspect, there is provided a second non-transitory computer readable medium storing instructions thereon that when executed by one or more processors cause the one or more processors to: generate a trainee avatar for a trainee operator based upon preliminary trainee operator characteristics; measure one or more trainee operator movements with one or more second operator sensors wherein the second operator sensors are configured to be placed on a trainee operator of the tool and the one or more second operator sensors comprise a second augmented reality headset, and one or more trainee operator tool movements with one or more second tool sensors of a second tool when a trainee operator is performing the action with the second tool, and represent these as the trainee avatar performing the action; calculate a difference between an expert operator avatar performing the action and the trainee avatar performing the action when the trainee operator is performing the action; display guidance at a display of the second augmented reality headset, wherein the guidance comprises instructions to guide the trainee operator to adapt movements to minimize the calculated difference when the trainee operator is performing the action.

[0034] Optionally, the ninth aspect can include the optional features of the first aspect, as appropriate. Brief Description of Drawings

[0035] Embodiments of the invention are now described, by way of example, with reference to the drawings, in which:

[0036] Figure 1 is a block diagram of a tool operator training system;

[0037] Figure 2 is a diagram of an operator, with an augmented reality headset, performing an action with a tool;

[0038] Figure 3 is a flow diagram of a process for training a trainee operator of the tool using the tool operator training system;

[0039] Figure 4 is a flow diagram of a process in which training data of an expert operator is gathered;

[0040] Figure 5 is a flow diagram of a process in which data from a trainee operator is gathered;

[0041] Figure 6 is a flow diagram of a process in which guidance is provided to a trainee operator when performing an action with the tool;

[0042] Figure 7 is a flow diagram of an exemplary execution of the process of Figure 3; and

[0043] Figure 8 is a high-level block diagram of an apparatus suitable for implementing various aspects of the disclosure

[0044] Detailed Description

[0045] The present invention provides a tool operator training system that tracks the movement of a tool, learns and extracts the details in each movement, and displays in real-time the correct movement to perform, on an augmented reality headset. This allows the trainee operator of the tool (for example an athlete or sportsperson) to follow it independently during the activity. The activity can be a sporting activity, for example golf or other sports, particularly those that use a tool such as a club, bat or racket. Using augmented reality, the system can learn and record best practices and habits from highly qualified professional players or athletes and enable the new learner to benefit from it.

[0046] Figure 1 shows a block diagram of the tool operator training system 100. The tool operator training system 100 comprises a first augmented reality headset 101 and first tool 103 that can be used by the expert operator (i.e., a trainer). The expert operator can be considered as an expert in the use of the tool.

[0047] The first tool 103 can comprise any type of tool that is used in a physical activity. For example, the tool could be sports equipment, such as a golf club, tennis racket, cricket bat or the like.

[0048] The first augmented reality headset 101 can include one or more first augmented reality headset sensors 104. The first augmented reality headset sensors 104 can be configured to detect the location in space and movement of the first augmented reality headset 101 . The first augmented reality headset sensors 104 can include one or more of an accelerometer, a gyroscope and / or a magnetometer. The first augmented reality headset sensors 104 can also include a camera configured to capture video from the first augmented reality headset 101 (for example, in a similar manner to traditional augmented reality headsets). This video can be used as an additional sensor to provide data to the process that will be subsequently described.

[0049] The first augmented reality headset 101 also includes a first controller 105. The first controller 105 can, for example, be a microcontroller.

[0050] The system 100 can also comprise first auxiliary sensors 102 that can be placed in various positions on the body of the expert operator. Figure 2 shows an exemplary diagram of an operator 200, with the augmented reality headset 201 , performing an action with a tool 203 (in this case a golf player swinging a golf club). In this example, the auxiliary sensors can be placed on a cap 231 , a shirt 232, shorts / trousers 234, and shoe(s) 235 of the operator.

[0051] As mentioned, in the example of Figure 2, the operator is a golfer and is shown to be in the follow through position after completing a golf swing with a golf club 203. The tool operation training system includes a plurality of sensors, such as accelerometers or magnetometers, and associated processors which are arranged in positions such that movement data relevant to golf (or any other suitable activity) can be captured. In the example of golf, the movement of the golf club 203 has a significant effect on the outcome of the golf swing. As such, a sensor may be positioned on the golf club 203 to capture data related to the position and / or movement of the golf club 203. In golf, it is typically taught that a user’s head should remain relatively still until after the ball has been hit. Accordingly, a sensor and associated processor may be placed on an operator’s cap 231 and / or glasses (i.e. , the augmented reality headset 201) to capture data related to the position and / or movement of the user’s head. Furthermore, the twisting or opening of a user’s body during a golf swing is typically considered to be important in a golf swing. Therefore, a sensor may be positioned on a user’s shirt 232 and / or shorts 234 or trousers to capture data related to the position and / or movement of the user’s body and legs. Furthermore, the position and movement, or lack thereof, of a user’s feet during a golf swing is typically considered to be important. Accordingly, a sensor or sensors may be positioned on the user’s shoe 235 or shoes to capture data related to the position and / or movement of the user’s feet during the golf swing.

[0052] The first augmented reality headset sensors 104, and the first auxiliary sensors 102, can collectively be considered as first operator sensors. These first operator sensors are used to measure the movement of the body of the expert operator when performing an action with the first tool 103.

[0053] The first tool 103 can include first tool sensors 106. The first tool sensors 106 can be embedded into and / or attached onto the tool. The first tool sensors 106 can include one or more of an accelerometer, a gyroscope and / or a magnetometer. The first tool sensors 106 are configured to detect and measure the location of the first tool 103 in space, and the movement of the first tool 103. These first tool sensors 106 are used to measure the movement of the first tool 103 as the expert operator performs an action with the first tool 103. The first tool 103 can include a wireless transmitter / receiver (for example, a Bluetooth transmitter / receiver) that connects to the first augmented reality headset 101 to transmit the measurements of the first tool sensors 106 to the first controller 105 of the first augmented reality headset 101. Likewise, the first augmented reality headset 101 can include a wireless transmitter / receiver (for example, a Bluetooth transmitter / receiver) that connects to the first tool 103 to receive the measurements of the first tool sensors 106. The first tool 103 may transmit raw sensor data to the first augmented reality headset 101.

[0054] The tool operator training system 100 also comprises a second augmented reality headset 111 and second tool 113 that can be used by the trainee operator (i.e., a trainee). The trainee operator can be considered as a non-expert in the use of the tool, and who is to be trained in the use of the tool.

[0055] The second tool 113 can be the same type of tool as the first tool 103. In some cases the second tool 113 may be the same tool as the first tool 103. In other cases the second tool 113 may be a different tool that is the same type of tool (for example, both the first tool 103 and second tool 113 could be a golf club) as the first tool 103.

[0056] In some cases, the second augmented reality headset 111 can be the same augmented reality headset as the first augmented reality headset 101. In other cases the second augmented reality headset 111 may be a different augmented reality headset to the first augmented reality headset 101.

[0057] The second augmented reality headset 111 and second tool 113 can include the same features as those described respectively for the first augmented reality headset 101 and first tool 103.

[0058] The second augmented reality headset 111 can include one or more second augmented reality headset sensors 114; the second augmented reality headset sensors 114 can be consistent with the first augment reality headset sensors 104 and so for brevity the detail is not repeated.

[0059] The second augmented reality headset 111 can also include a second controller 115; the second controller 115 can be consistent with the first controller 105 and so for brevity the detail is not repeated.

[0060] The system 100 can also comprise second auxiliary sensors 112 that can be placed in various positions on the body of the trainee operator; the second auxiliary sensors 112 can be consistent with the first auxiliary sensors 102 and so for brevity the detail is not repeated.

[0061] The second augmented reality headset sensors 114, and the second auxiliary sensors 112, can collectively be considered as second operator sensors. These second operator sensors are used to measure the movement of the body of the trainee operator when performing an action with the second tool 113.

[0062] The second tool 113 can include second tool sensors 116; these second tool sensors 116 can be consistent with the first tool sensors 106 and so for brevity the detail is not repeated. These second tool sensors 116 are used to measure the movement of the second tool 113 as the trainee operator performs an action with the second tool 113. The second tool 113 and second augmented reality headset 111 can also include wireless transmitters / receivers consistent with those of the first tool 103 and first augmented reality headset 101. The tool operator training system 100 can also include an avatar datastore 120. The avatar datastore 120 can be configured as electronic storage in which generated avatars of expert operators can be stored. The first controller 105 can communicate with the avatar datastore 120 (for example through a wireless protocol such as the internet) to transmit parameters measured with the first operator sensors and / or expert operator avatars to the avatar datastore 120. The second controller 115 can communicate with the avatar datastore 120 to retrieve the expert operator avatar(s) from the avatar datastore 120. The avatar datastore 120 may be local to or remotely accessible by the first / second controllers.

[0063] The components of the tool operator training system 100 will be described in more detail as follows.

[0064] Figure 3 shows a flow diagram of a process for training a trainee operator of the tool using the tool operator training system 100 described with reference to Figure 1.

[0065] The process can be considered to have three phases.

[0066] In a first phase, at step S301 , training data of the expert operator is gathered. At this step, the first controller 105 is configured to measure one or more expert operator movements (i.e., movements of the expert operator when the expert operator performs an action with the first tool 103) with the one or more first operator sensors, and one or more expert operator tool movements (i.e., movements of the first tool 103 when the expert operator performs the action with the first tool 103) with the one or more first tool sensors 106, for the expert operator performing an action with the first tool 103. The first controller 105 is then configured to generate an expert operator avatar (also referred to as an expert avatar) based upon the measured expert operator movements and expert operator tool movements.

[0067] In a second phase, at step S302, data from the trainee operator is gathered. At this step, the second controller 115 is configured to generate a trainee avatar (also referred to as a trainee operator avatar) for the trainee operator based upon preliminary trainee operator characteristics. The data can be gathered in real-time, processed and used to find a closest expert avatar representation that the system will try to match actively via providing real-time guidance to allow the trainee to match the expert’s movements and best possible positions.

[0068] In a third phase, at step S303, guidance is provided to the trainee when performing the action with the tool. At this step, the second controller 115 is configured to measure one or more trainee operator movements (i.e., movements of the trainee operator when the trainee operator performs an action with the second tool 113) with the one or more second operator sensors, and one or more trainee operator tool movements (i.e., movements of the second tool 113 when the trainee operator performs the action with the second tool 113) measured with the one or more second tool sensors 116 when the trainee operator is performing the action with the second tool 113, and represent these as the trainee avatar performing the action. The second controller 115 is then configured to calculate a difference between the expert avatar performing the action and the trainee avatar performing the action when the trainee operator is performing the action. The second controller 115 is then configured to display guidance at a display of the second augmented reality headset 111 , wherein the guidance comprises instructions to guide the trainee operator to adapt movements to minimize the calculated difference when the trainee operator is performing the action.

[0069] Steps S301 , S302 and S303 are described in more detail with regard to Figures 4, 5 and 6 respectively.

[0070] Figure 4 shows a flow diagram of the process of step S301 for the first phase (step S301 of Figure 3) in which training data of the expert operator is gathered.

[0071] In this phase, an expert operates the tool, and data is collected to create an expert operator avatar. Data is collected from the first operator sensors, and first tool sensors 106, which can include movement and visual sensors placed on the tool, placed on the expert operator and / or around the expert operator. For example these can include photo / video data of the expert operator, expert operator posture, hand position, and / or pressure applied on the tool grip. In an example in which the action comprises hitting a ball, the data can also include ball trajectory.

[0072] The expert operator avatar can be considered as an operator configuration in temporal parameters (for example position, target, setting of the tool and usage of the tool by an operator, etc.) in a virtual representation via augmented reality.

[0073] A plurality of expert operator avatars can be generated from data from a plurality of expert operators so that matches can be made for a wide variety of trainees. For example, if the tool is a golf club and the action is a golf swing, a plurality of expert avatars can be generated from data from multiple expert golf players to match a wide variety of trainee golfers and playing settings.

[0074] The process of gathering training data of the expert operator can begin at step S301-1. At step S301-1 , the first controller 105 can receive a first activation signal, wherein the first activation signal triggers the first controller 105 to measure the expert operator movements and the expert operator tool movements.

[0075] The first activation signal can comprise, for example, the pressing of a physical button on the first augmented reality headset 101 , or on the first tool 103, or the pressing of a virtual button on a smartphone that is connected to the first augmented reality headset 101 and / or the first tool 103.

[0076] In another example, the first activation signal can comprise a voice activation signal that can be received at a microphone of the first augmented reality headset 101.

[0077] In another example, the first activation signal can be the detection, by the first controller 105, of a predetermined position of the first augmented reality headset 101 , using sensor(s) of the first augmented reality headset 101 , such as the starting position of an operator at the beginning of a predetermined action (e.g., the starting position of the head of a golf player at the beginning of a golf swing).

[0078] In another example, the first activation signal can comprise the detection, by the first tool 103 or the first controller 105, of a predetermined position of the first tool 103, using sensor(s) of the first tool 103, such as the starting position of the tool at the beginning of a predetermined action (e.g., the starting position of a golf club at the beginning of a golf swing).

[0079] In response to receiving the first activation signal, the process can proceed to step S301- 2.

[0080] At step S301-2, the first controller 105 measures the one or more expert operator movements with the first operator sensors, and the one or more expert operator tool movements with the first tool sensors 106, for the expert operator performing an action with the first tool 103. Then, the first controller 105 generates the expert operator avatar based upon the measured expert operator movements and expert operator tool movements.

[0081] The one or more expert operator movements can be movements of the operator, as measured by the first operator sensors, when the expert operator is performing the action with the first tool 103. In the case of the action being a golf swing, the first operator sensors are used to measure the movement of the expert operator’s body when performing the golf swing. The one or more expert operator tool movements can be movements of the tool, as measured by the first tool sensors 106, when the expert operator is performing the action with the first tool 103. In the case of the action being a golf swing, the first tool sensors 106 are used to measure the movement of the first tool 103 when the expert operator is performing the golf swing.

[0082] The first controller 105 can continue measuring the expert operator movements and expert operator tool movements until a predetermined measurement ending condition is met.

[0083] In an example, the predetermined measurement ending condition can be the determination by the first controller 105 that a predetermined time has elapsed after the first controller 105 receives the first activation signal.

[0084] In another example, the predetermined measurement ending condition can be the detection, by the first controller 105, of the completion of the action with the first operator sensors and / or the first tool sensors 106. For example, if the action is the swing of a golf club, the predetermined measurement ending condition can comprise detecting that the end of the swing trajectory has been reached.

[0085] In another example, the predetermined measurement ending condition can be the detection, by the first controller 105, of the pressing of a physical button on the first augmented reality headset 101 or the first tool 103 by an operator.

[0086] In another example, the predetermined measurement ending condition can be the detection, by the first controller 105 using a microphone of the first augmented reality headset 101 , of predetermined voice prompt.

[0087] The process can then continue to step S301-3.

[0088] At step S301-3, the first controller 105 generates the expert operator avatar based upon the measured expert operator movements and expert operator tool movements.

[0089] The expert operator avatar can be considered as a 3D avatar that represents the movements and positions of the expert operator when performing the action (e.g., a golf swing).

[0090] At step S301-4, the first controller 105 can store the generated expert operator avatar in the avatar datastore 120. The avatar datastore 120 can, for example, be non-volatile flash memory (e.g., flash memory, network attached storage, cloud storage). The stored expert avatar can then be made available for the next phases.

[0091] In some cases, a plurality of expert operator avatars can be stored in the avatar datastore 120. In such cases, the process described with reference to Figure 4 can be repeated for a plurality of expert operators, and an avatar for each of these can be generated.

[0092] For a plurality of expert operators, the process can involve measuring one or more expert operator movements with the one or more first sensors, and one or more expert operator tool movements of the first tool 103, respectively for each of a plurality of expert operators when each of the plurality of expert operators performs the action with the first tool 103. The process can then involve generating an expert operator avatar for each of the plurality of expert operators based upon the measured respective expert operator movements and expert operator tool movements, and storing these in the avatar database.

[0093] Following step S301-4, the process can then continue to the second phase (step S302 of Figure 3), described in more detail at Figure 5 as follows.

[0094] Figure 5 shows a flow diagram of the process of step S302 for the second phase (step S302 of Figure 3) in which data from the trainee operator is gathered.

[0095] In this phase, when the trainee player is training to improve their skills in performing the action with the tool, a temporary avatar (or trainee avatar) can be created for example using sensors in the augmented reality headset and sensors on the tool.

[0096] The process of Figure 5 (i.e. , the second phase) can take place with the trainee operator using an identical or similar augmented reality headset and tool to that used by the expert operator. Alternatively, the second phase can take place with the trainee operator using the same augmented reality headset and tool as that used by the expert operator. In other words, the second augmented reality headset 111 (and associated components, sensors etc) can be identical, similar or the same as the first augmented reality headset 101 . Likewise, the second tool 113 (and associated sensors etc) can be identical, similar or the same as the first tool 103.

[0097] The process of gathering trainee operator can begin at step S302-1 . At step S302-1 , the second controller 115 can receive a second activation signal, wherein the second activation signal can trigger the second controller 115 to measure preliminary trainee operator characteristics.

[0098] The second activation signal can comprise, for example, the pressing of a physical button on the second augmented reality headset 111 , or on the second tool 113, or the pressing of a virtual button on a smartphone that is connected to the second augmented reality headset 111 and / or the second tool 113.

[0099] In another example, the second activation signal can comprise a voice activation signal that can be received at a microphone of the second augmented reality headset 111.

[0100] In another example, the second activation signal can be the detection, by the second controller 115, of a predetermined position of the second augmented reality headset 111 , using sensor(s) of the second augmented reality headset 111 , such as the starting position of an operator at the beginning of a predetermined action (e.g., the starting position of the head of a golf player at the beginning of a golf swing).

[0101] In another example, the second activation signal can comprise the detection, by the second tool 113 or the second controller 115, of a predetermined position of the second tool 113, using sensors of the second tool 113, such as the starting position of the tool at the beginning of a predetermined action (e.g., the starting position of a golf club at the beginning of a golf swing).

[0102] In response to this triggering, the process can then continue to step S302-2.

[0103] At step S302-2, the second controller 115 can measure the preliminary trainee operator characteristics by measuring one or more preliminary trainee operator movements with the second operator sensors and / or measuring one or more preliminary trainee operator tool movements with the second tool 113 sensors.

[0104] The preliminary trainee operator movements can be an initial performance of the action with the second tool (e.g., an initial swing of a golf club) by the trainee operator information could be based on a first swing of the trainee, for which guidance is not displayed in real-time, but data is only recorded. Then, the avatar can be created and in following performance(s) of the action, the avatar created from data from the previous performance of the action can be adjusted in real-time to match the current performance of the action.

[0105] The trainee avatar can be generated in the same manner as the expert avatar. The second controller 115 can then generate the trainee avatar, at step S302-3.

[0106] At step S302-3, the second controller 115 generates the trainee avatar for the trainee operator based upon preliminary trainee characteristics.

[0107] The trainee operator avatar can be considered as a 3D avatar of the trainee operator. In some cases generating the trainee avatar comprises creating an avatar of the trainee operator if one does not already exist. In other cases, generating the trainee avatar can comprise updating an already existing trainee avatar based on the latest data measured by the sensors. The trainee avatar may be stored in the avatar datastore 120, or can be stored in storage local to the second controller 115.

[0108] The process can then continue to the third phase (step S303 of Figure 3), described in more detail at Figure 6 as follows.

[0109] Figure 6 shows a flow diagram of the process of step S303 for the third phase (step S303 of Figure 3) in which guidance is provided to the trainee when performing the action with the tool.

[0110] In this phase, the system can provide real-time guidance to the non-expert player and can keep suggesting moves to match their avatar (trainee avatar) with a closest expert avatar via augmented reality.

[0111] At step S303-1 , the second controller 115 can determine an expert avatar that is a closest match to the trainee operator based on one or more trainee operator parameters.

[0112] The second controller 115 can access the avatar datastore 120 and compare measured parameters of the trainee operator to predetermined parameters of the plurality of expert operators to determine an avatar that is a closest match. Such parameters may include grip type on the tool, height of the operator, and style of using the tool (i.e., style of play for sports equipment), for example. The second controller 115 can then load the closest matching expert avatar for use in subsequent steps of the process.

[0113] That is, the system can determine the closest expert avatar based on similarity measure and can run an optimisation function which can minimise the distance between the trainee’s avatar and expert’s avatar while providing real-time guidance for best closest matching.

[0114] In some examples, there may only be one expert avatar, rather than a plurality; in this case the one expert avatar can be considered as the closest matching expert avatar. With the closest matching expert avatar determined, the process can continue to step S303-2.

[0115] At step S303-2, the second controller 115 measures trainee operator movements and trainee operator tool movements when the trainee operator is performing the action. The second controller 115 then represents these measurements of movement as the trainee avatar performing the action. With this, the second controller 115 calculates a difference between the expert avatar performing the action and the trainee avatar performing the action.

[0116] The second controller 115 can actively calculate, in real-time, the difference between the expert avatar performing the action and the trainee avatar performing the action, and provides indications and guidance to minimize the distance between avatars by a multicriteria optimization to minimize the distance real-time. That is, there is a loop that repeatedly computes, provides guidance, computes again and checks if the distance is the minimum possible between avatars characteristics such positions and setting, etc.

[0117] In an example of calculating the difference, for each avatar point (e.g., joints of the avatar) a difference score can be calculated in space between the point under consideration and the corresponding point of the expert avatar. This results in a 3D vector for every avatar point. From this, the difference can be calculated in different ways for different purposes. For example, for an arrow indicating a portion of the tool (e.g., a club head) is too high or too low, the magnitude of the vertical component of the difference vector for the point corresponding to the portion of the tool (e.g., club head) can be considered. To calculate how close the trainee avatar is to the expert avatar, the average of the magnitudes of all the difference vectors can be taken.

[0118] The difference can be also be calculated using machine learning methods. For example, clustering algorithms can be used to detect which expert avatar most closely matches the trainee avatar.

[0119] The second controller 115 then displays guidance at a display of the second augmented reality headset 111. The guidance comprises instructions to guide the trainee operator to adapt movements to minimize the calculated difference when the trainee operator is performing the action.

[0120] In other words, the second controller 115 repeatedly measures the trainee operator movements and trainee operator tool movements, throughout the trainee performing the action, repeatedly calculates the difference, and can then display the guidance repeatedly (i.e., constantly in real time) to the trainee while the trainee is performing the action.

[0121] At step S303-2, the second augmented reality headset 111 can display both the trainee avatar and the expert avatar in real time on the headset display. The second augmented reality headset 111 also displays real-time hints or guidance on how to make the trainee avatar match the expert avatar when performing the action.

[0122] The displayed guidance can be, for example, a vertical arrow indicating whether a swing of the second tool 113 (e.g., a golf club) is too high or low compared to the ideal trajectory (i.e., that of the expert avatar). The size of the arrow can depend on the difference between the ideal and current trajectory. The displayed guidance can also include, for example, colours that change between a “too low” colour, a “correct trajectory” colour, and a “too high” colour.

[0123] Different configurations for displaying the guidance can be implemented. The operator (e.g., the trainee) may select their preferred display option.

[0124] In some examples, the augmented reality headset can display both the expert avatar and the trainee avatar at the same time, overlayed on one-another. These avatars can also overlay the "real" image of the operators arms for example.

[0125] In some examples, the expert and trainee avatars can be both displayed to show the trainee how are they close to match expert avatar.

[0126] In some examples, only the expert avatar may be displayed and the trainee will see how they are close to match the same positions, movements etc of the expert avatar.

[0127] The process happens in real-time and the trainee avatar is changing when guidance is provided so that the trainee can follow the guidance.

[0128] At step S303-3, the second controller 115 repeats step S303-2 until a predetermined training ending condition is met.

[0129] In repeating step S302-2, the trainee avatar can be repeatedly updated using the most up-to-date data measured by the sensors, and the difference can then be repeatedly calculated between the updated trainee avatar, so that the guidance can be updated in real-time based upon the continually updated trainee avatar. In an example, the predetermined training ending condition can be the determination by the second controller 115 that a predetermined time has elapsed after the second controller 115 receives the second activation signal.

[0130] In another example, the predetermined training ending condition can be the detection, by the second controller 115, of the completion of the action with the second operator sensors and / or the second tool sensors 116. For example, if the action is the swing of a golf club, the predetermined measurement ending condition can comprise detecting that the end of the swing trajectory has been reached.

[0131] In another example, the predetermined training ending condition can be the detection, by the second controller 115, of the pressing of a physical button on the second augmented reality headset 111 or the second tool 113 by an operator.

[0132] In another example, the predetermined training ending condition can be the detection, by the second controller 115 using a microphone of the second augmented reality headset 111 , of predetermined voice prompt.

[0133] When generating avatars, the data collection, model training and inferencing can be performed via various artificial intelligence approaches based on learning systems, an expert rule-based systems, a knowledge graphs and reasoning, etc.

[0134] In one example, a learning approach could be based on one or more ensembled models and approaches such as the use of reinforcement learning (RL) system, a combination of RL with a mixed inputs deep learning neural network to generate guidance in realtime while the trainee operator is training, a deep learning inferencing model, a reasoning via link prediction from a modelled player’s domain knowledge graph, etc.

[0135] A reinforcement learning system can be used to find the best actions to match the trainee avatar with the closest best learned avatar from an expert operator using all the learned facts, settings and characteristics of an expert operator when they were performing the action. The learning agent in RL can make use of the rewards to achieve quickly in realtime their tasks in findings and matching the expert avatar when providing simultaneously the guidelines via augmented reality to the trainee operator.

[0136] In another example, the learning approach models can be also performed by a machine I deep learning model that learns from data and takes in consideration the settings, characteristics and all the additional details about an expert operator when performing an action, to learn and create avatars for the expert operator. A deep learning simple or mixed input based model could be used here where the model needs to undergo a model training procedure before being able to be used in the product. During the model training phase, an expert operator uses the system as a normal user would, however no hints or ideal trajectories are calculated. The system, instead, collects data and stores it in memory. Additionally, during the model training phase, data can be collected from a ball tracking system (e.g., Trackman) that provides information on the ball trajectory (e.g., distance, height, spin) in an example in which the action comprises hitting a ball. This information determines whether the action (e.g., a swing) was good or not and why (for example, good distance but not enough height). After the collection of this data, the mixed input model is trained using machine learning techniques to learn and create an expert’s avatar that will be further used to match to the trainee avatar when providing guidance real time to enable the matching quickly via augmented reality. Alternatively, a modelled operator’s domain in a knowledge graph (game ontology, game and players captured concepts, a semantic reasoning) can be used to learn from different concepts facts and ideal settings to learn and identify an expert avatar where a semantic reasoning from rules and modelled knowledge enables matching and providing real time guidance to match the trainee avatar to the closest similar expert avatar via augmented reality in the augmented reality headset.

[0137] Figure 7 is a flow diagram of an exemplary execution of the tool operator training system described with reference to Figure 3 (and as further defined with respect to Figures 4, 5 and 6).

[0138] At step S701 , the process begins.

[0139] At step S702, the first phase in which the training data is gathered begins. The steps of this phase are carried out in sub-steps S703, S704, S705, S706 and S707.

[0140] At sub-step S703, the first activation signal is received, as described with reference to step S301-1 of Figure 4. At sub-step S704, data is collected with the sensors, as described with reference to step S301-2 of Figure 4. At sub-step S705, it is determined whether a stopping condition has been met, as described with reference to the first ending condition at step S301-2 of Figure 4. At sub-step S706, the expert avatar is created, as described with reference to step S301-3 of Figure 4. At sub-step S707, the expert avatar is stored in the avatar datastore 120, as described with reference to step S301-4 of Figure 4. The process then continues to step S708 for the second phase in which data from a trainee operator is gathered. The steps of this phase are carried out in sub-steps S709, S710 and S711.

[0141] At sub-step S709, the second activation signal is received, as described with reference to step S302-1 in Figure 5. At sub-step S710, data is collected with the sensors, as described with reference to step S302-2 of Figure 5. At sub-step S711 , the trainee avatar is created or updated, as described with reference to step S302-3 of Figure 5.

[0142] The process then continues to step S712 for the third phase in which the closest matching expert avatar is determined and guidance is provided to the trainee. The steps of this phase are carried out in sub-steps S713, S714, S715 and S716.

[0143] At sub-step S713, the expert avatar that is the closest match to the trainee operator is determined, and loaded, as described with reference to step S303-1 of Figure 6. At substep S714, a difference between the trainee avatar and the expert avatar is calculated when the trainee operator is performing the action, as described with reference to step S303-2 of Figure 6. At sub-step S715, the expert avatar and the trainee avatar can be displayed, and hints or guidance are displayed to the trainee operator while performing the action, as described with reference to step S303-2. At sub-step S716, it is determined whether the predetermined training ending condition has been met, as described with reference to step S303-3 of Figure 6. When the predetermined training ending condition has been met, the process continues to step S717 where the process ends. When the predetermined training ending condition has not been met, the process loops back to step S714, and is repeated until the predetermined training ending condition has been met. When looping back to step S714 the trainee avatar can be updated based upon the latest measured sensor data (e.g., trainee movements), and then continues on through the repeat of steps S714, S715 and back to S716, as described, but using this updated trainee avatar.

[0144] Whilst each of the first controller 105 and second controller 115 have each been described as a single controller, the skilled person will readily understand that they can comprise more than one controller or processing module each configured to perform the described methodology. The controller described herein may be realised as one or more processors such as a central processing unit in a computer system, or as one or more microprocessor units. The controller may be dedicated hardware, or brought about by executing software stored on a medium such as a hard disk or semiconductor memory. In some examples, the methodology herein may be stored as instructions on a non- transitory computer-readable medium. Said instructions may be performed by one or more processors of a tool operator training system. This may be implemented on a computing device, for example. A computer-readable medium can include non-volatile media and volatile media. Volatile media can include semiconductor memories and dynamic memories, amongst others. Non-volatile media can include optical disks and magnetic disks, amongst others.

[0145] Figure 8 depicts a high-level block diagram of an apparatus 800 suitable for implementing various aspects of the disclosure. Although illustrated in a single block, in other embodiments the apparatus 800 may also be implemented using parallel and distributed architectures. Thus, for example, various steps such as those illustrated in the methods described above by reference to Figures 3 to 6 may be executed using apparatus 800 sequentially, in parallel, or in a different order based on particular implementations. Either or both the first augmented reality headset 101 and first controller 105 therein, or the second augmented reality headset 111 and second controller 115 therein, can be implemented in the form of apparatus 800.

[0146] According to an exemplary embodiment, depicted in Figure 8, apparatus 800 comprises a printed circuit board 801 on which a communication bus 802 connects a processor 803 (e.g., a central processing unit "CPU"), a random access memory 804, a storage medium 811 , an interface 805 for connecting a display 806, a series of connectors 807 for connecting one or more user interface devices or modules 808, 809, a wireless network interface 810 and / or a wired network interface 812. Depending on the functionality required, the apparatus may implement only part of the above. Certain modules of Figure 8 may be internal or connected externally, in which case they do not necessarily form integral part of the apparatus itself. Memory 811 contains software code which, when executed by processor 803, causes the apparatus to perform the methods described herein. In an exemplary embodiment, a detachable storage medium 813 such as a USB stick may also be connected. For example the detachable storage medium 813 can hold the software code to be uploaded to memory 811.

[0147] The processor 803 may be any type of processor such as a general purpose central processing unit ("CPU") or a dedicated microprocessor such as an embedded microcontroller or a digital signal processor ("DSP").

[0148] In addition, apparatus 800 may also include other components typically found in computing systems, such as an operating system, queue managers, device drivers, or one or more network protocols that are stored in memory 811 and executed by the processor 803.

[0149] Although aspects herein have been described with reference to particular embodiments, it is to be understood that these embodiments are merely illustrative of the principles and applications of the present disclosure. It is therefore to be understood that numerous modifications can be made to the illustrative embodiments and that other arrangements can be devised without departing from the spirit and scope of the disclosure as determined based upon the claims and any equivalents thereof.

[0150] For example, the data disclosed herein may be stored in various types of data structures which may be accessed and manipulated by a programmable processor (e.g., CPU or FPGA) that is implemented using software, hardware, or combination thereof.

[0151] It should be appreciated by those skilled in the art that any block diagrams herein represent conceptual views of illustrative circuitry embodying the principles of the disclosure. Similarly, it will be appreciated that any flow charts, flow diagrams, state transition diagrams, and the like represent various processes which may be substantially implemented by circuitry.

[0152] Each described function, engine, block, step can be implemented in hardware, software, firmware, middleware, microcode, or any suitable combination thereof. If implemented in software, the functions, engines, blocks of the block diagrams and / or flowchart illustrations can be implemented by computer program instructions I software code, which may be stored or transmitted over a computer-readable medium, or loaded onto a general purpose computer, special purpose computer or other programmable processing apparatus and I or system to produce a machine, such that the computer program instructions or software code which execute on the computer or other programmable processing apparatus, create the means for implementing the functions described herein.

[0153] When provided by a processor, the functions may be provided by a single dedicated processor, by a single shared processor, or by a plurality of individual processors, some of which may be shared. Moreover, explicit use of the term "processor" or "controller" should not be construed to refer exclusively to hardware capable of executing software, and may implicitly include, without limitation, digital signal processor (DSP) hardware, network processor, application specific integrated circuit (ASIC), field programmable gate array (FPGA), read only memory (ROM) for storing software, random access memory (RAM), and non-volatile storage. Other hardware, conventional or custom, may also be included. Their function may be carried out through the operation of program logic, through dedicated logic, through the interaction of program control and dedicated logic, or even manually, the particular technique being selectable by the implementer as more specifically understood from the context. 1

Claims

Claims1 . A tool operator training system, the system comprising: one or more first operator sensors configured to be placed on an expert operator of a tool, wherein the one or more first operator sensors comprise a first augmented reality headset; a first tool with one or more first tool sensors; a first controller configured to measure one or more expert operator movements with the one or more first operator sensors, and one or more expert operator tool movements with the one or more first tool sensors, for the expert operator performing an action with the first tool, and generate an expert operator avatar based upon the measured expert operator movements and expert operator tool movements; one or more second operator sensors configured to be placed on a trainee operator of the tool, wherein the one or more second operator sensors comprise a second augmented reality headset; a second tool with one or more second tool sensors; a second controller configured to: generate a trainee avatar for the trainee operator based upon preliminary trainee operator characteristics; measure one or more trainee operator movements with the one or more second operator sensors, and one or more trainee operator tool movements with the one or more second tool sensors when the trainee operator is performing the action with the second tool and represent these as the trainee avatar performing the action; calculate a difference between the expert operator avatar performing the action and the trainee avatar performing the action when the trainee operator is performing the action; and display guidance at a display of the second augmented reality headset, wherein the guidance comprises instructions to guide the trainee operator to adapt movements to minimize the calculated difference when the trainee operator is performing the action.

2. The tool operation training system of claim 1 , wherein the first tool and the second tool are a same type of tool.

3. The tool operation training system of any preceding claim, wherein the first tool and the second tool are a golf club, and the action comprises a golf swing.

4. The tool operation training system of any preceding claim, wherein the first operator sensors comprise one or more of an accelerometer, a gyroscope, and a magnetometer, and the second operator sensors comprise one or more of an accelerometer, a gyroscope, and a magnetometer.

5. The tool operation training system of any preceding claim, wherein the second operator sensors comprise a camera of the second augmented reality headset configured to capture video.

6. The tool operator training system of any preceding claim, wherein the first controller is configured to measure the one or more expert operator movements with the one or more first operator sensors, and the one or more expert operator tool movements with the one or more first tool sensors, for the expert operator performing an action with the first tool, until a predetermined measurement ending condition is met.

7. The tool operator training system of any preceding claim, wherein the measuring one or more expert operator movements with the one or more first operator sensors, and the one or more expert operator tool movements with the one or more first tool sensors, for the expert operator performing the action with the first tool, and the generating the expert operator avatar based upon the measured expert operator movements and expert operator tool movements comprises: measuring one or more expert operator movements with the one or more first sensors, and one or more expert operator tool movements of the first tool, respectively for each of a plurality of expert operators when each of the plurality of expert operators performs the action with the first tool; and generating an expert operator avatar for each of the plurality of expert operators based upon the measured respective expert operator movements and expert operator tool movements.

8. The tool operator training system of claim 7, wherein the second controller is configured to determine a closest matching expert operator avatar of the pluralityof expert operator avatars to the trainee avatar, such that the calculated difference is calculated between the closest matching expert operator avatar and the trainee avatar.

9. The tool operator training system of any preceding claim, wherein the first controller is configured to receive a first activation signal, wherein the first activation signal triggers the first controller to measure the expert operator movements and the expert operator tool movements.

10. The tool operator training system of any preceding claim, wherein the first controller is configured to store the generated expert operator avatar in an avatar datastore.

11. The tool operator training system of any preceding claim, wherein the second controller is configured to receive a second activation signal, wherein the second activation signal triggers the second controller to measure the preliminary trainee operator characteristics.

12. The tool operator training system of any preceding claim, wherein the second controller is configured to measure the preliminary trainee operator characteristics by measuring one or more preliminary trainee operator movements with the second operator sensors and / or measuring one or more preliminary trainee operator tool movements with the second tool sensors.

13. The tool operator training system of any preceding claim, wherein the second controller is configured to, until a predetermined ending condition is met, repeatedly: measure the one or more trainee operator movements with the second operator sensors, and the one or more trainee operator tool movements with the second tool sensors when the trainee operator is performing the action and represent these as the trainee avatar performing the action; calculate the difference between the expert operator avatar performing the action and the trainee avatar performing the action when the trainee operator is performing the action; and display the guidance at the display of the second augmented reality headset when the trainee operator is performing the action.

14. A tool operator training method, the method comprising: measuring one or more expert operator movements with one or more first operator sensors, wherein the one or more first operator sensors are configured to be placed on an expert operator of a tool and comprise a first augmented reality headset, and measuring one or more expert operator tool movements with one or more first tool sensors of a first tool, for an expert operator performing an action with the first tool, and generating an expert operator avatar based upon the measured expert operator movements and expert operator tool movements; generating a trainee avatar for a trainee operator based upon preliminary trainee operator characteristics; measuring one or more trainee operator movements with one or more second operator sensors wherein the second operator sensors are configured to be placed on a trainee operator of the tool and the one or more second operator sensors comprise a second augmented reality headset, and one or more trainee operator tool movements with one or more second tool sensors of a second tool when a trainee operator is performing the action with the second tool, and representing these as the trainee avatar performing the action; calculating a difference between the expert operator avatar performing the action and the trainee avatar performing the action when the trainee operator is performing the action; displaying guidance at a display of the second augmented reality headset, wherein the guidance comprises instructions to guide the trainee operator to adapt movements to minimize the calculated difference when the trainee operator is performing the action.

15. A first non-transitory computer readable medium storing instructions thereon that when executed by one or more processors cause the one or more processors to: measure one or more expert operator movements with one or more first operator sensors, wherein the one or more first operator sensors are configured to be placed on an expert operator of a tool and comprise a first augmented reality headset, and measure one or more expert operator tool movements with one or more first tool sensors of a first tool, for an expert operator performing an action with the first tool; and generate an expert operator avatar based upon the measured expert operator movements and expert operator tool movements;a second non-transitory computer readable medium storing instructions thereon that when executed by one or more processors cause the one or more processors to: generate a trainee avatar for a trainee operator based upon preliminary trainee operator characteristics; measure one or more trainee operator movements with one or more second operator sensors wherein the second operator sensors are configured to be placed on a trainee operator of the tool and the one or more second operator sensors comprise a second augmented reality headset, and one or more trainee operator tool movements with one or more second tool sensors of a second tool when a trainee operator is performing the action with the second tool, and represent these as the trainee avatar performing the action; calculate a difference between the expert operator avatar performing the action and the trainee avatar performing the action when the trainee operator is performing the action; display guidance at a display of the second augmented reality headset, wherein the guidance comprises instructions to guide the trainee operator to adapt movements to minimize the calculated difference when the trainee operator is performing the action.

Images