Training system, method and program

Abstract

To enable efficient enhancement of user's learning level and to evaluate user's movement to a training object with high accuracy.SOLUTION: A training system includes: a three-dimensional display control part for making it possible to display a frame for identifying an operation object part to be operated according to a progress of a training scenario by three-dimensionally displaying a training object in a virtual space; an operation detection part for detecting operation information including a user's operation position and an operation direction; and an operation determination part for calculating a user's learning level on the basis of the operation object part of the training object and the operation information. The training system has at least two modes such as a learning mode for reproducing operation procedures on a virtual space according to the training scenario without user's operations, and a scoring mode for allowing the user to perform operations along the training scenario to perform scoring by recording operation contents. A three-dimensional display part determines display / non-display of the frame on the basis of a mode selected from the modes or the user's learning level calculated by the operation determination part.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a training system, method, and program that uses technology such as VR (Virtual Reality), AR (Augmented Reality), or MR (Mixed Reality) (hereinafter simply referred to as "VR"). [Background technology]

[0002] Conventionally, systems have been proposed that use VR technology to train trainees (users). For example, Patent Document 1 proposes a training system that uses a simulation means to develop a training scenario that includes a trap that induces human error, thereby simulating an event to be trained, so that even skilled workers can be effectively trained to prevent human error, and notifies the user of the occurrence of a human error when a series of tasks in which the human error occurred has ended or when the task cannot be continued due to the human error. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2006-072193 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the technology of Patent Document 1 aims to prevent the occurrence of human error through training, and not to efficiently learn how to operate the equipment or device to be operated (hereinafter referred to as the "training object"). In particular, to achieve efficient training, the problem is what information to display in the virtual space, or how to determine the user's operation content and improve proficiency.

[0005] The present invention has been made in response to such conventional circumstances, and aims to provide a training system, method, and program that can efficiently improve a user's proficiency and accurately evaluate the user's movements relative to a training object. [Means for solving the problem]

[0006] In order to achieve the above object, the training system according to the present invention comprises: The training object is displayed in three dimensions in the virtual space, and the training scenario progresses. , user operation Should It is an equipment a three-dimensional display control unit (3D-CG display control unit) capable of displaying a frame for identifying an operation target portion (operation target identification frame); an operation detection unit that detects operation information including a user's operation position and operation direction; A training system comprising: The system has at least two modes: a learning mode in which the user does not operate the system but reproduces the operation procedures in a virtual space according to a training scenario, and a scoring mode in which the user operates the system according to the training scenario, records the operation details, and scores the operation details, and the user can select which mode to operate in; The three-dimensional display control unit is configured to If the selected mode is the learning mode, the frame is displayed, and if the selected mode is the marking mode, the frame is not displayed. Here, the training scenario is information that associates operation target parts with judgment conditions such as time limits so that the operation sequence can be recognized. In addition, an operation determination unit may be provided that calculates the user's proficiency based on the operation target part of the training object and operation information, and the three-dimensional display control unit may decide whether to display or hide the frame based on the user's proficiency calculated by the operation determination unit instead of the mode.

[0007] Preferably, the operation detection unit detects operation information by at least one tracking method selected from head tracking, position tracking, hand tracking, and eye tracking.

[0008] The training system according to the present invention also includes: a storage unit that stores coordinates of the frame in association with each operation target portion; The three-dimensional display control unit determines whether there is a change in the area to be operated as the training scenario progresses, and if there is no change, displays a normal frame as the frame identifying the area to be operated, while if there is a change, displays a warning frame different from the normal frame as the frame identifying the area to be operated.

[0009] The training system according to the present invention further comprises means for saving a judgment area where the operation means identifying image should or should not pass, depending on the distance from each operation target portion of the training object, and the operation judgment unit judges whether the user has correctly accessed the operation target portion based on whether the position of the operation means identifying image has passed through the judgment area. This enables training in a virtual space with high accuracy similar to training in a real space.

[0013] The training system according to the present invention is When in scoring mode , the above When the operation detection unit detects the user's line of sight, the three-dimensional display control unit detects that the line of sight vector intersects with a predetermined range of the operation target portion of the training object for a certain period of time or more, The aforementioned Show Frame It is characterized by:

[0014] In addition, the operation detection unit may detect the user's line of sight, and the operation determination unit may determine that the operation target area has been selected or operated when the line of sight vector intersects with a predetermined range of the operation target area of ​​the training object for a certain period of time or more. This allows the user to easily identify, select, and operate the area to be operated.

[0015] The operation target identification frame for identifying the operation target portion can be displayed in a different identification manner depending on various conditions other than those described above. For example, when it is determined that the operation target identification image has reached the operation target portion based on the positional relationship between the operation means identification image and the operation target portion, the operation target identification frame may be displayed in an identification manner such as by changing the color to notify that the operation target portion has been selected, or when it is determined that the operation means identification image has reached the operation target portion via the correct route based on the positional relationship between the operation means identification image and the operation target portion, the operation target identification frame may be displayed in an even different identification manner.

[0016] In addition, the operation judgment means of the training system according to the present invention is characterized by comprising a statistical analysis unit that calculates statistical data on the correctness / incorrectness of the operation content for each operation target part and the time it takes to perform the operation based on the judgment results of multiple users, and outputs the calculation results, and a scenario change editing unit that changes the training scenario based on the output of the statistical analysis unit.

[0017] The training method according to the present invention is a method for educating and training a user on how to operate a training object displayed in three dimensions in a virtual space, The training object is displayed in three dimensions in the virtual space, and the training scenario progresses. , user operation Should It is an equipment a three-dimensional display process for displaying a frame for identifying an operation target portion; an operation detection process for detecting operation information including a user's operation position and operation direction; Including, The system has at least two modes: a learning mode in which the user does not operate the system but reproduces the operation procedures in a virtual space according to a training scenario, and a scoring mode in which the user operates the system according to the training scenario, records the operation details, and scores the operation details, and the user can select which mode to operate in; The three-dimensional display process is performed in a selected mode from among the modes. If the selected mode is learning mode, the frame is displayed, and if the selected mode is grading mode, the frame is not displayed. It is characterized by: In addition, the present invention may include an operation determination process that calculates the user's proficiency level based on the operation target part of the training object and operation information, and the three-dimensional display process may determine whether to display or hide the frame based on the user's proficiency level calculated by the operation determination process instead of the mode.

[0018] The program according to the present invention is a program for educating and training a user on how to operate a training object displayed in three dimensions in a virtual space, The training object is displayed in three dimensions in the virtual space, and the training scenario progresses. , user operation Should It is an equipment a three-dimensional display process for displaying a frame for identifying an operation target portion; an operation detection process for detecting operation information including a user's operation position and operation direction; Run The system has at least two modes: a learning mode in which the user does not operate the system but reproduces the operation procedures in a virtual space according to a training scenario, and a scoring mode in which the user operates the system according to the training scenario, records the operation details, and scores the operation details, and the user can select which mode to operate in; The three-dimensional display process is performed in a selected mode from among the modes. If the selected mode is learning mode, the frame is displayed, and if the selected mode is grading mode, the frame is not displayed. It is characterized by: In addition, an operation determination process may be executed to calculate the user's proficiency level based on the operation target part of the training object and operation information, and the three-dimensional display process may determine whether to display or hide the frame based on the user's proficiency level calculated by the operation determination process instead of the mode. [Effects of the Invention]

[0020] According to the present invention, it is possible to efficiently improve the user's proficiency and to accurately evaluate the user's movements relative to a training object. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a functional block diagram of a training system according to an embodiment of the present invention. [Figure 2] 1 is an illustration of a training object according to an embodiment of the present invention; FIG. [Figure 3] FIG. 2 is a data structure diagram of the training scenario storage unit in FIG. 1. [Figure 4] 4A and 4B are diagrams showing the data structure of the operation determination storage unit of FIG. 1, where FIG. 4A shows the operation determination records of individual users, and FIG. 4B shows the statistically processed operation records of multiple users. [Figure 5] 1. FIG. 4 is an explanatory diagram of the display of an operation target identification frame by the 3D-CG display control unit in FIG. [Figure 6] 10 is a flowchart showing the procedure of an operation target identification frame display process performed by the 3D-CG display control unit in FIG. 1. [Figure 7] 2 is an explanatory diagram of a guidance display by the 3D-CG display control unit of FIG. 1. [Figure 8] FIG. 2 is an explanatory diagram of an operation using a controller according to an embodiment of the present invention. [Figure 9] 1A and 1B are explanatory diagrams of operations using hand tracking technology according to an embodiment of the present invention. [Figure 10] 1 is an explanatory diagram of an operation using gaze tracking technology according to an embodiment of the present invention; [Figure 11] 1. FIG. 4 is a conceptual explanatory diagram of an access route determination process performed by an operation determination unit in FIG. [Figure 12] 1. FIG. 4 is a detailed explanatory diagram of the access route determination process of the operation determination unit of FIG. [Figure 13] 10 is a flowchart of an access route determination process of an operation determination unit in FIG. [Figure 14] 13 is a modified example of the access route determination process of the operation determination unit in FIG. [Figure 15] 2 is a flowchart showing the processing steps of the statistical analysis unit and the scenario modification editing unit of FIG. 1. [Figure 16] 10 is a flowchart showing a processing procedure in an operation mode of the 3D-CG display control unit of FIG. [Figure 17] 16 is a flowchart showing the processing steps of the statistical analysis unit and the scenario modification editing unit according to another embodiment of FIG. 15. [Figure 18] 17 is a flowchart showing the processing procedures of the 3D-CG control unit, the operation determination unit, and the operation determination output unit according to another embodiment of FIG. 16. DETAILED DESCRIPTION OF THE INVENTION

[0022] A training system according to an embodiment of the present invention will be described below with reference to the drawings. 1, the training system 1 is composed of a virtual content display device 10, an operation device 50, and a training scenario / operation record determination device 60. The training scenario / operation record determination device 60 and the HMD 10 are connected via a wired or wireless communication line.

[0023] For example, a head mounted display (hereinafter simply referred to as "HMD") can be used as the virtual content display device 10. Below, an example will be explained in which this HMD is used.

[0024] The training scenario / operation record determination device (hereinafter simply referred to as the "determination device") 60 stores three-dimensional data of the training object and operation procedure data of the training object to be transmitted to the HMD 10. The determination device 60 also performs determination and statistical analysis of the user's operation content based on the operation information transmitted from the HMD 10. Then, the determination device 60 changes the training scenario, such as the operation procedure and determination criteria, based on the results of the statistical analysis.

[0025] The operation device 50 can be configured as a controller held or worn by the user, a hand tracking device that detects the movement of the user's fingers, etc. This operation device 50 is sufficient as long as it can detect the movement of the user, and it also includes a device that detects the user's line of sight if it is possible to operate the training object with the user's line of sight.

[0026] The HMD 10 displays the training object in a three-dimensional virtual space based on the three-dimensional data of the training object passed from the determination device 60, and obtains operation information such as the position of the means for operating the training object (for example, a finger or a wrench, which will be described later) and the operation content (for example, pressing a button or rotating a wrench) based on the detection signal passed from the operation device 50. This operation information is sent to the determination device 60, which determines the operation content.

[0027] More specifically, the judgment device 60 includes a 3D model memory unit 61 that stores three-dimensional data of the training object, a training scenario memory unit 62 that stores scenarios (training scenarios) for training the user, such as operation procedures, and an operation judgment memory unit 63 that stores the results of operation judgment. The judgment device 60 also includes a 3D-CG display control unit 71 that virtually displays information such as training objects and operation instructions according to the training scenario on the HMD 10 based on data stored in the 3D model storage unit 61 and the training scenario storage unit 62, an operation judgment unit 72 that evaluates the user's operation content based on operation information sent from the HMD 10 and writes the evaluation results in the operation judgment storage unit 63, an operation judgment output unit 73 that outputs the evaluation results stored in the operation judgment storage unit 63, a statistical analysis unit 74 that performs statistics and analysis based on the evaluation results, and a scenario modification and editing unit 75 that edits the training scenario data based on the results of this statistical analysis. The scenario modification and editing unit 75 may manually edit the training scenario using a user interface (not shown), or may automatically perform editing processing based on the results of the statistical analysis using a procedure described below.

[0028] The HMD 10 is equipped with a 3D-CG display unit 11 that virtually displays information sent from the 3D-CG display control unit 71 of the judgment device 60, an operation content display unit 12 that displays an image representing a specified operation means (operation means identification image) in three-dimensional virtual space based on data related to operations such as the user's movements passed from the operation device 50, an operation coordinate detection unit 13 that detects coordinates used for operation judgment (e.g., coordinates of the operation means), and an operation content detection unit 14 that detects the operation content (e.g., the position of the user's hand and the approach route to the training object).

[0029] The operation device 50 includes an operation detection unit 51 that detects data necessary for determining an operation, such as a user's movement, and passes the data to the operation content display unit 12 of the HMD 10. Here, each of the means 12 to 14, 51, and 71 to 75 can be realized by a program as a function of a computer. The determination device 60 can be realized by a general-purpose computer, but may also be realized as an integrated part of the HMD 10. The operation device 50 can also be realized using the functions of the HMD 10.

[0030] An example of the device configuration of the training system 1 and an image of the training object displayed on the HMD 10 are shown in FIG. For example, when training is given to operate the driver's cab of a railway vehicle, the equipment in the driver's cab is displayed in three dimensions as a training object 11a on the HMD 10. Known methods such as head tracking, position tracking, hand tracking, and eye tracking can be used to track the user's movements.

[0031] An example of data stored in the training scenario storage unit 62 is shown in Figure 3. A training scenario consists of judgment conditions for each operation procedure. Multiple training scenarios are provided for each learning subject. The training scenario may be branched depending on the judgment result for each work item. In FIG. 3, task items are stored in the order of the operation procedure, and each task item is associated with and stored the following: the content of the action (guidance), a flag that sets whether or not it is subject to operation judgment, a flag that sets whether or not it is subject to gaze judgment, the coordinates of the operation target part, the behavior during operation (change in the operation target part), the coordinates of the operation target identification frame, the coordinates of the judgment area, the time limit, and a flag that sets whether or not scoring of the judgment target is enabled. The judgment area is an area that indicates the range through which the user's operating means passes depending on the distance from the operation target part, and is information (route information) that specifies the route the user should take to access each operation target part. Note that this judgment area may also be set as a route that the user should not access. Also, in FIG. 3, a flag that sets whether or not to perform an action (guidance) for each task item may be provided.

[0032] 4 shows an example of the data structure of the operation determination storage unit 63. The results of each work item are stored for each user ID and training scenario ID. Three-dimensional coordinates for displaying a 3D model of a training object are stored in the 3D model storage unit 61. The coordinates of the operation object identification frame, the determination area, or the coordinates of the operation means identification image shown in FIG. 3 may be stored in the 3D model storage unit 61.

[0033] Next, the operation of the training system 1 having the above-described configuration will be described. (1) Training Mode In this embodiment, the training system 1 has three training modes: 1) a learning mode, 2) an operation mode, and 3) a scoring mode.

[0034] The learning mode is an operation mode in which the operating procedures are reproduced in a virtual space according to a scenario without user operation. In other words, the user does not operate the virtual space, but rather memorizes the operation content by looking at the operation object, the operation object identification frame, and the guidance. In this case, an operating means identification image or avatar that moves based on the training scenario may be displayed.

[0035] The operation mode is an operation mode in which the user operates according to a training scenario. In this operation mode, the user's operations are not scored. The scoring mode is also an operation mode in which the user performs operations according to a training scenario, but unlike the operation mode, the user's operations are recorded, scored, and evaluated.

[0036] It is preferable that training be carried out in stages such as 1) learning mode → 2) operation mode → 3) scoring mode. Furthermore, effective training can be carried out by controlling the display of an operation object identification frame that indicates which part of the training object to operate, such that it is displayed in the learning mode and operation mode but not displayed in the scoring mode. Of course, whether the operation object identification frame is displayed or not may be determined based on conditions such as the user's proficiency as well as the training mode.

[0037] (2) Operation target guide function (Fig. 5) At the start of training, the 3D-CG display control unit 71 displays a three-dimensional image of the training object on the 3D-CG display unit 11. Then, in the learning mode and operation mode, as the training scenario progresses, a frame (operation object identification frame) 21 is displayed superimposed on the 3D model of the training object to make it easier to visually identify the operation object portion at that time, as shown in Fig. 5. When the user finishes operating the operation object portion, that operation object identification frame is erased, and an operation object identification frame for the next operation object portion is displayed.

[0038] The procedure of the operation target identification frame display process of the 3D-CG display control unit 71 will be described in detail below with reference to FIG. When training begins, the 3D-CG display control unit 71 is activated and displays the training object on the 3D-CG display unit 11 of the HMD 10 (S501). Then, it determines whether the training mode is a scoring mode (S502). If it is a scoring mode, the process ends. If the training mode is not a scoring mode, i.e., if it is a learning mode or an operation mode, the following processing is performed for all operation target parts of the training scenario (S503a, S503b). First, the training scenario storage unit 62 is accessed to extract operation target identification frame information for the operation target item (S504). This operation target identification frame information determines the display mode and position of the operation target identification frame, such as its color and operation (e.g., flicker). The 3D-CG display control unit 71 displays the operation target identification frame in the virtual space of the 3D-CG display unit 11 according to this operation target identification frame information (S505). The coordinates of the operation target identification frame are associated with the operation target part of the operation item and are stored in advance in the training scenario storage unit 62 or the 3D model storage unit 61.

[0039] Next, the 3D-CG display control unit 71 determines the training mode (S506), and if the training mode is the learning mode, it erases (S510) the operation object identification frame after a set time (time limit) stored in advance in the training scenario storage unit 62 has elapsed (S507).On the other hand, if the training mode is the operation mode, it determines whether the operation has ended, and if the operation has ended (S508) or after a timeout (S509), it erases (S510) the operation object identification frame.

[0040] The above processing is a procedure for displaying / hiding the operation target identification frame, and for example, guidance display (FIG. 7) and voice instructions according to the training scenario are displayed on the 3D-CG display unit 11 in separate steps. The operation target identification frame, whose display is controlled according to the above procedure, allows the user to easily recognize the operation target area. Furthermore, even if the user's viewpoint or posture changes, the display can be easily understood, which improves training efficiency.

[0041] The operation object identification frame is not limited to the training mode described above, and other elements such as the user's operation content and proficiency may be added to distinguishably display the operation object identification frame so that the user can understand the situation. Furthermore, instead of displaying / hiding each individual operation object part, the display of operation object identification frames for multiple operation object parts may be controlled for each series of training units.

[0042] (3) Operation of training equipment (Figure 8) According to the training scenario, the user operates the equipment (operation target part) in the virtual space, and this equipment operation can be performed using the controller shown in FIG.

[0043] When a controller is used, the operation detection unit 51 provided in the controller sends information regarding the position and orientation (direction) of the controller to the HMD 10, and the operation content display unit 12 displays an image (operation means identification image) in the virtual space indicating the means (tool) used to operate the operation target area.

[0044] The display position of the operation means identification image at this time may be based on a preset default position (for example, a relative distance from the position of the HMD 10), and the operation means identification image may be moved from that position depending on the amount and direction of movement of the controller, or may be displayed in a different way depending on the type of controller. For example, if the controller is a device that emits light such as a laser, the trajectory of the emitted light may be calculated based on the position and orientation of the controller, and the operation means identification image may be displayed at the intersection of that trajectory and the training object.

[0045] The following describes the procedure for selecting and operating a training object using the controller. The data such as the position coordinates and orientation of the controller in the virtual space can be acquired using existing position tracking technology. The operation detection unit 51 of the controller 50 passes the acquired data such as the position coordinates and orientation to the operation content display unit 12 of the HMD 10.

[0046] The operation content display unit 12 draws an extension line in a certain direction in the virtual space from the position and orientation of the controller 50. This extension line may or may not be displayed in the virtual space, but by displaying the extension line in the virtual space from the controller 50 to the intersection of the training object, the user will recognize the extension line as if it were a laser beam, and improved operability can be expected.

[0047] Then, a pointer 50a is displayed at the intersection of this extension line and the 3D model of the training object. The coordinates of this pointer 50 a are sent by the operation coordinate detection unit 13 to the operation determination unit 72 of the determination device 60 . Furthermore, when the pointer 50a reaches a desired position on the training object, the user performs a predetermined action (for example, pressing a button on the controller 50). This action is detected by the operation content detection unit 14 and sent to the operation determination unit 72 of the determination device 60.

[0048] The operation determination unit 72 determines that the determination of the user's operation on the operation target portion has been completed successfully when the pointer 50a enters the determination coordinates (plane or solid) and a predetermined action is performed. In addition to this determination result, the operation determination unit 72 calculates an evaluation of the operation target portion by the user, taking into account the operation content such as the time required for the operation and the trajectory of the pointer 50a, i.e., the approach route to the operation target portion, and the like, and stores the evaluation result in the operation determination storage unit 63.

[0049] (Operation of equipment by other means) Although the above description has been given of a case where operation is performed using a controller, the operation device 50 is not limited to this, and known techniques can be used. For example, a hand tracking device may be used as the operation device 50 to track the movements of the user's fingers and display them in a three-dimensional virtual space. In this case, too, as described above, a default display position in the three-dimensional virtual space is determined in advance, and the operation means identification image in the three-dimensional virtual space can be moved according to the amount and direction of movement detected by the hand tracking device.

[0050] Specifically, the operation content display unit 12 takes in the operation data passed from the hand tracking device and calculates coordinates in the virtual space (virtual hand coordinates) using the operation coordinate detection unit 13. These virtual hand coordinates are passed to the operation determination unit 72. When the operation determination unit 72 detects that the virtual hand coordinates enter a predetermined determination coordinate (plane or solid) associated with the operation target part and that a predetermined action (hand gesture / button) has been executed via the operation content detection unit 14, it determines that the operation is complete.

[0051] In the above example of operation determination, a virtual hand is displayed in a virtual space using hand tracking technology, and an object in the virtual space is operated and determined. Alternatively, as shown in Fig. 9, instead of a virtual hand, a tool 22 may be displayed in the virtual space based on the position (coordinates and direction) of the user's own hand in the real space captured by hand tracking technology, and an object 11a in the virtual space may be operated and determined. In this operation determination, the position and orientation of the user's hand are tracked using hand tracking technology to obtain coordinates, directions, and angles. Based on the obtained data, the operation content display unit 12 determines that a tool is being held by the hand in real space, and displays an image of the tool (operation means identification image) 22 in virtual space.

[0052] FIG. 9 shows an example of a device assembly / disassembly task in which a wrench is used as jig 22 to remove and install nuts (operation target parts) on a device (training object) displayed in virtual space. While FIG. 9 depicts a wrench as jig 22, jig 22 is not limited to a wrench. Jig 22 is operated by moving and rotating a controller, pressing a button, or performing other predetermined actions such as hand gestures (e.g., clenching one's hand). The image of jig 22 is preferably a composite image of the so-called jig body (B in the figure) combined with an image (circle C in the figure) showing the part that engages with the operation target part of the training object (A in the figure). This allows the user to accurately recognize the position of jig 22 that engages with the operation target part, enabling them to set jig 22 in the correct position relative to the training object in virtual space. In some cases, different jigs are used for each work item or each part to be operated. The 3D coordinates of the jig (operation means identification image) 22 used for that work can be stored in association with the work item number in the training scenario memory unit 62, and the image of the jig 22 that differs for each work item can be displayed in the virtual space using the 3D coordinates and the position of the fingers obtained by hand tracking, etc.

[0053] This processing procedure will be described in detail below with reference to FIG. (Step 1) The 3D-CG display control unit 71 displays a 3D model of the device to be operated in the virtual space. (Step 2) The part (A) to be operated is set in advance on the equipment. A guide (frame) may be displayed to indicate the part to be operated, but it is not necessary to display it. This data is saved in advance in the training scenario storage unit 62 shown in Figure 3. The setting information includes, for example, the coordinates of the part to be operated, and the direction and angle of the jig relative to the part to be operated. In addition, the operation content determined by the operating means, such as the behavior during operation and the correct operation (e.g., turning a wrench clockwise), is set in advance. (Step 3) Based on the coordinate information of the user's hand acquired in the real space by hand tracking technology, a virtual tool (B) is displayed in the virtual space at a relative position (pre-set) from the coordinates of the hand. An operation detection part (C) is set on the tool. The process described below determines whether this operation detection part (C) has accessed the operation target area via the correct route. (Step 4) If a specified action (such as turning right) is performed while the coordinates (within a certain range) of the operation detection part (C) of the jig 22 and the part to be operated and the angle of the jig (pre-set: relative angle to the part to be operated) match, it is recorded as a correct operation and a virtual model is displayed indicating that the operation was performed correctly. Actions can be performed using hand movements or controller buttons.

[0054] (Eye tracking) (Figure 10) The operation means identification image 22 can also be moved using the head tracking function and gaze (eye) tracking function of the HMD 10. That is, the head position and orientation in the virtual space are acquired using the head tracking function, and the eye movement (gaze) is tracked using the gaze tracking function to move the operation means identification image 22. In this case, the operation detection unit 51 that tracks the head movement and gaze movement can be incorporated into the HMD 10.

[0055] The procedure for eye-gaze tracking will be described below with reference to FIG. The operation content display unit 12 displays an extension line (laser pointer) 23 in a certain direction based on the line of sight or head position and orientation information, based on the tracking data passed from the operation detection unit 51. This display is not necessary, but by displaying it, the user can easily know which direction they are facing. Then, a pointer image 22 is displayed at the intersection of the extension line (laser pointer) 23 and the 3D model of the training object. The operation determination unit 72 determines that the operation is complete when it detects that the coordinates of the pointer image enter a determination coordinate (plane or solid) and a predetermined action (button, gaze for a certain period of time or more) has been performed. The coordinates of the operation target identification frame 21 can be used as the determination coordinate, but separate determination coordinates may also be provided.

[0056] When the training mode is the scoring mode, the operation judgment unit 72 judges each task item in the training scenario for which the scoring valid flag is set, and stores the judgment result in the operation judgment storage unit 63. An example of the judgment result is shown in FIG. 10(b). The judgment result may include not only pass / fail but also information used for user evaluation (calculation of proficiency), such as operation time. Furthermore, a point value may be calculated based on the judgment result, and the user may be evaluated using this as proficiency. At this time, the points may be weighted for each task item.

[0057] (4) Operation detection process In the scoring mode, the operation determination unit 72 determines whether the user accessed and operated the operation target area via the correct route within a preset time (access route determination process), and performs an evaluation based on the result.

[0058] FIG. 11 is a conceptual explanatory diagram of the access route determination process of the operation determination unit 72. In this diagram, if the operation tool identification image 22 reaches the operation target portion via a route defined by multiple determination areas, it is determined that the approach has been correct. In this embodiment, arrival at the operation target portion is determined based on whether or not the operation target identification frame 21 has been reached. This operation target identification frame 21 can be represented in a plane as shown in FIG. 11(a), but it may also be represented in a three-dimensional form as shown in FIG. 11(b). In addition, instead of the coordinates of the operation target identification frame, other determination coordinates, for example, the coordinates of the image of the operation target portion, can be used.

[0059] FIG. 12 shows a specific example of a judgment area used in the access route judgment process. As shown in this figure, multiple judgment areas of a predetermined size are provided according to the vertical distance (height) from the operation object identification frame 21 that is set to surround the operation object part. Each judgment area is provided parallel to the operation object identification frame 21. In FIG. 12, judgment areas 1 to n are provided. In this figure, H1 is the distance from the operation object identification frame to judgment area 1 that is located farthest from the operation object identification frame, t is the coordinate (point coordinate) of the position of the operation tool identification image 22, and D is the distance from the operation object identification frame 21 to the operation tool identification image 22.

[0060] Next, the procedure of the access route determination process performed by the operation determination unit 72 will be described with reference to FIG. The operation determination unit 72 acquires the coordinates (t) of the operation tool identification image (S601). It also calculates the distance (Dt) between the operation tool identification image and the operation object identification frame (S602). It then compares the distance Dt with the distance H1 (S603). If the distance Dt is equal to or greater than the distance H1, it returns to step S601 and repeats the subsequent processes. On the other hand, if the distance Dt is less than the distance H1 in step S603, it determines whether the next determination area i (i = 1 to n) has been passed through (S604). This determination process can be performed, for example, by determining for each determination area whether the height of the operation tool identification image is equal to or greater than the height of the determination area, calculating a movement vector from the position information of the height just after and just before the image falls below the determination area, and checking whether the vector intersects with the determination area. If the determination result in step S604 is YES, it records (sets) a passage flag indicating that the determination area i has been passed through (S605). The above process is repeated until the operation tool identification image reaches the operation object identification frame (S606). Thereafter, it is determined whether the pass flag is set for all judgment areas i (i = 1 to n) (S607), and if it is set, it is determined that the user has passed (S608), that is, that the user has correctly accessed the operation target portion. On the other hand, if the result of the determination in step S607 is that there is a judgment area i for which the pass flag is not set, it is determined that the user has failed (S609). Note that in this determination, for example, a weight may be assigned to each judgment area, and the weight may be increased for judgment areas closer to the operation target identification frame, and the user's operation may be evaluated based on this score.

[0061] (Other Examples) In the above-described access route determination process, multiple determination areas are set parallel to the operation object identification frame, but as shown in Fig. 14, multiple determination areas may be set according to the distance from the operation object part, rather than the operation object identification frame. In this case, each determination area forms a curved surface at approximately the same distance from the operation object part. In this case, the processing procedure is as follows: in step S602 of Fig. 13, the distance from the operation tool identification image to the operation object part is calculated.

[0062] (5) Statistical processing and training scenario change processing Next, the statistical processing of the training system 1 and the training scenario change processing based on the statistical results will be described with reference to FIG. The statistical analysis unit extracts the operation time for each operation target item (S301). Next, items whose variation (standard deviation) in the operation time for each operation target item is greater than a certain value are extracted (S302). Then, in the training scenario storage unit 62, a scenario adjustment flag (not shown) is assigned to the operation target item extracted in step S302 (S303).

[0063] Next, the scenario change editing unit 75 writes change information into the operation item to which the scenario adjustment flag has been assigned (S304). The change information may be, for example, changing the color of the operation target identification frame 21. The above process involves, for example, changing the training scenario to resolve the situation when a person successfully (passed) operates an operation item but takes a long time to perform the operation, or when there are people who can operate the operation quickly and others who take a long time (variation).

[0064] Furthermore, the above-described steps S301 to S303 are an example of performing statistical processing on a population of people who answered correctly to the operation, but it is also possible to use all users as a population and assign a scenario adjustment flag to items with a low rate of correct answers. In particular, the learning effect can be improved by changing the display of the operation object identification frame displayed in the learning mode and operation mode based on the scoring data (operation data) measured in the scoring mode.

[0065] As an example, FIG. 16 shows a processing procedure of the 3D-CG display control unit 71 in the operation mode. The 3D-CG display control unit 71 executes the following processing for all operation target parts in the training scenario (S401a, S401b). First, change information for the operation target item is extracted (S402). Then, it is determined whether or not there is change information (S403), and if there is change information, it displays the operation target identification frame 21 as a warning frame (S404). On the other hand, if there is no change information, it displays the operation target identification frame 21 as a normal frame (S405). Here, it is sufficient that the user can distinguish between the warning frame and the normal frame by the difference in display mode, such as color or blinking.

[0066] Incidentally, in the learning mode, the branching decision in step S406 is made by a timer. For example, the transition may be made depending on whether or not the display time for each operation item in the training scenario storage unit 62 has elapsed.

[0067] (Other Examples) Next, another embodiment of the statistical processing and training scenario change processing will be described with reference to Figures 17 and 18. This embodiment calculates the correlation of the evaluation results between operation target items and effectively displays an operation target identification frame or guidance during training.

[0068] In Fig. 17, when the statistical analysis unit 74 of the training system 1 is activated after the user's training has ended or at a preset cycle, it accesses the operation judgment storage unit 63 and extracts the evaluation results of each user. Then, based on the extracted data, it calculates the correlation between the evaluation results of each operation target part in the training scenario (S101). For example, if the evaluation of operation target part a is poor, the evaluation of operation target part b will also be poor. In this way, a correlation coefficient is calculated for the evaluation results of each operation target part. Then, a group number is assigned to a combination of operation target parts whose correlation coefficient is equal to or greater than a certain value (S102). At this time, multiple group numbers may be assigned to one operation target part. Then, the scenario change editing unit 75 writes the group number in association with the operation target body part in the training scenario storage unit 62 (S103).

[0069] (6) Training image display processing Next, the procedure for the training image display process will be explained, focusing on the processing by the 3D-CG control unit. When the 3D-CG control unit is activated by the start of training, it displays a 3D image of the training object on the 3D-CG display unit and performs the following processing for all operation target parts listed in the training scenario memory unit 62 (S201a, S201b). The 3D-CG control unit first extracts (S202) the warning display flag (not shown) for the operation target body part from the training scenario storage unit 62. This warning display flag is provided for each user and is reset beforehand at the time of the user's first training.

[0070] If the warning display flag is not set, a normal identification frame is displayed for the operation target portion (S204). If the warning display flag is set, a warning display identification frame is displayed for the operation target portion (S205). Here, the warning display is displayed, for example, in a frame of a different color from the normal display, to particularly call the user's attention.

[0071] The operation determination unit then determines whether the user's operation has been completed within a predetermined time (S206, S207), and if the operation has been completed, determines whether the operation result is pass or not (S208). If the result is pass, the processing from step S202 onwards is carried out for the next operation target part. On the other hand, if the result of the determination in step S208 is fail, the warning display flag for the operation target part in the training scenario storage unit 62 that is correlated with the operation target part, i.e., that belongs to the same group as the operation target part, is set (S209), and the processing proceeds to the next operation target part. In the above, steps S206 to S208 are executed as processing by the operation determination unit, and step S209 is executed as processing by the operation determination output unit.

[0072] With the above processing procedure, the user is warned when operating a certain operation target area that has a low rating and is correlated with the operation of other operation target areas, so the user can operate the area with particular care, thereby improving training efficiency.

[0073] As described above, according to this embodiment, the display / non-display of the operation target identification frame displayed superimposed on the training object is switched or the display mode of the operation target identification frame is changed depending on the training mode and the user's proficiency, and the user's movements, such as the route taken to approach the operation target portion of the training object, are evaluated in detail using a judgment area set according to the distance from the operation target identification frame or the operation target portion, thereby improving the effectiveness of training. Furthermore, since the training scenario is changed according to the evaluation results of the user's operation content, not only can the user's proficiency be efficiently improved, but it can also be expected to be effective in preventing recurrence when a human error occurs. [Explanation of symbols]

[0074] 1 Training System 10 Virtual Content Display Device (HMD) 11 3D-CG display section 12 Operation content display area 13 Operation coordinate detection unit 14 Operation content detection unit 50 Operating equipment 51 Operation detection unit 60 Training scenario / operation record judgment device (judgment device) 61 3D model storage section 62 Training scenario memory section 63 Operation judgment storage unit 71 3D-CG display control unit (three-dimensional display control unit) 72 Operation judgment section 73 Operation judgment output section 74 Statistical Analysis Department 75 Scenario Change Editorial Department

Claims

1. a three-dimensional display control unit that displays a training object in three dimensions in a virtual space and that displays a frame that identifies an operation target portion, which is a device that the user should operate, as the training scenario progresses; an operation detection unit that detects operation information including a user's operation position and operation direction; A training system comprising: The system has at least two modes: a learning mode in which the user does not operate the system but reproduces the operation procedure in a virtual space according to a training scenario, and a scoring mode in which the user operates the system according to the training scenario, records the operation details, and scores the operation details, and the user can select which mode to operate in; A training system characterized in that the three-dimensional display control unit displays the frame when the selected mode among the modes is a learning mode, and does not display the frame when the selected mode is a scoring mode.

2. An operation determination unit that calculates a user's proficiency level based on the operation target portion of the training object and the operation information, The training system according to claim 1, characterized in that the three-dimensional display control unit determines whether to display or hide the frame based on the user's proficiency calculated by the operation determination unit instead of the mode.

3. a storage unit that stores coordinates of the frame in association with each operation target portion; The training system described in claim 1 or 2, characterized in that the three-dimensional display control unit determines whether there is a change in the operation target area to be operated as the training scenario progresses, and if there is no change, displays a normal frame as the frame identifying the operation target area, while if there is a change, displays a warning frame different from the normal frame as the frame identifying the operation target area.

4. A training system as described in any one of claims 1 to 3, characterized in that when the operation detection unit detects the user's gaze during the scoring mode, the three-dimensional display control unit displays the frame by causing the gaze vector to intersect with a predetermined range of the operation target portion of the training object for a certain period of time or more.

5. A method for educating and training a user on how to operate a training object displayed in three dimensions in a virtual space, comprising: a three-dimensional display process for displaying a training object in a virtual space in three dimensions and displaying a frame identifying an operation target portion, which is a device to be operated by the user, as the training scenario progresses; an operation detection process for detecting operation information including a user's operation position and operation direction; Including, The system has at least two modes: a learning mode in which the user does not operate the system but reproduces the operation procedure in a virtual space according to a training scenario, and a scoring mode in which the user operates the system according to the training scenario, records the operation details, and scores the operation details, and the user can select which mode to operate in; A training method characterized in that the three-dimensional display processing displays the frame when the selected mode among the modes is a learning mode, and does not display the frame when the selected mode is a scoring mode.

6. A program for educating and training how to operate a training object displayed in three dimensions in a virtual space, a three-dimensional display process for displaying a training object in a virtual space in three dimensions and displaying a frame identifying an operation target portion, which is a device to be operated by the user, as the training scenario progresses; an operation detection process for detecting operation information including a user's operation position and operation direction; Run The system has at least two modes: a learning mode in which the user does not operate the system but reproduces the operation procedure in a virtual space according to a training scenario, and a scoring mode in which the user operates the system according to the training scenario, records the operation details, and scores the operation details, and the user can select which mode to operate in; The three-dimensional display processing is characterized in that, when a selected mode among the modes is a learning mode, the frame is displayed, and when the selected mode is a scoring mode, the frame is not displayed.

Images