Maintenance training support system, maintenance training support method, and program

Abstract

To support implementation of training related to maintenance of electronic equipment more efficiently and effectively in virtual reality.SOLUTION: A display device transmits motion information including motions of the head and hands of a user wearing the display device, to a maintenance training support apparatus. The maintenance training support apparatus is configured to: generate display information for displaying a three-dimensional virtual reality space in which maintenance-specific objects are disposed; identify motion of the user in the virtual reality space based on the motion information acquired from the display device; detect a composite gesture based on a hand gesture detected based on analyzing the motion information, and a point-of-view position of the user; and acknowledges, when detecting a predetermined composite gesture, that the user has performed pointing and calling on an object pointed by the point-of-view position and the hand gesture.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a maintenance training support system, a maintenance training support method, and a program. [Background technology]

[0002] In recent years, there has been an increase in video content, such as games, that utilizes VR (Virtual Reality) technology. VR is a virtual reality visual world created using, for example, 3DCG (Dimensional Computer Graphics) technology, and VR users, such as gamers, can have a variety of experiences within the virtual reality.

[0003] There are also vocational training contents that utilize the characteristics of VR. For example, Patent Document 1 discloses agricultural crop cultivation training in a VR space. Specifically, Patent Document 1 discloses a system for educating and training agricultural crop cultivation techniques, which includes a training scenario storage unit that stores work target areas for the crops displayed in a virtual space and work implements, which are tools used on the work target areas, in association with work procedures; a three-dimensional display control unit that displays the crops and work implements in three dimensions on a display device as the work procedures progress; a detection signal output unit that detects the position and movement of a user or an operating device and outputs a detection signal; an operation detection unit that changes the work implement displayed in the virtual space based on the detection signal; and an operation determination unit that determines the result of the operation based on the coordinates of the work target areas, the coordinates of the work implement, and the operation content in the virtual space. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2022-149636 Summary of the Invention [Problem to be solved by the invention]

[0005] In VR content for vocational training, display information for the virtual reality world that changes in response to user actions is usually generated based on information defining the objects and their movements specific to the training subject, and information acquired from the VR goggles and controller. Therefore, for example, when using VR content to train in maintenance work for electronic devices, information defining the objects specific to the maintenance work is required.

[0006] In addition, it may be desirable to display information in the VR space during training to assist the user in performing maintenance work, and even when displaying such information, predefined information is required.

[0007] The technology in Patent Document 1 is for providing education and training on agricultural crop cultivation in a VR space, and therefore does not take into consideration objects specific to maintenance work such as electronic devices and their movements.

[0008] The present invention has been made in view of the above points, and aims to support the implementation of training on maintenance work of electronic devices more efficiently and effectively in virtual reality. [Means for solving the problem]

[0009] The present application includes a number of means for solving at least part of the above problems, examples of which are as follows.

[0010] In order to solve the above-mentioned problems, one aspect of the present invention provides a maintenance training support system having a maintenance training support device and a display device, wherein the display device transmits motion information, including head and hand movements, of a user wearing the display device to the maintenance training support device, and the maintenance training support device comprises: a display information generation unit that generates display information for displaying a three-dimensional virtual reality space in which objects specific to maintenance work are arranged; a motion identification unit that identifies the user's motion in the virtual reality space based on the motion information acquired from the display device; and a gesture detection unit that detects a composite gesture based on a hand gesture detected based on an analysis of the motion information and a position of the user's viewpoint; and when the gesture detection unit detects a predetermined composite gesture, the motion identification unit recognizes the performance of pointing and calling out to the object pointed to by the position of the viewpoint and the hand gesture.

[0011] In addition, in the above-described maintenance training support system, when the object is pointed to by the viewpoint position and the hand gesture, the display information generation unit may display a gauge indicating a time until the pointing and calling is confirmed.

[0012] In the maintenance training support system, when the action identifying unit recognizes that the pointing and calling has been performed, the action identifying unit may cause the display device to output display information or a sound indicating the recognition.

[0013] Furthermore, in the above-described maintenance training support system, the display information generation unit may display a viewpoint pointer of a predetermined shape indicating the position of the viewpoint on the display information, and when the composite gesture is detected by the gesture detection unit, may display an icon shaped like an eye at the position of the viewpoint instead of the viewpoint pointer.

[0014] Furthermore, in the above-described maintenance training support system, the gesture detection unit may detect a composite gesture based on the position of the user's viewpoint, the hand gesture, and the user's speech, and when the gesture detection unit detects the composite gesture based on the position of the viewpoint, the hand gesture, and the speech, the action identification unit may recognize the performance of the pointing and calling.

[0015] In addition, in the above-described maintenance training support system, the action identification unit may recognize the performance of the pointing and calling when the position of the viewpoint and the object pointed to by the hand gesture match the content of the spoken voice.

[0016] Furthermore, in the above-described maintenance training support system, when the action identification unit recognizes that the pointing and calling has been performed, it may cause the display device to output display information or sound indicating the recognition, and when the position of the viewpoint and the object pointed to by the hand gesture do not match the content of the spoken voice, it may not recognize that the pointing and calling has been performed, and cause the display device to output display information or sound indicating the reason for this.

[0017] Furthermore, in the above-described maintenance training support system, when a pointing hand gesture is detected by the gesture detection unit, the display information generation unit may generate the display information showing a raycast, which is a beam-like line of light, being output from near the tip of the finger making the pointing hand gesture.

[0018] Furthermore, in the above-described maintenance training support system, the display information generation unit may generate the display information in which the raycast is drawn so that the output starting point of the raycast is near the base of the palm of the hand making the pointing hand gesture and passes through the output starting point of the raycast, which is set near the position of the fingertip.

[0019] In addition, in the above-mentioned maintenance training support system, the display information generation unit may set multiple data points within a certain period of time, average the changes in the direction of the raycast obtained at each data point, and generate the display information that depicts the raycast.

[0020] Furthermore, in the above-described maintenance training support system, when the user uses the raycast to point to a location in the virtual reality space that is more than a predetermined distance away, the display information generation unit may set a number of data points greater than the plurality of data points, average the change in direction of the raycast obtained at each data point, and generate the display information that depicts the raycast.

[0021] Furthermore, in the above-described maintenance training support system, when the user's viewpoint overlaps with a predetermined object in the virtual reality space, the display information generation unit may generate the display information fixed to the angle of view at that time, and may continue to generate the display information based on that angle of view until the user's viewpoint moves out of the angle of view.

[0022] Furthermore, in the above-described maintenance training support system, when the head of the user wearing the display device is tilted upward or downward by a predetermined angle or more, the display information generation unit may move the user's viewpoint upward or downward from a predetermined position and generate the display information so that the viewpoint is positioned approximately in the center of the angle of view.

[0023] Furthermore, a maintenance training support method according to another aspect of the present invention is a maintenance training support method performed by a maintenance training support device, in which the maintenance training support device performs the following steps: a display information generation step of generating display information for displaying a three-dimensional virtual reality space in which objects specific to maintenance work are arranged; a motion identification step of identifying a user's motion within the virtual reality space based on motion information acquired from a display device worn by the user and capable of detecting the user's head and hand movements; and a gesture detection step of detecting a composite gesture based on a hand gesture detected based on an analysis of the motion information and the user's viewpoint position; and when a predetermined composite gesture is detected in the gesture detection step, the motion identification step recognizes the performance of pointing and calling out to the object pointed to by the viewpoint position and the hand gesture.

[0024] Furthermore, a program according to another aspect of the present invention is a program that causes a computer to function as a maintenance training support device, and causes the computer to function as: a display information generation unit that generates display information for displaying a three-dimensional virtual reality space in which objects specific to maintenance work are arranged; a motion identification unit that identifies a user's motion within the virtual reality space based on motion information acquired from a display device worn by the user and capable of detecting the user's head and hand movements; and a gesture detection unit that detects a composite gesture based on a hand gesture detected based on an analysis of the motion information and the user's viewpoint position; and when the gesture detection unit detects a predetermined composite gesture, the motion identification unit recognizes the performance of pointing and calling out to the object pointed to by the viewpoint position and the hand gesture. [Effects of the Invention]

[0025] According to the present invention, it is possible to support the implementation of training on maintenance work of electronic devices more efficiently and effectively in virtual reality.

[0026] Problems, configurations, effects, and the like other than those described above will become clear from the following description of the embodiments. [Brief explanation of the drawings]

[0027] [Figure 1] 1 is a diagram illustrating an example of a schematic configuration of a maintenance training support system. [Figure 2] FIG. 2 is a block diagram showing an example of a functional configuration of a maintenance training support device. [Figure 3] FIG. 10 is a diagram illustrating an example of object management information. [Figure 4] FIG. 10 is a diagram illustrating an example of interference information. [Figure 5] FIG. 10 is a diagram illustrating an example of pair information. [Figure 6] FIG. 10 is a diagram showing an example of location information. [Figure 7] FIG. 10 is a diagram illustrating an example of gesture definition information. [Figure 8] FIG. 10 is a diagram showing an example of VR display information. [Figure 9] FIG. 10 is a flow diagram showing an example of a maintenance training support process. [Figure 10] FIG. 10 is a diagram for explaining a maintenance training support process related to pointing and calling. [Figure 11] FIG. 10 is a diagram for explaining a maintenance training support process related to pointing and calling. [Figure 12] FIG. 10 is a diagram showing an example of a viewpoint pointer according to a first modified example. [Figure 13] FIG. 10 is a diagram for explaining certification of pointing and calling in accordance with a second modified example. [Figure 14] FIG. 10 is a diagram illustrating control related to raycast output. [Figure 15] 10A and 10B are diagrams for explaining a flow from temporary fixing of the angle of view to its release. [Figure 16] 10A and 10B are diagrams for explaining control of the angle of view of VR display information according to the tilt of the head in the up and down direction. [Figure 17] FIG. 2 is a diagram illustrating an example of a hardware configuration of a maintenance training support device. DETAILED DESCRIPTION OF THE INVENTION

[0028] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0029] <General Configuration of Maintenance Training Support System 1000> 1 is a diagram showing an example of a schematic configuration of a maintenance training support system 1000 according to this embodiment. As shown in the figure, the maintenance training support system 1000 includes a maintenance training support device 100, a computer 150, VR goggles 160 and a display device 180, and a computer 151 and a display device 181. It is assumed that the computer 150, the VR goggles 160 and the display device 180, and the computer 151 and the display device 181 will be used by different users of the maintenance training support system 1000, respectively.

[0030] The VR goggles 160 are connected to the computer 150 via wireless communication using a predetermined wireless communication standard (e.g., Bluetooth (registered trademark)) or via a wired cable so that they can communicate with each other, and the computers 150 and 151 are connected to the maintenance training support device 100 via the network N so that they can communicate with each other.

[0031] The network N may be, for example, the Internet, an intranet, a wide area network (WAN), a mobile phone network, or a communication network that combines these. The network N may also be a virtual private network (VPN) on a wireless communication network such as a mobile phone communication network.

[0032] In the maintenance training support system 1000, a computer 150 acquires VR display information of the three-dimensional image generated by the maintenance training support device 100, performs 3D rendering, and outputs the rendered VR display information to a VR goggle 160, thereby displaying a virtual reality space for training in maintenance work on the VR goggle 160.

[0033] The maintenance training assistance device 100 may generate display information corresponding to various types of augmented reality (so-called XR: Cross Reality), such as AR (Augmented Reality) and MR (Mixed Reality). That is, the maintenance training assistance device 100 can be applied to XR, which includes VR, AR, and MR. In the present embodiment, the following description will be given taking VR as an example.

[0034] <VRゴーグル160> The VR goggles 160 are a display device (head-mounted display) that displays VR display information of three-dimensional images created by, for example, 3DCG (Three-Dimensional Computer Graphics). The VR goggles 160 display the VR display information on a display mounted on the VR goggles 160 so that objects in the VR space appear three-dimensional by utilizing the parallax between the user's eyes.

[0035] The VR goggles 160 also have a sensor that detects the movement of the user's head and a plurality of cameras that are used to detect the movement of the user's hands, and perform head tracking and hand tracking using the sensors and cameras.

[0036] Specifically, the VR goggles 160 perform head tracking and transmit tracking data indicating the movement of the user's head as operation information to the maintenance training assistance device 100. In addition, the VR goggles 160 perform hand tracking and transmit tracking data indicating the movement of the user's hand (hand movement direction, movement amount, fingertip movement, predetermined hand gesture, etc.) as operation information to the maintenance training assistance device 100.

[0037] There are various types of VR goggles 160, such as a standalone type that displays a VR space by rendering VR display information by itself, or a type that is used by inserting a smartphone into it. In this embodiment, the following explanation will be given using an example of a type that displays a VR space on a display using VR display information rendered by a computer 150 that is connected so that communication is possible.

[0038] By using the VR goggles 160, the user can grasp an object, move the grasped object, or perform various other maintenance work-related actions using hand gestures. In addition, the user can practice pointing and calling, an action specific to maintenance work, for a specific object placed in the VR space, using a composite gesture based on the viewpoint position and hand gesture (hereinafter sometimes referred to as a "composite gesture"). Details of the maintenance training support process related to pointing and calling will be described later.

[0039] <Calculator 150, 151> The computers 150 and 151 are devices that display information about the VR space in which maintenance work training is performed on the VR goggles 160 or the display devices 180 and 181, and are realized by, for example, a PC (Personal Computer). Note that the computers 150 and 151 may also be devices such as tablet terminals or smartphones as long as they satisfy the required processing specifications. The computer 150 may also be, for example, an information processing device built into the standalone VR goggles 160.

[0040] The computer 150 performs 3D rendering when it acquires VR display information, which is a three-dimensional image, from the maintenance training assistance device 100. Specifically, the computer 150 performs real-time rendering to generate a three-dimensional image in real time using the VR display information acquired from the maintenance training assistance device 100. The computer 150 also outputs the VR display information, which is a three-dimensional image generated by the 3D rendering, to the VR goggles 160.

[0041] Furthermore, when the computers 150 and 151 display the user's training status in the VR space on the display devices 180 and 181, they convert the rendered VR display information into display information for display (two-dimensional video) and output this to the display devices 180 and 181.

[0042] The computer 150 also performs processing to transmit the operation information acquired from the VR goggles 160 to the maintenance training assistance device 100 .

[0043] <Display device 180, 181> The display devices 180 and 181 are display devices such as displays that display two-dimensional images. The display devices 180 and 181 display display information acquired from the computers 150 and 151 or the maintenance training support device 100.

[0044] <Maintenance training support device 100> The maintenance training support device 100 is a device that supports a user's maintenance training in a VR space, and is realized by, for example, a server computer, a cloud server, or a personal computer. Specifically, the maintenance training support device 100 generates VR display information of a three-dimensional image in which a user performs maintenance work training, and displays the information via a computer 150 on VR goggles 160 worn by the user performing the training.

[0045] In addition, when the maintenance training support device 100 acquires user movement information from the VR goggles 160, it generates VR display information that depicts the appearance of the VR space that changes in conjunction with the user's movements, and displays it on the VR goggles 160 via the computer 150.

[0046] <<Details of the maintenance training support device 100>> 2 is a block diagram showing an example of the functional configuration of the maintenance training support device 100. As shown in the figure, the maintenance training support device 100 has a processing unit 110, a storage unit 120, and a communication unit 130.

[0047] The processing unit 110 is a functional unit that performs various processes executed by the maintenance training assistance device 100. Specifically, the processing unit 110 has an action identification unit 111, a gesture detection unit 112, and a display information generation unit 113 as functional units for executing each process.

[0048] The motion identification unit 111 is a functional unit that analyzes motion information (tracking data) to identify the motion of the user's head and hands. The motion identification unit 111 also uses various information related to objects in the storage unit 120 to identify (calculate) the motion of objects, such as interference between objects, contact with the user, or movement accompanying the user's motion.

[0049] Specifically, the action identification unit 111 identifies the user's actions and object movements in the VR space using action information and various information in the memory unit 120 that manages the object positions, physical relationships such as interference, and linked objects based on the global coordinate system set in the VR space and the local coordinate system set for each object.

[0050] The method for identifying the user's actions and the object's movements in the VR space is not particularly limited, and any known VR technology may be used.

[0051] The gesture detection unit 112 is a functional unit that analyzes operation information (hand tracking data) and detects a predetermined hand gesture when the user is making the gesture. The gesture detection unit 112 also detects a composite gesture based on a combination of the user's viewpoint position and hand gesture. Specifically, the gesture detection unit 112 identifies the user's viewpoint position based on information acquired from the display information generation unit 113, which sets the angle of view of VR display information using the user's viewpoint position. The gesture detection unit 112 also detects a composite gesture based on a combination of the viewpoint position and hand gesture based on gesture definition information 126.

[0052] The display information generating unit 113 is a functional unit that generates VR display information. Specifically, the display information generating unit 113 generates VR display information that depicts, in three-dimensional video, the state of the VR space that changes in accordance with the user's movement, using the user's movement, object movement, position, and posture identified by the movement identifying unit 111, and various information in the storage unit 120.

[0053] More specifically, the display information generating unit 113 determines the position of the viewpoint at a predetermined distance (e.g., 3 m) from the user, in front of the user's head identified by the action identifying unit 111, and generates VR display information so that the viewpoint is located approximately in the center of the angle of view. At this time, the display information generating unit 113 may display a viewpoint pointer (e.g., a circle and the word "Self") indicating the position of the viewpoint approximately in the center of the VR display information.

[0054] Moreover, the display information generating unit 113 transmits the generated VR display information to the computer 150 via the communication unit 130.

[0055] Furthermore, the display information generating unit 113 displays the user's hands drawn in 3DCG in the VR display information, reflecting the user's hand movements identified by the action identifying unit 111. That is, the VR display information displays the user's hands performing maintenance work or making hand gestures such as pointing in the VR space.

[0056] Furthermore, the display information generation unit 113 generates VR display information according to the user's hand gesture and the motion corresponding to the composite gesture. For example, when the gesture detection unit 112 detects a hand gesture of pinching a target object with the thumb, index finger, and middle finger, the display information generation unit 113 uses predetermined information (gesture definition information described later) to identify the corresponding motion (in this case, the motion of grasping the selected object), and generates VR display information according to the identified motion (in this case, drawing information showing how the hand and the grasped object move (move) together).

[0057] Furthermore, for example, when gesture detection unit 112 detects a composite gesture in which the user hits an object at the viewpoint with a raycast output by a pointing hand gesture, display information generation unit 113 identifies a corresponding action (in this case, confirmation of pointing and calling) using predetermined information (gesture definition information, described below). Display information generation unit 113 also generates VR display information corresponding to the identified action (in this case, a series of drawing information, such as displaying a gauge indicating the time until confirmation of pointing and calling, filling the gauge, and displaying predetermined output information indicating confirmation of pointing and calling when the gauge is full). The content of the VR display information corresponding to the identified action, generated by display information generation unit 113, is stored in advance in storage unit 120 as information associated with each action (not shown).

[0058] Next, the storage unit 120 will be described. The storage unit 120 is a functional unit that stores various types of information. Specifically, the storage unit 120 has an object management information DB (Database) 121 in which objects specific to maintenance work and their operations are defined, interference information 122 in which information on objects having physical relationships such as interference is registered, pair information 123 in which objects having linked operations are registered, position information 124 in which the latest positions of each object and user in the VR space are stored, scenario information 125 in which procedures and work instructions for maintenance work training are registered, and gesture definition information 126 in which operations corresponding to hand gestures and composite gestures are defined.

[0059] 3 is a diagram showing an example of object management information stored in the object management information DB 121. The object information is information that defines the type, position, and behavior of an object. Specifically, the object management information has records in which items such as identification information, type, coordinate position, and special object flag are associated with each other.

[0060] The identification information is information that uniquely identifies an object. The type is information that identifies the type of object, and there are various types such as racks, which are objects specific to maintenance work, electronic devices, rack doors, cables, power cords, and warning labels.

[0061] The coordinate position is coordinate information that indicates the position of an object in the VR space. Note that the coordinate position includes both the position of the object specified in the global coordinate system set in the VR space and the position of the object specified in the local coordinate system set for each object.

[0062] The special object flag is information that indicates objects that can be moved during maintenance work training (for example, electronic devices, racks, rack doors, cables, power cords, warning labels, etc.). Specifically, an object with a "1" registered in the special object flag corresponds to a special object. Note that objects that are not subject to movement during maintenance work training have a "0" registered in the flag.

[0063] In addition to these items, the object management information also includes information defining operational characteristics (for example, in the case of a rack door, opening and closing around one of the X to Z axes in the local coordinate system) (not shown).

[0064] FIG. 4 is a diagram showing an example of the interference information 122. The interference information 122 is information indicating the state of interference between objects. Specifically, the interference information 122 has records in which items such as identification information, interfering objects, and interfering portions are associated with each other. The identification information is information that uniquely identifies an object. The interfering object is an object that is colliding when viewed from the object identified by the identification information, and the identification information of the object is registered. The interfering portion is information that identifies a portion that is interfering (in contact) with the interfering object, and is, for example, information identified using a local coordinate system.

[0065] 5 is a diagram showing an example of pair information 123. Pair information 123 is information in which objects in a linked relationship are registered. Specifically, pair information 123 registers information about a parent object and a child object that moves in conjunction with the parent object. More specifically, pair information 123 has records in which items such as identification information of the parent object, identification information of the child object, and coordinate positions are associated with each other.

[0066] The identification information of the parent object and the child object is information that uniquely identifies the parent object, such as a rack or a rack door (for example, the object on which the child object is pasted or attached), and the child object, such as an electronic device or a warning label (for example, the object on which the child object is pasted or attached). The coordinate position is coordinate information that specifies the position of the child object relative to the parent object, and is information specified by, for example, a local coordinate system. The type of object that corresponds to the parent object and the type of object that corresponds to the child object are defined by predetermined information (not shown), and this information may be stored in advance in the storage unit 120. Also, for example, a special object may be treated as a child object, and a non-special object may be treated as a parent object.

[0067] Whether linked objects are parent objects or child objects can be determined based on priority information (not shown) that is preset for each object according to a predetermined viewpoint (e.g., difficulty of operation).

[0068] FIG. 6 is a diagram showing an example of the position information 124. The position information 124 is information indicating the position of an object after it has moved due to maintenance work training by the user, i.e., the latest position in the VR space. Specifically, the position information 124 has records in which items such as identification information and coordinate positions are associated. Note that the identification information is information that uniquely identifies an object. Also, the coordinate position is coordinate information indicating the latest position of the object, and is information specified by a global coordinate system and a local coordinate system.

[0069] It is assumed that the position information 124 also includes coordinate information that identifies the position of the user in the VR space and the postures of the user and objects.

[0070] The scenario information 125 is information in which maintenance work training procedures, work instructions, etc. are registered. Specifically, the scenario information 125 registers scenarios corresponding to multiple types of training objectives, such as replacing electronic devices or changing mount positions, as well as information such as objects, object placement, maintenance work procedures, and work instructions corresponding to each scenario.

[0071] The work instructions are information indicating work instructions that follow maintenance work procedures, such as "Please identify the rack to be worked on by pointing and calling" or "Please attach a work-in-progress label (warning label) to the rack to be worked on."

[0072] 7 is a diagram showing an example of gesture definition information 126. Gesture definition information 126 is information that defines predetermined hand gestures (including combinations of hand gestures) and composite gestures (gestures that are any combination of hand gestures, gaze, and voice), as well as actions corresponding to each gesture. Specifically, gesture definition information 126 has records in which gestures are associated with actions.

[0073] Note that a gesture is information indicating various hand gestures or composite gestures that serve as execution triggers for corresponding operations. An operation is information on an operation to be executed when a corresponding gesture is detected. Note that there are various types of gestures and operations, and FIG. 7 shows an example thereof.

[0074] Note that each piece of information in the storage unit 120 is an example, and the type and content of the information and the association relationship of the information are not limited to the above example.

[0075] Next, the communication unit 130 will be described. The communication unit 130 is a functional unit that performs information communication with an external device. Specifically, the communication unit 130 performs information communication with the computers 150 and 151 and acquires operation information output from the VR goggles 160. Further, the communication unit 130 transmits the VR display information generated by the display information generation unit 113 to the computers 150 and 151.

[0076] <An example of VR display information> FIG. 8 is a diagram showing an example of VR display information displayed on the VR goggles 160. As shown in the drawing, the VR display information is display information that depicts a VR space in which the rack 200 and the electronic device 201 are arranged from the first-person perspective of the user. Further, the VR display information displays a caution label 202, which is an object specific to maintenance work, a work instruction window 203, the user's hand 204 drawn in 3DCG, and a viewpoint pointer 205 indicating the user's viewpoint position.

[0077] Note that each object such as the rack 200 and the work instruction window 203 is drawn based on various information stored in the storage unit 120. Further, the user's hand 204 and the viewpoint pointer 205 indicating the viewpoint position are drawn based on operation information acquired from the VR goggles 160.

[0078] By wearing VR goggles 160 displaying such VR display information, a user can use hand gestures and complex gestures to perform various types of maintenance training in the VR space, such as replacing broken electronic equipment.

[0079] <Maintenance training support processing> 9 is a flow diagram showing an example of the maintenance training support process. This process starts, for example, when the processing unit 110 receives a processing start instruction from the user. At this time, the user selects a predetermined scenario in which the user wants to train from the scenario information 125 and starts the process.

[0080] When the process starts, the gesture detection unit 112 determines whether or not a predetermined gesture has been detected based on an analysis of the motion information (step S10). Specifically, the gesture detection unit 112 determines whether or not a hand gesture or a composite gesture registered in the gesture definition information 126 has been detected.

[0081] If it is determined that a predetermined gesture has not been detected (No in step S10), the gesture detection unit 112 performs the process of step S10 again. On the other hand, if it is determined that a predetermined gesture has been detected (Yes in step S10), the gesture detection unit 112 proceeds to step S20.

[0082] Note that gesture detection unit 112 may detect multiple gestures simultaneously. For example, when different hand gestures are made with the right and left hands and each is associated with a different action, gesture detection unit 112 detects each hand gesture at the same time. Alternatively, when different hand gestures are made with the right and left hands and, for example, a hand gesture with one hand and a composite gesture made by combining the other hand with a gaze and / or voice are indicated, gesture detection unit 112 detects the hand gesture and the composite gesture at the same time.

[0083] In step S20, the gesture detection unit 112 identifies an action corresponding to the detected gesture based on the gesture definition information 126, and the process proceeds to step S30.

[0084] In step S30, the display information generation unit 113 generates VR display information corresponding to the identified action, and then the process proceeds to step S40.

[0085] In step S40, the display information generation unit 113 transmits the generated VR display information to the computer 150 via the communication unit 130, thereby causing the VR goggles 160 to display the VR display information.

[0086] Furthermore, after transmitting the VR display information to the computers 150 and 151, the display information generating unit 113 shifts the process to step S10, and repeatedly executes the processes of steps S10 to S40.

[0087] When the position or posture of the object or the position of the user himself / herself changes in response to the user's action, the action identification unit 111 updates the information by registering the latest information in the position information 124 .

[0088] Next, the above-mentioned maintenance training support process will be explained using a specific example of pointing and calling.

[0089] 10 and 11 are diagrams for explaining the maintenance training support process for pointing and calling. The user points the viewpoint pointer 205 at an object to be pointed and called (for example, a warning label 202, a rack 200, or an electronic device 201). Specifically, the user points the viewpoint pointer 205 at the object to be pointed and called by moving their head so that the object to be pointed and called is located approximately in the center of the angle of view of the VR display information displayed on the VR goggles 160 (FIG. 10).

[0090] The user also makes a "pointing" hand gesture (a gesture in which the palm of the hand is turned sideways and the index finger is extended to point at an object) with one hand 204. At this time, the gesture detection unit 112 detects that the user is making a pointing hand gesture based on an analysis of the motion information (step S10).

[0091] Next, the gesture detection unit 112 identifies an action corresponding to the pointing hand gesture from the gesture definition information 126 (step S20). Specifically, the gesture detection unit 112 identifies an action called "raycast output" that corresponds to the hand gesture. Note that a raycast is a beam-like line of light, and outputting a raycast means an action of outputting a beam-like line of light from near the tip of the finger (e.g., the index finger) making the hand gesture in the direction pointed by the hand gesture.

[0092] When the gesture detection unit 112 identifies an action corresponding to the hand gesture, the display information generation unit 113 generates VR display information corresponding to the action (step S30). Specifically, the display information generation unit 113 displays a raycast output start point 220 near the tip of the finger making the pointing hand gesture, generates VR display information in which a raycast 230 is output from the output start point 220 (step S30), and displays the VR display information on the VR goggles 160 (step S40).

[0093] Next, while keeping viewpoint pointer 205 pointed at the target object, the user uses a pointing hand gesture to point to the target object with a raycast 230 output from near the fingertip (FIG. 11). At this time, gesture detection unit 112 detects a composite gesture of "directing a raycast at an object at the viewpoint position with a pointing gesture" based on gesture definition information 126 (step S10). Gesture detection unit 112 also identifies "perform pointing and calling" as an action corresponding to this composite gesture from gesture definition information 126 (step S20).

[0094] Next, the display information generation unit 113 generates VR display information corresponding to the identified action. In this case, the display information generation unit 113 generates VR display information in which a gauge 240 of a predetermined shape (in this example, a circular gauge) is displayed at the location pointed to by the user using the raycast 230 (step S30), and displays the VR display information on the VR goggles 160 (step S40). Note that while an enlarged gauge 241 is shown in FIG. 11, this is to clearly show an example of the gauge 240, and is not displayed in the VR space visually recognized by the user through the VR goggles 160.

[0095] In addition, the display information generation unit 113 generates VR display information corresponding to the identified action, in which the gauge 240 continues to increase while the state of the composite gesture is maintained (step S30), and displays it on the VR goggles 160 (step S40).

[0096] Next, action identification unit 111 recognizes that pointing and calling has been performed when gauge 240 becomes full. Specifically, when the state of the detected composite gesture continues (is maintained) for a predetermined time (for example, any time between 1 second and 3 seconds), the drawing of gauge 240 by display information generation unit 113 becomes full, and action identification unit 111 recognizes that pointing and calling has been performed at this time. Furthermore, when the performance of pointing and calling has been recognized, display information generation unit 113 ends the display of gauge 240 (proceeding from step S40 to step S10).

[0097] In order to indicate that the pointing and calling action has been recognized, the action identification unit 111 may include a predetermined sound effect (for example, "ping," "pong," or a predetermined beep) in the VR display information and transmit it to the VR goggles 160, so that the sound effect is output from a speaker included in the VR goggles 160. The output of such a sound effect allows the user to recognize that the pointing and calling action has been correctly performed.

[0098] In addition to outputting sound effects, the action identification unit 111 may also indicate to the user that the pointing and calling has been approved by displaying text information (e.g., "Pointing and calling OK") or other display information (e.g., by flashing the VR space for a moment) in the VR space via the display information generation unit 113.

[0099] The composite gesture related to pointing and calling is preferably the above example that reproduces the actual pointing and calling movement, but is not limited to this, and any of a variety of hand gestures or composite gestures can be set.

[0100] By executing such maintenance training support processing, the maintenance training support device can support maintenance training related to pointing and calling actions by a user. For example, if maintenance training in a VR space is conducted by a user holding a controller, it is difficult for the user to faithfully reproduce actual pointing and calling actions. In contrast, the maintenance training support device according to this embodiment recognizes the execution of pointing and calling based on the user's viewpoint and pointing gestures, allowing the user to faithfully reproduce the pointing and calling actions performed in actual maintenance work sites, thereby further enriching the content of the maintenance training. As a result, the maintenance training support device can more efficiently and effectively support the implementation of training related to electronic device maintenance work.

[0101] <First Modification> Note that various modifications of the above-described embodiment can be implemented in the maintenance training assistance device 100. For example, the maintenance training assistance device 100 according to a first modification changes the viewpoint pointer 205, which is usually drawn as a circle, to an eye icon.

[0102] FIG. 12 is a diagram showing an example of a viewpoint pointer according to a first modified example. As shown in the figure, in this modified example, when the gesture detection unit 112 detects a composite gesture of "directing a raycast at an object at the viewpoint position using a pointing gesture," the display information generation unit 113 draws an "eye" icon 215, generates VR display information in which the viewpoint pointer 205, which had been drawn as a circle, is displayed at the viewpoint position, and displays the generated VR display information on the VR goggles 160. That is, the viewpoint pointer 205 is switched from a circle to the "eye" icon 215 at the same time that the gauge 240 is displayed. Note that while FIG. 12 shows an enlarged "eye" icon 216, this is for the purpose of making the icon 215 easier to see, and is not displayed in the VR space viewed by the user through the VR goggles 160.

[0103] According to such a maintenance training assistance device 100, when pointing and calling, the user's viewpoint position can be displayed in an intuitive and easy-to-understand manner, and a more realistic sense of work can be given to the user.

[0104] <Second Modification> Furthermore, the maintenance training assistance device 100 according to the second modification certifies that the user has performed pointing and calling based on a composite gesture that combines the user's viewpoint, hand gestures, and voice in the above-described embodiment. Specifically, when the user focuses their viewpoint on the target object to be pointed and called, makes a pointing gesture to hit the target object with the raycast 230, and yells "(type or name of target object) Good!" when the gauge 240 is full, the maintenance training assistance device 100 certifies that the user has performed pointing and calling.

[0105] In this case, the premise is that the gesture definition information 126 should correspond to the action of "certifying the pointing and calling out" such as a gesture of "casting a ray at an object at the viewpoint position with a pointing gesture and calling out '(type or name of object) Good!'".

[0106] Furthermore, the object management information only needs to have the name of each object registered in advance.

[0107] The VR goggles 160 are also equipped with a microphone that collects the user's voice, and when the VR goggles 160 acquire the user's call (voice), they transmit the voice to the maintenance training assistance device 100 as operation information.

[0108] FIG. 13 is a diagram for explaining the certification of the pointing and calling in accordance with the second modified example. As shown in the figure, when the action identification unit 111 detects a call such as "(type or name of target object) Good!" at the timing when the gauge 240 is full, the action identification unit 111 performs a determination process for certification of the pointing and calling. Specifically, the action identification unit 111 uses object management information to determine whether the type or name of the object targeted by the pointing and calling included in the call by the user matches the type or name of the object on which the user's viewpoint pointer 205 is placed or the object hit by the raycast 230. If it is determined that they match, the action identification unit 111 certifies that the user has performed the pointing and calling.

[0109] Note that if the type or name of the target object included in the call does not match the object on which viewpoint pointer 205 is placed or the object hit by raycast 230, action identification unit 111 will not approve the performance of pointing and calling. In this case, action identification unit 111 may display, via display information generation unit 113, text information indicating the reason for not approving the performance of pointing and calling (for example, "Pointing and calling not allowed") or other display information (for example, flashing the VR space multiple times) in the VR space.

[0110] Alternatively, the action identification unit 111 may include a predetermined sound effect (for example, a beep such as a buzzer indicating an error) in the VR display information and transmit it to the VR goggles 160, so that the sound effect is output from a speaker included in the VR goggles 160. By outputting such a display or sound effect, the user can recognize that the object being pointed at and called does not match the type or name of the object included in the call.

[0111] The maintenance training support device 100 has various features in addition to those described in the above embodiment. Each of these features will be described below.

[0112] <Raycast output control> 14 is a diagram for explaining control related to the output of a raycast. As described above, a raycast 230 is a beam-like line of light output from near the fingertip when a user makes a pointing hand gesture.

[0113] The maintenance training assistance device 100 controls so that a raycast is output from the base of the user's palm (near the wrist) (hereinafter sometimes referred to as the "raycast output origin") 250. Specifically, the gesture detection unit 112, which has detected the pointing hand gesture, analyzes the motion information to identify the position of the base of the user's palm 250. Furthermore, based on the analysis of the motion information by the gesture detection unit 112, the position of the fingertip (index finger) 221 making the pointing hand gesture is identified.

[0114] The display information generation unit 113 uses this identified information to draw a raycast 230. Specifically, the display information generation unit 113 displays a spherical raycast output start point 220 near the identified position of the fingertip 221. The display information generation unit 113 also draws the raycast 230 so that it starts from the identified position of the base of the palm 250 and passes through the raycast output start point 220.

[0115] The display information generation unit 113 generates drawing information that makes the raycast 230 appear to be output from the raycast output starting point 220 by hiding the raycast 230 between the base of the palm 250 and the output starting point 220.

[0116] This process allows the maintenance training assistance device to perform raycast output control that is easy for the user to control. For example, if the raycast output start point is set to the fingertip, the fingertip may bend slightly, making it difficult for the user to determine the direction of the raycast or hit the target object. Therefore, the maintenance training assistance device sets the raycast start point at the base of the user's palm, allowing for stable raycast output control that is easy for the user to control.

[0117] Such raycast control is useful, for example, when a user is training in pointing and calling, and is pointing to a target object.

[0118] <Controlling the amount of movement of raycast> The maintenance training assistance device 100 calculates the average value of multiple data points over a certain period of time to control the amount of movement of the raycast 230. Specifically, the display information generation unit 113 sets multiple data points (e.g., every 0.1 seconds) over a certain period of time (e.g., 0.5 seconds) at the raycast output origin 250, the raycast output start point (near the fingertip) 220, or both, and averages the change in the direction of the raycast 230 acquired at each data point.

[0119] More specifically, for example, if the orientation of the raycast 230 is defined by the x, y, and z axes, the display information generation unit 113 calculates the average value of the amount of change in the x-axis, the amount of change in the y-axis, and the amount of change in the z-axis acquired at each data point.The display information generation unit 113 then obtains the calculated values ​​for each axis as the change in the orientation of the raycast 230 over a certain period of time, and reflects this in the rendering information of the raycast 230.

[0120] Note that the number of data points and the averaging method are not particularly limited, and any well-known method may be adopted as long as it can stably display the ray cast 230 without linearly reflecting the movement of the finger.

[0121] Through such processing, the maintenance training support device can stabilize the unstable movement of the ray cast due to the reflection of minute movements of the user's hand, and perform output control of the ray cast that is easier for the user to control.

[0122] Such control of the ray cast is useful, for example, when the user points to a target object during the operation training of finger-pointing designation.

[0123] Note that when the maintenance training support device 100 ray-casts and indicates a location far from the user's own location within the VR space, that is, a location more than a predetermined distance (for example, 7 m or more) ahead of the user's position, the ray cast 230 may move more slowly than when indicating a nearby location (a location less than the predetermined distance).

[0124] Specifically, the display information generation unit 113 provides more data points (for example, every 0.05 seconds) than the data points (for example, every 0.1 seconds) when a nearby location is indicated within a certain period (for example, 0.5 seconds), and averages the changes in the direction of the ray cast 230 acquired at each data point.

[0125] Through such processing, the maintenance training support device can control the ray cast to move more slowly when the user is indicating a far location. As a result, even when the user indicates a far location (or object), it becomes easier to control the ray cast.

[0126] <Temporary Fixing of the Viewing Angle of VR Display Information> When the user's viewpoint position overlaps with a predetermined object, the maintenance training assistance device 100 performs control to temporarily fix the angle of view of the VR display information.

[0127] 15 is a diagram for explaining the flow from temporary fixation of the angle of view to its release. When the user's viewpoint moves in the VR space and overlaps with a predetermined object OJ, the display information generation unit 113 generates VR display information fixed to the angle of view 300 at the timing when the overlap is identified (detected), and continues to generate VR display information based on the angle of view 300 until the user makes a movement to release the fixation of the angle of view 300. In other words, as shown in the figure, even if the viewpoint position moves within the range of the fixed angle of view 300 due to movement of the user's head, the VR display information based on the fixed angle of view 300 is displayed on the VR goggles 160.

[0128] Furthermore, when the user's viewpoint position moves outside the fixed angle of view 300, the display information generating unit 113 releases the fixation of the angle of view 300 and generates VR display information of an angle of view 310 with the user's viewpoint position at the time of release as the approximate center. Note that the predetermined object that is the target of such processing is, for example, an object that is the target of maintenance work, and more specifically, an object whose special object flag is registered as "1".

[0129] Through this processing, the maintenance training support device can generate VR display information that makes it easier for the user to perform maintenance work. Typically, a user's head constantly moves slightly, and when training in maintenance work on a specific object, these slight movements are reflected, causing the angle of view of the VR display information to move slightly, which can make it difficult for the user to concentrate on the pointing and calling or maintenance work. Therefore, the maintenance training support device temporarily fixes the angle of view of the VR display information when the user's viewpoint overlaps with the specific object that is the target of the pointing and calling or maintenance work, thereby stopping the small movements of the VR display information and making it easier for the user to concentrate on the pointing and calling movement training or maintenance work.

[0130] In addition, since the maintenance training support device releases the fixed viewing angle when the user's viewpoint moves outside the fixed viewing angle, the user does not need to use a special gesture or the like to release it.

[0131] <Vertical Angle Control of VR Display Information> When the user's head is tilted upward, the maintenance training support device 100 moves the viewpoint closer to the upper side than the normal position and generates VR display information so that the viewpoint position is located at approximately the center of the viewing angle. Similarly, when the user's head is tilted downward, the maintenance training support device 100 moves the viewpoint closer to the lower side than the normal position and generates VR display information so that the viewpoint position is located at approximately the center of the viewing angle.

[0132] FIG. 16 is a diagram for explaining the control of the viewing angle of VR display information according to the vertical tilt of the head. As shown in the figure, when the user's head (VR goggles 160) is not tilted in the vertical direction (horizontal case), the display information generation unit 113 sets the viewpoint at a position in front of the head and at a predetermined distance from the user, and generates VR display information of the viewing angle 350 with the position 351 of the viewpoint as the approximate center.

[0133] Also, when the user's head is tilted upward by a predetermined angle or more (for example, 21 degrees or more), the normal viewpoint position 352 is at the same angle as the tilt of the head (21 degrees), but the display information generation unit 113 moves it closer to the upper side by α degrees (α is preferably within the range of, for example, 0.3 degrees to 0.5 degrees, and is described as 0.4 degrees in this example), and generates VR display information so that the viewpoint position (in this case, for example, the position 21.4 degrees upward) 361 is located at approximately the center of the viewing angle 360. Note that the display information generation unit 113 moves the viewpoint position to a position obtained by adding α degrees every time the upward tilt of the user's head exceeds 21 degrees and increases by 1 degree.

[0134] Specifically, when the user's head is tilted upward by 25 degrees, the display information generation unit 113 moves the viewpoint to a position of 25 degrees + (5 × 0.4 degrees) = 27.0 degrees, and generates VR display information so that the viewpoint position (in this case, for example, a position 27.0 degrees upward) 361 is located approximately in the center of the angle of view 360.

[0135] It is preferable that α is set to about 10 degrees when the user's head is tilted upward at an angle of about 45 degrees.

[0136] The same applies to a downward tilt of the head. For example, if the user's head is tilted downward by 25 degrees, the display information generating unit 113 moves the viewpoint to a position of 25 degrees + (5 × 0.4 degrees) = 27.0 degrees, and generates VR display information so that the viewpoint position (in this case, for example, a position 27.0 degrees downward) 361 is located approximately in the center of the angle of view 360.

[0137] Through this processing, the maintenance training assistance device can generate VR display information with a more natural angle of view and display it on the VR goggles when the user tilts their head up or down. Generally, when a person tilts their head to look diagonally upward or downward, they naturally tend to look up or down. Therefore, when the head tilt is greater than a predetermined angle, the maintenance training assistance device sets the viewpoint position slightly above or below the angle of head tilt and generates VR display information such that the viewpoint position is located approximately in the center of the angle of view, thereby generating VR display information with a more natural angle of view. This allows the user to concentrate on maintenance work training without feeling uncomfortable when looking up or down in the VR space through the VR goggles, for example, during pointing and calling training.

[0138] Such a maintenance training support system 1000 may have a controller that detects the user's hand movements in the VR space and receives various operation instructions from the user, such as grabbing an object.

[0139] <Hardware configuration> 17 is a diagram showing an example of the hardware configuration of the maintenance training assistance device 100. As shown in the figure, the maintenance training assistance device 100 has a processor 410, a memory 420, a storage 430, and a communication device 440, and each component is connected by a bus 450. The power source (not shown) may be a primary battery or a rechargeable secondary battery, which can be replaced when worn out or deteriorated.

[0140] The processor 410 is an arithmetic device such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), and executes processing according to a program recorded in the memory 420 or the storage 430. In the maintenance training assistance device 100, processing is performed by the processor 410 that operates according to a program read onto the memory 420 or the storage 430. The processing unit 110 realizes each function by the processor 410 executing the program.

[0141] The memory 420 is a storage device such as RAM (Random Access Memory) or flash memory, and functions as a storage area from which programs and data are temporarily read. The storage 430 is a writable and readable storage device. The functions of the storage unit 120 are realized by the memory 420 or the storage 430. Note that the functions of the storage unit 120 may also be realized by a storage device connected via the communication device 440.

[0142] The communication device 440 is an interface for connecting the maintenance training support device 100 to external devices (including the computers 150 and 151) for communication.

[0143] The processing of each component of the maintenance training assistance device 100 may be executed by one piece of hardware or by multiple pieces of hardware. Furthermore, the processing of each component of the maintenance training assistance device 100 may be realized by one program or by multiple programs.

[0144] Furthermore, the above-described exemplary embodiments have been described in detail to facilitate understanding of the present invention, and the present invention is not limited to those including all of the configurations described herein. Furthermore, part of the configuration of one embodiment can be replaced with the configuration of another embodiment. Furthermore, the configuration of one embodiment can be added to the configuration of another embodiment. Furthermore, part of the configuration of each embodiment can be added, deleted, or replaced with other configurations. Furthermore, some or all of the above-described configurations, functions, processing units, processing means, etc. may be realized in hardware, for example, by designing them as integrated circuits. Furthermore, the control lines and information lines in the figures are those considered necessary for explanation, and are not necessarily all shown. It can be assumed that almost all of the configurations are interconnected. [Explanation of symbols]

[0145] 1000... Maintenance training support system, 100... Maintenance training support device, 110... Processing unit, 111... Action identification unit, 112... Gesture detection unit, 113... Display information generation unit, 120... Storage unit, 121... Object management information DB, 122... Interference information, 123... Pair information, 124... Position information, 125... Scenario information, 126... Gesture definition information, 130... Communication unit, 150, 151... Computer, 160... VR goggles, 180, 181... Display device, 410... Processor, 420... Memory, 430... Storage, 440... Communication device, 450... Bus, N... Network

Claims

1. A maintenance training support system having a maintenance training support device and a display device, the display device transmits motion information including head and hand movements of a user wearing the display device to the maintenance training assistance device; The maintenance training support device includes: a display information generating unit that generates display information for displaying a three-dimensional virtual reality space in which an object specific to the maintenance work is arranged; a motion identification unit that identifies a motion of the user in the virtual reality space based on the motion information acquired from the display device; a gesture detection unit that detects a composite gesture based on a hand gesture detected based on an analysis of the operation information and a position of a user's viewpoint, When the gesture detection unit detects a predetermined composite gesture, The action identification unit identifies the position of the viewpoint and the pointing and calling of the object pointed to by the hand gesture. A maintenance training support system characterized by:

2. 2. The maintenance training support system according to claim 1, The display information generation unit When the object is pointed to by the position of the viewpoint and the hand gesture, a gauge indicating the time until the pointing and calling is confirmed is displayed. A maintenance training support system characterized by:

3. 2. The maintenance training support system according to claim 1, The action identification unit When the pointing and calling is recognized, display information or sound indicating the recognition is output from the display device. A maintenance training support system characterized by:

4. 2. The maintenance training support system according to claim 1, The display information generation unit displaying a viewpoint pointer of a predetermined shape indicating the position of the viewpoint on the display information; When the gesture detection unit detects the composite gesture, an icon in the shape of an eye is displayed at the position of the viewpoint instead of the viewpoint pointer. A maintenance training support system characterized by:

5. 2. The maintenance training support system according to claim 1, the gesture detection unit detects a composite gesture based on the position of the user's gaze point, the hand gesture, and the user's speech; When the gesture detection unit detects the composite gesture based on the position of the viewpoint, the hand gesture, and the speech sound, The action identification unit recognizes the execution of the pointing and calling. A maintenance training support system characterized by:

6. 6. The maintenance training support system according to claim 5, The action identification unit If the position of the viewpoint and the object pointed to by the hand gesture match the content of the spoken voice, the pointing and calling is recognized. A maintenance training support system characterized by:

7. 7. The maintenance training support system according to claim 6, when the action identification unit recognizes the pointing and calling, it causes the display device to output display information or sound indicating the recognition; If the position of the viewpoint and the object pointed to by the hand gesture do not match the content of the utterance, the pointing and calling is not recognized, and display information or sound indicating the reason is output from the display device. A maintenance training support system characterized by:

8. 2. The maintenance training support system according to claim 1, The display information generation unit When the gesture detection unit detects a pointing hand gesture, the display information is generated to show a raycast, which is a beam-like line of light, being output from the vicinity of the fingertip making the pointing hand gesture. A maintenance training support system characterized by:

9. 9. The maintenance training support system according to claim 8, The display information generation unit The base of the palm of the hand making the pointing hand gesture is set as the starting point of the raycast output, and the display information is generated by drawing the raycast so that the raycast passes through the output starting point of the raycast set near the fingertip. A maintenance training support system characterized by:

10. 9. The maintenance training support system according to claim 8, The display information generation unit A plurality of data points are set within a certain period of time, and the change in the direction of the raycast acquired at each data point is averaged to generate the display information depicting the raycast. A maintenance training support system characterized by:

11. The maintenance training support system according to claim 10, The display information generation unit When the user points with the raycast at a location that is at least a predetermined distance away in the virtual reality space, a number of data points greater than the plurality of data points is set, and the change in direction of the raycast acquired at each data point is averaged to generate the display information that depicts the raycast. A maintenance training support system characterized by:

12. 2. The maintenance training support system according to claim 1, The display information generation unit When the user's viewpoint overlaps with a predetermined object in the virtual reality space, the display information is generated with a fixed angle of view at that time; The display information is generated based on the angle of view until the user's viewpoint moves out of the angle of view. A maintenance training support system characterized by:

13. 2. The maintenance training support system according to claim 1, The display information generation unit When the head of the user wearing the display device is tilted upward or downward by a predetermined angle or more, The user's viewpoint is moved upward or downward from a predetermined position, and the display information is generated so that the viewpoint is positioned approximately at the center of the angle of view. A maintenance training support system characterized by:

14. A maintenance training support method performed by a maintenance training support device, The maintenance training support device includes: a display information generating step of generating display information for displaying a three-dimensional virtual reality space in which an object specific to the maintenance work is arranged; a motion identification step of identifying a motion of the user in the virtual reality space based on motion information acquired from a display device worn by the user and capable of detecting motions of the user's head and hands; a gesture detection step of detecting a composite gesture based on the hand gesture detected based on the analysis of the motion information and the position of the user's gaze point; When the predetermined composite gesture is detected in the gesture detection step, In the action identification step, the position of the viewpoint and the pointing and calling of the object pointed by the hand gesture are identified. A maintenance training support method comprising:

15. A program that causes a computer to function as a maintenance training support device, The computer a display information generating unit that generates display information for displaying a three-dimensional virtual reality space in which an object specific to the maintenance work is arranged; a motion identification unit that identifies a motion of the user in the virtual reality space based on motion information acquired from a display device that is worn by the user and that can detect the motion of the user's head and hands; a gesture detection unit that detects a composite gesture based on the hand gesture detected based on the analysis of the motion information and the position of the user's gaze point; When the gesture detection unit detects a predetermined composite gesture, The action identification unit identifies the position of the viewpoint and the pointing and calling of the object pointed to by the hand gesture. A program characterized by:

Images