Maintenance training support system, maintenance training support method, and program

Abstract

To support implementation of training related to maintenance of electronic equipment more efficiently and effectively in virtual reality.SOLUTION: A display device transmits motion information including motions of the head and hands of a user wearing the display device, to a maintenance training support apparatus. The maintenance training support apparatus is configured to: generate display information for displaying a three-dimensional virtual reality space in which maintenance-specific objects are disposed; identify motion of the user in the virtual reality space based on the motion information acquired from the display device; detect a hand gesture based on analyzing the motion information; when detecting a predetermined hand gesture, move the location of the user in the virtual reality space to a location indicated by the hand gesture; and cause the display device to display the display information of the virtual reality space in the location after the relocation.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a maintenance training support system, a maintenance training support method, and a program. [Background technology]

[0002] In recent years, there has been an increase in video content, such as games, that utilizes VR (Virtual Reality) technology. VR is a virtual reality visual world created using, for example, 3DCG (Dimensional Computer Graphics) technology, and VR users, such as gamers, can have a variety of experiences within the virtual reality.

[0003] There are also vocational training contents that utilize the characteristics of VR. For example, Patent Document 1 discloses agricultural crop cultivation training in a VR space. Specifically, Patent Document 1 discloses a system for educating and training agricultural crop cultivation techniques, which includes a training scenario storage unit that stores work target areas for the crops displayed in a virtual space and work implements, which are tools used on the work target areas, in association with work procedures; a three-dimensional display control unit that displays the crops and work implements in three dimensions on a display device as the work procedures progress; a detection signal output unit that detects the position and movement of a user or an operating device and outputs a detection signal; an operation detection unit that changes the work implement displayed in the virtual space based on the detection signal; and an operation determination unit that determines the result of the operation based on the coordinates of the work target areas, the coordinates of the work implement, and the operation content in the virtual space. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2022-149636 Summary of the Invention [Problem to be solved by the invention]

[0005] In VR content for vocational training, display information for the virtual reality world that changes in response to user actions is usually generated based on information defining the objects and their movements specific to the training subject, and information acquired from the VR goggles and controller. Therefore, for example, when using VR content to train in maintenance work for electronic devices, information defining the objects specific to the maintenance work is required.

[0006] In addition, it may be desirable to display information in the VR space during training to assist the user in performing maintenance work, and even when displaying such information, predefined information is required.

[0007] The technology in Patent Document 1 is for providing education and training on agricultural crop cultivation in a VR space, and therefore does not take into consideration objects specific to maintenance work such as electronic devices and their movements.

[0008] The present invention has been made in view of the above points, and aims to support the implementation of training on electronic equipment maintenance work more efficiently and effectively in virtual reality. [Means for solving the problem]

[0009] The present application includes a number of means for solving at least part of the above problems, examples of which are as follows.

[0010] In order to solve the above-mentioned problems, one embodiment of the present invention provides a maintenance training support system having a maintenance training support device and a display device, wherein the display device transmits motion information, including head and hand movements, of a user wearing the display device to the maintenance training support device, and the maintenance training support device comprises: a display information generation unit that generates display information for displaying a three-dimensional virtual reality space in which objects specific to maintenance work are arranged; a motion identification unit that identifies the user's motion within the virtual reality space based on the motion information acquired from the display device; and a gesture detection unit that detects hand gestures based on an analysis of the motion information, wherein when the gesture detection unit detects a predetermined hand gesture, the motion identification unit moves the user's location within the virtual reality space to the location indicated by the hand gesture, and the display information generation unit displays the display information of the virtual reality space at the moved location on the display device.

[0011] In addition, in the above-mentioned maintenance training support system, the hand gesture corresponding to the movement may be a predetermined gesture in which one hand is used to point to a destination location and the other hand is used to indicate the time until the movement is performed.

[0012] In addition, in the above-mentioned maintenance training support system, the display information generation unit may display a gauge indicating the time until the movement is executed at the destination location when a predetermined gesture is made with the other hand.

[0013] In addition, in the above-mentioned maintenance training support system, the action identification unit may, upon detecting the hand gesture, instantly move the user's location in the virtual reality space by switching the location of the user to the location indicated by the hand gesture.

[0014] Furthermore, in the above-described maintenance training support system, when a pointing hand gesture is detected by the gesture detection unit, the display information generation unit may generate the display information showing a raycast, which is a beam-like line of light, being output from near the tip of the finger making the pointing hand gesture.

[0015] Furthermore, in the above-described maintenance training support system, the display information generation unit may generate the display information in which the raycast is drawn so that the output starting point of the raycast is near the base of the palm of the hand making the pointing hand gesture and passes through the output starting point of the raycast, which is set near the position of the fingertip.

[0016] In addition, in the above-mentioned maintenance training support system, the display information generation unit may set multiple data points within a certain period of time, average the changes in the direction of the raycast obtained at each data point, and generate the display information that depicts the raycast.

[0017] Furthermore, in the above-described maintenance training support system, when the user uses the raycast to point to a location in the virtual reality space that is more than a predetermined distance away, the display information generation unit may set a number of data points greater than the plurality of data points, average the change in direction of the raycast obtained at each data point, and generate the display information that depicts the raycast.

[0018] Furthermore, in the above-described maintenance training support system, when the user's viewpoint overlaps with a predetermined object in the virtual reality space, the display information generation unit may generate the display information fixed to the angle of view at that time, and may continue to generate the display information based on that angle of view until the user's viewpoint moves out of the angle of view.

[0019] Furthermore, in the above-described maintenance training support system, when the head of the user wearing the display device is tilted upward or downward by a predetermined angle or more, the display information generation unit may move the user's viewpoint upward or downward from a predetermined position and generate the display information so that the viewpoint is positioned approximately in the center of the angle of view.

[0020] Another aspect of the present invention provides a maintenance training support system having a maintenance training support device and a display device, wherein the display device transmits motion information, including head and hand movements, of a user wearing the display device to the maintenance training support device, and the maintenance training support device comprises: a display information generation unit that generates display information for displaying a three-dimensional virtual reality space in which objects specific to maintenance work are arranged; and a motion identification unit that identifies the user's motion within the virtual reality space based on the motion information acquired from the display device, and when the head of the user wearing the display device is tilted upward or downward by more than a predetermined angle, the display information generation unit moves the user's viewpoint above or below the predetermined position, and generates the display information so that the viewpoint is positioned approximately in the center of the angle of view, and displays it on the display device.

[0021] Another aspect of the present invention provides a maintenance training support method performed by a maintenance training support device, wherein the maintenance training support device performs the following steps: a display information generation step of generating display information for displaying a three-dimensional virtual reality space in which objects specific to maintenance work are arranged; a motion identification step of identifying a user's motion within the virtual reality space based on motion information acquired from a display device worn by the user and capable of detecting the user's head and hand movements; and a gesture detection step of detecting a hand gesture based on an analysis of the motion information; when a predetermined hand gesture is detected in the gesture detection step, the motion identification step instantaneously moves the user's location to the location indicated by the hand gesture, and the display information generation step generates the display information of the virtual reality space at the location after the move.

[0022] Furthermore, a program according to another aspect of the present invention is a program that causes a computer to function as a maintenance training support device, and causes the computer to function as a display information generation unit that generates display information for displaying a three-dimensional virtual reality space in which objects specific to maintenance work are arranged, a motion identification unit that identifies the user's motion within the virtual reality space based on motion information acquired from a display device worn by the user and capable of detecting the user's head and hand movements, and a gesture detection unit that detects hand gestures based on an analysis of the motion information. When the gesture detection unit detects a predetermined hand gesture, the motion identification unit instantly moves the user's location to the location indicated by the hand gesture, and the display information generation unit displays the display information of the virtual reality space at the new location on the display device. [Effects of the Invention]

[0023] According to the present invention, it is possible to support the implementation of training on maintenance work of electronic devices more efficiently and effectively in virtual reality.

[0024] Problems, configurations, effects, and the like other than those described above will become clear from the following description of the embodiments. [Brief explanation of the drawings]

[0025] [Figure 1] 1 is a diagram illustrating an example of a schematic configuration of a maintenance training support system. [Figure 2] FIG. 2 is a block diagram showing an example of a functional configuration of a maintenance training support device. [Figure 3] FIG. 10 is a diagram illustrating an example of object management information. [Figure 4] FIG. 10 is a diagram illustrating an example of interference information. [Figure 5] FIG. 10 is a diagram illustrating an example of pair information. [Figure 6] FIG. 10 is a diagram showing an example of location information. [Figure 7] FIG. 10 is a diagram illustrating an example of gesture definition information. [Figure 8] FIG. 10 is a diagram showing an example of VR display information. [Figure 9] FIG. 10 is a flow diagram showing an example of a maintenance training support process. [Figure 10] FIG. 10 is a diagram for explaining the maintenance training support process related to warp movement. [Figure 11] FIG. 10 is a diagram for explaining the maintenance training support process related to warp movement. [Figure 12] FIG. 10 is a diagram for explaining the maintenance training support process related to warp movement. [Figure 13] FIG. 10 is a diagram illustrating control related to raycast output. [Figure 14] 10A and 10B are diagrams for explaining a flow from temporary fixing of the angle of view to its release. [Figure 15] 10A and 10B are diagrams for explaining control of the angle of view of VR display information according to the tilt of the head in the up and down direction. [Figure 16] FIG. 2 is a diagram illustrating an example of a hardware configuration of a maintenance training support device. DETAILED DESCRIPTION OF THE INVENTION

[0026] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0027] <General Configuration of Maintenance Training Support System 1000> 1 is a diagram showing an example of a schematic configuration of a maintenance training support system 1000 according to this embodiment. As shown in the figure, the maintenance training support system 1000 includes a maintenance training support device 100, a computer 150, VR goggles 160 and a display device 180, and a computer 151 and a display device 181. It is assumed that the computer 150, the VR goggles 160 and the display device 180, and the computer 151 and the display device 181 will be used by different users of the maintenance training support system 1000, respectively.

[0028] The VR goggles 160 are connected to the computer 150 via wireless communication using a predetermined wireless communication standard (e.g., Bluetooth (registered trademark)) or via a wired cable so that they can communicate with each other, and the computers 150 and 151 are connected to the maintenance training support device 100 via the network N so that they can communicate with each other.

[0029] The network N may be, for example, the Internet, an intranet, a wide area network (WAN), a mobile phone network, or a communication network that combines these. The network N may also be a virtual private network (VPN) on a wireless communication network such as a mobile phone communication network.

[0030] In the maintenance training support system 1000, a computer 150 acquires VR display information of the three-dimensional image generated by the maintenance training support device 100, performs 3D rendering, and outputs the rendered VR display information to a VR goggle 160, thereby displaying a virtual reality space for training in maintenance work on the VR goggle 160.

[0031] The maintenance training assistance device 100 may generate display information corresponding to various types of augmented reality (so-called XR: Cross Reality), such as AR (Augmented Reality) and MR (Mixed Reality). That is, the maintenance training assistance device 100 can be applied to XR, which includes VR, AR, and MR. In the present embodiment, the following description will be given taking VR as an example.

[0032] <VRゴーグル160> The VR goggles 160 are a display device (head-mounted display) that displays VR display information of three-dimensional images created by, for example, 3DCG (Three-Dimensional Computer Graphics). The VR goggles 160 display the VR display information on a display mounted on the VR goggles 160 so that objects in the VR space appear three-dimensional by utilizing the parallax between the user's eyes.

[0033] The VR goggles 160 also have a sensor that detects the movement of the user's head and a plurality of cameras that are used to detect the movement of the user's hands, and perform head tracking and hand tracking using the sensors and cameras.

[0034] Specifically, the VR goggles 160 perform head tracking and transmit tracking data indicating the movement of the user's head as operation information to the maintenance training assistance device 100. In addition, the VR goggles 160 perform hand tracking and transmit tracking data indicating the movement of the user's hand (hand movement direction, movement amount, fingertip movement, predetermined hand gesture, etc.) as operation information to the maintenance training assistance device 100.

[0035] There are various types of VR goggles 160, such as a standalone type that displays a VR space by rendering VR display information by itself, or a type that is used by inserting a smartphone into it. In this embodiment, the following explanation will be given using an example of a type that displays a VR space on a display using VR display information rendered by a computer 150 that is connected so that communication is possible.

[0036] By using the VR goggles 160, the user can grasp an object, move the grasped object, or perform various other maintenance-related actions using hand gestures. The user can also warp to a desired location in the VR space using hand gestures. Note that warp movement refers to instantaneous movement from the user's current location to a location specified by the user. Details of warp movement will be described later.

[0037] <Calculator 150, 151> The computers 150 and 151 are devices that display information about the VR space in which maintenance work training is performed on the VR goggles 160 or the display devices 180 and 181, and are realized by, for example, a PC (Personal Computer). Note that the computers 150 and 151 may also be devices such as tablet terminals or smartphones as long as they satisfy the required processing specifications. The computer 150 may also be, for example, an information processing device built into the standalone VR goggles 160.

[0038] The computer 150 performs 3D rendering when it acquires VR display information, which is a three-dimensional image, from the maintenance training assistance device 100. Specifically, the computer 150 performs real-time rendering to generate a three-dimensional image in real time using the VR display information acquired from the maintenance training assistance device 100. The computer 150 also outputs the VR display information, which is a three-dimensional image generated by the 3D rendering, to the VR goggles 160.

[0039] Furthermore, when the computers 150 and 151 display the user's training status in the VR space on the display devices 180 and 181, they convert the rendered VR display information into display information for display (two-dimensional video) and output this to the display devices 180 and 181.

[0040] The computer 150 also performs processing to transmit the operation information acquired from the VR goggles 160 to the maintenance training assistance device 100 .

[0041] <Display device 180, 181> The display devices 180 and 181 are display devices such as displays that display two-dimensional images. The display devices 180 and 181 display display information acquired from the computers 150 and 151 or the maintenance training support device 100.

[0042] <Maintenance training support device 100> The maintenance training support device 100 is a device that supports a user's maintenance training in a VR space, and is realized by, for example, a server computer, a cloud server, or a personal computer. Specifically, the maintenance training support device 100 generates VR display information of a three-dimensional image in which a user performs maintenance work training, and displays the information via a computer 150 on VR goggles 160 worn by the user performing the training.

[0043] In addition, when the maintenance training support device 100 acquires user movement information from the VR goggles 160, it generates VR display information that depicts the appearance of the VR space that changes in conjunction with the user's movements, and displays it on the VR goggles 160 via the computer 150.

[0044] <<Details of the maintenance training support device 100>> 2 is a block diagram showing an example of the functional configuration of the maintenance training assistance device 100. As shown in the figure, the maintenance training assistance device 100 has a processing unit 110, a storage unit 120, and a communication unit 130.

[0045] The processing unit 110 is a functional unit that performs various processes executed by the maintenance training assistance device 100. Specifically, the processing unit 110 has an action identification unit 111, a gesture detection unit 112, and a display information generation unit 113 as functional units for executing each process.

[0046] The motion identification unit 111 is a functional unit that analyzes motion information (tracking data) to identify the motion of the user's head and hands. The motion identification unit 111 also uses various information related to objects in the storage unit 120 to identify (calculate) the motion of objects, such as interference between objects, contact with the user, or movement accompanying the user's motion.

[0047] Specifically, the action identification unit 111 identifies the user's actions and object movements in the VR space using action information and various information in the memory unit 120 that manages the object positions, physical relationships such as interference, and linked objects based on the global coordinate system set in the VR space and the local coordinate system set for each object.

[0048] The method for identifying the user's actions and the object's movements in the VR space is not particularly limited, and any known VR technology may be used.

[0049] The gesture detection unit 112 is a functional unit that analyzes motion information (tracking data of hand tracking) and detects a predetermined hand gesture when the user is making the gesture.

[0050] The display information generating unit 113 is a functional unit that generates VR display information. Specifically, the display information generating unit 113 generates VR display information that depicts, in three-dimensional video, the state of the VR space that changes in accordance with the user's movement, using the user's movement, object movement, position, and posture identified by the movement identifying unit 111, and various information in the storage unit 120.

[0051] More specifically, the display information generating unit 113 determines the position of the viewpoint at a predetermined distance (e.g., 3 m) from the user, in front of the user's head identified by the action identifying unit 111, and generates VR display information so that the viewpoint is located approximately in the center of the angle of view. At this time, the display information generating unit 113 may display a viewpoint pointer (e.g., a circle and the word "Self") indicating the position of the viewpoint approximately in the center of the VR display information.

[0052] Moreover, the display information generating unit 113 transmits the generated VR display information to the computer 150 via the communication unit 130.

[0053] Furthermore, the display information generating unit 113 displays the user's hands drawn in 3DCG in the VR display information, reflecting the user's hand movements identified by the action identifying unit 111. That is, the VR display information displays the user's hands performing maintenance work or making hand gestures such as pointing in the VR space.

[0054] Furthermore, the display information generation unit 113 generates VR display information according to an action corresponding to a user's hand gesture. For example, when a hand gesture of pinching a target object with the thumb, index finger, and middle finger is detected by the gesture detection unit 112, the display information generation unit 113 identifies the corresponding action (in this case, the action of grasping the selected object) using predetermined information (gesture definition information described later), and generates VR display information according to the identified action (in this case, drawing information showing how the hand and the grasped object move (move) together).

[0055] Next, the storage unit 120 will be described. The storage unit 120 is a functional unit that stores various types of information. Specifically, the storage unit 120 has an object management information DB (Database) 121 in which objects specific to maintenance work and their operations are defined, interference information 122 in which information on objects having physical relationships such as interference is registered, pair information 123 in which objects having interlocking operations are registered, position information 124 in which the latest positions of each object and user in the VR space are stored, scenario information 125 in which maintenance work training procedures, work instructions, etc. are registered, and gesture definition information 126 in which predetermined operations corresponding to hand gestures are defined.

[0056] 3 is a diagram showing an example of object management information stored in the object management information DB 121. The object information is information that defines the type, position, and behavior of an object. Specifically, the object management information has records in which items such as identification information, type, coordinate position, and special object flag are associated with each other.

[0057] The identification information is information that uniquely identifies an object. The type is information that identifies the type of object, and there are various types such as racks, which are objects specific to maintenance work, electronic devices, rack doors, cables, power cords, and warning labels.

[0058] The coordinate position is coordinate information that indicates the position of an object in the VR space. Note that the coordinate position includes both the position of the object specified in the global coordinate system set in the VR space and the position of the object specified in the local coordinate system set for each object.

[0059] The special object flag is information that indicates objects that can be moved during maintenance work training (for example, electronic devices, racks, rack doors, cables, power cords, warning labels, etc.). Specifically, an object with a "1" registered in the special object flag corresponds to a special object. Note that objects that are not subject to movement during maintenance work training have a "0" registered in the flag.

[0060] In addition to these items, the object management information also includes information defining operational characteristics (for example, in the case of a rack door, opening and closing around one of the X to Z axes in the local coordinate system) (not shown).

[0061] FIG. 4 is a diagram showing an example of the interference information 122. The interference information 122 is information indicating the state of interference between objects. Specifically, the interference information 122 has records in which items such as identification information, interfering objects, and interfering portions are associated with each other. The identification information is information that uniquely identifies an object. The interfering object is an object that is colliding when viewed from the object identified by the identification information, and the identification information of the object is registered. The interfering portion is information that identifies a portion that is interfering (in contact) with the interfering object, and is, for example, information identified using a local coordinate system.

[0062] 5 is a diagram showing an example of pair information 123. Pair information 123 is information in which objects in a linked relationship are registered. Specifically, pair information 123 registers information about a parent object and a child object that moves in conjunction with the parent object. More specifically, pair information 123 has records in which items such as identification information of the parent object, identification information of the child object, and coordinate positions are associated with each other.

[0063] The identification information of the parent object and the child object is information that uniquely identifies the parent object, such as a rack or a rack door (for example, the object on which the child object is pasted or attached), and the child object, such as an electronic device or a warning label (for example, the object on which the child object is pasted or attached). The coordinate position is coordinate information that specifies the position of the child object relative to the parent object, and is information specified by, for example, a local coordinate system. The type of object that corresponds to the parent object and the type of object that corresponds to the child object are defined by predetermined information (not shown), and this information may be stored in advance in the storage unit 120. Also, for example, a special object may be treated as a child object, and a non-special object may be treated as a parent object.

[0064] Whether linked objects are parent objects or child objects can be determined based on priority information (not shown) that is preset for each object according to a predetermined viewpoint (e.g., difficulty of operation).

[0065] FIG. 6 is a diagram showing an example of the position information 124. The position information 124 is information indicating the position of an object after it has moved due to maintenance work training by the user, i.e., the latest position in the VR space. Specifically, the position information 124 has records in which items such as identification information and coordinate positions are associated. Note that the identification information is information that uniquely identifies an object. Also, the coordinate position is coordinate information indicating the latest position of the object, and is information specified by a global coordinate system and a local coordinate system.

[0066] It is assumed that the position information 124 also includes coordinate information that identifies the position of the user in the VR space and the postures of the user and objects.

[0067] The scenario information 125 is information in which maintenance work training procedures, work instructions, etc. are registered. Specifically, the scenario information 125 registers scenarios corresponding to multiple types of training objectives, such as replacing electronic devices or changing mount positions, as well as information such as objects, object placement, maintenance work procedures, and work instructions corresponding to each scenario.

[0068] The work instructions are information indicating work instructions that follow maintenance work procedures, such as "Please identify the rack to be worked on by pointing and calling" or "Please attach a work-in-progress label (warning label) to the rack to be worked on."

[0069] 7 is a diagram showing an example of gesture definition information 126. Gesture definition information 126 is information that defines predetermined hand gestures (including combinations of hand gestures) and actions corresponding to each hand gesture. Specifically, gesture definition information 126 has records in which hand gestures are associated with actions.

[0070] Note that the hand gesture is information indicating various hand gestures that serve as execution triggers for the associated operations. An operation is information indicating an operation to be executed when an associated hand gesture is detected. Note that there are various types of hand gestures and operations, and FIG. 7 shows an example thereof.

[0071] Note that each piece of information in the storage unit 120 is an example, and the type, content, and association relationship of the information are not limited to the above example.

[0072] Next, the communication unit 130 will be described. The communication unit 130 is a functional unit that performs information communication with an external device. Specifically, the communication unit 130 performs information communication with the computers 150 and 151 and acquires operation information output from the VR goggles 160. In addition, the communication unit 130 transmits the VR display information generated by the display information generation unit 113 to the computers 150 and 151.

[0073] <An example of VR display information> FIG. 8 is a diagram showing an example of VR display information displayed on the VR goggles 160. As shown in the figure, the VR display information is display information that depicts the VR space in which the rack 200 and the electronic device 201 are arranged from the first-person perspective of the user. In addition, the VR display information includes a caution label 202, which is an object specific to maintenance work, a work instruction window 203, the user's hand 204 drawn in 3DCG, and a viewpoint pointer 205 indicating the user's viewpoint position.

[0074] Note that each object such as the rack 200 and the work instruction window 203 is drawn based on various pieces of information stored in the storage unit 120. In addition, the user's hand 204 and the viewpoint pointer 205 indicating the viewpoint position are drawn based on the operation information acquired from the VR goggles 160.

[0075] By wearing the VR goggles 160 on which such VR display information is displayed, the user can perform various types of maintenance training, such as replacing a malfunctioning electronic device, in the VR space by means of hand gestures.

[0076] <Maintenance training support processing> 9 is a flow diagram showing an example of the maintenance training support process. This process starts, for example, when the processing unit 110 receives a processing start instruction from the user. At this time, the user selects a predetermined scenario in which the user wants to train from the scenario information 125 and starts the process.

[0077] When the process starts, the gesture detection unit 112 determines whether or not a hand gesture has been detected based on an analysis of the motion information (step S10). If it determines that a hand gesture has not been detected (No in step S10), the gesture detection unit 112 performs the process of step S10 again. On the other hand, if it determines that a hand gesture has been detected (Yes in step S10), the gesture detection unit 112 proceeds to step S20.

[0078] Note that gesture detection unit 112 may detect multiple hand gestures simultaneously. For example, when different hand gestures are made with the right hand and the left hand and each gesture is associated with a different action, gesture detection unit 112 detects each hand gesture at the same time.

[0079] In step S20, gesture detection unit 112 identifies an action corresponding to the detected hand gesture based on gesture definition information 126, and proceeds to step S30.

[0080] In step S30, the display information generation unit 113 generates VR display information corresponding to the identified action, and then the process proceeds to step S40.

[0081] In step S40, the display information generation unit 113 transmits the generated VR display information to the computer 150 via the communication unit 130, thereby causing the VR goggles 160 to display the VR display information.

[0082] Furthermore, after transmitting the VR display information to the computers 150 and 151, the display information generating unit 113 shifts the process to step S10, and repeatedly executes the processes of steps S10 to S40.

[0083] When the position or posture of the object or the position of the user himself / herself changes in response to the user's action, the action identification unit 111 updates the information by registering the latest information in the position information 124 .

[0084] Next, the above-mentioned maintenance training support process will be explained using a specific example of warp movement.

[0085] 10 to 12 are diagrams for explaining the maintenance training support process related to warp movement. When the user wishes to warp to a location in the VR space, the user makes a "pointing" hand gesture (a gesture in which the palm is turned sideways and the index finger is extended to point to a target) with one hand 210 (FIG. 10). At this time, the gesture detection unit 112 detects that the user is making a pointing hand gesture based on an analysis of the motion information (step S10).

[0086] Next, the gesture detection unit 112 identifies an action corresponding to the pointing hand gesture from the gesture definition information 126 (step S20). Specifically, the gesture detection unit 112 identifies an action called "raycast output" that corresponds to the hand gesture. Note that a raycast is a beam-like line of light, and outputting a raycast means an action of outputting a beam-like line of light from near the tip of the finger (e.g., the index finger) making the hand gesture in the direction pointed by the hand gesture.

[0087] When the gesture detection unit 112 identifies an action corresponding to the hand gesture, the display information generation unit 113 generates VR display information corresponding to the action (step S30). Specifically, the display information generation unit 113 displays a raycast output start point 220 near the tip of the finger making the pointing hand gesture, generates VR display information in which a raycast 230 is output from the output start point 220 (step S30), and displays the VR display information on the VR goggles 160 (step S40).

[0088] Next, the user uses a pointing hand gesture to indicate the location to which they wish to warp using a raycast 230 output from near their fingertip, and while maintaining this state, turns their other hand 211 upward with the palm open (FIG. 11). At this time, the gesture detection unit 112 detects the new hand gesture (step S10), and identifies from the gesture definition information 126 the action corresponding to this hand gesture: "display a gauge indicating the time remaining until the warp movement is executed in the warp movement destination area" (step S20).

[0089] Next, the display information generation unit 113 generates VR display information corresponding to the identified action. In this case, the display information generation unit 113 generates VR display information in which a gauge of a predetermined shape (in this example, a circular gauge) 240 is displayed at the location pointed to by the user using the raycast 230 (step S30), and displays the VR display information on the VR goggles 160 (step S40). Note that the hand gestures up to this point are hand gestures in a preliminary stage before executing warp movement, and therefore the gauge indicating the time remaining until warp movement is displayed at 0 (zero).

[0090] Next, the user makes a gesture of making a fist with hand 211 with the palm facing up (FIG. 12). Note that the other hand 210 maintains the pointing hand gesture. At this time, gesture detection unit 112 detects the new hand gesture (step S10), and identifies the action corresponding to this hand gesture from gesture definition information 126 as "continue to fill up the gauge displayed in the warp destination area, and execute warp movement when the gauge is full" (step S20).

[0091] Furthermore, the display information generating unit 113 generates VR display information corresponding to the identified action. In this case, the display information generating unit 113 generates VR display information in which the gauge 240 keeps increasing while the identified hand gesture is maintained (step S30), and displays the VR display information on the VR goggles 160 (step S40).

[0092] Furthermore, the user's warp movement is executed when the gauge 240 becomes full. Specifically, the action identification unit 111 instantaneously moves the user by switching the user's location from the current position to the location indicated by the raycast 230. Furthermore, the display information generation unit 113 generates VR display information from a first-person perspective of the VR space as seen from the destination location, and displays it on the VR goggles 160.

[0093] The time until warp movement is executed, i.e., the time from the start of the hand gesture for filling the gauge 240 until the gauge 240 is full, may be set appropriately (for example, about 3 to 5 seconds). Also, the hand gestures corresponding to each operation, such as "displaying a gauge in the warp movement destination area that indicates the time until warp movement is executed" and "continuing to fill the gauge displayed in the warp movement destination area, and executing warp movement when the gauge is full," are not limited to the above examples, and any gestures may be set in advance.

[0094] By executing such maintenance training support processing, the maintenance training support device realizes warp movement in VR space using hand gestures. This prevents the user from experiencing so-called VR sickness, which occurs when moving through VR space. For example, in the case of normal movement rather than warp movement, the state of the user's head shaking slightly during movement is reflected in the VR display information, which can cause the user to experience screen sickness (VR sickness). On the other hand, with such warp movement, the user can instantly move to the desired location, so the user does not experience VR sickness and can concentrate on the maintenance training.

[0095] The maintenance training support device 100 has various features in addition to those described in the above embodiment. Each of these features will be described below.

[0096] <Raycast output control> 13 is a diagram for explaining control related to the output of a raycast. As described above, a raycast 230 is a beam-like line of light output from near the fingertip when the user makes a pointing hand gesture.

[0097] The maintenance training assistance device 100 controls so that a raycast is output from the base of the user's palm (near the wrist) (hereinafter sometimes referred to as the "raycast output origin") 250. Specifically, the gesture detection unit 112, which has detected the pointing hand gesture, analyzes the motion information to identify the position of the base of the user's palm 250. Furthermore, based on the analysis of the motion information by the gesture detection unit 112, the position of the fingertip (index finger) 221 making the pointing hand gesture is identified.

[0098] The display information generation unit 113 uses this identified information to draw a raycast 230. Specifically, the display information generation unit 113 displays a spherical raycast output start point 220 near the identified position of the fingertip 221. The display information generation unit 113 also draws the raycast 230 so that it starts from the identified position of the base of the palm 250 and passes through the raycast output start point 220.

[0099] The display information generation unit 113 generates drawing information that makes the raycast 230 appear to be output from the raycast output starting point 220 by hiding the raycast 230 between the base of the palm 250 and the output starting point 220.

[0100] This process allows the maintenance training assistance device to perform raycast output control that is easy for the user to control. For example, if the raycast output start point is set to the fingertip, the fingertip may bend slightly, making it difficult for the user to determine the direction of the raycast or hit the target object. Therefore, the maintenance training assistance device sets the raycast start point at the base of the user's palm, allowing for stable raycast output control that is easy for the user to control.

[0101] Such raycast control is useful, for example, when the user points to a warp destination.

[0102] <Controlling the amount of movement of raycast> The maintenance training assistance device 100 calculates the average value of multiple data points over a certain period of time to control the amount of movement of the raycast 230. Specifically, the display information generation unit 113 sets multiple data points (e.g., every 0.1 seconds) over a certain period of time (e.g., 0.5 seconds) at the raycast output origin 250, the raycast output start point (near the fingertip) 220, or both, and averages the change in the direction of the raycast 230 acquired at each data point.

[0103] More specifically, for example, if the orientation of the raycast 230 is defined by the x, y, and z axes, the display information generation unit 113 calculates the average value of the amount of change in the x-axis, the amount of change in the y-axis, and the amount of change in the z-axis acquired at each data point.The display information generation unit 113 then obtains the calculated values ​​for each axis as the change in the orientation of the raycast 230 over a certain period of time, and reflects this in the rendering information of the raycast 230.

[0104] The number of data points and the averaging method are not particularly limited, and any known method may be adopted as long as it can stably display the raycast 230 without linearly reflecting the finger movement.

[0105] By performing such processing, the maintenance training support device can stabilize the unstable movement of the raycast, which is caused by the minute movements of the user's hand being reflected, and perform raycast output control that is easier for the user to control.

[0106] Such raycast control is useful, for example, when the user points to a warp destination.

[0107] In addition, when the user points with the raycast 230 at a location far from the user's location in the VR space, i.e., a location that is more than a predetermined distance (for example, more than 7 m) away from the user's position, the maintenance training support device 100 may make the raycast 230 move more slowly than when pointing to a nearby location (a location less than the predetermined distance).

[0108] Specifically, the display information generation unit 113 sets more data points (e.g., every 0.05 seconds) within a certain period (e.g., 0.5 seconds) than the data points (e.g., every 0.1 seconds) when a nearby location is indicated, and averages the change in direction of the raycast 230 obtained at each data point.

[0109] Through such processing, when the user points to a distant location, the maintenance training support device can control the ray casting to move more slowly. As a result, even when the user points to a distant location, it becomes easier to control the ray casting.

[0110] <Temporary Fixation of the Viewing Angle of VR Display Information> When the viewpoint position of the user overlaps with a predetermined object, the maintenance training support device 100 performs control to temporarily fix the viewing angle of the VR display information.

[0111] FIG. 14 is a diagram for explaining the flow from the temporary fixation of the viewing angle to its release. When the viewpoint of the user moves within the VR space and overlaps with a predetermined object OJ, the display information generation unit 113 generates VR display information fixed at the viewing angle 300 at the timing when the overlap is identified (detected), and generates VR display information based on the viewing angle 300 until the user makes a movement to release the fixation of the viewing angle 300. That is, as shown in the figure, even if the viewpoint position accompanying the movement of the user's head moves within the range of the fixed viewing angle 300, the VR display information based on the fixed viewing angle 300 is displayed on the VR goggles 160.

[0112] Further, when the viewpoint position of the user moves outside the fixed viewing angle 300, the display information generation unit 113 releases the fixation of the viewing angle 300 and generates VR display information of a viewing angle 310 with the viewpoint position of the user at the time of release as the approximate center. The predetermined object targeted for such processing is, for example, an object to be the target of maintenance work, and specifically, an object in which "1" is registered in the special object flag.

[0113] Through such processing, the maintenance training support device can generate VR display information that makes it easier for the user to perform maintenance work. Usually, the user's head is constantly moving slightly. When training for maintenance work on a predetermined object, this slight movement is reflected, causing the viewing angle of the VR display information to move in small increments, and in some cases, the user may not be able to concentrate on the maintenance work. Therefore, the maintenance training support device temporarily fixes the viewing angle of the VR display information at the timing when the user's viewpoint overlaps the object to be maintained, thereby stopping the small increments of movement of the VR display information and making it easier for the user to concentrate on the maintenance work.

[0114] In addition, since the maintenance training support device releases the fixation of the viewing angle when the user's viewpoint moves outside the fixed viewing angle, the user does not need to use a special gesture or the like to release it.

[0115] <Control of the Vertical Viewing Angle of VR Display Information> When the user's head is tilted upward, the maintenance training support device 100 moves the viewpoint closer to the upper side than the normal position and generates VR display information so that the viewpoint position is located at approximately the center of the viewing angle. Similarly, when the user's head is tilted downward, the maintenance training support device 100 moves the viewpoint closer to the lower side than the normal position and generates VR display information so that the viewpoint position is located at approximately the center of the viewing angle.

[0116] FIG. 15 is a diagram for explaining the control of the viewing angle of VR display information according to the vertical tilt of the head. As shown in the figure, when the user's head (VR goggles 160) is not tilted vertically (in the horizontal case), the display information generation unit 113 sets the viewpoint at a position in front of the head and at a predetermined distance from the user, and generates VR display information with a viewing angle 350 having the position 351 of the viewpoint as approximately the center.

[0117] Furthermore, when the user's head is tilted upward by a predetermined angle or more (for example, 21 degrees or more), the normal viewpoint position 352 is at the same angle (21 degrees) as the tilt of the head, but the display information generation unit 113 moves it upward by α degrees (α is preferably within a range of, for example, 0.3 degrees to 0.5 degrees, and is described as 0.4 degrees in this example) from the predetermined angle, and generates VR display information so that the viewpoint position (in this case, for example, a position 21.4 degrees upward) 361 is located approximately in the center of the angle of view 360. Note that the display information generation unit 113 moves the viewpoint position to a position adding α degrees for each additional degree that the upward tilt of the user's head exceeds 21 degrees.

[0118] Specifically, when the user's head is tilted upward by 25 degrees, the display information generation unit 113 moves the viewpoint to a position of 25 degrees + (5 × 0.4 degrees) = 27.0 degrees, and generates VR display information so that the viewpoint position (in this case, for example, a position 27.0 degrees upward) 361 is located approximately in the center of the angle of view 360.

[0119] It is preferable that α is set to about 10 degrees when the user's head is tilted upward at an angle of about 45 degrees.

[0120] The same applies to a downward tilt of the head. For example, if the user's head is tilted downward by 25 degrees, the display information generating unit 113 moves the viewpoint to a position of 25 degrees + (5 × 0.4 degrees) = 27.0 degrees, and generates VR display information so that the viewpoint position (in this case, for example, a position 27.0 degrees downward) 361 is located approximately in the center of the angle of view 360.

[0121] Through this processing, the maintenance training assistance device can generate VR display information with a more natural angle of view when the user tilts their head up or down and display it on the VR goggles. Generally, when a person tilts their head to look diagonally upward or downward, they naturally tend to look up or down. Therefore, when the head tilt is greater than a predetermined angle, the maintenance training assistance device sets the viewpoint position slightly above or below the angle of head tilt and generates VR display information such that the viewpoint position is located approximately in the center of the angle of view, thereby generating VR display information with a more natural angle of view. This allows the user to concentrate on maintenance work training without feeling uncomfortable when looking up or down in the VR space through the VR goggles.

[0122] Such a maintenance training support system 1000 may have a controller that detects the user's hand movements in the VR space and receives various operation instructions from the user, such as grabbing an object.

[0123] <Hardware configuration> 16 is a diagram showing an example of the hardware configuration of the maintenance training support device 100. As shown in the figure, the maintenance training support device 100 has a processor 410, a memory 420, a storage 430, and a communication device 440, and each component is connected by a bus 450. In addition, the power source (not shown) may be a primary battery or a rechargeable secondary battery, which can be replaced when worn out or deteriorated.

[0124] The processor 410 is an arithmetic device such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), and executes processing according to a program recorded in the memory 420 or the storage 430. In the maintenance training assistance device 100, processing is performed by the processor 410 that operates according to a program read onto the memory 420 or the storage 430. The processing unit 110 realizes each function by the processor 410 executing the program.

[0125] The memory 420 is a storage device such as RAM (Random Access Memory) or flash memory, and functions as a storage area from which programs and data are temporarily read. The storage 430 is a writable and readable storage device. The functions of the storage unit 120 are realized by the memory 420 or the storage 430. Note that the functions of the storage unit 120 may also be realized by a storage device connected via the communication device 440.

[0126] The communication device 440 is an interface for connecting the maintenance training support device 100 to external devices (including the computers 150 and 151) for communication.

[0127] The processing of each component of the maintenance training assistance device 100 may be executed by one piece of hardware or by multiple pieces of hardware. Furthermore, the processing of each component of the maintenance training assistance device 100 may be realized by one program or by multiple programs.

[0128] Furthermore, the above-described exemplary embodiments have been described in detail to facilitate understanding of the present invention, and the present invention is not limited to those including all of the configurations described herein. Furthermore, part of the configuration of one embodiment can be replaced with the configuration of another embodiment. Furthermore, the configuration of one embodiment can be added to the configuration of another embodiment. Furthermore, part of the configuration of each embodiment can be added, deleted, or replaced with other configurations. Furthermore, some or all of the above-described configurations, functions, processing units, processing means, etc. may be realized in hardware, for example, by designing them as integrated circuits. Furthermore, the control lines and information lines in the figures are those considered necessary for explanation, and are not necessarily all shown. It can be assumed that almost all of the configurations are interconnected. [Explanation of symbols]

[0129] 1000... Maintenance training support system, 100... Maintenance training support device, 110... Processing unit, 111... Action identification unit, 112... Gesture detection unit, 113... Display information generation unit, 120... Storage unit, 121... Object management information DB, 122... Interference information, 123... Pair information, 124... Position information, 125... Scenario information, 126... Gesture definition information, 130... Communication unit, 150, 151... Computer, 160... VR goggles, 180, 181... Display device, 410... Processor, 420... Memory, 430... Storage, 440... Communication device, 450... Bus, N... Network

Claims

1. A maintenance training support system having a maintenance training support device and a display device, the display device transmits motion information including head and hand movements of a user wearing the display device to the maintenance training assistance device; The maintenance training support device includes: a display information generating unit that generates display information for displaying a three-dimensional virtual reality space in which an object specific to the maintenance work is arranged; a motion identification unit that identifies a motion of the user in the virtual reality space based on the motion information acquired from the display device; a gesture detection unit that detects a hand gesture based on an analysis of the motion information, When the gesture detection unit detects a predetermined hand gesture, the action identification unit moves the location of the user in the virtual reality space to a location indicated by the hand gesture; The display information generation unit causes the display device to display the display information of the virtual reality space at the location after the movement. A maintenance training support system characterized by:

2. 2. The maintenance training support system according to claim 1, The hand gesture corresponding to the movement is Point to the destination with one hand, A predetermined gesture with the other hand to allow time to elapse before the movement is performed. A maintenance training support system characterized by:

3. 3. The maintenance training support system according to claim 2, The display information generation unit When a predetermined gesture is made with the other hand, a gauge indicating the time until the movement is executed is displayed at the movement destination location. A maintenance training support system characterized by:

4. 2. The maintenance training support system according to claim 1, The action identification unit Upon detecting the hand gesture, the location of the user in the virtual reality space is switched to the location indicated by the hand gesture, thereby instantly moving the user. A maintenance training support system characterized by:

5. 2. The maintenance training support system according to claim 1, The display information generation unit When the gesture detection unit detects a pointing hand gesture, the display information is generated to show a raycast, which is a beam-like line of light, being output from the vicinity of the fingertip making the pointing hand gesture. A maintenance training support system characterized by:

6. 6. The maintenance training support system according to claim 5, The display information generation unit The base of the palm of the hand making the pointing gesture is set as the starting point of the raycast output, and the display information is generated by drawing the raycast so that the raycast passes through the output starting point of the raycast set near the position of the fingertip. A maintenance training support system characterized by:

7. 6. The maintenance training support system according to claim 5, The display information generation unit A plurality of data points are set within a certain period of time, and the change in the direction of the raycast acquired at each data point is averaged to generate the display information depicting the raycast. A maintenance training support system characterized by:

8. 8. The maintenance training support system according to claim 7, The display information generation unit When the user points with the raycast at a location that is at least a predetermined distance away in the virtual reality space, a number of data points greater than the plurality of data points is set, and the change in direction of the raycast acquired at each data point is averaged to generate the display information that depicts the raycast. A maintenance training support system characterized by:

9. 2. The maintenance training support system according to claim 1, The display information generation unit When the user's viewpoint overlaps with a predetermined object in the virtual reality space, the display information is generated with a fixed angle of view at that time; The display information is generated based on the angle of view until the user's viewpoint moves out of the angle of view. A maintenance training support system characterized by:

10. 2. The maintenance training support system according to claim 1, The display information generation unit When the head of the user wearing the display device is tilted upward or downward by a predetermined angle or more, The user's viewpoint is moved upward or downward from a predetermined position, and the display information is generated so that the viewpoint is positioned approximately at the center of the angle of view. A maintenance training support system characterized by:

11. A maintenance training support system having a maintenance training support device and a display device, the display device transmits motion information including head and hand movements of a user wearing the display device to the maintenance training assistance device; The maintenance training support device includes: a display information generating unit that generates display information for displaying a three-dimensional virtual reality space in which an object specific to the maintenance work is arranged; a motion identification unit that identifies a motion of the user in the virtual reality space based on the motion information acquired from the display device, The display information generation unit When the head of the user wearing the display device is tilted upward or downward by a predetermined angle or more, The user's viewpoint is moved above or below a predetermined position, and the display information is generated so that the viewpoint is positioned approximately at the center of the angle of view, and the display information is displayed on the display device. A maintenance training support system characterized by:

12. A maintenance training support method performed by a maintenance training support device, The maintenance training support device includes: a display information generating step of generating display information for displaying a three-dimensional virtual reality space in which an object specific to the maintenance work is arranged; a motion identification step of identifying a motion of the user in the virtual reality space based on motion information acquired from a display device worn by the user and capable of detecting motions of the user's head and hands; a gesture detection step of detecting a hand gesture based on an analysis of the motion information; When a predetermined hand gesture is detected in the gesture detection step, In the action identification step, a location of the user in the virtual reality space is moved to a location indicated by the hand gesture; In the display information generating step, the display information of the virtual reality space at the location after movement is generated. A maintenance training support method comprising:

13. A program that causes a computer to function as a maintenance training support device, The computer a display information generating unit that generates display information for displaying a three-dimensional virtual reality space in which an object specific to the maintenance work is arranged; a motion identification unit that identifies a motion of the user in the virtual reality space based on motion information acquired from a display device that is worn by the user and that can detect the motion of the user's head and hands; a gesture detection unit that detects a hand gesture based on an analysis of the motion information; When the gesture detection unit detects a predetermined hand gesture, the action identification unit moves the location of the user in the virtual reality space to a location indicated by the hand gesture; The display information generation unit causes the display device to display the display information of the virtual reality space at the location after the movement. A program characterized by:

Images