Maintenance training support system, maintenance training support method, and program

Abstract

To support execution of training related to maintenance of an electronic device more efficiently and effectively in virtual reality.SOLUTION: A maintenance training support apparatus includes: a display information generation unit which generates display information for a three-dimensional VR space in which objects specific to maintenance work are arranged; an action specifying unit which specifies action of a user in maintenance work based on action information acquired from a VR display apparatus and an action instruction receiving unit; and a movement command detection unit which detects, from the action information, a movement command indicating a movement instruction of the user in the VR space. The action specifying unit specifies predetermined status corresponding to an action state of the user. The movement command detection unit detects, from the action information, a different movement command according to the specified status. The display information generation unit causes the VR display apparatus to display VR display information for displaying the VR space that changes along with the movement of the user.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a maintenance training support system, a maintenance training support method, and a program. [Background technology]

[0002] In recent years, video content such as games that use VR (Virtual Reality) technology is on the rise. VR is a virtual reality video world created using, for example, 3DCG (Dimensional Computer Graphics) technology, and VR users such as gamers can have various experiences in the virtual reality.

[0003] There is also content that utilizes the characteristics of VR to provide vocational training. For example, Patent Document 1 discloses that agricultural crop cultivation training is provided in a VR space. Specifically, Patent Document 1 discloses a system for educating and training agricultural crop cultivation techniques, which includes a training scenario storage unit that stores a work target part for the agricultural crop displayed in a virtual space and a work means that is a tool used for the work target part in association with a work procedure, a three-dimensional display control unit that displays the agricultural crop and the work means in three dimensions on a display device as the work procedure progresses, a detection signal output unit that detects the position and movement of a user or an operating device and outputs a detection signal, an operation detection unit that changes the work means displayed in the virtual space based on the detection signal, and an operation determination unit that determines the result of the operation based on the coordinates of the work target part in the virtual space, the coordinates of the work means, and the operation content. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2022-149636 A Summary of the Invention [Problem to be solved by the invention]

[0005] In VR content for vocational training, display information for the virtual reality world that changes in response to user actions is usually generated based on information that defines objects and their movements specific to the training subject, and information acquired from the VR goggles and controller. Therefore, for example, when training on maintenance work for electronic devices, etc. is conducted using VR content, information that defines objects specific to the maintenance work is required.

[0006] In addition, it may be preferable to display information in the VR space during training for maintenance work to assist the user in performing maintenance work-specific actions, and even when displaying such information, predefined information is required.

[0007] The technology of Patent Document 1 is for carrying out education and training on agricultural crop cultivation in a VR space. Therefore, the technology of this document does not take into consideration objects specific to maintenance work such as electronic devices and their movements.

[0008] The present invention has been made in consideration of the above points, and has an object to support the implementation of training on maintenance work for electronic devices more efficiently and effectively in virtual reality. [Means for solving the problem]

[0009] The present application includes a number of means for solving at least part of the above problems, examples of which are as follows.

[0010] In order to solve the above problems, a maintenance training support system according to one embodiment of the present invention is a maintenance training support system having a maintenance training support device, a VR display device, and an operation instruction receiving device, wherein the maintenance training support device is equipped with a display information generation unit that generates VR display information for displaying a three-dimensional VR (Virtual Reality) space in which objects specific to the maintenance work are arranged, an action identification unit that identifies a user's action in the maintenance work based on action information acquired from the VR display device and the action instruction receiving device, and a movement command detection unit that detects a movement command indicating a user's movement instruction in the VR space from the action information, wherein the action identification unit identifies a predetermined status corresponding to the user's action state, the movement command detection unit detects the movement command from the action information that differs depending on the identified status, and the display information generation unit causes the VR display device to display the VR display information for displaying the VR space that changes in accordance with the user's movement when the user moves in the direction indicated by the movement command.

[0011] Moreover, in the above-described maintenance training support system, the maintenance training support device may include a memory unit that stores movement command information in which a correspondence between the status and the movement command of the input means that differs depending on the status is registered, and the movement command detection unit may detect the movement command from the operation information using the status and the movement command information identified by the operation identification unit.

[0012] In the above maintenance training assistance device, the status may be information indicating the usage status of both hands used by the user during maintenance training in the VR space.

[0013] In addition, in the above-mentioned maintenance training support device, the movement command detection unit may detect the movement command acquired from the operation instruction receiving device from the operation information when the status indicating that at least one of the user's hands is not being used is identified during maintenance training in a VR space.

[0014] In addition, in the above-mentioned maintenance training support device, the movement command detection unit may detect the movement command by a hand gesture detected by the VR display device from the operation information when the status indicating that at least one of the user's hands is not being used is identified during maintenance training in a VR space.

[0015] In addition, in the above-mentioned maintenance training support device, when the status indicating that both hands of the user are being used in maintenance training in a VR space is specified, the movement command detection unit may detect the movement command caused by the tilt of the VR display device from the operation information.

[0016] In addition, in the above-mentioned maintenance training support device, when the status indicating that both of the user's hands are being used in maintenance training in a VR space is specified, the movement command detection unit may detect the movement command from the user's voice acquired by a microphone provided in the VR display device from the operation information.

[0017] Furthermore, a maintenance training support method according to another aspect of the present invention is a maintenance training support method performed by a maintenance training support device, in which the maintenance training support device performs a display information generating step of generating VR display information for displaying a three-dimensional VR (Virtual Reality) space in which objects specific to the maintenance work are arranged, an action identifying step of identifying a user's action in the maintenance work based on action information acquired from a VR display device that displays the VR display information and an action instruction receiving device that receives the user's action instructions in the VR space, and a movement command detecting step of detecting a movement command indicating a user's movement instruction in the VR space from the action information, in which the action identifying step identifies a predetermined status corresponding to the user's action state, the movement command detecting step detects from the action information the movement command that differs depending on the identified status, and in the display information generating step, when the user moves in a direction indicated by the movement command, the VR display device is caused to display the VR display information for displaying the VR space that changes in accordance with the user's movement.

[0018] Furthermore, a program according to another aspect of the present invention is a program that causes a computer to function as a maintenance training support device, and causes the computer to function as a display information generation unit that generates VR display information for displaying a three-dimensional VR (Virtual Reality) space in which objects specific to maintenance work are arranged, a VR display device that displays the VR display information, an action identification unit that identifies a user's action in the maintenance work based on action information acquired from an action instruction receiving device that receives the user's action instructions in the VR space, and a movement command detection unit that detects a movement command indicating a user's movement instruction in the VR space from the action information, wherein the action identification unit is caused to identify a predetermined status corresponding to the user's action state, the movement command detection unit is caused to detect the movement command that differs according to the identified status from the action information, and the display information generation unit is caused to display the VR display information on the VR display device for displaying the VR space that changes in accordance with the user's movement when the user moves in the direction indicated by the movement command. Effect of the Invention

[0019] According to the present invention, it is possible to support the implementation of training on maintenance work of electronic devices more efficiently and effectively in virtual reality.

[0020] Problems, configurations, effects, and the like other than those described above will become apparent from the description of the following embodiments. [Brief description of the drawings]

[0021] [Figure 1] FIG. 1 is a diagram illustrating an example of a schematic configuration of a maintenance training support system. [Diagram 2] FIG. 2 is a block diagram showing an example of a functional configuration of a maintenance training support device. [Diagram 3] FIG. 11 is a diagram showing an example of object management information. [Figure 4] FIG. 11 is a diagram showing an example of interference information. [Diagram 5] FIG. 11 is a diagram showing an example of pair information. [Figure 6] FIG. 4 is a diagram showing an example of location information. [Figure 7] FIG. 11 is a diagram showing an example of operation state definition information. [Figure 8] FIG. 11 is a diagram showing an example of movement command information. [Figure 9] FIG. 13 shows an example of a VR space for conducting maintenance work training. [Figure 10] FIG. 11 is a diagram showing an example of an exchange operation of an electronic device. [Figure 11] FIG. 2 is a diagram illustrating an example of a hardware configuration of a maintenance training support device. [Figure 12] FIG. 11 is a flow diagram showing an example of a maintenance training support process. [Figure 13] FIG. 1 is a diagram illustrating an example of a user's movement in a VR space. [Figure 14] FIG. 1 is a diagram illustrating an example of a user's movement in a VR space. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0023] <Outline of the configuration of the maintenance training support system 1000> 1 is a diagram showing an example of a schematic configuration of a maintenance training support system 1000 according to this embodiment. As shown in the figure, the maintenance training support system 1000 includes a maintenance training support device 100, a computer 150, VR goggles 160, and controllers 170L and 170R.

[0024] In addition, the VR goggles 160 and the controllers 170L, 170L are connected to the computer 150 so as to be able to communicate with each other via wireless communication based on a predetermined wireless communication standard (e.g., Bluetooth (registered trademark)) or via a wired cable, and the computer 150 is connected to the maintenance training support device 100 so as to be able to communicate with each other via the network N.

[0025] The network N may be, for example, the Internet, an intranet, a Wide Area Network (WAN), a mobile phone network, or a combination of these. The network N may also be a Virtual Private Network (VPN) on a wireless communication network such as a mobile phone network.

[0026] In such a maintenance training support system 1000, the computer 150 acquires the VR display information generated by the maintenance training support device 100, performs 3D rendering, and outputs the rendered VR display information to the VR goggles 160, thereby displaying a VR space for training in maintenance work on the VR goggles 160. Note that the maintenance training support device 100 may generate display information for expressing various types of augmented reality (so-called XR (Cross Reality)) such as AR (Augmented Reality) and MR (Mixed Reality) in addition to the VR display information. That is, the maintenance training support device 100 can be applied to various types of augmented reality (XR: Cross Reality) including VR, AR, and MR. In this embodiment, a case where VR is the target will be described as an example.

[0027] Furthermore, the VR goggles 160 and the controllers 170L and 170R each transmit information indicating the user's actions (hereinafter, sometimes referred to as "action information") acquired from a camera or sensor mounted thereon to the maintenance training assistance device 100, and provide feedback of the user's movements in the VR space. The maintenance training assistance device 100, which has acquired the feedback, generates display information of the VR space that has changed in conjunction with the user's actions, and displays it on the VR goggles 160 via the computer 150.

[0028] By repeatedly executing such processing, the user can practice maintenance work on electronic devices and the like in the VR space.

[0029] <VRゴーグル160> The VR goggles 160 are a VR display device (head-mounted display) that displays VR display information of three-dimensional images created by, for example, 3DCG. The VR goggles 160 display the VR display information on a display mounted on the VR goggles 160 so that objects in the VR space can be seen three-dimensionally by utilizing the parallax between the user's eyes.

[0030] In addition, the VR goggles 160 have a sensor that detects the movement of the user's head and a plurality of cameras that are used to detect the movement of the user's hands, etc. In addition, the VR goggles 160 perform head tracking using the sensors and cameras, and transmit tracking data indicating the movement of the user's head to the maintenance training assistance device 100 as operation information.

[0031] In addition, when the user does not use the controllers 170L, 170R, the VR goggles 160 performs hand tracking using a camera, and transmits tracking data indicating the user's hand movements due to hand gestures (hand movement direction, movement amount, fingertip movement, etc.) to the maintenance training support device 100 as operation information.

[0032] The VR goggles 160 also have a microphone for collecting the user's voice. When the VR goggles 160 acquires the user's voice via the microphone, the VR goggles 160 transmits the voice to the maintenance training assistance device 100 as operation information together with tracking data.

[0033] In addition, the VR goggles 160 accept movement commands that instruct the user to move in the VR space. Specifically, the VR goggles 160 accept input of movement commands by tilting the VR goggles 160, hand gestures, and voice.

[0034] For example, when the VR goggles 160 detects a large (deep) tilt angle that exceeds a predetermined angle (e.g., an angle based on normal head movement), the VR goggles 160 determines that the user has input a movement command. Alternatively, when the VR goggles 160 detects a change in the tilt angle at a speed that exceeds a predetermined speed (e.g., a speed based on normal head movement), the VR goggles 160 determines that the user has input a movement command. In other words, the VR goggles 160 does not determine that a tilt angle that is smaller (shallower) than the predetermined angle or slow head movement less than the predetermined speed is an input of a movement command.

[0035] Furthermore, when the VR goggles 160 detects a predetermined hand gesture of the user pointing in the direction in which the user wants to move, the VR goggles 160 determines that a movement command has been input. Furthermore, when the VR goggles 160 detects a voice utterance of the user indicating the direction in which the user wants to move, the VR goggles 160 determines that the user has input a movement command.

[0036] In addition, the VR goggles 160 includes information indicating the user's movement when it determines that a movement command has been input (head tracking data, hand gestures, voice data) in the movement information as a movement command and transmits it to the maintenance training support device 100.

[0037] There are various types of VR goggles 160, such as a standalone type that displays a VR space by rendering VR display information by itself, and a type that is used by inserting a smartphone into it. In this embodiment, the following explanation will be given using as an example a type that displays a VR space on a display using VR display information rendered by a computer 150 that is communicatively connected.

[0038] <Controller 170L, 170R> The controllers 170L and 170R are operation instruction receiving devices that receive various operation instructions related to maintenance training from the user. The controller 170L is a controller that the user uses with the left hand, and the controller 170R is a controller that the user uses with the right hand. The controllers 170L and 170R each have multiple types of sensors such as a motion sensor and an acceleration sensor, and multiple instruction receiving buttons that receive operation instructions.

[0039] The controllers 170L, 170R detect the user's hand movements in the VR space (such as the direction and amount of hand movement) via these sensors, and accept operation instructions from the user indicating movements / operations using the fingertips, such as grasping or releasing an object, via the instruction acceptance button.

[0040] The controllers 170L and 170R also have sticks (levers) and accept input of movement commands that instruct the user to move in the VR space via the sticks. Specifically, the user inputs movement commands to the controllers 170L and 170R by tilting the stick in the direction in which the user wants to move.

[0041] In addition, the controllers 170L, 170R transmit operation information including the user's hand movements detected by the sensors, operation instructions obtained via the instruction acceptance button, and movement commands obtained via the stick to the maintenance training support device 100 via the computer 150.

[0042] By using such VR goggles 160 and controllers 170L, 170R, a user can perform various types of maintenance work operations in the VR space in which the object to be worked on is located, such as grasping an object, moving the object while grasping it, or releasing the grasped object.

[0043] Furthermore, by using the VR goggles 160 and the controllers 170L and 170R, the user can realize movement in the VR space. Normally, when the user moves in the VR space, the user inputs a movement command using the stick of the controller 170L or the controller 170R. Furthermore, when the controllers 170L and 170R are not used, the user inputs a movement command by a hand gesture.

[0044] On the other hand, when a maintenance training task requires, for example, grabbing a target device with both hands or performing an action using both hands, the user may not be able to input movement commands using a controller or hand gestures. Therefore, the maintenance training support system 1000 makes it possible to detect movement commands input by tilting the VR goggles 160 or by voice from the motion information according to the user's motion state.

[0045] Specifically, the maintenance training support device 100 detects user movements that are not normally detected as movement commands as movement commands depending on the user's motion state (for example, both hands are being used, one hand is being used, or both hands are free, etc.), and realizes the user's movement in the VR space.

[0046] For example, when the user is using one hand or both hands are free, the maintenance training assistance device 100 does not detect motion information or voice indicating the tilt of the VR goggles 160 as a movement command. This is because in such a motion state, the user can input a movement command by a controller or hand gestures. However, even when using one hand or both hands are free, it may be possible to input and detect a movement command by tilting the VR goggles 160 or by voice.

[0047] On the other hand, even when the user is performing maintenance training work using both hands, the user can input a movement command by tilting the VR goggles 160 or by voice. Therefore, when the user is using both hands in the motion state, the maintenance training assistance device 100 detects motion information indicating the tilt of the VR goggles 160 or voice as a movement command, and performs processing to move the user's position in the direction indicated by the movement command.

[0048] In this embodiment, the following description will focus on the process related to detection of such a movement command.

[0049] <Calculator 150> The computer 150 is a device that displays display information of the VR space in which maintenance work training is performed on the VR goggles 160, and is realized by, for example, a PC (Personal Computer). Note that the computer 150 may be a device such as a tablet terminal or a smartphone as long as it satisfies the required processing specifications.

[0050] The computer 150 performs 3D rendering when it acquires the VR display information, which is a three-dimensional image, from the maintenance training support device 100. Specifically, the computer 150 performs real-time rendering to generate a three-dimensional image in real time using the VR display information acquired from the maintenance training support device 100. The computer 150 also outputs the VR display information, which is a three-dimensional image generated by the 3D rendering, to the VR goggles 160.

[0051] In addition, the computer 150 performs a process of transmitting the operation information acquired from the VR goggles 160 and the controllers 170L and 170R to the maintenance training assistance device 100.

[0052] <Maintenance training support device 100> The maintenance training support device 100 is a device that supports a user's maintenance training in a VR space, and is realized by, for example, a server computer, a cloud server, or a personal computer. Specifically, the maintenance training support device 100 generates VR display information of a three-dimensional image in which a user trains in maintenance work, and displays the VR display information on VR goggles 160 worn by the user who performs the training, via a computer 150.

[0053] In addition, the maintenance training support device 100 generates display information for the VR space that changes in conjunction with the user's actions based on operation information acquired from the VR goggles 160 and the controllers 170L and 170R, and displays the information on the VR goggles 160 via the computer 150.

[0054] <<Details of the maintenance training support device 100>> 2 is a block diagram showing an example of a functional configuration of the maintenance training assistance device 100. As shown in the figure, the maintenance training assistance device 100 has a processing unit 110, a storage unit 120, and a communication unit .

[0055] The processing unit 110 is a functional unit that performs various processes executed by the maintenance training support device 100. Specifically, the processing unit 110 has a display information generating unit 111, an action identifying unit 112, and a movement command detecting unit 113 as functional units for executing each process.

[0056] <<<Display information generation unit 111>>> The display information generating unit 111 is a functional unit that generates VR display information. Specifically, the display information generating unit 111 generates VR display information that shows a state of training for maintenance work in a VR space in a three-dimensional image, using various information related to objects and the like stored in the storage unit 120 and operation information acquired from the VR goggles 160 and the controllers 170L and 170R. In addition, the display information generating unit 111 transmits the VR display information to the computer 150 via the communication unit 130.

[0057] <<<Operation identification unit 112>>> The motion identification unit 112 is a functional unit that identifies the motion of the user and the motion of the object in the VR space. Specifically, the motion identification unit 112 identifies the motion of the user in the VR space (for example, the motion of the hand, the motion of grabbing or releasing an object, the motion of the head, etc.) by using the motion information acquired from the controllers 170L, 170R or the VR goggles 160.

[0058] Furthermore, the operation identifying unit 112 identifies a predetermined status indicating the operation state of the user. Specifically, the operation identifying unit 112 identifies a status indicating the operation state of the user by using the operation information and information in the storage unit 120 (operation state definition information described later). Furthermore, the operation identifying unit 112 registers the identified status in information in the storage unit 120 (status management information described later).

[0059] Furthermore, the motion identification unit 112 uses the motion information to identify motions of objects, such as the position of an object that changes with the motion of the user, the distance between objects, or interference between objects. More specifically, the motion identification unit 112 uses the motion information and various information in the storage unit 120 that manages the position of an object, physical relationships such as interference, and interlocking objects based on a global coordinate system set in the VR space and a local coordinate system set for each object, to identify the motion of the user and the motion of an object in the VR space.

[0060] The method of identifying the user's actions and object movements in the VR space is not particularly limited, and any known VR technology may be used, for example.

[0061] <<<Movement command detection unit 113>>> The movement command detection unit 113 is a functional unit that detects a movement command from the operation information. The movement command detection unit 113 detects a movement command that instructs the user to move in the VR space from the operation information by using information in the storage unit 120 (movement command information described later).

[0062] Specifically, the movement command detection unit 113 identifies the input means and movement command of the movement command according to the user's current (latest) status from the information in the storage unit 120 (movement command information described later). For example, when the user's status is one-handed or both hands free, the movement command detection unit 113 identifies the stick of the controller or hand gesture as the input means of the movement command. Furthermore, when the operation information includes a movement command associated with the identified input means, the movement command detection unit 113 detects the user's movement direction indicated by the movement command. Note that even when one-handed or both hands are free, the movement command detection unit 113 may be able to detect the movement command by the inclination of the VR goggles 160 or by voice according to the registered (set) contents of the information (movement command information) in the storage unit 120.

[0063] Similarly, for example, when the user's status is "using both hands," the movement command detection unit 113 identifies VR goggles or voice as the input means for the movement command. Also, when the movement command associated with the identified input means is included in the operation information, the movement command detection unit 113 detects the movement command from the operation information.

[0064] Furthermore, the movement command detection unit 113 identifies the user's movement direction in the VR space based on the detected movement command. The user's movement amount (movement distance) may be a predetermined amount of movement corresponding to one detection of a movement command. The user's movement amount may be different for each type of movement command, or may be the same amount of movement. The user's movement speed may be preset to be high in proportion to, for example, the inclination of the VR goggles 160 or the volume of the voice (utterance).

[0065] <<<Storage section 120>>> Next, the storage unit 120 will be described. The storage unit 120 is a functional unit that stores various information. Specifically, the storage unit 120 has an object management information DB (Database) 121 in which objects specific to the maintenance work and their operations are defined, interference information 122 in which information on objects having physical relationships such as interference is registered, pair information 123 in which objects having interlocking operations are registered, position information 124 in which the latest positions of each object and user in the VR space are stored, scenario information 125 in which procedures and work instructions for maintenance work training are registered, operation state definition information 126 in which a predetermined status corresponding to the operation state of the user is defined, status management information 127 in which the status of the operation state at times is registered, and movement command information 128 in which information on movement commands is registered.

[0066] 3 is a diagram showing an example of object management information stored in the object management information DB 121. The object information is information that defines the type, position, and action of an object. Specifically, the object management information has records in which items such as identification information, type, coordinate position, and special object flag are associated with each other.

[0067] The identification information is information that uniquely identifies an object. The type is information that identifies the type of object, and there are various types such as a rack, which is an object specific to a maintenance work, an electronic device, a rack door, a cable, a power cord, and a warning label.

[0068] The coordinate position is coordinate information that indicates the position of an object in a VR space. The coordinate position includes both the position of an object specified in a global coordinate system set in the VR space and the position of an object specified in a local coordinate system set for each object.

[0069] The special object flag is information indicating objects that can be moved during maintenance work training (for example, electronic devices, racks, rack doors, cables, power cords, warning labels, etc.). Specifically, an object with a "1" registered in the special object flag corresponds to a special object. Note that objects that are not targets for movement during maintenance work training have a "0" registered in the flag.

[0070] In addition to these items, the object management information also includes information defining operational characteristics (for example, in the case of a rack door, opening and closing around one of the X to Z axes in the local coordinate system) (not shown).

[0071] 4 is a diagram showing an example of the interference information 122. The interference information 122 is information indicating an interference state between objects. Specifically, the interference information 122 has records in which items such as identification information, an interfering object, and an interfering portion are associated with each other. The identification information is information for uniquely identifying an object. The interfering object is an object that is a colliding target when viewed from an object identified by the identification information, and the identification information of the object is registered. The interfering portion is information for identifying a portion that is interfering (in contact) with the interfering object, and is, for example, information identified by a local coordinate system.

[0072] 5 is a diagram showing an example of pair information 123. Pair information 123 is information in which objects in a linked relationship are registered. Specifically, pair information 123 registers information on a parent object and a child object that moves in conjunction with the movement of the parent object. More specifically, pair information 123 has records in which items such as identification information of a parent object, identification information of a child object, and a coordinate position are associated with each other.

[0073] The identification information of the parent object and the child object are information that uniquely identifies the parent object such as a rack or a rack door (e.g., the object on which the child object is affixed or attached) and the child object such as an electronic device or a warning label (e.g., the object on which the child object is affixed or attached). The coordinate position is coordinate information that specifies the position of the child object relative to the parent object, and is information that is specified by a local coordinate system, for example. The type of object that corresponds to the parent object and the type of object that corresponds to the child object are defined by predetermined information (not shown), and the information may be stored in advance in the storage unit 120. Also, for example, a special object may be treated as a child object, and a non-special object may be treated as a parent object.

[0074] In addition, whether objects that are linked to each other are parent objects or child objects can be determined based on priority information (not shown) that is preset for each object according to a predetermined viewpoint (e.g., difficulty of performing an action, etc.).

[0075] FIG. 6 is a diagram showing an example of the position information 124. The position information 124 is information indicating the position of an object after it has moved due to maintenance work training by a user, that is, the latest position in the VR space. Specifically, the position information 124 has records in which items such as identification information and coordinate positions are associated. The identification information is information that uniquely identifies an object. The coordinate position is coordinate information that indicates the latest position of an object, and is information specified by a global coordinate system and a local coordinate system.

[0076] It should be noted that the position information 124 also includes a coordinate position that specifies the user's position in the VR space.

[0077] The scenario information 125 is information in which maintenance work training procedures, work instructions, etc. are registered. Specifically, the scenario information 125 registers scenarios corresponding to a plurality of types of training objectives, such as replacing electronic devices and changing mount positions, as well as information such as objects, object placement, maintenance work procedures, and work instructions corresponding to each scenario.

[0078] In addition, the work instructions are information that indicates work instructions that follow maintenance work procedures, such as "Please identify the rack to be worked on by pointing and calling out" or "Please attach a work-in-progress label (warning label) to the rack to be worked on."

[0079] 7 is a diagram showing an example of the motion state definition information 126. The motion state definition information 126 is information that defines motion states corresponding to the characteristics of a user's motion. Specifically, the motion state definition information 126 has records in which a status, a motion characteristic (in the case of using a controller), and a motion characteristic (in the case of a hand gesture) are associated with each other.

[0080] The status is information indicating a predetermined status of the motion state identified from the motion characteristics. Specifically, the status indicates the usage status of the user's both hands, such as one hand in use, both hands in use, and both hands free, depending on the associated motion characteristics. The motion characteristics (when using a controller) and the motion characteristics (when using a hand gesture) are information indicating the motion characteristics of the user identified from the motion information, and are used to identify the status. For example, when the motion information indicates a "state in which a motion instruction (such as gripping) is input from a controller in one hand to an object", a status of one hand in use is identified. Also, when the motion information indicates a "state in which a motion instruction is not input from the controllers in both hands to an object", a status of both hands free is identified.

[0081] The status management information 127 is information for managing the user's status from time to time. Specifically, a predetermined status determined based on the latest operation information is registered in the status management information 127. Note that the status is determined again every time operation information is acquired, so that the status determined based on the latest operation information is always registered in the status management information 127.

[0082] 8 is a diagram showing an example of the movement command information 128. The movement command information 128 is information in which the linking relationship between the status and the input means of the movement command, and the movement command for each input means are registered. Specifically, the movement command information 128 has records in which the status, the movement command for each input means, and the movement direction are associated with each other.

[0083] Note that the status is a predetermined status indicating the operation state of the user. The movement commands for each input means indicate the movement commands for each input means such as the control stick, hand gesture, VR goggles, and voice. The movement direction is information for specifying the movement direction of the user indicated by the corresponding movement command.

[0084] Such movement command information 128 is used when the movement command detection unit 113 detects a movement command from the operation information.

[0085] Note that each piece of information in these storage units 120 is an example, and as long as it is information that can specify physical relationships such as the position, movement, and interference of objects in the VR space, associated objects with linkage, the position after movement, the operation state of the user, etc., the type and content of the information and the association relationship of the information are not limited to the above examples.

[0086] <<<Communication unit 130>>> Returning to FIG. 2 for explanation. The communication unit 130 is a functional unit that performs information communication with an external device. Specifically, the communication unit 130 performs information communication with the computer 150 and acquires the operation information output from the VR goggles 160, controllers 170L, and 170R. Further, the communication unit 130 transmits the VR display information generated by the display information generation unit 111 to the computer 150.

[0087] <An example of the VR space displayed on the VR goggles 160> FIG. 9 is a diagram showing an example of a VR space for performing maintenance work training. As shown in the figure, in the VR space for maintenance work training, for example, objects specific to maintenance work such as a rack 200, electronic equipment 201, and a warning label 202, an operation instruction window 203, and the user's hand 204 (the shape of the hand 204 is simplified in the illustrated example) are displayed.

[0088] Each object such as the rack 200 and the work instruction window 203 are displayed in the VR space based on the information stored in the storage unit 120. Also, the user's hand 204 is represented at a corresponding position in the VR space based on the operation information acquired from the VR goggles 160 and the controllers 170L and 170R.

[0089] <Display of maintenance operation specific operations and auxiliary information> 10 is a diagram showing an example of a replacement operation of the electronic device 201. The maintenance work includes, for example, a work of replacing a broken electronic device 201, and a work of changing the mounting position of the electronic device 201. The processing unit 110 enables the user to practice such a maintenance work-specific operation in the VR space.

[0090] Specifically, the display information generating unit 111 generates display information of the VR space in which the rack 200 on which the electronic device 201 is mounted is arranged, using each piece of information in the storage unit 120. Furthermore, the action identifying unit 112 calculates the distance from the user's hand 204 to the electronic device 201 to be worked on, based on the action information and each piece of information in the storage unit 120. Furthermore, when an action instruction to grip is input when the distance is less than a predetermined value, the action identifying unit 112 generates VR display information of the action in which the user's hand 204 grips the electronic device 201 to be worked on, via the display information generating unit 111.

[0091] Furthermore, the action identification unit 112 calculates the distance from the user's hand 204 holding the electronic device 201 to the surface of an object such as the rack 200 based on the action information and each piece of information in the storage unit 120. Furthermore, when the action identification unit 112 detects that an action instruction to release the electronic device 201 has been input when the distance is less than a predetermined value, the display information generation unit 111 generates VR display information in a state in which the electronic device 201 is mounted in a mountable position in the rack 200 close to the user's hand 204.

[0092] Through such processing, the user can practice operations specific to maintenance work, such as replacing or changing the position of the electronic device 201, in the VR space.

[0093] Note that the object related to the operation specific to the maintenance work is not limited to the electronic device 201, but may also be a cable, a power cord, or other objects. Even in this case, the maintenance training support device 100 (particularly, the processing unit 110) can perform the same processing as above to enable training of the user in the operation specific to the maintenance work, and can support the implementation of efficient and effective training.

[0094] <Hardware configuration> 11 is a diagram showing an example of a hardware configuration of the maintenance training support device 100. As shown in the figure, the maintenance training support device 100 has a processor 310, a memory 320, a storage 330, and a communication device 340, and each component is connected by a bus 350. In addition, the power source (not shown) may be a primary battery or a rechargeable secondary battery, which can be replaced when worn out or deteriorated.

[0095] The processor 310 is an arithmetic device such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), and executes processing according to a program recorded in the memory 320 or the storage 330. In the maintenance training support device 100, processing is performed by the processor 310 that operates according to a program read onto the memory 320 or the storage 330. The processing unit 110 realizes each function by the processor 310 executing the program.

[0096] The memory 320 is a storage device such as a RAM (Random Access Memory) or a flash memory, and functions as a storage area from which programs and data are temporarily read. The storage 330 is a writable and readable storage device. The functions of the storage unit 120 are realized by the memory 320 or the storage 330. The functions of the storage unit 120 may be realized by a storage device connected via the communication device 340.

[0097] The communication device 340 is an interface for connecting the maintenance training support device 100 to external devices (including the computer 150) for communication.

[0098] The processing of each component of the maintenance training assistance device 100 may be executed by one piece of hardware or by multiple pieces of hardware. Also, the processing of each component of the maintenance training assistance device 100 may be realized by one program or by multiple programs.

[0099] <Maintenance training support processing> 12 is a flow diagram showing an example of a maintenance training support process. The process is started, for example, when the processing unit 110 receives an instruction to start the process from a user. At this time, the user selects a predetermined scenario in which the user wants to train from the scenario information 125 and starts the process.

[0100] When the process starts, the motion identification unit 112 determines whether or not motion information has been acquired (step S10). If it is determined that motion information has not been acquired (No in step S10), the motion identification unit 112 performs the process of step S10 again. On the other hand, if it is determined that motion information has been acquired (Yes in step S10), the motion identification unit 112 transitions the process to step S20.

[0101] In step S20, the motion identification unit 112 identifies the motion state of the user. Specifically, the motion identification unit 112 identifies a status indicating the motion state of the user using the motion state definition information 126. For example, when the motion information indicates a state in which the user is holding an object with one hand, the motion identification unit 112 identifies the motion state of the user as a status of "one hand in use" using the motion state definition information 126. Also, for example, when the motion information indicates a state in which the user is holding an object with both hands, the motion identification unit 112 identifies the motion state of the user as a status of "both hands in use" using the motion state definition information 126.

[0102] Next, the behavior identifying unit 112 registers the identified behavior state of the user (step S30). Specifically, the behavior identifying unit 112 registers the identified user's status in the status management information 127.

[0103] Next, the movement command detection unit 113 determines whether or not the movement information includes a movement command corresponding to the user's movement state (step S40). Specifically, the movement command detection unit 113 uses the status management information 127 to identify a status indicating the current movement state of the user.

[0104] Furthermore, the movement command detection unit 113 identifies the input means linked to the identified status from the movement command information 128. For example, if the identified status is "one handed", the movement command detection unit 113 identifies "controller stick" and "hand gesture" as the input means for the movement command linked to the status. Furthermore, if the identified status is "both hands used", the movement command detection unit 113 identifies "VR goggles" and "voice" as the input means for the movement command linked to the status.

[0105] Furthermore, when the movement command of the identified input means is included in the action information, the movement command detection unit 113 determines that the action information includes a movement command.

[0106] If it is determined that a movement command is included (Yes in step S40), movement command detection unit 113 proceeds to step S50. On the other hand, if it is determined that a movement command is not included (No in step S40), movement command detection unit 113 proceeds to step S60.

[0107] In step S50, the movement command detection unit 113 identifies the movement direction from the detected movement command. Specifically, the movement command detection unit 113 identifies the movement direction indicated by the detected movement command using the movement command information 128. For example, when a stick of a controller is used as an input means and a movement command indicating that the stick has been tilted upward is detected, the movement command detection unit 113 identifies the movement direction "forward" from the movement command information 128.

[0108] Also, for example, when the VR goggles 160 are identified as the input means and a movement command indicating that the VR goggles 160 have been tilted downward is detected, the movement command detection unit 113 identifies the movement direction as “forward” from the movement command information 128.

[0109] Next, the display information generating unit 111 generates display information of the VR space that changes by reflecting the user's actions and movements identified by the action identifying unit 112 and the movement command detecting unit 113 (step S60). In addition, the display information generating unit 111 causes the VR display information to be displayed on the VR goggles 160 of the corresponding user via the communication unit 130 and the computer 150 (step S70).

[0110] Furthermore, the movement command detection unit 113 updates the position information 124 by registering the coordinate position of the user after the movement in the VR space in the position information 124 (step S80), and returns the process to step S10.

[0111] 13 is a diagram illustrating an example of a user's movement in a VR space. The left side of the arrow AR indicates the user's motion state before the movement in the VR space. The right side of the arrow indicates the user's motion after the movement. The illustrated example is a scene in which the user is about to attach a label 202 to a rack 200 located further back than the user's own position.

[0112] As shown in the figure, the user's operating state before movement is a state in which one hand holds a label 202 with "working" written on it. In this case, a status of "one-handed use" is specified. Here, when the user inputs a movement command by tilting the stick of the controller 170L with the hand (for example, the left hand) that does not hold the label 202 in the direction in which the user wants to move, or by pointing in the direction in which the user wants to move with a hand gesture, operation information including the movement command is transmitted to the maintenance training assistance device 100. Also, when the maintenance training assistance device 100 acquires the operation information, the movement command detection unit 113 detects the movement command if the movement command input by the stick of the controller or the hand gesture, which is an input means corresponding to the status, is included in the operation information. Also, the display information generation unit 111 generates VR display information of the VR space that changes when the user moves in the movement direction specified by the movement command detection unit 113, and displays the VR display information on the VR goggles 160.

[0113] Based on the input and detection of such a movement command, as shown to the right of the arrow AR, the user moves to the front of the rack 200 that is the target of the work, and VR display information corresponding to the series of movements is displayed on the VR goggles 160.

[0114] Fig. 14 is a diagram illustrating an example of a user's movement in a VR space. As in Fig. 13, the left side of the arrow AR indicates the user's motion state before the movement in the VR space. The right side of the arrow indicates the user's motion after the movement. The illustrated example is a scene in which the user is attempting to mount a target device 201 on a rack 200 located further back than the user's own position.

[0115] As shown in the figure, the user's operating state before movement is a state in which the target device 201 is held with both hands. In this case, a status of using both hands is specified. Here, since the user is using both hands, when the user inputs a movement command by tilting the VR goggles 160 in a predetermined direction corresponding to the direction to which the user wants to move, or by speaking an utterance indicating the direction to which the user wants to move, operation information including the movement command is transmitted to the maintenance training assistance device 100. In addition, when the maintenance training assistance device 100 acquires the operation information, the movement command detection unit 113 detects the movement command if the movement command input by the VR goggles 160 or voice, which is an input means corresponding to the status, is included in the operation information. In addition, the display information generation unit 111 generates VR display information of the VR space that changes when the user moves in the movement direction specified by the movement command detection unit 113, and displays the VR display information on the VR goggles 160.

[0116] Based on the input and detection of such a movement command, as shown to the right of the arrow AR, the user moves to the front of the rack 200 that is the target of the work, and VR display information corresponding to the series of movements is displayed on the VR goggles 160.

[0117] The maintenance training support process has been described above.

[0118] According to such a maintenance training support device, it is possible to more efficiently and effectively support the implementation of training on maintenance work of electronic devices in virtual reality.

[0119] In particular, the user can input movement commands that instruct movement in the VR space by various input means (methods) according to the motion state of the maintenance training. For example, if the user has both hands occupied by holding a tool or a target device, there are cases where the user cannot input a movement command by a controller or hand gestures. The maintenance training support device identifies such a motion state of the user, and, for example, according to a motion state in which both hands cannot be used, detects a user's movement (e.g., tilt of VR goggles or voice) that is not normally detected as a movement command as a movement command, and realizes the user's movement in the VR space. This allows the user to input movement commands by multiple input means (methods) according to the motion state, and to move within the VR space.

[0120] The maintenance training assistance device 100 may display the VR display information on a display device other than the VR goggles 160. For example, the maintenance training assistance device 100 may display display information obtained by converting the VR display information into a two-dimensional image on a display device such as a display that displays a two-dimensional image.

[0121] The present invention is not limited to the above embodiment, and various modifications are possible. For example, the maintenance training support device may use a standalone VR goggle instead of a computer if the VR goggle has a function capable of executing the same processing as a computer. The controller communicates information with the standalone VR goggle and transmits user operation information to the maintenance training support device via the VR goggle.

[0122] In addition, the above-mentioned embodiment examples have been described in detail to make the present invention easier to understand, and the present invention is not limited to those having all the configurations described here. In addition, it is possible to replace a part of the configuration of one example of an embodiment with the configuration of another example. In addition, it is also possible to add the configuration of another example to the configuration of one example of an embodiment. In addition, it is also possible to add, delete, or replace other configurations with respect to a part of the configuration of each example of an embodiment. In addition, each of the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in hardware by designing them as integrated circuits, for example, in part or in whole. In addition, the control lines and information lines in the figures show those that are considered necessary for explanation, and do not necessarily show all of them. It may be considered that almost all of the configurations are connected to each other. [Explanation of symbols]

[0123] 1000...maintenance training support system, 100...maintenance training support device, 110...processing unit, 111...display information generating unit, 112...operation identifying unit, 113...movement command detecting unit, 120...storage unit, 121...object management information DB, 122...interference information, 123...pair information, 124...position information, 125...scenario information, 126...operation state definition information, 127...status management information, 128...movement command information, 130...communication unit, 310...processor, 320...memory, 330...storage, 340...communication device, 350...bus, N...network

Claims

1. A maintenance training support system having a maintenance training support device, a VR display device, and an operation instruction receiving device, The maintenance training support device includes: a display information generating unit that generates VR display information for displaying a three-dimensional virtual reality (VR) space in which objects specific to the maintenance work are arranged; an action identification unit that identifies a user's action during maintenance work based on action information acquired from the VR display device and the action instruction receiving device; a movement command detection unit that detects a movement command indicating a movement instruction of a user in a VR space from the operation information, The motion identification unit is Identifying a predetermined status corresponding to an operating state of the user; The movement command detection unit Detecting the movement command, which differs depending on the identified status, from the operation information; The display information generating unit When the user moves in a direction indicated by the movement command, the VR display device displays the VR display information for displaying the VR space that changes in accordance with the user's movement. A maintenance training support system comprising:

2. 2. The maintenance training support system according to claim 1, The maintenance training support device includes: a storage unit that stores movement command information in which a correspondence between the status and the movement command of the input means that differs depending on the status is registered; The movement command detection unit detects the movement command from the operation information by using the status specified by the operation specification unit and the movement command information. A maintenance training support system comprising:

3. 3. The maintenance training support system according to claim 2, The status is: This is information showing the usage status of both hands used by a user during maintenance training in a VR space. A maintenance training support system comprising:

4. The maintenance training support system according to claim 3, In the maintenance training in the VR space, the movement command detection unit When the status indicating that at least one hand of the user is not being used is specified, the movement command acquired from the operation instruction receiving device is detected from the operation information. A maintenance training support system comprising:

5. The maintenance training support system according to claim 3, In the maintenance training in the VR space, the movement command detection unit When the status indicating that at least one hand of the user is not being used is specified, the movement command by a hand gesture detected by the VR display device is detected from the operation information. A maintenance training support system comprising:

6. The maintenance training support system according to claim 3, In the maintenance training in the VR space, the movement command detection unit If the status indicating that both hands of the user are being used is identified, the movement command caused by tilting the VR display device is detected from the motion information. A maintenance training support system comprising:

7. The maintenance training support system according to claim 3, In the maintenance training in the VR space, the movement command detection unit When the status indicating that both hands of the user are being used is specified, the movement command by the user's voice acquired by a microphone included in the VR display device is detected from the motion information. A maintenance training support system comprising:

8. A maintenance training support method performed by a maintenance training support device, comprising: The maintenance training support device includes: a display information generating step of generating VR display information for displaying a three-dimensional virtual reality (VR) space in which objects specific to the maintenance work are arranged; an action identification step of identifying a user's action in the maintenance work based on action information acquired from a VR display device that displays the VR display information and an action instruction receiving device that receives an action instruction from the user in the VR space; a movement command detection step of detecting a movement command indicating a movement instruction of a user in a VR space from the operation information; In the action identification step, Identifying a predetermined status corresponding to an operating state of the user; In the movement command detection step, Detecting the movement command, which differs depending on the identified status, from the operation information; In the display information generating step, When the user moves in a direction indicated by the movement command, the VR display device displays the VR display information for displaying the VR space that changes in accordance with the user's movement. A maintenance training support method comprising:

9. A program for causing a computer to function as a maintenance training support device, The computer, a display information generating unit that generates VR display information for displaying a three-dimensional virtual reality (VR) space in which objects specific to the maintenance work are arranged; a VR display device that displays the VR display information; and an action identification unit that identifies a user's action in the maintenance work based on action information acquired from an action instruction receiving device that receives an action instruction from the user in the VR space; a movement command detection unit that detects a movement command indicating a movement instruction of a user in a VR space from the operation information; The motion identification unit includes: Identifying a predetermined status corresponding to an operating state of the user; The movement command detection unit includes: detecting, from the operation information, the movement command that differs depending on the identified status; The display information generating unit includes: When the user moves in a direction indicated by the movement command, the VR display device displays the VR display information for displaying the VR space that changes in accordance with the user's movement. A program characterized by:

Images