Maintenance practice support device, maintenance practice support method and program

Abstract

To enable supporting an implementation of a practice on a maintenance work of an electronic device more efficiently and more effectively in a virtual reality.SOLUTION: A maintenance training support device comprises a display information generation unit that generates display information displaying a three-dimensional VR (Virtual Reality) in which an object unique to a maintenance work is arranged, and serving as field-of-view pictures of a plurality of users sharing the VR space. The display information generation unit is configured to: generate a first field-of-view picture serving as a field-of-view picture of a first user, and a second field-of-view picture serving as a field-of-view picture when viewing from a field of view different from the first field-of-view picture; generate VR display information in which the second field-of-view picture is included in the first field-of-view picture; and cause the VR display information to be displayed on a VR display device the first user owns.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a maintenance training support device, a maintenance training support method, and a program. [Background technology]

[0002] In recent years, video content such as games that use VR (Virtual Reality) technology is on the rise. VR is a virtual reality video world created using, for example, 3DCG (Dimensional Computer Graphics) technology, and VR users such as gamers can have various experiences in the virtual reality.

[0003] There is also content that utilizes the characteristics of VR to provide vocational training. For example, Patent Document 1 discloses that agricultural crop cultivation training is provided in a VR space. Specifically, Patent Document 1 discloses a system for educating and training agricultural crop cultivation techniques, which includes a training scenario storage unit that stores a work target part for the agricultural crop displayed in a virtual space and a work means that is a tool used for the work target part in association with a work procedure, a three-dimensional display control unit that displays the agricultural crop and the work means in three dimensions on a display device as the work procedure progresses, a detection signal output unit that detects the position and movement of a user or an operating device and outputs a detection signal, an operation detection unit that changes the work means displayed in the virtual space based on the detection signal, and an operation determination unit that determines the result of the operation based on the coordinates of the work target part in the virtual space, the coordinates of the work means, and the operation content. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2022-149636 A Summary of the Invention [Problem to be solved by the invention]

[0005] In VR content for vocational training, display information for the virtual reality world that changes in response to user actions is usually generated based on information that defines objects and their movements specific to the training subject, and information acquired from the VR goggles and controller. Therefore, for example, when training on maintenance work for electronic devices, etc. is conducted using VR content, information that defines objects specific to the maintenance work is required.

[0006] In addition, it may be preferable to display information in the VR space during training for maintenance work to assist the user in performing maintenance work-specific actions, and even when displaying such information, predefined information is required.

[0007] The technology of Patent Document 1 is for carrying out education and training on agricultural crop cultivation in a VR space. Therefore, the technology of this document does not take into consideration objects specific to maintenance work such as electronic devices and their movements.

[0008] The present invention has been made in consideration of the above points, and aims to support the implementation of training on maintenance work for electronic devices more efficiently and effectively in virtual reality and augmented reality. [Means for solving the problem]

[0009] The present application includes a number of means for solving at least part of the above problems, examples of which are as follows.

[0010] In order to solve the above problems, a maintenance training support device according to one embodiment of the present invention is provided with a display information generating unit that generates viewpoint images of multiple users sharing the VR space, which is display information displaying a three-dimensional VR (Virtual Reality) space in which objects specific to maintenance work are arranged, and the display information generating unit generates a first viewpoint image that is a viewpoint image of a first user and a second viewpoint image that is a viewpoint image seen from a viewpoint different from the first viewpoint image, generates VR display information in which the second viewpoint image is included in the first viewpoint image, and displays the VR display information on a VR display device owned by the first user.

[0011] In the above maintenance training assistance device, the display information generating section may generate a viewpoint video of a second user different from the first user as the second viewpoint video.

[0012] In addition, in the above maintenance training support device, the display information generation unit may generate, as the second viewpoint image, an overhead image in which the object is viewed from a predetermined height and angle in the VR space.

[0013] In addition, in the above-mentioned maintenance training support device, the display information generation unit may generate, as the second viewpoint image, a viewpoint image of the objects when viewed from a viewpoint closer to a certain target object than a viewpoint of the first user toward the target object.

[0014] In the above maintenance training assistance device, the display information generating section may generate the second viewpoint video as viewed from an angle specified by the first user.

[0015] In addition, in the above-mentioned maintenance training support device, the display information generation unit may generate the second viewpoint video that replays an action of the first user or a second user other than the first user, with respect to the object.

[0016] In the above maintenance training assistance device, the display information generating section may generate the second viewpoint video replayed at a playback speed designated by the first user.

[0017] In addition, in the above maintenance training support device, the display information generation unit may generate the VR display information in which a plurality of the second viewpoint images are simultaneously included in the first viewpoint image.

[0018] In addition, in the above-mentioned maintenance training support device, the display information generation unit may generate display information corresponding to at least one of the first viewpoint video, the first viewpoint video including the second viewpoint video, and the second viewpoint video, and output the generated display information to a type of device different from the VR display device.

[0019] Furthermore, a maintenance training support method according to another aspect of the present invention is a maintenance training support method executed by a maintenance training support device, wherein the maintenance training support device performs a display information generation step of generating display information displaying a three-dimensional VR (Virtual Reality) space in which objects specific to maintenance work are arranged, and generating viewpoint images of multiple users sharing the VR space, the display information generation step including the steps of generating a first viewpoint image which is a viewpoint image of a first user and a second viewpoint image which is a viewpoint image seen from a viewpoint different from the first viewpoint image, and generating VR display information in which the second viewpoint image is included in the first viewpoint image, and displaying the VR display information on a VR display device owned by the first user.

[0020] Furthermore, a program according to another aspect of the present invention is a program that causes a computer to function as a maintenance training support device, and causes the computer to function as a display information generation unit that generates display information displaying a three-dimensional VR (Virtual Reality) space in which objects specific to maintenance work are arranged, and generates viewpoint images of multiple users sharing the VR space, causes the display information generation unit to generate a first viewpoint image that is a viewpoint image of a first user and a second viewpoint image that is a viewpoint image seen from a viewpoint different from the first viewpoint image, generates VR display information in which the second viewpoint image is included in the first viewpoint image, and displays the VR display information on a VR display device owned by the first user. Effect of the Invention

[0021] According to the present invention, it is possible to support the implementation of training on maintenance work of electronic equipment more efficiently and effectively in virtual reality and augmented reality.

[0022] Problems, configurations, effects, and the like other than those described above will become apparent from the description of the following embodiments. [Brief description of the drawings]

[0023] [Figure 1] FIG. 1 is a diagram illustrating an example of a schematic configuration of a maintenance training support system. [Diagram 2] FIG. 2 is a block diagram showing an example of a functional configuration of a maintenance training support device. [Diagram 3] FIG. 11 is a diagram showing an example of object management information. [Figure 4] FIG. 11 is a diagram showing an example of interference information. [Diagram 5] FIG. 11 is a diagram showing an example of pair information. [Figure 6] FIG. 4 is a diagram showing an example of position information. [Figure 7] FIG. 11 is a diagram showing an example of VR space information. [Figure 8] 11 is a diagram showing an example of login space management information; [Figure 9]FIG. 13 shows an example of a VR space for conducting maintenance work training. [Figure 10] FIG. 11 is a diagram showing an example of an exchange operation of an electronic device. [Figure 11] FIG. 2 is a diagram illustrating an example of a hardware configuration of a maintenance training support device. [Figure 12] FIG. 11 is a flow diagram showing an example of a maintenance training support process. [Figure 13] FIG. 13 shows an example of a VR space in which a different viewpoint window is displayed. [Figure 14] FIG. 13 shows an example of a VR space in which a plurality of different viewpoint windows are displayed. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0025] <Outline of the configuration of the maintenance training support system 1000> 1 is a diagram showing an example of a schematic configuration of a maintenance training support system 1000 according to this embodiment. As shown in the figure, the maintenance training support system 1000 has a maintenance training support device 100, a computer 150, a VR goggles 160, a controller 170 and a display device 180, and a computer 151 and a display device 181. It is assumed that the computer 150, the VR goggles 160, the controller 170 and the display device 180, and the computer 151 and the display device 181 are used by different users of the maintenance training support system 1000, respectively.

[0026] The VR goggles 160 and the controller 170 are connected to the computer 150 so as to be able to communicate with each other via wireless communication based on a predetermined wireless communication standard (e.g., Bluetooth (registered trademark) or the like) or a wired cable. In addition, the computers 150 and 151 are each connected to the maintenance training support device 100 via the network N so as to be able to communicate with each other.

[0027] The network N may be, for example, the Internet, an intranet, a Wide Area Network (WAN), a mobile phone network, or a combination of these. The network N may also be a Virtual Private Network (VPN) on a wireless communication network such as a mobile phone network.

[0028] In such a maintenance training support system 1000, the computer 150 acquires the VR display information (three-dimensional image) generated by the maintenance training support device 100 and performs 3D rendering. The computer 150 outputs the rendered VR display information to the VR goggles, so that the VR space for training the maintenance work is displayed on the VR goggles 160. The maintenance training support device 100 may generate display information for expressing various types of augmented reality (so-called XR (Cross Reality)) such as AR (Augmented Reality) and MR (Mixed Reality) in addition to the VR display information. That is, the maintenance training support device 100 can be applied to various types of augmented reality (XR: Cross Reality) including VR, AR, and MR. In this embodiment, a case of VR is taken as an example for explanation.

[0029] The VR goggles 160 and the controller 170 each transmit information indicating the user's actions (hereinafter, sometimes referred to as "action information") acquired from a camera or a sensor mounted thereon to the maintenance training assistance device 100, and feed back the user's movements in the VR space. In addition, the maintenance training assistance device 100 that has acquired the feedback generates display information of the VR space that has changed in conjunction with the user's actions, and displays it on the VR goggles 160 via the computer 150.

[0030] By repeatedly executing such processing, the user can practice maintenance work on electronic devices and the like in the VR space.

[0031] <VRゴーグル160> The VR goggles 160 are head-mounted displays that display VR display information of three-dimensional images created by, for example, 3DCG. The VR goggles 160 display the VR display information on a display mounted on the VR goggles 160 so that objects in the VR space can be seen three-dimensionally by utilizing the parallax between the user's eyes.

[0032] The VR goggles 160 also has a sensor that detects the user's head movement (e.g., direction, tilt, etc.) and a plurality of cameras that are used to detect the user's hand movement, and transmits the detected operation information to the maintenance training assistance device 100 via the computer 150. Specifically, the VR goggles 160 performs head tracking using a sensor or a camera, and transmits the tracking data as operation information to the maintenance training assistance device 100. When the user does not use the controller 170, the VR goggles 160 performs hand tracking using a camera, and transmits operation information indicating the user's hand movement (hand gesture) to the maintenance training assistance device 100.

[0033] There are various types of such VR goggles 160, such as a standalone type in which a computer (information processing device) built into the VR goggles 160 itself renders VR display information to display a VR space, and a type in which a smartphone is fitted and used. Note that in this embodiment, as shown in FIG. 1, the following description will be given using as an example a type in which VR display information rendered by a computer 150 connected in a communicable manner is displayed on a display mounted on the VR goggles 160.

[0034] <Controller 170> The controller 170 is a device that detects the movement of the user's hand in the VR space (the direction and amount of movement of the hand, etc.) and receives various operation instructions from the user, such as grabbing, releasing, or touching an object. In addition, the controller 170 receives, for example, an operation instruction from the user for a predetermined window displayed in the user's field of view displayed on the VR goggles 160 as VR display information. Specifically, the controller 170 has multiple types of sensors, such as a motion sensor and an acceleration sensor, for detecting the user's movements and operations. In addition, the controller 170 has multiple instruction receiving buttons for receiving operation instructions.

[0035] Furthermore, the controller 170 transmits, via the computer 150, operation information including the user's hand movements detected by the sensor and operation instructions acquired via the instruction acceptance button to the maintenance training assistance device 100.

[0036] By using such VR goggles 160 and controller 170, a user can perform various types of maintenance work operations in the VR space in which the object to be worked on is located, such as grasping an object, moving the object while grasping it, or releasing the grasped object.

[0037] Furthermore, the user can perform various operations on a window (another viewpoint window) displayed within the user's field of view by using the VR goggles 160 and the controller 170. For example, the user can perform various operations on the other viewpoint window, which displays an image seen from a viewpoint different from the user's viewpoint, via the VR goggles 160 and the controller 170, such as selecting the type of image displayed in the window, adjusting the angle, and moving and resizing the other viewpoint window.

[0038] <Calculator 150, 151> The computers 150 and 151 are devices that display information on the VR space in which maintenance work training is performed on the VR goggles 160 or the display devices 180 and 181, and are realized by, for example, a PC (Personal Computer). The computer 150 may be a device such as a tablet terminal or a smartphone as long as it satisfies the required processing specifications. The computer 150 may also be, for example, an information processing device built into the standalone VR goggles 160.

[0039] The computer 150 performs 3D rendering when it acquires the VR display information, which is a three-dimensional image, from the maintenance training support device 100. Specifically, the computer 150 performs real-time rendering to generate a three-dimensional image in real time using the VR display information acquired from the maintenance training support device 100. The computer 150 also outputs the VR display information, which is a three-dimensional image generated by the 3D rendering, to the VR goggles 160.

[0040] In addition, when the computers 150 and 151 display the user's training status in the VR space on the display devices 180 and 181, they convert the rendered VR display information into display information for display (two-dimensional image) and output this to the display devices 180 and 181.

[0041] In addition, the computer 150 performs a process of transmitting the operation information acquired from the VR goggles 160 and the controller 170 to the maintenance training assistance device 100.

[0042] <Display device 180, 181> The display devices 180 and 181 are display devices such as displays that display two-dimensional images. The display devices 180 and 181 display display information acquired from the computers 150 and 151 or the maintenance training support device 100.

[0043] <Maintenance training support device 100> The maintenance training support device 100 is a device that supports a user's maintenance training in a VR space, and is realized by, for example, a server computer, a cloud server, or a personal computer. Specifically, the maintenance training support device 100 generates VR display information of a three-dimensional image in which a user trains in maintenance work, and displays the VR display information on VR goggles 160 worn by the user who performs the training, via a computer 150.

[0044] In addition, the maintenance training assistance device 100 generates display information for the VR space that changes in conjunction with the user's actions based on operation information acquired from the VR goggles 160 and the controller 170, and displays the information on the VR goggles 160 via the computer 150.

[0045] <<Details of the maintenance training support device 100>> 2 is a block diagram showing an example of a functional configuration of the maintenance training assistance device 100. As shown in the figure, the maintenance training assistance device 100 has a processing unit 110, a storage unit 120, and a communication unit .

[0046] The processing unit 110 is a functional unit that performs various processes executed by the maintenance training support device 100. Specifically, the processing unit 110 has a display information generating unit 111, an action identifying unit 112, a login space managing unit 113, a viewpoint setting unit 114, and an action storage unit 115 as functional units for executing each process.

[0047] <<<Display information generation unit 111>>> The display information generating unit 111 is a functional unit that generates VR display information. Specifically, the display information generating unit 111 generates VR display information that shows, in a three-dimensional image, a state in which training for maintenance work is performed in a VR space, using various information related to objects and the like stored in the storage unit 120 and operation information acquired from the VR goggles 160 and the controller 170. In addition, the display information generating unit 111 transmits the VR display information to the computer 150 via the communication unit 130.

[0048] The display information generating unit 111 may convert the VR display information into display information for display on the display devices 180 and 181, and transmit this to the computers 150 and 151 via the communication unit .

[0049] Furthermore, the display information generating unit 111 displays, in addition to the user's own viewpoint video (a video of the VR space seen from the user's viewpoint, hereinafter sometimes referred to as the "first viewpoint video"), a video of the same VR space seen from a viewpoint other than the first viewpoint video (hereinafter sometimes referred to as the "second viewpoint video") within the VR display information. Specifically, the display information generating unit 111 generates VR display information in which a different viewpoint window showing the second viewpoint video is included at a predetermined position within the user's field of view. The types of information displayed in the different viewpoint window will be described later.

[0050] <<<Operation identification unit 112>>> The motion identification unit 112 is a functional unit that identifies the motion of the user and the motion of the object in the VR space. The motion identification unit 112 accepts an input from the controller 170 or an input from the VR goggles 160 (indicating the contents of the hand gesture recognized by the VR goggles 160) to identify the motion of the user. Specifically, the motion identification unit 112 uses the motion information accepted from the controller 170 or the VR goggles 160 to identify the motion of the user in the VR space (for example, a motion of the hand or a motion of grabbing or releasing an object) and the motion of the object, such as the position of the object that changes with the motion of the user, the distance between the objects, or the interference between the objects.

[0051] More specifically, the action identification unit 112 identifies the user's action and the object's movement in the VR space using the action information and various information in the storage unit 120 that manages the object's position, physical relationships such as interference, linked objects, etc. based on a global coordinate system set in the VR space and a local coordinate system set for each object. Also, the action identification unit 112 identifies the user's operation on the another viewpoint window based on the display position of the another viewpoint window displayed in the user's field of view and the user's action information.

[0052] The method of identifying the user's actions and object movements in the VR space is not particularly limited, and any known VR technology may be used, for example.

[0053] <<<Login space management unit 113>>> The login space management unit 113 is a functional unit that manages the VR space in which the user is logged in. Specifically, the login space management unit 113 registers the identification information of the VR space in which the user is logged in and the identification information of the user in association with each other in the login space management information 127.

[0054] <<<Viewpoint setting unit 114>>> The viewpoint setting unit 114 is a functional unit that sets a second viewpoint in a predetermined case. Specifically, the viewpoint setting unit 114 sets the second viewpoint for obtaining an overhead image or an image showing the vicinity of the target device of the maintenance training.

[0055] <<<Operation Memory Unit 115>>> The action storage unit 115 is a functional unit that stores the actions of the user in the VR space. Specifically, the action storage unit 115 stores the actions of the user in the VR space indicated by the action information as action storage information. That is, the action storage unit 115 stores the history of the action information for each user in the action storage information.

[0056] <<<Storage section 120>>> Next, the storage unit 120 will be described. The storage unit 120 is a functional unit that stores various information. Specifically, the storage unit 120 has an object management information DB (Database) 121 in which objects specific to the maintenance work and their operations are defined, interference information 122 in which information on objects having physical relationships such as interference is registered, pair information 123 in which objects having interlocking operations are registered, position information 124 in which the latest positions of each object in the VR space are stored, scenario information 125 in which procedures and work instructions for maintenance work training are registered, VR space information 126 in which information on individual VR spaces is registered, login space management information 127 in which users logged in to the VR space are managed, and operation storage information 128 in which operation history for each user in the VR space is stored.

[0057] 3 is a diagram showing an example of object management information stored in the object management information DB 121. The object information is information that defines the type, position, and action of an object. Specifically, the object management information has records in which items such as identification information, type, coordinate position, and special object flag are associated with each other.

[0058] The identification information is information that uniquely identifies an object. The type is information that identifies the type of object, and there are various types such as a rack, which is an object specific to a maintenance work, an electronic device, a rack door, a cable, a power cord, and a warning label.

[0059] The coordinate position is coordinate information that indicates the position of an object in a VR space. The coordinate position includes both the position of an object specified in a global coordinate system set in the VR space and the position of an object specified in a local coordinate system set for each object.

[0060] The special object flag is information indicating objects that can be moved during maintenance work training (for example, electronic devices, racks, rack doors, cables, power cords, warning labels, etc.). Specifically, an object with a "1" registered in the special object flag corresponds to a special object. Note that objects that are not targets for movement during maintenance work training have a "0" registered in the flag.

[0061] In addition to these items, the object management information also includes information defining operational characteristics (for example, in the case of a rack door, opening and closing around one of the X to Z axes in the local coordinate system) (not shown).

[0062] 4 is a diagram showing an example of the interference information 122. The interference information 122 is information indicating an interference state between objects. Specifically, the interference information 122 has records in which items such as identification information, an interfering object, and an interfering portion are associated with each other. The identification information is information for uniquely identifying an object. The interfering object is an object that is a colliding target when viewed from an object identified by the identification information, and the identification information of the object is registered. The interfering portion is information for identifying a portion that is interfering (in contact) with the interfering object, and is, for example, information identified by a local coordinate system.

[0063] 5 is a diagram showing an example of pair information 123. Pair information 123 is information in which objects in a linked relationship are registered. Specifically, pair information 123 registers information on a parent object and a child object that moves in conjunction with the movement of the parent object. More specifically, pair information 123 has records in which items such as identification information of a parent object, identification information of a child object, and a coordinate position are associated with each other.

[0064] The identification information of the parent object and the child object are information that uniquely identifies the parent object such as a rack or a rack door (e.g., the object on which the child object is affixed or attached) and the child object such as an electronic device or a warning label (e.g., the object on which the child object is affixed or attached). The coordinate position is coordinate information that specifies the position of the child object relative to the parent object, and is information that is specified by a local coordinate system, for example. The type of object that corresponds to the parent object and the type of object that corresponds to the child object are defined by predetermined information (not shown), and the information may be stored in advance in the storage unit 120. Also, for example, a special object may be treated as a child object, and a non-special object may be treated as a parent object.

[0065] In addition, whether objects that are linked to each other are parent objects or child objects can be determined based on priority information (not shown) that is preset for each object according to a predetermined viewpoint (e.g., difficulty of performing an action, etc.).

[0066] FIG. 6 is a diagram showing an example of the position information 124. The position information 124 is information indicating the position of an object after it has moved due to maintenance work training by a user, that is, the latest position in the VR space. Specifically, the position information 124 has records in which items such as identification information and coordinate positions are associated. The identification information is information that uniquely identifies an object. The coordinate position is coordinate information that indicates the latest position of an object, and is information specified by a global coordinate system and a local coordinate system.

[0067] The scenario information 125 is information in which maintenance work training procedures, work instructions, etc. are registered. Specifically, the scenario information 125 registers, in association with each other, scenarios corresponding to a plurality of types of training purposes, such as replacement of the electronic device 201 or change of the mounting position, maintenance work procedures and work instructions corresponding to each scenario, and identification information of a plurality of VR spaces corresponding to each scenario.

[0068] In addition, the work instructions are information that indicates work instructions that follow maintenance work procedures, such as "Please identify the rack to be worked on by pointing and calling out" or "Please attach a work-in-progress label (warning label) to the rack to be worked on."

[0069] FIG. 7 is a diagram showing an example of the VR space information 126. The VR space information 126 is information about the configuration of the VR space. Specifically, the VR space information 126 has a record in which VR space identification information and items related to object information are associated with each other. The VR space identification information is information that uniquely identifies the VR space. The object information is information about an object specific to the associated VR space, and has items such as identification information, type, coordinate position, and special object flag, similar to the object management information.

[0070] 8 is a diagram showing an example of the login space management information 127. The login space management information 127 is information for managing a VR space in which a user is logged in, and is updated by the login space management unit 113 every time a user logs in or logs out. Specifically, the login space management information 127 has a record in which user identification information and login space identification information are associated with each other. The user identification information is information that uniquely identifies a user. The login space identification information is identification information of a VR space in which a user is logged in.

[0071] The motion memory information 128 is information that stores the user's motions in the VR space. Specifically, the motion memory information 128 stores, as its history, motion information indicating the user's past motions in the VR space. That is, the motion memory information 128 has the identification information of the user as the motion subject and the past motion information associated and registered therein. Note that the motion memory information 128 is used when the display information generation unit 111 generates a replay video that reproduces the user's motions.

[0072] Note that each piece of information in the storage unit 120 is an example, and the type, content, and association relationship of the information are not limited to the above examples as long as the information can specify physical relationships such as the position, movement, and interference of objects in the VR space, associated objects, and the position after movement.

[0073] <<<Communication unit 130>>> Returning to FIG. 2 for explanation, the communication unit 130 is a functional unit that performs information communication with an external device. Specifically, the communication unit 130 performs information communication with the computers 150 and 151 and acquires the motion information output from the VR goggles 160 and the controller 170. Further, the communication unit 130 transmits the VR display information or the display information for the display generated by the display information generation unit 111 to the computers 150 and 151.

[0074] <An example of the VR space displayed on the VR goggles 160> FIG. 9 is a diagram showing an example of a VR space for performing maintenance work training. As shown in the figure, in the VR space for maintenance work training, for example, objects specific to maintenance work such as a rack 200, an electronic device 201, and a warning label 202, a work instruction window 203, and the user's hand 204 (the shape of the hand 204 in the illustrated example is simplified) are displayed.

[0075] Each object such as the rack 200 and the work instruction window 203 are displayed in the VR space based on the information stored in the storage unit 120. Also, the user's hand 204 is expressed at a corresponding position in the VR space based on the operation information acquired from the VR goggles 160 and the controller 170.

[0076] 10 is a diagram showing an example of a replacement operation of the electronic device 201. The maintenance work includes, for example, a work of replacing a broken electronic device 201, and a work of changing the mounting position of the electronic device 201. The processing unit 110 enables the user to practice such a maintenance work-specific operation in the VR space.

[0077] Specifically, the display information generating unit 111 generates display information of the VR space in which the rack 200 on which the electronic device 201 is mounted is arranged, using each piece of information in the storage unit 120. Furthermore, the action identifying unit 112 calculates the distance from the user's hand 204 to the electronic device 201 to be worked on, based on the action information and each piece of information in the storage unit 120. Furthermore, when an action instruction to grip is input when the distance is less than a predetermined value, the action identifying unit 112 generates VR display information of the action in which the user's hand 204 grips the electronic device 201 to be worked on, via the display information generating unit 111.

[0078] Furthermore, the action identification unit 112 calculates the distance from the user's hand 204 holding the electronic device 201 to the surface of an object such as the rack 200 based on the action information and each piece of information in the storage unit 120. Furthermore, when the action identification unit 112 detects that an action instruction to release the electronic device 201 has been input when the distance is less than a predetermined value, the display information generation unit 111 generates VR display information in a state in which the electronic device 201 is mounted in a mountable position in the rack 200 close to the user's hand 204.

[0079] Through such processing, the user can practice operations specific to maintenance work, such as replacing or changing the position of the electronic device 201, in the VR space.

[0080] Note that the object related to the operation specific to the maintenance work is not limited to the electronic device 201, but may also be a cable, a power cord, or other objects. Even in this case, the maintenance training support device 100 (particularly, the processing unit 110) can perform the same processing as above to enable training of the user in the operation specific to the maintenance work, and can support the implementation of efficient and effective training.

[0081] <Hardware configuration> 11 is a diagram showing an example of a hardware configuration of the maintenance training support device 100. As shown in the figure, the maintenance training support device 100 has a processor 310, a memory 320, a storage 330, and a communication device 340, and each component is connected by a bus 350. In addition, the power source (not shown) may be a primary battery or a rechargeable secondary battery, which can be replaced when worn out or deteriorated.

[0082] The processor 310 is an arithmetic device such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), and executes processing according to a program recorded in the memory 320 or the storage 330. In the maintenance training support device 100, processing is performed by the processor 310 that operates according to a program read onto the memory 320 or the storage 330. The processing unit 110 realizes each function by the processor 310 executing the program.

[0083] The memory 320 is a storage device such as a RAM (Random Access Memory) or a flash memory, and functions as a storage area from which programs and data are temporarily read. The storage 330 is a writable and readable storage device. The functions of the storage unit 120 are realized by the memory 320 or the storage 330. The functions of the storage unit 120 may be realized by a storage device connected via the communication device 340.

[0084] The communication device 340 is an interface for connecting the maintenance training support device 100 to an external device for communication.

[0085] The processing of each component of the maintenance training assistance device 100 may be executed by one piece of hardware or by multiple pieces of hardware. Also, the processing of each component of the maintenance training assistance device 100 may be realized by one program or by multiple programs.

[0086] <Maintenance training support processing> 12 is a flow diagram showing an example of the maintenance training support process. The process starts when the display information generating unit 111 receives an instruction to start the process from a user. The user starts the process by selecting, for example, a combination of a scenario and a VR space that the user wants to train in from the scenario information 125.

[0087] When the process starts, the login space management unit 113 updates the login space management information 127 (step S10). Specifically, the login space management unit 113 identifies the VR space into which the user has logged in from the selected scenario information 125. In addition, the login space management unit 113 updates the login space management information 127 by registering the user's identification information and the identification information of the identified VR space in the login space management information 127.

[0088] Next, the display information generating unit 111 generates VR display information from the user's viewpoint and displays it on the VR goggles 160 of the corresponding user (step S20). Specifically, the display information generating unit 111 uses the VR space information 126 and the scenario information 125 to identify information such as objects, object placement, maintenance work procedures, and work instructions corresponding to the VR space selected by the user. In addition, the display information generating unit 111 uses the identified information to generate VR display information in which various objects, etc. are placed at predetermined positions defined in the VR space information 126.

[0089] At this time, the display information generating unit 111 uses the login space management information 127 to determine whether or not another user is logged in to the same VR space as the VR space into which the user is logged in. If the display information generating unit 111 determines that another user is logged in to the same VR space, it identifies the coordinate position of the other user in the VR space using the VR space information 126 and the motion information acquired from the other user. If the other user is within the user's field of view, the display information generating unit 111 generates VR display information including the avatar of the other user.

[0090] Furthermore, the display information generating unit 111 transmits the generated VR display information to the computer 150 via the communication unit 130, and causes the VR display information as seen from each user's viewpoint to be displayed on each user's VR goggles 160. As a result, when multiple users are simultaneously logged into the same VR space, that is, when multiple users share the same VR space, when each user comes into view of another user, the avatar of the other user is displayed on the VR goggles 160 of each user in the VR space as seen from the user's viewpoint.

[0091] In addition, when the display information generation unit 111 acquires operation information via the communication unit 130, it generates display information of the VR space for each user based on the operation information and each information in the memory unit 120, and transmits the VR display information to the computer 150 via the communication unit 130.

[0092] As a result, an image from the user's viewpoint that reflects the user's movements in the VR space is displayed on each user's VR goggles 160. Also, when another user sharing the same VR space moves, VR display information that reflects the movement within the user's field of view is displayed on the VR goggles 160.

[0093] Next, the motion identification unit 112 judges whether or not a first motion of the user has been detected based on the acquired motion information and each piece of information in the storage unit 120 (step S30). Specifically, the motion identification unit 112 judges whether or not motion information indicating a display instruction for a different viewpoint window that displays a video (hereinafter, sometimes referred to as a "second viewpoint video") seen from a viewpoint (second viewpoint) other than the user's own viewpoint (first viewpoint) has been acquired. Note that the method of issuing the display instruction for the different viewpoint window is not particularly limited. For example, it may be an operation that is assigned in advance as a command operation indicating a display instruction for the different viewpoint window, such as a predetermined operation using the controller 170 or a hand gesture.

[0094] If it is determined that the motion information has been acquired (Yes in step S30), the motion identification unit 112 proceeds to step S40. On the other hand, if it is determined that the motion information has not been acquired (No in step S30), the motion identification unit 112 returns the process to step S10.

[0095] In step S40, the display information generating unit 111 accepts the type of the second viewpoint video to be displayed in the different viewpoint window. The types of the second viewpoint video include viewpoint video of another user, an overhead video including the user and the target device of the maintenance training, a viewpoint video of the target device of the maintenance training, and a video replay of the maintenance training.

[0096] The display information generating unit 111 displays the type candidates of the second viewpoint video in a selectable manner in the VR space and accepts the selection from the user. Note that the type candidate of the second viewpoint video selected by the user may be identified by the action identifying unit 112 based on the action information.

[0097] Next, the display information generating unit 111 generates VR display information including a different viewpoint window displaying the second viewpoint video of the accepted type, and displays it on the user's VR goggles 160 (step S50).

[0098] The different viewpoint window is not arranged as an object in the VR space, but is display information displayed at a predetermined position in the user's field of view. For example, when the user moves in the VR space, the user's field of view changes accordingly, but the different viewpoint window is always displayed at a predetermined position in the user's field of view (for example, the lower left or lower right of the field of view) regardless of the user's movement. However, the different viewpoint window may be an object arranged at a predetermined coordinate position in the VR space.

[0099] When the type of the second viewpoint video is a viewpoint video of another user, the display information generating unit 111 first identifies the other user. Specifically, the display information generating unit 111 identifies the identification information of the other user who is simultaneously logged in to the same VR space, for example, by using the login space management information 127. In addition, the display information generating unit 111 displays the identified identification information in the VR space in a selectable manner, thereby accepting the selection by the user. However, the method of accepting the selection from the user is not limited to this, and may be, for example, a voice input or another method.

[0100] The display information generating unit 111 also generates VR display information in which a viewpoint image of another user selected by the user, i.e., VR display information displayed on the VR goggles of the other user, is displayed in a different viewpoint window as a second viewpoint image, and the different viewpoint window is displayed at a predetermined position within the user's field of view. The display information generating unit 111 also displays the generated VR display information on the VR goggles 160 of the corresponding user.

[0101] Furthermore, when the type of the second viewpoint video is an overhead video including the user and the training target equipment, the viewpoint setting unit 114 first sets the viewpoint (second viewpoint) of the overhead video. When setting the viewpoint, the viewpoint setting unit 114 acquires, for example, information identifying the target equipment of the maintenance training from the action identification unit 112. The action identification unit 112 identifies the target equipment of the maintenance training on which the user is working, based on the action information.

[0102] Moreover, the viewpoint setting unit 114 sets the viewpoint of the overhead video at a predetermined height and angle for overlooking both the user and the identified target device based on the coordinate position of the user in the VR space and the coordinate position of the identified target device. Furthermore, the display information generating unit 111 generates VR display information in which the overhead video seen from the set viewpoint is displayed in a different viewpoint window as a second viewpoint video, and displays this on the VR goggles 160 of the user.

[0103] In addition, the overhead video may be displayed transparently to allow the user to see the state of the target device at hand. In this case, the display information generating unit 111 identifies the object to be displayed transparently from the object management information, and displays the identified object transparently.

[0104] Moreover, when the type of the second viewpoint video is a viewpoint video near the target device of the maintenance training, the viewpoint setting unit 114 first sets the second viewpoint near the target device. Specifically, the viewpoint setting unit 114 acquires information for identifying the target device from the action identification unit 112. Moreover, the viewpoint setting unit 114 sets the second viewpoint at a predetermined position (for example, a position facing the target device at a distance where the distance from the second viewpoint to the target device is less than a predetermined distance) that is closer to the target device than at least the viewpoint (first viewpoint) of the user. Moreover, the display information generating unit 111 generates VR display information in which the state of the maintenance training for the target device as seen from the set viewpoint is displayed in a different viewpoint window as the second viewpoint video, and displays this on the VR goggles 160 of the user.

[0105] Furthermore, when the type of the second viewpoint video is a video of a replay of a maintenance training, the display information generating unit 111 first identifies the subject of the video to be replayed. Specifically, the display information generating unit 111 identifies the identification information of other users who are simultaneously logged in to the same VR space, for example, by using the login space management information 127. Furthermore, the display information generating unit 111 selectively displays the identification information of the other users and the user's own identification information in the VR space, and accepts the selection of the subject of the replay video, including the user himself.

[0106] The display information generating unit 111 also generates a video for replay playback using the action storage information 128 associated with the identification information of the selected user. The display information generating unit 111 also displays the generated replay video in a different viewpoint window, and displays VR display information including the different viewpoint window on the VR goggles 160 of the user.

[0107] The replay video may be played back at a predetermined speed set by the user (for example, 1.1 to 1.5 times the normal speed). By playing back at a speed of 1.0 times or faster, the user's viewing time can be shortened, and the efficiency of maintenance training can be improved.

[0108] 13 is a diagram showing an example of a VR space in which a different viewpoint window is displayed. As shown in the figure, in addition to a first viewpoint video seen from the user's own viewpoint, a different viewpoint window (the displayed video content is omitted) displaying a second viewpoint video at a predetermined position in the user's field of view (lower left in the illustrated example) is displayed on the user's VR goggles.

[0109] The shape and display position of the different viewpoint window are not limited.

[0110] Next, the action identifying unit 112 determines whether or not a second action has been detected (step S60). Specifically, when the user action identified based on the action information is an operation on a different viewpoint window, the action identifying unit 112 determines that the second action has been detected.

[0111] Here, operations on the different viewpoint window include, for example, changing the display position of the different viewpoint window, changing the window size, enlarging (zooming) the displayed second viewpoint video, changing the angle, playing, stopping, rewinding, and fast-forwarding the replay video, etc.

[0112] It should be noted that when the action identification unit 112 identifies from the action information that a specific position (e.g., any of the four corners) of the another viewpoint window has been grasped, that the position has been moved while being grasped, and that the grasped state has been released, it determines that a second action related to a change in the display position of the another viewpoint window has been detected.

[0113] Furthermore, when the action identifying unit 112 identifies from the action information an action of widening or narrowing the window frame while the edge of the different viewpoint window is being grabbed, it determines that a second action related to changing the window size has been detected.

[0114] Furthermore, when the action identification unit 112 identifies from the action information that, for example, an operation equivalent to pinching in / out has been performed with the user's hand position overlapping a different viewpoint window, it determines that it has detected a second action related to the enlargement / reduction of the second viewpoint image displayed in that window.

[0115] Furthermore, when the action identification unit 112 identifies from the action information that, for example, the user's hand is overlapped with the different viewpoint window and an action of sliding vertically on the second viewpoint video is performed, it determines that a second action relating to changing the angle of the second viewpoint video displayed in the window has been detected.

[0116] Furthermore, when the action identification unit 112 identifies from the action information that a specified operation has been performed on a different viewpoint window while a replay video is being played or stopped in that window, the display information generation unit 111 determines that a second action related to replay playback has been detected in response to that operation.

[0117] If it is determined that the second movement has been detected (Yes in step S60), movement identification unit 112 proceeds to step S70. On the other hand, if it is determined that the second movement has not been detected (No in step S60), movement identification unit 112 returns the process to step S10.

[0118] In step S70, the display information generating unit 111 generates VR display information including a different viewpoint window reflecting the second operation, and displays it on the corresponding user's VR goggles 160. After performing the process of step S70, the display information generating unit 111 returns the process to step S10.

[0119] When the processing of steps S30 and S40 is repeated and an instruction to display another viewpoint window is received again, the display information generating unit 111 performs processing to add another viewpoint window. That is, a plurality of another viewpoint windows can be simultaneously displayed within the user's field of view.

[0120] 14 is a diagram showing an example of a VR space in which a plurality of different viewpoint windows are displayed. As shown in the figure, in addition to a first viewpoint video seen from the user's own viewpoint, a first different viewpoint window displaying a second viewpoint video and a second different viewpoint window (both images are omitted) are displayed at predetermined positions (lower left and lower right in the example shown) in the user's field of view on the user's VR goggles. Note that the different viewpoint windows are not objects placed in the VR space, so they are displayed in front of the first viewpoint video and are not hidden by objects placed in the VR space.

[0121] In this way, by making it possible to simultaneously display multiple different viewpoint windows within the user's field of vision, the user can further improve the efficiency of maintenance training. For example, if multiple users performing maintenance training and an instructor checking / guiding their progress are in the same VR space at the same time, the instructor can simultaneously check the progress of each user's work by simultaneously displaying multiple viewpoint images of each user as second viewpoint images.

[0122] In addition, different types of second viewpoint images can be displayed in each of the multiple different viewpoint windows. Therefore, for example, by simultaneously displaying the expert's viewpoint and the user's own replay video, the user can review his / her own work while checking the procedures and tips of the expert, and can carry out maintenance training more efficiently.

[0123] The maintenance training support process has been described above.

[0124] According to such a maintenance training support device, it is possible to support the implementation of training on maintenance work of electronic devices more efficiently and effectively in virtual reality and augmented reality.

[0125] In particular, since the user can check the video from a viewpoint other than the user's own viewpoint, the maintenance training can be carried out more efficiently. For example, the user who performs the maintenance training can train while checking the video from the viewpoint of another user, an expert, as the second viewpoint video, and therefore the level of mastery of the maintenance training can be improved.

[0126] In addition, a user who is undergoing maintenance training can display an overhead image or an image of the target device from a closer distance as a second viewpoint image, so that, for example, it becomes easier to confirm the operation of another user, an expert. In addition, a user who is undergoing maintenance training can train while checking replay images of the expert and himself as the second viewpoint image, so that he can repeatedly check the procedures and tips of the expert or the appearance of his own work.

[0127] Furthermore, an instructor or an expert user can check the viewpoint video of another user who is conducting maintenance training as the second viewpoint video. Also, an instructor or an expert user can check the overhead video of another user who is conducting maintenance training, a video of the target device seen from nearby, or a replay video as the second viewpoint video. Therefore, the user can give appropriate advice to another user who is conducting maintenance training.

[0128] In addition, the user can perform various operations on the second viewpoint video, such as changing the display position, changing the display size, enlarging (zooming), changing the angle, etc. Therefore, the user can display the second viewpoint video in a more easily viewable form.

[0129] In this way, the maintenance training support device can provide various types of users, such as maintenance trainers, experts, and instructors, with a variety of convenient functions for supporting maintenance training.

[0130] The present invention is not limited to the above embodiment, and various modifications are possible. For example, when a stand-alone VR goggle with a built-in computer is used as the maintenance training support device, the controller communicates with the stand-alone VR goggle and transmits user operation information to the maintenance training support device via the VR goggle.

[0131] Also, only the second viewpoint video may be displayed on the display devices 180 and 181. For example, the computers 150 and 151 or the maintenance training support device 100 (display information generating unit 111) generate display information for display of the second viewpoint video (two-dimensional video) based on a predetermined setting or instruction received from a user, and output this to the display devices 180 and 181. That is, the display devices 180 and 181 may display only the first viewpoint video, may display the first viewpoint video including the second viewpoint video, or may display only the second viewpoint video.

[0132] In addition, the above-mentioned embodiment examples have been described in detail to make the present invention easier to understand, and the present invention is not limited to those having all the configurations described here. In addition, it is possible to replace a part of the configuration of one example of an embodiment with the configuration of another example. In addition, it is also possible to add the configuration of another example to the configuration of one example of an embodiment. In addition, it is also possible to add, delete, or replace other configurations with respect to a part of the configuration of each example of an embodiment. In addition, each of the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in hardware by designing them as integrated circuits, for example, in part or in whole. In addition, the control lines and information lines in the figures show those that are considered necessary for explanation, and do not necessarily show all of them. It may be considered that almost all of the configurations are connected to each other. [Explanation of symbols]

[0133] 1000...maintenance training support system, 100...maintenance training support device, 110...processing unit, 111...display information generating unit, 112...action identifying unit, 113...login space managing unit, 114...viewpoint setting unit, 115...action memory unit, 120...memory unit, 121...object management information DB, 122...interference information, 123...pair information, 124...position information, 125...system Scenario information, 126...VR space information, 127...Login space management information, 128...Operation memory information, 130...Communication unit, 150, 151...Computer, 160...VR goggles, 170...Controller, 180, 181...Display device, 310...Processor, 320...Memory, 330...Storage, 340...Communication device, 350...Bus, N...Network

Claims

1. A display information generating unit that generates viewpoint images of a plurality of users sharing the VR space, the display information generating unit generating display information for displaying a three-dimensional VR (Virtual Reality) space in which objects specific to the maintenance work are arranged, The display information generating unit generating a first viewpoint video which is a viewpoint video of a first user and a second viewpoint video which is a viewpoint video seen from a viewpoint different from the first viewpoint video; generating VR display information including the second viewpoint video in the first viewpoint video, and displaying the VR display information on a VR display device owned by the first user; A maintenance training support device comprising:

2. The maintenance training support device according to claim 1, The display information generating unit A viewpoint image of a second user different from the first user is generated as the second viewpoint image. A maintenance training support device comprising:

3. The maintenance training support device according to claim 1, The display information generating unit A bird's-eye view image of the object viewed from a predetermined height and angle in the VR space is generated as the second viewpoint image. A maintenance training support device comprising:

4. The maintenance training support device according to claim 1, The display information generating unit A viewpoint image when a certain target object is viewed from a viewpoint closer to the target object than a viewpoint of the first user is generated as the second viewpoint image. A maintenance training support device comprising:

5. The maintenance training support device according to any one of claims 1 to 4, The display information generating unit generating the second viewpoint image as viewed from an angle specified by the first user; A maintenance training support device comprising:

6. The maintenance training support device according to any one of claims 1 to 4, The display information generating unit generating the second viewpoint video in which an action of the first user or a second user other than the first user with respect to the object is replayed; A maintenance training support device comprising:

7. The maintenance training support device according to claim 6, The display information generating unit generating the second viewpoint video replayed at a playback speed designated by the first user; A maintenance training support device comprising:

8. The maintenance training support device according to any one of claims 1 to 4, The display information generating unit The VR display information is generated such that a plurality of the second viewpoint images are simultaneously included in the first viewpoint image. A maintenance training support device comprising:

9. The maintenance training support device according to any one of claims 1 to 4, The display information generating unit Generate display information corresponding to at least one of the first viewpoint video, the first viewpoint video including the second viewpoint video, and the second viewpoint video, and output the generated display information to a device of a type different from the VR display device. A maintenance training support device comprising:

10. A maintenance training support method executed by a maintenance training support device, comprising: The maintenance training support device includes: A display information generating step is performed to generate display information for displaying a three-dimensional virtual reality (VR) space in which objects specific to the maintenance work are arranged, the display information generating step generating viewpoint images of a plurality of users who share the VR space; The display information generating step includes: A step of generating a first viewpoint video which is a viewpoint video of a first user and a second viewpoint video which is a viewpoint video seen from a viewpoint different from the first viewpoint video; generating VR display information in which the second viewpoint video is included in the first viewpoint video, and displaying the VR display information on a VR display device owned by the first user. A maintenance training support method comprising:

11. A program for causing a computer to function as a maintenance training support device, The computer, a display information generating unit that generates a three-dimensional virtual reality (VR) space in which an object specific to the maintenance work is arranged, the display information generating unit generating viewpoint images of a plurality of users who share the VR space; The display information generating unit, generating a first viewpoint video which is a viewpoint video of a first user and a second viewpoint video which is a viewpoint video seen from a viewpoint different from the first viewpoint video; generating VR display information including the second viewpoint video in the first viewpoint video, and displaying the VR display information on a VR display device owned by the first user; A program characterized by:

Images