Maintenance training support device, maintenance training support system, maintenance training support method and program

Abstract

To support more efficient and effective training for maintenance work of electronic apparatuses within a virtual reality.SOLUTION: A maintenance training device comprises: a display information generation unit configured to generate augmented reality display information for displaying a three-dimensional augmented reality space in which objects specific to maintenance work are arranged; a viewpoint setting unit configured to selectively accept and set a viewpoint position of the augmented reality display information; and a storage unit configured to store motion information which records the three-dimensional motion of a worker performing maintenance work in the augmented reality space in a time-series manner. The display information generation unit uses the motion information to generate augmented reality display information that reproduces the three-dimensional motion of the worker as viewed from the viewpoint position set by the viewpoint setting unit within the augmented reality space in a time-series manner.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a maintenance training support device, a maintenance training support system, a maintenance training support method, and a program. [Background technology]

[0002] In recent years, video content such as games that use VR (Virtual Reality) technology is on the rise. VR is a virtual reality video world created using, for example, 3DCG (Dimensional Computer Graphics) technology, and VR users such as gamers can have various experiences in the virtual reality.

[0003] There is also content that utilizes the characteristics of VR to provide vocational training. For example, Patent Document 1 discloses that agricultural crop cultivation training is provided in a VR space. Specifically, Patent Document 1 discloses a system for educating and training agricultural crop cultivation techniques, which includes a training scenario storage unit that stores a work target part for the agricultural crop displayed in a virtual space and a work means that is a tool used for the work target part in association with a work procedure, a three-dimensional display control unit that displays the agricultural crop and the work means in three dimensions on a display device as the work procedure progresses, a detection signal output unit that detects the position and movement of a user or an operating device and outputs a detection signal, an operation detection unit that changes the work means displayed in the virtual space based on the detection signal, and an operation determination unit that determines the result of the operation based on the coordinates of the work target part in the virtual space, the coordinates of the work means, and the operation content. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2022-149636 A Summary of the Invention [Problem to be solved by the invention]

[0005] In VR content for vocational training, display information for the virtual reality world that changes in response to user actions is usually generated based on information that defines objects and their movements specific to the training subject, and information acquired from the VR goggles and controller. Therefore, for example, when training on maintenance work for electronic devices, etc. is conducted using VR content, information that defines objects specific to the maintenance work is required.

[0006] In addition, it may be preferable to display information in the VR space during training for maintenance work to assist the user in performing maintenance work-specific actions, and even when displaying such information, predefined information is required.

[0007] The technology of Patent Document 1 is for carrying out education and training on agricultural crop cultivation in a VR space. Therefore, the technology of this document does not take into consideration objects specific to maintenance work such as electronic devices and their movements.

[0008] The present invention has been made in consideration of the above points, and has an object to support the implementation of training on maintenance work for electronic devices more efficiently and effectively in virtual reality. [Means for solving the problem]

[0009] The present application includes a number of means for solving at least part of the above problems, examples of which are as follows.

[0010] In order to solve the above problems, a maintenance training support device according to one embodiment of the present invention comprises a display information generation unit that generates augmented reality display information for displaying a three-dimensional augmented reality space in which objects specific to maintenance work are arranged, a viewpoint setting unit that selectively accepts and sets a viewpoint position of the augmented reality display information, and a memory unit that stores action information that chronologically records three-dimensional actions of a worker performing the maintenance work in the augmented reality space, and the display information generation unit uses the action information to generate the augmented reality display information that chronologically reproduces the three-dimensional actions of the worker as viewed from the viewpoint position set by the viewpoint setting unit in the augmented reality space.

[0011] In addition, in the above-mentioned maintenance training support device, the operation information may be stored in the memory unit for each step of the maintenance work, and the display information generation unit may reproduce the operation in the step of the specified maintenance work.

[0012] In the above-mentioned maintenance training support device, one or more of the viewpoint positions received by the viewpoint setting unit may be provided in advance for each step of the maintenance work.

[0013] In addition, in the above-mentioned maintenance training support device, when the maintenance work is an on-site hazard prediction activity, the display information generation unit may reproduce and display three-dimensional movements of a worker performing the maintenance work as movements of at least the fingers of a humanoid avatar.

[0014] In addition, in the above-mentioned maintenance training support device, when the maintenance work step is disassembly, assembly, or replacement of a specified device, the operation information also records the position, appearance, and shape of the device or part held by the worker performing the maintenance work, and the display information generation unit may generate the augmented reality display information that transparently displays some or all of the three-dimensional operations of the worker performing the maintenance work when the maintenance work step is disassembly, assembly, or replacement of the specified device or part, and reproduces the state of the device or part held by the worker in a manner that does not obstruct visibility from the viewpoint position.

[0015] In addition, the above-mentioned maintenance training support device may include an action identification unit that identifies three-dimensional actions of a user in the augmented reality space in a time series, and a trace determination unit that determines whether or not the three-dimensional action of the worker related to the action information and the three-dimensional action of the user identified by the action identification unit are similar to each other to a predetermined degree or more.

[0016] In addition, in the above-mentioned maintenance training support device, the trace determination unit may divide the three-dimensional movement of the worker related to the movement information into a plurality of intervals on a time series, and if the three-dimensional movement of the worker and the three-dimensional movement of the user in a section are similar to each other by a predetermined amount or more, reproduce the three-dimensional movement of the worker in the subsequent section in the time series, and if the three-dimensional movement of the worker and the three-dimensional movement of the user in a section are not similar to each other by a predetermined amount or more, reproduce the three-dimensional movement of the worker in that section again.

[0017] Another aspect of the present invention provides a maintenance training support system having a maintenance training support device and augmented reality goggles used by a user, the maintenance training support device comprising: a display information generation unit that generates augmented reality display information for displaying a three-dimensional augmented reality space in which objects specific to the maintenance work are arranged; a viewpoint setting unit that selectively accepts and sets a viewpoint position of the augmented reality display information; and a memory unit that stores action information that chronologically records three-dimensional actions of a worker performing the maintenance work within the augmented reality space, and the display information generation unit uses the action information to generate the augmented reality display information that chronologically reproduces the three-dimensional actions of the worker as viewed from the viewpoint position set by the viewpoint setting unit in the augmented reality space, and displays it to the user.

[0018] Furthermore, a maintenance training support method according to another aspect of the present invention is a maintenance training support method performed by a maintenance training support device, the maintenance training support device having a memory unit that stores operation information that records three-dimensional operations of a worker performing maintenance work in a chronological order, and performs a display information generating step of generating augmented reality display information for displaying a three-dimensional augmented reality space in which objects specific to the maintenance work are arranged, and a viewpoint setting step of selectively accepting and setting a viewpoint position of the augmented reality display information, in which the operation information is used to generate augmented reality display information that reproduces the three-dimensional operations of the worker in the chronological order as viewed from the viewpoint position set in the viewpoint setting step in the augmented reality space, and displays it to a user.

[0019] In addition, a maintenance training support program according to another aspect of the present invention is a program that causes a computer to function as a maintenance training support device, and causes the computer to function as a display information generation unit that generates augmented reality display information for displaying a three-dimensional augmented reality space in which objects specific to maintenance work are arranged, a viewpoint setting unit that selectively accepts and sets a viewpoint position of the augmented reality display information, and a memory unit that stores action information that chronologically records three-dimensional actions of a worker performing the maintenance work in the augmented reality space, and causes the display information generation unit to use the action information to generate the augmented reality display information that chronologically reproduces the three-dimensional actions of the worker viewed from the viewpoint position set by the viewpoint setting unit in the augmented reality space. Effect of the Invention

[0020] According to the present invention, it is possible to support the implementation of training on maintenance work of electronic devices more efficiently and effectively in virtual reality.

[0021] Problems, configurations, effects, and the like other than those described above will become apparent from the description of the following embodiments. [Brief description of the drawings]

[0022] [Figure 1] FIG. 1 is a diagram illustrating an example of a schematic configuration of a maintenance training support system. [Diagram 2] FIG. 2 is a block diagram showing an example of a functional configuration of a maintenance training support device. [Diagram 3] FIG. 11 is a diagram showing an example of object management information. [Figure 4] FIG. 11 is a diagram showing an example of operation information. [Diagram 5] FIG. 4 is a diagram showing an example of location information. [Figure 6] FIG. 13 shows an example of a VR space for conducting maintenance work training. [Figure 7] FIG. 11 is a diagram showing an example of an exchange operation of an electronic device. [Figure 8] FIG. 2 is a diagram illustrating an example of a hardware configuration of a maintenance training support device. [Figure 9] FIG. 11 is a flow diagram showing an example of a maintenance training support process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0024] <Outline of the configuration of the maintenance training support system 1000> 1 is a diagram showing an example of a schematic configuration of a maintenance training support system 1000 according to this embodiment. As shown in the figure, the maintenance training support system 1000 includes a maintenance training support device 100, a computer 150, an augmented reality goggles 160, and a controller 170.

[0025] The augmented reality goggles 160 and the controller 170 are connected to the computer 150 so as to be able to communicate with each other via wireless communication based on a predetermined wireless communication standard (e.g., Bluetooth (registered trademark)) or wired communication via a cable, and the computer 150 is connected to the maintenance training support device 100 so as to be able to communicate with each other via the network N.

[0026] The network N may be, for example, the Internet, an intranet, a Wide Area Network (WAN), a mobile phone network, or a combination of these. The network N may also be a Virtual Private Network (VPN) on a wireless communication network such as a mobile phone network.

[0027] In such a maintenance training support system 1000, the computer 150 acquires the augmented reality display information generated by the maintenance training support device 100, performs 3D rendering, and outputs the rendered augmented reality display information to the augmented reality goggles 160, thereby displaying an augmented reality space for training in maintenance work on the augmented reality goggles 160. Note that the maintenance training support device 100 is not limited to VR display information, and may generate information on various augmented realities (so-called XR) such as AR (Augmented Reality) and MR (Mixed Reality). Therefore, the application to augmented reality shown in this embodiment can be applied to various XRs such as VR, AR, and MR.

[0028] Furthermore, the augmented reality goggles 160 and the controller 170 each transmit information indicating the actions of a user (user: a person receiving training using the maintenance training support system 1000) (hereinafter, sometimes referred to as "action information") acquired from a camera or sensor mounted thereon to the maintenance training support device 100, and feed back the user's movements in the augmented reality space. Furthermore, the maintenance training support device 100, which has acquired the feedback, generates display information of the augmented reality space that has changed in conjunction with the user's actions, and displays it on the augmented reality goggles 160 via the computer 150.

[0029] By repeatedly executing such processing in succession, the user can practice maintenance work on electronic devices and the like in the augmented reality space.

[0030] <Augmented Reality Goggles 160> The augmented reality goggles 160 are head-mounted displays that display augmented reality display information of three-dimensional images created, for example, by 3DCG (3 Dimensional Computer Graphics). The augmented reality goggles 160 display the augmented reality display information on a display mounted on the augmented reality goggles 160 so that objects in an augmented reality space appear three-dimensional by utilizing the parallax between the user's eyes.

[0031] The augmented reality goggles 160 also have a sensor for detecting the user's head movement (e.g., direction, tilt, etc.), a sensor for detecting the user's line of sight, and multiple cameras used for detecting the user's hand movement, and transmit the detected operation information to the maintenance training assistance device 100 via the computer 150. Specifically, the augmented reality goggles 160 perform head tracking using a sensor or a camera, and transmit the tracking data as operation information to the maintenance training assistance device 100. When the user does not use the controller 170, the augmented reality goggles 160 perform hand tracking using a camera, and transmit operation information indicating the user's hand movement (hand gesture) to the maintenance training assistance device 100.

[0032] There are various types of augmented reality goggles 160, such as a standalone type that displays an augmented reality space by rendering augmented reality display information by itself, and a type that is used by inserting a smartphone into the goggles. In this embodiment, the following explanation will be given using as an example a type that displays an augmented reality space on a display using augmented reality display information rendered by a computer 150 that is communicatively connected.

[0033] <Controller 170> The controller 170 is a device that detects the movement of the user's hand in the augmented reality space (such as the direction and amount of hand movement) and receives operation instructions from the user, such as grabbing or releasing an object. Specifically, the controller 170 has multiple types of sensors, such as a motion sensor and an acceleration sensor. The controller 170 also has multiple instruction receiving buttons that receive operation instructions.

[0034] Furthermore, the controller 170 transmits, via the computer 150, operation information including the user's hand movements detected by the sensor and operation instructions acquired via the instruction acceptance button to the maintenance training assistance device 100.

[0035] By using such augmented reality goggles 160 and controller 170, a user can perform various types of maintenance work-related actions in the augmented reality space in which the object to be worked on is located, such as grasping an object, moving the object while grasping it, or releasing the grasped object.

[0036] <Calculator 150> The computer 150 is a device that displays display information of the augmented reality space in which maintenance work training is performed on the augmented reality goggles 160, and is realized by, for example, a PC (Personal Computer). Note that the computer 150 may be a device such as a tablet terminal or a smartphone as long as it satisfies the required processing specifications.

[0037] The computer 150 performs 3D rendering when it acquires the augmented reality display information, which is a three-dimensional image, from the maintenance training support device 100. Specifically, the computer 150 performs real-time rendering to generate a three-dimensional image in real time using the augmented reality display information acquired from the maintenance training support device 100. The computer 150 also outputs the augmented reality display information, which is a three-dimensional image generated by the 3D rendering, to the augmented reality goggles 160.

[0038] In addition, the computer 150 performs a process of transmitting operation information acquired from the augmented reality goggles 160 and the controller 170 to the maintenance training assistance device 100.

[0039] <Maintenance training support device 100> The maintenance training support device 100 is a device that supports a user's maintenance training in an augmented reality space, and is realized by, for example, a server computer, a cloud server, or a personal computer. Specifically, the maintenance training support device 100 generates augmented reality display information of a three-dimensional image in which a user trains in maintenance work, and displays the augmented reality display information on augmented reality goggles 160 worn by the user performing the training via a computer 150.

[0040] In addition, the maintenance training support device 100 generates display information of the augmented reality space (e.g., an avatar that resembles the user) that changes in conjunction with the user's actions based on operation information acquired from the augmented reality goggles 160 and the controller 170, and displays it on the augmented reality goggles 160 via the calculator 150.

[0041] <<Details of the maintenance training support device 100>> 2 is a block diagram showing an example of a functional configuration of the maintenance training assistance device 100. As shown in the figure, the maintenance training assistance device 100 has a processing unit 110, a storage unit 120, and a communication unit .

[0042] The processing unit 110 is a functional unit that performs various processes executed by the maintenance training support device 100. Specifically, the processing unit 110 has a display information generating unit 111, an action identifying unit 112, a viewpoint setting unit 113, and a trace determining unit 114 as functional units for executing each process.

[0043] <<<Display information generation unit 111>>> The display information generating unit 111 is a functional unit that generates augmented reality display information. Specifically, the display information generating unit 111 uses various information related to objects and the like stored in the storage unit 120 and operation information acquired from the augmented reality goggles 160 and the controller 170 to generate augmented reality display information that time-series displays a state of training for maintenance work in an augmented reality space as a three-dimensional image viewed from a predetermined viewpoint position. The display information generating unit 111 also transmits the augmented reality display information to the computer 150 via the communication unit 130. The display information generating unit 111 also reproduces the operation of the worker recorded in the designated maintenance work step. Note that the worker here is a person who performs an operation that serves as a model for a person (user) who is to be trained using the maintenance training support system 1000.

[0044] Furthermore, when the maintenance work is an on-site danger prediction activity, the display information generating unit 111 reproduces and displays the three-dimensional movements of the worker performing the maintenance work as at least the movements of the fingers of the humanoid avatar. On-site danger prediction activity (also called on-site KY) is a danger prediction performed while observing the actual object and current state on-site immediately before the actual work begins, and a detailed countermeasure is taken by setting a scene or situation in which a specific on-site object changes. Therefore, since a specific object is often specified by pointing or line of sight, and the direction of movement or amount of change of the object is often indicated by gesture, the effect can be enhanced by reproducing a specific pointing, line of sight (for example, a method of indicating the pointing direction or line of sight direction with a straight light such as an arrow or a laser pointer), gesture, etc. As a method of performing this with augmented reality, a humanoid avatar is displayed and the avatar's movements are visually recognized from the outside, thereby obtaining an amount of information close to reality.

[0045] In addition, when the maintenance work step is the replacement of a specified device or part, the display information generation unit 111 generates augmented reality display information that reproduces the state of the device or part held by the worker by displaying part or all of the worker performing the maintenance work transparently, for example by displaying the part semi-transparently, that is, by using a so-called ghost display.

[0046] <<<Operation identification unit 112>>> The motion identification unit 112 is a functional unit that identifies a user's motion or object motion in the augmented reality space. The motion identification unit 112 accepts an input from the controller 170 or an input from the augmented reality goggles 160 (indicating the contents of a hand gesture recognized by the augmented reality goggles 160) to identify the user's motion. Specifically, the motion identification unit 112 uses motion information accepted from the controller 170 or the augmented reality goggles 160 to identify the user's motion in the augmented reality space (for example, a hand motion or a motion such as grabbing or releasing an object) and the object motion such as the position of an object that changes with the user's motion, the distance between objects, or interference between objects.

[0047] More specifically, the action identification unit 112 identifies the user's actions and object movements in the augmented reality space using the action information and various information in the memory unit 120 that manages the positions of objects, physical relationships such as interference, and linked objects based on a global coordinate system set in the augmented reality space.

[0048] The method for identifying the user's actions and object movements in the augmented reality space is not particularly limited, and any known VR technology may be used, for example.

[0049] <<<Viewpoint setting unit 113>>> The viewpoint setting unit 113 selectively accepts and sets the viewpoint position of the augmented reality display information. Specifically, one or more viewpoint positions accepted by the viewpoint setting unit 113 are set in advance for each step of the maintenance work, or are set in the process in which the display information generating unit 111 generates the augmented reality display information.

[0050] <<<Trace determination unit 114>>> The trace determination unit 114 determines whether the three-dimensional motion of the worker (the person who performs the model motion of the work) related to the motion information is similar to the three-dimensional motion of the user (the person who receives training using the maintenance training support system 1000) identified by the motion identification unit 112 by a predetermined amount or more. The trace determination unit 114 divides the three-dimensional motion of the worker related to the motion information into a plurality of sections in a time series, and when the three-dimensional motion of the worker and the three-dimensional motion of the user in the section are similar to each other by a predetermined amount or more, reproduces the three-dimensional motion of the worker in the subsequent section in a time series. When the three-dimensional motion of the worker and the three-dimensional motion of the user in the section are not similar to each other by a predetermined amount or more, the three-dimensional motion of the worker in the section is reproduced again. That is, when the person receiving the training cannot trace the model motion, the trace determination unit 114 does not allow the trainee to proceed to the next section and allows the trainee to repeatedly train the motion of the same section.

[0051] <<<Storage section 120>>> Next, the storage unit 120 will be described. The storage unit 120 is a functional unit that stores various information. Specifically, the storage unit 120 has an object management information DB (Database) 121 in which objects specific to the maintenance work and their operations are defined, operation information 122 in which the three-dimensional operations of a worker performing the maintenance work in the augmented reality space are recorded in time series for each step of the maintenance work, position information 123 in which the latest positions of each object in the augmented reality space are stored, and scenario information 124 in which the procedures and work instructions for the maintenance work training are registered.

[0052] 3 is a diagram showing an example of object management information stored in the object management information DB 121. The object information is information that defines the type, position, and action of an object. Specifically, the object management information has records in which items such as identification information, type, coordinate position, and special object flag are associated with each other.

[0053] The identification information is information that uniquely identifies an object. The type is information that identifies the type of object, and there are various types such as a rack, which is an object specific to a maintenance work, an electronic device, a rack door, a cable, a power cord, and a warning label.

[0054] The coordinate position is coordinate information that indicates the position of an object in the augmented reality space, and includes the position of the object specified in a global coordinate system that is set in the augmented reality space.

[0055] The special object flag is information that indicates special objects (for example, electronic devices, rack doors, cables, power cords, warning labels, etc.) that can be moved during maintenance work training. Specifically, an object with a "1" registered in the special object flag corresponds to a special object. Note that for objects such as racks that are not subject to movement during maintenance work training, a "0" is registered in the flag.

[0056] In addition to these items, the object management information also includes information defining operational characteristics (for example, in the case of a rack door, opening and closing around one of the X to Z axes in the local coordinate system) (not shown).

[0057] 4 is a diagram showing an example of the motion information 122. The motion information 122 is information in which three-dimensional motions of a worker performing maintenance work in an augmented reality space are recorded in time series for each step of the maintenance work. Specifically, the motion information 122 has records in which items for specifying a motion, such as a motion step, a left arm motion, a head motion, a line of sight, and a right arm motion, and an item of a playback viewpoint for specifying a standard viewpoint for playing back and displaying the motion are associated with each other.

[0058] A work step is information that uniquely identifies a series of predetermined actions into which a work is subdivided. The left arm action, head action, line of sight, and right arm action are information that records changes in the position, orientation, moving direction, moving speed, etc., of each in a time series, and are, for example, information that records the actions of a worker acquired by the augmented reality goggles 160 and the controller 170. When a maintenance work step is disassembly / assembly or replacement work of a specified device, the action information also records the position, appearance, and shape of the device or part held by the worker performing the maintenance work.

[0059] The playback viewpoint includes not only the viewpoint position when the worker's actions are played back and displayed, but also information specifying whether or not to display the worker's avatar transparently. When the worker is holding a work object, there is a risk that the work object may be obstructed from being seen, and therefore it is effective to display part or all of the action transparently using the worker's avatar.

[0060] 5 is a diagram showing an example of the position information 123. The position information 123 is information indicating the position of an object after it has moved due to maintenance work training by a user, that is, the latest position in an augmented reality space. Specifically, the position information 123 has records in which items such as identification information and coordinate positions are associated with each other. The identification information is information that uniquely identifies an object. The coordinate position is coordinate information that indicates the latest position of an object, and is information specified by a global coordinate system.

[0061] The scenario information 124 is information in which maintenance work training procedures, work instructions, etc. are registered. Specifically, the scenario information 124 registers scenarios corresponding to a plurality of types of training objectives, such as replacement of the electronic device 201 or change of the mounting position, and information such as objects, object placement, maintenance work procedures, and work instructions corresponding to each scenario.

[0062] The work instructions are information indicating work instructions following the procedures of the maintenance work, such as, for example, "Please identify the rack to be worked on by pointing and calling it out," "Please attach a work-in-progress label (attention label) to the rack to be worked on," etc. The work instructions may basically be the same as the work steps, but are not limited to this, and a work instruction may be made up of multiple work steps, or a work step may be made up of multiple work instructions.

[0063] Furthermore, each piece of information in the memory unit 120 is merely an example, and the type, content, and linking relationship of the information are not limited to the above examples as long as the information can identify physical relationships such as the position, movement, and interference of objects in the augmented reality space, objects that are linked, and their positions after movement.

[0064] <<<Communication unit 130>>> Returning to Fig. 2, the communication unit 130 is a functional unit that performs information communication with an external device. Specifically, the communication unit 130 performs information communication with the computer 150, and acquires operation information output from the augmented reality goggles 160 and the controller 170. In addition, the communication unit 130 transmits the augmented reality display information generated by the display information generation unit 111 to the computer 150.

[0065] <An example of an augmented reality space displayed on the augmented reality goggles 160> 6 is a diagram showing an example of an augmented reality space for carrying out maintenance work training. As shown in the figure, the augmented reality space for maintenance work training displays objects specific to the maintenance work, such as a rack 200, an electronic device 201, and a warning label 202, a work instruction window 203, and a user's hand 204 (in the example shown in the figure, the shape of the hand 204 is simplified, and the hand is shown in a distinguishable state, such as whether it is open, clenched, or gripped).

[0066] Each object such as the rack 200 and the work instruction window 203 are displayed in the augmented reality space based on the information stored in the above-mentioned storage unit 120. Also, the user's hand 204 is represented in a corresponding position in the augmented reality space based on the motion information acquired from the augmented reality goggles 160 and the controller 170, in a transparent (semi-transparent) manner so as not to obstruct the visibility of the object being held.

[0067] <Display of maintenance operation specific operations and auxiliary information> FIG. 7 is a diagram showing an example of the replacement operation of the electronic device 201. The maintenance work includes, for example, the operation of replacing a broken electronic device 201 and the operation of changing the mounting position of the electronic device 201. The processing unit 110 enables the user to practice such an operation specific to the maintenance work in the augmented reality space. Specifically, the display information generating unit 111 uses each piece of information in the storage unit 120 to generate augmented reality display information in which the electronic device 201 is placed at a predetermined position (for example, in the rack 200) in the augmented reality space. In addition, the action identifying unit 112 calculates the distance from the user's left hand 204L and the user's right hand 204R to the electronic device 201 based on the action information and each piece of information in the storage unit 120. In addition, when an action instruction to grip is input when the distance is less than a predetermined distance, the action identifying unit 112 generates augmented reality display information of the action of the user's left hand 204L and the user's right hand 204R gripping the electronic device 201 via the display information generating unit 111.

[0068] Furthermore, the action identification unit 112 calculates the distance from the user's left hand 204L and the user's right hand 204R holding the electronic device 201 to an object such as the rack 200 based on the action information and each piece of information in the storage unit 120. Furthermore, when the action identification unit 112 detects that an action instruction to release the electronic device 201 has been input when the distance is less than a predetermined value, the display information generation unit 111 generates augmented reality display information of a state in which the electronic device 201 is attached to an object such as the rack 200 close to the user's left hand 204L and the user's right hand 204R.

[0069] Furthermore, the motion identification unit 112 calculates the distance from a predetermined feature point (an invisible part such as a button, switch, corner, side, or center of gravity) of the electronic device 201 to the user's right hand 204R based on the motion information and each piece of information in the storage unit 120. Furthermore, when the distance is less than a predetermined distance, the display information generation unit 111 generates augmented reality display information in a state in which the user's right hand 204R is pointing at a position of the electronic device 201 close to the user's right hand 204R.

[0070] Through such processing, the user can practice, within the augmented reality space, the actions specific to maintenance work, such as installing the electronic device 201 and pointing and confirming the work.

[0071] Note that the objects relating to operations specific to maintenance work are not limited to the attention label 202 and the electronic device 201, but may also include cables, power cords, or other objects. Even in this case, the maintenance training support device 100 (particularly, the processing unit 110) can perform the same processing as above to enable training of the user in operations specific to maintenance work, and can support more efficient and effective training by displaying auxiliary information.

[0072] Through such processing, the processing unit 110 can express more realistic object movements in the augmented reality space.

[0073] <Hardware configuration> 8 is a diagram showing an example of a hardware configuration of the maintenance training support device 100. As shown in the figure, the maintenance training support device 100 has a processor 310, a memory 320, a storage 330, and a communication device 340, and each component is connected by a bus 350. In addition, the power source (not shown) may be a primary battery or a rechargeable secondary battery, which can be replaced when worn out or deteriorated.

[0074] The processor 310 is an arithmetic device such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), and executes processing according to a program recorded in the memory 320 or the storage 330. In the maintenance training support device 100, processing is performed by the processor 310 that operates according to a program read onto the memory 320 or the storage 330. The processing unit 110 realizes each function by the processor 310 executing the program.

[0075] The memory 320 is a storage device such as a RAM (Random Access Memory) or a flash memory, and functions as a storage area from which programs and data are temporarily read. The storage 330 is a writable and readable storage device. The functions of the storage unit 120 are realized by the memory 320 or the storage 330. The functions of the storage unit 120 may be realized by a storage device connected via the communication device 340.

[0076] The communication device 340 is an interface for connecting the maintenance training support device 100 to an external device for communication.

[0077] The processing of each component of the maintenance training assistance device 100 may be executed by one piece of hardware or by multiple pieces of hardware. Also, the processing of each component of the maintenance training assistance device 100 may be realized by one program or by multiple programs.

[0078] <Maintenance training support processing> 9 is a flow diagram showing an example of the maintenance training support process. The process is started when the display information generating unit 111 receives an instruction to start the process from a user. At this time, the user selects a specific scenario that the user wants to train from the scenario information 124 and starts the process.

[0079] When the process starts, the display information generating unit 111 generates augmented reality display information (step S10). Specifically, the display information generating unit 111 uses the scenario information 124 to identify information such as objects, object placement, maintenance work steps, and work instructions corresponding to the scenario selected by the user. The display information generating unit 111 also generates display information of the augmented reality space in which the identified objects, etc. are placed.

[0080] In addition, when the display information generation unit 111 acquires operation information via the communication unit 130, it generates display information of an augmented reality space that changes in response to the user's operation based on the operation information and each piece of information in the memory unit 120, and transmits the augmented reality display information to the computer 150 via the communication unit 130.

[0081] Next, the display information generating unit 111 determines whether or not an instruction to play back the work contents has been issued (step S20). Specifically, the display information generating unit 111 determines whether or not an instruction to play back the work contents has been accepted via the controller 170 or the computer 150. If an instruction to play back the work contents has not been issued ("No" in step S20), the display information generating unit 111 executes step S20 again.

[0082] When an instruction to play back the work content is given ("Yes" in step S20), the display information generating unit 111 reads out the record of the corresponding work step from the action information 122, and converts it into augmented reality display information from the specified viewpoint (step S30). At that time, the display information generating unit 111 displays the worker's action (teacher action) on coordinates on the augmented reality display information using 3DCG as seen from the playback viewpoint of the action information 122 read out by the viewpoint setting unit 113, or from a selectively accepted playback viewpoint. Furthermore, when the playback viewpoint is "with transparency," the display information generating unit 111 displays the worker's body transparently (displayed semi-transparently or as a line drawing).

[0083] Then, the display information generating unit 111 judges whether or not an instruction to start tracing judgment has been given (step S40). Specifically, the display information generating unit 111 judges whether or not an instruction to start tracing judgment has been received via the controller 170 or the computer 150. If an instruction to start tracing judgment has not been given (if "No" in step S40), the display information generating unit 111 executes step S20 again to continue playing back the teacher action in 3DCG as seen from the playback viewpoint of the action information 122 or a selectively received playback viewpoint.

[0084] When an instruction to start tracing determination has been given ("Yes" in step S40), the motion identification unit 112 identifies the motion of the user (student) based on information acquired from the augmented reality goggles 160 and the controller 170 (step S50).

[0085] Then, the trace determination unit 114 determines whether the student's motion is similar to the teacher's motion by a predetermined amount or more (step S60). Specifically, the trace determination unit 114 divides the three-dimensional motion of the worker (teacher's motion) related to the motion information 122 into a plurality of sections in a time series, and determines whether the three-dimensional motion of the worker and the three-dimensional motion of the user (student's motion) in the sections are similar to each other by a predetermined amount or more. In this process, the trace determination unit 114 determines that the motion is similar if the difference between the motion of the worker (teacher's motion) and the amount of change in the movement direction, movement speed, head and line of sight direction, etc. of the user's arm is less than a predetermined amount. However, the present invention is not limited to this, and the trace determination unit 114 may determine the similarity of the motions by various algorithms.

[0086] If the student's actions are not similar to the teacher's actions to a certain extent ("No" in step S60), the tracing determination unit 114 displays a message such as "Retry is required", re-creates the three-dimensional movements of the worker in the section displayed in step S60, and returns control to step S50.

[0087] If the student's action is similar to the teacher's action to a certain degree ("Yes" at step S60), the trace determination unit 114 proceeds to the next task step (step S70). Specifically, the trace determination unit 114 displays a message such as "You have mastered it sufficiently. Let's proceed to the next task step," reproduces the three-dimensional action of the worker in the section following the section displayed at step S60, and returns control to step S50.

[0088] In this way, if the user can execute an action similar to the teacher's action in the augmented reality space, the user can proceed with the work steps, and if the user cannot execute an action similar to the teacher's action in the augmented reality space, the user can repeat the action. Therefore, the user can efficiently train in maintenance work. For example, the user can actually move his / her body alone without an instructor and proceed with the training by repeating the action until the user can execute the appropriate action, so that more detailed training is possible. In addition, since the training can be performed in the augmented reality space, the training can be performed without being restricted by the physical environment (time, place, equipment, opportunity, etc.).

[0089] The maintenance training support process has been described above.

[0090] The present invention is not limited to the above embodiment, and various modifications are possible. For example, the maintenance training support device may use standalone augmented reality goggles instead of a computer when the augmented reality goggles have a function of executing the same processing as a computer. The controller communicates information with the standalone augmented reality goggles and transmits user operation information to the maintenance training support device via the augmented reality goggles.

[0091] In addition, the above-mentioned embodiment examples have been described in detail to make the present invention easier to understand, and the present invention is not limited to those having all the configurations described here. In addition, it is possible to replace a part of the configuration of one example of an embodiment with the configuration of another example. In addition, it is also possible to add the configuration of another example to the configuration of one example of an embodiment. In addition, it is also possible to add, delete, or replace other configurations with respect to a part of the configuration of each example of an embodiment. In addition, each of the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in hardware by designing them as integrated circuits, for example, in part or in whole. In addition, the control lines and information lines in the figures show those that are considered necessary for explanation, and do not necessarily show all of them. It may be considered that almost all of the configurations are connected to each other. [Explanation of symbols]

[0092] 1000...maintenance training support system, 100...maintenance training support device, 110...processing unit, 111...display information generation unit, 112...action identification unit, 113...viewpoint setting unit, 114...trace determination unit, 120...storage unit, 121...object management information DB, 122...action information, 123...position information, 124...scenario information, 130...communication unit, 310...processor, 320...memory, 330...storage, 340...communication device, 350...bus, N...network

Claims

1. a display information generating unit that generates augmented reality display information for displaying a three-dimensional augmented reality space in which an object specific to the maintenance work is arranged; a viewpoint setting unit that selectively receives and sets a viewpoint position of the augmented reality display information; a storage unit that stores motion information that records, in time series, three-dimensional motions of a worker performing the maintenance work in the augmented reality space; The display information generating unit generating, using the motion information, the augmented reality display information that reproduces, in time series, a three-dimensional motion of the worker as viewed from the viewpoint position set by the viewpoint setting unit in the augmented reality space; A maintenance training support device comprising:

2. The maintenance training support device according to claim 1, the operation information is stored in the storage unit for each step of the maintenance work, The display information generating unit reproduces the operation in the step of the designated maintenance work. A maintenance training support device comprising:

3. The maintenance training support device according to claim 1, One or more viewpoint positions accepted by the viewpoint setting unit are provided in advance for each step of the maintenance work. A maintenance training support device comprising:

4. The maintenance training support device according to claim 1, When the maintenance work is an on-site risk prediction activity, the display information generation unit reproduces and displays a three-dimensional movement of a worker performing the maintenance work as a movement of at least the fingers of a hand of a humanoid avatar. A maintenance training support device comprising:

5. The maintenance training support device according to claim 1, When the step of the maintenance work is a disassembly, assembly or replacement work of a predetermined device, the operation information also records the position, appearance, and shape of the device or part held by a worker performing the maintenance work, When the step of the maintenance work is disassembly, assembly, or replacement of the specified device or part, the display information generation unit generates the augmented reality display information that transparently displays a part or all of a three-dimensional movement of a worker performing the maintenance work, and reproduces a state of the device or part held by the worker in a manner that does not obstruct visibility from the viewpoint position. A maintenance training support device comprising:

6. The maintenance training support device according to claim 1, a motion identification unit that identifies a three-dimensional motion of a user in the augmented reality space in a time series; a trace determination unit that determines whether or not the three-dimensional motion of the worker related to the motion information and the three-dimensional motion of the user identified by the motion identification unit are similar to each other by a predetermined amount or more; A maintenance training support device comprising:

7. The maintenance training support device according to claim 6, the trace determination unit divides the three-dimensional movement of the worker related to the movement information into a plurality of sections in a time series, and when the three-dimensional movement of the worker and the three-dimensional movement of the user in a section are similar to each other by a predetermined amount or more, reproduces the three-dimensional movement of the worker in the subsequent section in the time series, and when the three-dimensional movement of the worker and the three-dimensional movement of the user in a section are not similar to each other by a predetermined amount or more, reproduces the three-dimensional movement of the worker in the section again. A maintenance training support device comprising:

8. A maintenance training support system having a maintenance training support device and augmented reality goggles used by a user, The maintenance training support device includes: a display information generating unit that generates augmented reality display information for displaying a three-dimensional augmented reality space in which an object specific to the maintenance work is arranged; a viewpoint setting unit that selectively receives and sets a viewpoint position of the augmented reality display information; a storage unit that stores motion information that records, in time series, three-dimensional motions of a worker performing the maintenance work in the augmented reality space; The display information generating unit using the motion information, generate augmented reality display information that reproduces in time series the three-dimensional motion of the worker as seen from the viewpoint position set by the viewpoint setting unit in the augmented reality space, and display the augmented reality display information to a user. A maintenance training support system comprising:

9. A maintenance training support method performed by a maintenance training support device, comprising: The maintenance training support device includes: a storage unit that stores motion information that records three-dimensional motions of a worker performing maintenance work in a time series; a display information generating step of generating augmented reality display information for displaying a three-dimensional augmented reality space in which the object specific to the maintenance work is arranged; a viewpoint setting step of selectively accepting and setting a viewpoint position of the augmented reality display information; In the display information generating step, using the motion information, generate augmented reality display information that reproduces in time series the three-dimensional motion of the worker as seen from the viewpoint position set in the viewpoint setting step in the augmented reality space, and display the augmented reality display information to a user. A maintenance training support method comprising:

10. A program for causing a computer to function as a maintenance training support device, The computer, a display information generating unit that generates augmented reality display information for displaying a three-dimensional augmented reality space in which an object specific to the maintenance work is arranged; a viewpoint setting unit that selectively receives and sets a viewpoint position of the augmented reality display information; a storage unit that stores motion information that records in time series three-dimensional motions of a worker performing the maintenance work in the augmented reality space; The display information generating unit, generating the augmented reality display information, which reproduces in time series a three-dimensional motion of the worker as viewed from the viewpoint position set by the viewpoint setting unit in the augmented reality space, using the motion information; A program characterized by:

Images