Work simulation device, work simulation system, and work simulation method

Abstract

The objective is to generate a knowledge graph from recovery work reports and simulations of skilled workers' work in a virtual space, and then use this knowledge graph and LLM to add expert knowledge to the recovery work reports, thereby enabling the inheritance of expert knowledge. [Solution] The work simulation system 110 uses the operation unit 105 to select a scene from the scenes stored in the scene storage unit 120 in which it wants to perform a work simulation in the virtual space 100, generates a knowledge graph from the work recovery reports and work simulations stored in the recovery work report storage unit 140, adds expert knowledge to the work recovery reports using the generated knowledge graph, and optionally uses all the knowledge graphs stored in the knowledge graph storage unit 130 to determine the standard work for the same case.

Description

Technical Field

[0001] The present invention relates to, for example, a work simulation device, a work simulation system, and a work simulation method.

Background Art

[0002] In recent years, engineers who have been responsible for maintaining railway infrastructure have been retiring, and there is concern about the inheritance of expert knowledge. Therefore, in recent years, there are many software that utilize Extended Reality (XR) such as Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) to promote the inheritance of expert knowledge.

[0003] Patent Document 1 discloses a 3D simulation system on an arbitrary virtual space using a metaverse space. Here, it enables the accumulation and extraction of expert knowledge triggered by voice. Also, Patent Document 2 discloses a system that synchronizes the real space on a virtual space and manages work progress information and the like. However, in these two patent documents, the actions of experts are not taken into account, and it is knowledge that experts regard as general work. On the other hand, there is no accumulation of knowledge that can be regarded as expert knowledge for new employees.

[0004] Furthermore, Non-Patent Document 1 discloses a technique for making explicit the relationship between people and objects by constructing a scene graph, which is a set of Triplets composed of Subject, Relation, and Object from a video, and thereby enabling Visual Question Answering (VQA) for the video. However, when using this technique with domain-specific tools such as in the railway or manufacturing industries, there is a problem that the scene graph cannot be properly constructed.

Prior Art Documents

Patent Documents

[0005] [Patent Document 1] U.S. Patent Application Publication No. 2023 / 0162736 [Patent Document 2] Japanese Patent Publication No. 2024-80494 [Non-patent literature]

[0006] [Non-Patent Document 1] Jingwei Ji, et al., "Action Genome: Actions as Composition of Spatio-temporal Scene Graphs", [online], December 15, 2019, Stanford University, Internet.<https: / / arxiv.org / abs / 1912.06992> [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] During simulations using XR, users primarily exchange opinions based on their existing knowledge, and log data such as recovery work reports are sometimes not utilized. Furthermore, recovery work reports are written after the work is completed, relying on the memories of the workers who performed the recovery work, and may not include detailed information at the level of expert knowledge, such as which tools were used and what actions were taken. In addition, ensuring safety relies on the experience of skilled workers, and there was a challenge in that it could not be quantitatively evaluated.

[0008] The present invention has been made in view of the above-mentioned problems, and one of its objectives is to provide a work simulation device, a work simulation system, and a work simulation method that can transfer and accumulate expert knowledge in railway infrastructure maintenance work. [Means for solving the problem]

[0009] To solve the above-mentioned problems and achieve the above objectives, one embodiment of the present invention is a work simulation device comprising: a reproduction unit that reproduces a scene in a virtual space; a generation unit that generates a knowledge graph from an input recovery work report and a work simulation; and a determination unit that understands the generated knowledge graph, adds expert knowledge to the recovery work report, and analyzes all of the knowledge graphs to determine whether the work is standard or non-standard.

[0010] Another embodiment of the present invention is a work simulation system comprising: a virtual space; a scene storage unit for storing scenes; a work recovery report storage unit for storing work recovery reports; a knowledge graph storage unit for storing knowledge graphs; a selection unit for selecting a scene from among the scenes stored in the scene storage unit in which work simulations are to be performed in the virtual space; a knowledge graph generation unit for generating a knowledge graph from the work recovery reports stored in the work recovery report storage unit and the work simulations; and a determination unit for adding expert knowledge to the work recovery reports using the knowledge graphs generated by the knowledge graph generation unit, and optionally using all the knowledge graphs stored in the knowledge graph storage unit to determine whether the same case is a standard operation or a non-standard operation.

[0011] Furthermore, another embodiment of the present invention is a work simulation method for a device that simulates work, characterized by including: a reproduction step of reproducing a scene in a virtual space; a generation step of generating a knowledge graph from a recovery work report and a work simulation stored in memory; and a determination step of understanding the knowledge graph, adding expert knowledge to the recovery work report, and analyzing all of the knowledge graphs to determine whether it is a standard or non-standard operation. [Effects of the Invention]

[0012] According to this invention, a knowledge graph is generated from recovery work reports and simulations of skilled workers' work in a virtual space. By using this knowledge graph and a large-scale language model to add expert knowledge to the recovery work reports, it becomes possible to pass on expert knowledge. Furthermore, quantitative evaluation of worker safety through musculoskeletal analysis becomes possible. [Brief explanation of the drawing]

[0013] [Figure 1] This block diagram shows an overview of the work simulation system according to Embodiment 1 of the present invention. [Figure 2] This block diagram shows an example of a network configuration to which the overall system for the work simulation system according to Embodiment 1 of the present invention is applied. [Figure 3] This is an explanatory diagram showing an example of a railway accident recovery work report in Embodiment 1 of the present invention. [Figure 4] This is a block diagram showing a functional configuration example of the overall system according to Embodiment 1 of the present invention. [Figure 5] This is an explanatory diagram showing an example in which a column for inheriting expert knowledge is added to a railway accident recovery work report in Embodiment 1 of the present invention. [Figure 6] This flowchart shows an example of the procedure by which the knowledge graph generation unit generates a knowledge graph in Embodiment 1 of the present invention. [Figure 7] This is an explanatory diagram showing an example of a knowledge graph related to the work flow extracted from a recovery work report in Embodiment 1 of the present invention. [Figure 8] This is an explanatory diagram illustrating an example of a case in which an unrecognizable object exists in the scene graph in Embodiment 1 of the present invention. [Figure 9] This flowchart shows an example of a procedure for predicting the name of an unrecognizable object using a Vision-Language Model (VLM) in Embodiment 1 of the present invention. [Figure 10] This is an explanatory diagram illustrating the process of extracting context from a knowledge graph in Embodiment 1 of the present invention. [Figure 11]It is an explanatory diagram showing an image when inferring an unrecognizable object using VLM in Example 1 of the present invention. [Figure 12] It is an explanatory diagram showing an image when complementing from an existing other scene graph based on graph similarity in Example 1 of the present invention. [Figure 13] It is an explanatory diagram showing an example of an additional training dataset of a scene graph generation AI in Example 1 of the present invention. [Figure 14] It is an explanatory diagram showing an image when adding expert knowledge in the knowledge graph understanding unit in Example 1 of the present invention. [Figure 15] It is an explanatory diagram showing an example when aggregating knowledge graphs to determine whether it is a standard operation in the knowledge graph analysis unit in Example 2 of the present invention. [Figure 16] It is an explanatory diagram showing an example of a user viewpoint screen on a virtual space in Example 2 of the present invention. [Figure 17] It is an explanatory diagram showing an image of a knowledge graph generated by the knowledge graph generation unit in Example 2 of the present invention.

Modes for Carrying Out the Invention

[0014] Hereinafter, embodiments of the present invention will be described while referring to the drawings. Note that the following description and drawings are merely examples for explaining the present invention, and for the sake of clarity of explanation, appropriate omissions and simplifications have been made. Also, the present invention can be implemented in various other forms. Also, unless otherwise limited, each component may be singular or plural.

[0015] In the following explanation, identical or similar structures may be denoted by the same symbol, and redundant explanations may be omitted. Also, in the following explanation, various types of information may be described using expressions such as "information" and "table," but these types of information may be represented by data structures other than these. Furthermore, while expressions such as "identification information," "identifier," "name," "ID," and "number" may be used to represent identification information, these can be substituted for each other. In the following explanation, "database" will be abbreviated as "DB" and "table" as "TBL." [Examples]

[0016] [Summary of the Embodiment] Hereinafter, Embodiment 1 of the present invention will be described with reference to the drawings. Note that Embodiment 1 described below does not limit the invention to the claims, and not all of the elements and combinations described in Embodiment 1 are necessarily essential to the solution of the invention.

[0017] Figure 1 is a diagram showing an overview of an embodiment in Embodiment 1 of the present invention. The work simulation system 110 consists of a work simulation device 190 having a virtual space 100 displayed on a display unit 106 and receiving operation input from an operation unit 105, a scene reproduction unit 150, a knowledge graph generation unit 160, a knowledge graph understanding unit 170, and a knowledge graph analysis unit 180, a scene storage unit 120, a knowledge graph storage unit 130, a recovery work report storage unit 140, and the like.

[0018] In the work simulation system 110, when a scene to be simulated is selected from the scenes stored in the scene storage unit 120 in response to user operation via the operation unit 105 in the virtual space 100, the work simulation device 190 sequentially performs processing in the scene reproduction unit 150, the knowledge graph generation unit 160, and the knowledge graph understanding unit 170 using the work simulation in that scene and the recovery work report stored in the recovery work report storage unit 140. The knowledge graph generated by the knowledge graph generation unit 160 is stored in the knowledge graph storage unit 130, and the knowledge graph analysis unit 180 performs analysis processing using all the knowledge graphs stored in the knowledge graph storage unit 130. In the work simulation system 110, the processing of the work simulation device 190 is implemented in software by executing a program described later. [Block diagram] Figure 2 is a block diagram showing an example of the network configuration to which the work simulation device 190 according to the embodiment of Example 1 is applied, and Figure 3 is an explanatory diagram showing an example of a railway accident recovery work report in Example 1.

[0019] The work simulation device 190 is connected to a network 200, which in turn is connected to one or more scene storage units 120, one or more knowledge graph storage units 130, one or more recovery work report storage units 140, one or more recovery work report generators 230, and one or more 3D measurement data generators 240. The recovery work report generators 230 and 3D measurement data generators 240 are part of the field system 250B acquired at the work site.

[0020] The work simulation device 190, the scene storage unit 120, the knowledge graph storage unit 130, the recovery work report storage unit 140, the recovery work report generator 230, and the 3D measurement data generator 240 are connected to the information collection system 210 via the network 200. Although the main component of the work simulation system 110 is the work simulation device 190, as shown in Figure 2, the entire system 250A is sometimes referred to as the "work simulation system" in a broader sense.

[0021] The work simulation device 190 is connected to VR hardware 220 for users to connect to the work simulation device 190 via data communication, etc. The VR hardware 220 is used by various users, such as experienced users who accumulate expert knowledge and young users who inherit expert knowledge, and consists of one or more information terminal devices. Furthermore, by connecting the work simulation device 190 and the VR hardware 220 via the network 200, the work simulation device 190 may be configured to be accessible from any point outside the overall system 250A.

[0022] The recovery work report generator 230 is a device that generates log data of recovery work created when an accident or failure occurs. The recovery work report generator 230 is, for example, a PC (Personal Computer) or a server. The recovery work report generator 230 may be centrally managed or distributed.

[0023] For example, as shown in Figure 3, the recovery work reports 400 stored in the recovery work report generator 230 have columns such as report number 410, date of occurrence 420, location of occurrence 430, business operator category 440, type of accident, etc. 450, summary 460, cause 470, countermeasures 480, and recovery work order 490. The recovery work reports generated or collected by the recovery work report generator 230 are transmitted to the recovery work report storage unit 140 via the network 200 and stored there.

[0024] The 3D measurement data generator 240 is a device that generates 3D measurement data for accurately recreating a site in a virtual space. 3D measurement data is data that accurately represents the shape and position of objects and spaces in three dimensions using measurement technologies such as laser surveying, photogrammetry, Light Detection and Ranging (LiDAR), and 3D scanners. It can also be a 3D CAD model created with a game engine such as Unity or Unreal Engine.

[0025] The scene storage unit 120 is, for example, a storage device such as a server or memory, and stores 3D measurement data received from the 3D measurement data generator 240 via the network 200. The knowledge graph storage unit 130 is, for example, a storage device such as a server or memory, and stores knowledge graphs generated by the knowledge graph generation unit via the network 200. The recovery work report storage unit 140 is, for example, a storage device such as a server or memory, and stores recovery work reports received from the recovery work report generator 230 via the network 200. [System Configuration] Next, the main functions of the work simulation device 190 will be described. As shown in Figure 2, the work simulation device 190 is composed of a processor 211 such as a Central Processing Unit (CPU) that controls the entire work simulation device 190, a storage device 212 that stores various processing programs for realizing the functions of the work simulation device 190, a network interface (I / F) 213, and the like. The storage device 212 is implemented using known storage devices such as ROM (Read Only Memory) for storing various processing programs, Random Access Memory (RAM) for temporarily storing information, and Hard Disk Drives (HDD).

[0026] The processor 211 executes various processing programs stored in the memory, thereby realizing the functions of the present invention described below. Note that the configuration of the work simulation device 190 is not limited to the illustrated example; some or all of the programs may be introduced from other devices via non-temporary storage media or communication lines.

[0027] Figure 4 is a block diagram illustrating the functions of the work simulation device 190. The work simulation device 190 uses a computer program (processor 2011 and memory device 212) to perform work simulations and acquire knowledge graphs, utilizing the functions of the scene reproduction unit 150, the knowledge graph generation unit 160, the knowledge graph understanding unit 170, and the knowledge graph analysis unit 180.

[0028] As mentioned above, the work simulation device 190 is connected to the operation unit 105. Here, the flow of information through the connection path is indicated by the direction of the arrows. The operation unit 105 provides a function to select a scene to be reproduced in the virtual space using the scene reproduction unit 150, and a function to, for example, manually modify the knowledge graph that is automatically generated from the work simulation using the knowledge graph generation unit 160.

[0029] First, the scene reproduction unit 150 of the work simulation device 190 reproduces a scene selected by the user using the operation unit 105 in a virtual space from the scene storage unit 120, which stores 3D measurement data of scenes corresponding to each recovery work report. The operation unit 105 is responsible for receiving operations from the user, such as touch operations on 3D models or selection operations on the UI, and issuing instructions to other functional units to execute processing according to the operations.

[0030] Next, the knowledge graph generation unit 160 generates a knowledge graph related to the work flow from the recovery work reports stored in the recovery work report storage unit 140 and the work simulations on the scene selected by the scene reproduction unit 150.

[0031] The knowledge graph is generated using a streaming method, and the user can manually modify the knowledge graph generated by the knowledge graph generation unit 160 using the operation unit 105. After manual modification, the knowledge graph is stored in the knowledge graph storage unit 130.

[0032] Next, the knowledge graph understanding unit 170 uses the knowledge graph generated by the knowledge graph generation unit 160 and the LLM to add and update the recovery work report stored in the recovery work report storage unit 140, which was selected by the operation unit 105 in the scene reproduction unit 150, with the expert knowledge 500.

[0033] Finally, the knowledge graph analysis unit 180 analyzes whether each identical case is a standard procedure using all the knowledge graphs stored in the knowledge graph storage unit 130. The execution of the knowledge graph analysis unit may be mandatory for each work simulation, or it may be optional for the user to execute. [Scene Recreation Section 150] The scene reproduction unit 150 reproduces the detailed site for each recovery work report in virtual space. The 3D model used to reproduce the detailed site in virtual space may be 3D measurement data (point cloud data) acquired with a 3D laser scanner or similar device, or it may be a 3D CAD model built with a game engine such as Unity or Unreal Engine. [Knowledge graph generation unit 160] Figure 6 shows an example flowchart of the knowledge graph generation unit 160, Figure 7 shows an example of extracting the work flow from the recovery work report 400 in Figure 4, and Figure 8 shows an example where an unrecognizable object exists in the scene graph. The knowledge graph generation unit 160 first extracts the work flow from the recovery work report (step 605).

[0034] From the accident type 450 of the recovery work report, an accident node with the property of "auxiliary overhead wire breakage" is generated; from the recovery work sequence 490, a work node 720 with the property of "tensioning the trolley wire to the appropriate tension" is generated; a work node 730 with the property of "correcting the twist of the trolley wire" is generated; and a work node 740 with the property of "connecting the trolley wire with a double ear" is generated.

[0035] Next, a scene graph is generated from user motions obtained through a virtual space work simulation (step 610). As shown in Figure 8, in the example where a scene graph is generated from user motions, a scene graph is generated for work node 720, which involves tensioning the trolley wire to the appropriate tension.

[0036] The scene graph consists of a worker (subject) node 810, an object node 830, business information 850, a relationship 820 indicating a predicate between the worker node and the object node, and a relationship 840 indicating an object and a predicate to identify the business information node between the object node and the business information node. Here, the scene graph may be generated as one scene graph for each frame, or multiple frames may be combined into one scene graph.

[0037] Next, UNKNOWN objects and business information are inferred (step 615). Step 615 is completed when it has been applied to all scene graphs (step 655). If there are objects in the scene graph that cannot be recognized during step 615 (step 625), the Vision-Language Model (VLM) is used to predict the UNKNOWN objects (step 630).

[0038] Figure 9 shows a flowchart for predicting UNKNOWN objects using a Vision-Language Model (VLM). First, sentences (context) describing the scene containing the UNKNOWN object are extracted from the knowledge graph (step 910).

[0039] Figure 10 shows an illustrative diagram of step 910. Figure 10 shows how context 1030 is generated by inputting the work knowledge graph 1010 into the Graph to Text model 1020. As a more concrete example, one could consider using LLM in the Graph to Text model 1020 and extracting context 1030 by providing the knowledge graph 1010 as a graph (Retrieval Augmented Generation).

[0040] Next, the UNKNOWN object image 1110 and the context 1030 generated in step 910 are input to VLM to predict the UNKNOWN object name (step 920).

[0041] Figure 11 shows an image of step 920, and Figure 12 shows an image of how graph similarity is used to supplement existing scene graphs. In Figure 11, the image 1110 of the frame containing the UNKNOWN object and the context 1030 generated in step 910 are input to VLM 1130, inferring that the UNKNOWN object name is trolley wire (1140).

[0042] Next, a check is performed to see if business information exists in the scene graph (step 635). If it does not exist, a check is performed to see if it can be supplemented from other existing scene graphs using graph similarity (step 640).

[0043] At this time, the graph similarity threshold used to determine whether the data can be supplemented from other scene graphs can be arbitrarily set by the user. If there is a graph that exceeds the graph similarity threshold set by the user, the business information is supplemented from an existing other scene graph (step 645). If there is no graph that exceeds the graph similarity threshold set by the user, the user performs the supplementation manually or through interaction with the generating AI (step 650).

[0044] Once UNKNOWN objects and business information inferences have been completed for all frames (step 655), the next step is to create an additional training dataset for the scene graph generation AI (step 660).

[0045] Figure 13 shows an example of an additional training dataset for a scene graph generation AI. The additional training dataset for the scene graph generation AI consists of image data 1310 and annotation data 1320 for each frame. The annotation data 1320 contains the following required columns: subject 1330, which indicates the subject's name; object 1340, which indicates the object's name; subject_bbox 1350, which indicates the subject's bounding box; object_bbox 1360, which indicates the object's bounding box; and relationship 1370, which indicates the relationship between the subject and the object. Alternatively, a column for business information can be added to the annotation data, allowing for simultaneous learning of business information.

[0046] Finally, the scene graph generation AI is further trained using the additional training dataset generated in step 660 (step 675). In this way, even if there are objects that the scene graph cannot recognize, the recognition accuracy of the scene graph improves over time as it learns from the automatically generated training data. [Knowledge Graph Understanding Section 170] Figure 5 shows an example of adding a column for inheriting expert knowledge to a railway accident recovery work report, and Figure 14 shows an image of how the knowledge graph understanding unit 170 adds expert knowledge about the task of tensioning the trolley wire to the appropriate tension 1410.

[0047] The knowledge graph understanding unit 170 understands the knowledge graph generated by the knowledge graph generation unit 160 and adds expert knowledge to the recovery work report. When the processing by the knowledge graph understanding unit 170 is completed, expert knowledge 500 is added as shown in Figure 5. The LLM 1420 receives the recovery work report and the knowledge graph 1400 generated by the work simulation of tensioning the trolley wire to the appropriate tension as graph RAG, outputs expert knowledge such as "climb the insulating tower, take the trolley wire, and pull it with the winder," and adds it to the recovery work report. The knowledge graph understanding unit 170 extracts expert knowledge 500 as shown in Figure 5 by performing this process for all work nodes 720, 730, and 740. [Knowledge Graph Analysis Department 180] The knowledge graph analysis unit 180 uses all the knowledge graphs stored in the knowledge graph storage unit 130 to distinguish between standard and non-standard tasks for each identical case. Figure 15 shows an example where all knowledge graphs for a case of accident type 450 being a broken auxiliary suspension wire are aggregated. By aggregating the individual knowledge graphs 1610, 1620, and 1630 to construct an aggregated knowledge graph 1640, it becomes possible to identify majority edges as standard tasks and minority edges as non-standard tasks. The knowledge graph understanding unit 170 and the knowledge graph analysis unit 180 described above constitute a task discrimination unit.

[0048] In this way, by using all the knowledge graphs stored in the knowledge graph storage unit 130 and aggregating the knowledge graphs for each of the 450 arbitrary accident types, it becomes possible to distinguish between standard and non-standard operations. Furthermore, since there is a concern that the UI will become complex as the number of nodes and edges increases, it is also possible to implement a format that avoids a cumbersome display by allowing the display of each node type on or off. [Display screen] Figure 16 shows an example of a user-viewpoint screen in a virtual space. Users performing simulations in the virtual space displayed on the display unit 106 can check the scene selected by the scene reproduction unit 150 at the top of the screen 1710, and check the knowledge graph created via streaming during the work simulation at the bottom of the screen 1720. The knowledge graph can also be manually modified by the user. The user can initiate / end the simulation of which task to start by triggering voice recognition, or by touching the work node 1730 on the display unit 106.

[0049] Thus, the work simulation system 110 is characterized by having a scene reproduction unit that reproduces the actual work site in a virtual space from 3D model information stored in the scene storage unit, a knowledge graph generation unit that generates a knowledge graph from the recovery work report and the expert's work simulation in the virtual space reproduced by the scene reproduction unit, a knowledge graph understanding unit in which the LLM understands the knowledge graph generated by the knowledge graph generation unit and adds expert knowledge to the recovery work report, and a knowledge graph analysis unit that statistically analyzes whether the same case is a standard operation using all the knowledge graphs stored in the knowledge graph storage unit 130.

[0050] As described above, according to Example 1, a knowledge graph can be constructed from recovery work reports and simulations of skilled workers' work in a virtual space. A large-scale language model then uses this knowledge graph to add expert knowledge to the recovery work reports, thereby assisting in the transfer of expert knowledge. Furthermore, quantitative evaluation of worker safety through musculoskeletal analysis becomes possible. [Examples]

[0051] A second embodiment of the present invention is a system for evaluating worker safety with added musculoskeletal analysis results. The overall configuration and function are the same as in Embodiment 1 described above, so the explanation will be omitted, and only the differences will be described.

[0052] Representative musculoskeletal analysis software includes OpenSim and Anybody. Figure 17 shows an image of the knowledge graph generated by the knowledge graph generation unit 160 in Example 2. The difference from the knowledge graph generated by the knowledge graph generation unit 160 in Example 1 is the addition of musculoskeletal nodes.

[0053] Figure 17 shows an image of how the results of the load analysis on the lower back, calculated using OpenSim's Static Optimization, are divided into time-series information and managed in relation to the work nodes. It can be seen that the task of holding the trolley wire and pulling it with a winder (1550) places a greater burden on the lower back than the task of climbing the insulation tower (1540).

[0054] Thus, in Embodiment 2, it becomes possible to evaluate worker safety by managing musculoskeletal analysis results in conjunction with a scene graph.

[0055] This invention is not limited to railway infrastructure maintenance work but can be applied to all types of maintenance work, including in manufacturing and construction. Furthermore, given the increasing concern about the aging workforce, it enables the quantitative evaluation of worker safety.

[0056] Furthermore, each of the above-mentioned configurations, functional units, processing units, processing means, etc., may be implemented in hardware, in whole or in part, for example, by designing them as integrated circuits. Alternatively, each of the above-mentioned configurations, functions, etc., may be implemented in software by having the processor interpret and execute programs that realize each function. Information such as programs, tables, and files that realize each function can be stored in memory, hard disks, SSDs (Solid State Drives), or other recording devices, or in recording media such as IC cards, SD cards, or DVDs.

[0057] Furthermore, the arrangement of the various functional units, processing units, and databases described above is merely an example. The arrangement of the various functional units, processing units, and databases can be changed to the optimal arrangement from the standpoint of the performance, processing efficiency, and communication efficiency of the hardware and software of these devices.

[0058] Furthermore, the configuration of the database (schema, etc.) that stores the various types of data mentioned above can be flexibly modified from the perspective of efficient resource utilization, improved processing efficiency, improved access efficiency, and improved search efficiency. [Explanation of symbols]

[0059] 100 Virtual Spaces 105 Operation section 110 Work Simulation System 120 Failure Mode and Effects Analysis Management System 130 Knowledge Graph Accumulation Section 140 Recovery Work Report Storage Department 150 Scene Recreation Section 160 Knowledge Graph Generation Unit 170 Knowledge Graph Understanding Section 180 Knowledge Graph Analysis Department 190 Work Simulation Device 200 Networks 210 Information Gathering System 220 VR Hardware 230 Recovery work report generator 240 3D measurement data generator 250A overall system 250B Field System

Claims

1. A work simulation device, A reproduction unit that recreates the scene in a virtual space, A generation unit that generates a knowledge graph from the input recovery work report and work simulation, A discrimination unit that understands the generated knowledge graph, adds expert knowledge to the recovery work report, analyzes all the knowledge graphs to determine whether the work is standard or non-standard, A work simulation device characterized by being equipped with the following features.

2. A work simulation apparatus according to claim 1, characterized in that the 3D measurement data of the actual site in the virtual space is constructed using a game engine.

3. A work simulation device according to claim 1, characterized in that it extracts the knowledge graph relating to the accident and the recovery work from the recovery work report and generates a scene graph by performing a work simulation for each work node.

4. The work simulation device according to claim 3, characterized in that the scene graph is composed of worker nodes, object nodes, business information nodes, the relationship between worker nodes and object nodes, and the relationship between object nodes and business information nodes.

5. A work simulation apparatus according to claim 1, characterized in that if there are unrecognizable objects in the scene graph generated during the work simulation, VLM (Vision-Language Model) is used to compensate for them.

6. A work simulation device according to claim 5, characterized in that when supplementing objects that cannot be recognized using the VLM, a descriptive text about the unrecognizable object is generated from the knowledge graph, the image and context of the unrecognizable object are input to the VLM, and the name of the unrecognizable object is inferred.

7. A work simulation device according to claim 1, characterized in that, if work information is not present in the scene graph, it is supplemented from other existing scene graphs based on graph similarity.

8. A work simulation device according to claim 7, characterized in that when interpolating from existing other scene graphs using graph similarity, the threshold for graph similarity to determine whether or not to interpolate is set according to the user's operation.

9. A work simulation device according to claim 1, characterized in that it is connected to an storage unit that stores recovery work reports, and adds expert knowledge to the recovery work reports of the storage unit by providing the scene graph as graph RAG (Retrievable Augmented Generation) to LLM (Large Language Model).

10. A work simulation device according to claim 1, characterized in that the safety of the worker is evaluated by linking the musculoskeletal analysis results to the scene graph as time-series information.

11. A work simulation device according to claim 1, characterized in that it is connected to a storage unit that stores knowledge graphs, and distinguishes between standard work and non-standard work by summing the knowledge graphs of the storage unit for each identical case using the knowledge graphs.

12. A work simulation device according to claim 1, characterized in that the knowledge graph is generated via streaming during the work simulation, and the knowledge graph is checked and modified in response to user operations in the virtual space.

13. A work simulation device according to claim 1, characterized in that when signaling which work to create a scene graph for, or when signaling the start or end of generating the scene graph, the trigger is either voice or touch operation on the displayed knowledge graph.

14. It is a work simulation system, Virtual space and, A scene storage unit that stores scenes, A recovery work report storage unit that stores work recovery reports, A knowledge graph storage unit that stores knowledge graphs, A selection unit for selecting a scene from among the scenes stored in the scene storage unit for which work simulations are to be performed in the virtual space, A knowledge graph generation unit that generates a knowledge graph from the work recovery reports and work simulations stored in the aforementioned recovery work report storage unit, A determination unit that uses the knowledge graph generated by the knowledge graph generation unit to add expert knowledge to the work recovery report, and optionally uses all the knowledge graphs stored in the knowledge graph storage unit to determine whether the same case is a standard operation or a non-standard operation, A work simulation system characterized by having the following features.

15. A method for simulating the work of a device that simulates work, The reproduction step involves recreating the scene in a virtual space, A generation step that generates a knowledge graph from recovery work reports and work simulations stored in memory, A determination step involves understanding the knowledge graph, adding expert knowledge to the aforementioned recovery work report, analyzing all the aforementioned knowledge graphs, and determining whether the work is standard or non-standard. A work simulation method characterized by including the following.

Images