Simulation program, simulation device, simulation system, and display method

Abstract

The simulation program causes the computer to function as a three-dimensional model display unit (103) that displays a three-dimensional model that reproduces a real space in which the facility equipment is placed, a request receiving unit (104) that receives requests from a user for the facility equipment included in the three-dimensional model, and a function identification unit (105) that interprets the requests received by the request receiving unit (104) using an interactive AI function to identify functions related to the facility equipment that satisfy the requests. The simulation program also causes the computer to function such that the three-dimensional model display unit (103) displays information on the functions identified by the function identification unit (105) in the three-dimensional model.

Description

[Technical field]

[0001] The present disclosure relates to a simulation program, a simulation device, a simulation system, and a display method. [Background technology]

[0002] Digital twin technology is known, which reproduces real space as a three-dimensional model in a digital space. Digital twin technology makes it possible to display various information in a three-dimensional model in a digital space or in an image of a real space according to a user's request.

[0003] On the other hand, there is known a technology for displaying information superimposed on a real-space image in response to a user's request. For example, Patent Document 1 discloses a technology for displaying information acquired using a remote expert system in response to a request from a worker in the real space, superimposed on a real-space image displayed on a display terminal used by the worker. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special Publication No. 2003-515294 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the technology using the above-mentioned expert system, if the user makes a request that is not included in the set of pre-prepared rules, it is not possible to provide information corresponding to the user's request. Since digital twin technology has a wide range of uses and is used by a variety of users who do not have specialized knowledge, there is a demand for displaying information in response to requests made by users in natural language.

[0006] The present disclosure has been made in consideration of the above circumstances, and aims to provide a simulation program, a simulation device, a simulation system, and a display method capable of displaying appropriate information that meets the requirements of a user in a three-dimensional model that reproduces real space. [Means for solving the problem]

[0007] In order to achieve the above object, the simulation program according to the present disclosure comprises: Computer, a three-dimensional model display means for displaying a three-dimensional model that reproduces a real space in which the facility equipment is arranged; a request receiving means for receiving a request for equipment included in the three-dimensional model; A function specifying means for specifying a function related to the facility device that satisfies the request by interpreting the request received by the request receiving means using an interactive AI function. 、 and functioning as an analysis means for performing an analysis for detecting a failure of the facility equipment based on the data collected from the facility equipment; The three-dimensional model display means displays information on the function specified by the function specifying means in the three-dimensional model. death, When the function cannot be identified based on the request accepted by the request accepting means, the function identifying means identifies a function related to facility equipment included in the three-dimensional model based on a result of the analysis by the analysis means and the request accepted by the request accepting means. . Effect of the Invention

[0008] According to the present disclosure, it is possible to provide a simulation program, a simulation device, a simulation system, and a display method capable of displaying appropriate information that meets user requirements in a three-dimensional model that reproduces real space. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 shows a simulation system according to an embodiment. [Diagram 2] FIG. 1 is a block diagram showing a hardware configuration of an information processing apparatus according to an embodiment. [Diagram 3] FIG. 2 is a diagram showing the functional configuration of a simulation device and a display terminal according to an embodiment; [Figure 4] FIG. 1 is a diagram showing a state in which a widget is displayed on a screen of a simulation device according to an embodiment. [Diagram 5] FIG. 1 is a diagram showing a state in which a widget is displayed on a screen of a simulation device according to an embodiment. [Figure 6] FIG. 1 is a diagram showing a state in which a widget is displayed on a screen of a simulation device according to an embodiment. [Figure 7] FIG. 1 is a diagram showing a state in which a widget is displayed on a screen of a simulation device according to an embodiment. [Figure 8] FIG. 1 is a diagram showing a state in which a widget is displayed on a screen of a simulation device according to an embodiment. [Figure 9] FIG. 1 is a diagram showing a widget displayed on a screen of a display device according to an embodiment; [Figure 10] FIG. 1 is a diagram showing a widget displayed on a screen of a display device according to an embodiment; [Figure 11] 1 is a flowchart showing a widget display process executed in response to a request made to a simulation device according to an embodiment. [Figure 12] A flowchart showing a widget update process according to an embodiment. [Figure 13] 1 is a flowchart showing a control process using a widget according to an embodiment. [Figure 14] 1 is a flowchart showing a widget display process executed in response to a request received from a display terminal according to an embodiment; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] (Embodiment) The simulation system according to the embodiment is a system that provides a digital twin environment for a real space in which facility equipment is arranged.

[0011] 1 shows a simulation system 1 of this embodiment. The simulation system 1 includes a simulation device 100 and a display terminal 200. The simulation device 100 is communicatively connected to the display terminal 200 via a wired or wireless communication network 500. In addition, the simulation device 100 and the display terminal 200 are communicatively connected to one or more facility devices 300 arranged in a real space 400 via the communication network 500.

[0012] In the following, a case where the simulation system 1 is applied to a factory automation (FA) system installed in a factory will be described as an example. For example, the real space 400 is a space in a factory, and the facility equipment 300 is a plurality of FA equipment installed in the factory. The FA equipment is, for example, a programmable logic controller, a programmable display, a motion controller, a servo amplifier, an inverter, a robot, etc.

[0013] The simulation device 100 is a device that reproduces a real space 400 in which the facility equipment 300 is arranged, in a digital space. The simulation device 100 expresses the real space 400 as a three-dimensional model. For example, the simulation device 100 generates a three-dimensional model of a factory space in which FA equipment is arranged. The simulation device 100 is generally arranged in a place different from the factory, and is used, for example, by someone who designs the line layout of the factory, someone who designs the three-dimensional model, a manager at the factory site, etc. Hereinafter, a user who uses the simulation device 100 is referred to as a "first user."

[0014] In the simulation device 100, a simulation application 100-1 and an engineering tool 100-2 are installed.

[0015] The simulation application 100-1 displays a three-dimensional model that reproduces the real space 400, collects data from the equipment 300 arranged in the real space 400, and reflects the collected data in the three-dimensional model. The simulation application 100-1 also has an interactive AI function, and uses the interactive AI function to output text or voice input from a user. Here, the interactive AI function is a generative AI function that uses language model technology, such as a large-scale language model, and is a function that appropriately interprets input data that includes ambiguous natural language requests and generates a response in natural language.

[0016] The engineering tool 100-2 has a function of controlling the facility equipment 300 arranged in the real space 400, and, for example, collects data from the facility equipment 300 and performs settings for the facility equipment 300. Note that the number of engineering tools installed in the simulation device 100 is not limited to one, and there may be multiple engineering tools.

[0017] The display terminal 200 is a device capable of AR (Augmented Reality) display, which displays a virtual object superimposed on an image of the real space 400 captured by a camera. The display terminal 200 can also use the functions of a simulation application 100-1 and an engineering tool 100-2 in the simulation device 100. The virtual object is generated based on information received from the simulation device 100, for example. The display terminal 200 is carried and used by, for example, a worker at a factory site. Hereinafter, a user who uses the display terminal 200 is referred to as a "second user."

[0018] An AR display application 200-1 is installed in the display terminal 200. The AR display application 200-1 transmits a request for the facility equipment 300 in the real space 400 to the simulation device 100. Then, the AR display application 200-1 displays information received from the simulation device 100 as a response to the request, superimposed on an image of the real space 400 captured by a camera.

[0019] FIG. 2 shows an example of a hardware configuration of an information processing device 10 in which the simulation device 100 and the display terminal 200 are realized.

[0020] The information processing device 10 has a processor 11 that executes various processes, a main memory unit 12 used as a working area for the processor 11, an auxiliary memory unit 13 that stores various data used in the processes of the processor 11, a communication unit 14 for communicating with external devices, an input unit 15 that acquires input information, and an output unit 16 that presents various information. The main memory unit 12, the auxiliary memory unit 13, the communication unit 14, the input unit 15, and the output unit 16 are all connected to the processor 11 via a bus 17.

[0021] The processor 11 includes a CPU (Central Processing Unit). The processor 11 executes programs stored in the auxiliary storage unit 13 to realize various functions of the information processing device 10.

[0022] The main memory unit 12 includes a RAM (Random Access Memory). Programs are loaded into the main memory unit 12 from the auxiliary memory unit 13. The main memory unit 12 is used as a working area for the processor 11.

[0023] The auxiliary storage unit 13 includes a non-volatile memory such as an EEPROM (Electrically Erasable Programmable Read-Only Memory). In addition to programs, the auxiliary storage unit 13 stores various data used in the processing of the processor 11. In accordance with instructions from the processor 11, the auxiliary storage unit 13 supplies the processor 11 with data used by the processor 11 and stores the data supplied from the processor 11.

[0024] The communication unit 14 includes a network interface circuit for communicating with an external device. The communication unit 14 receives a signal from the external device and outputs data indicated by the signal to the processor 11. The communication unit 14 also transmits a signal indicating the data output from the processor 11 to the external device.

[0025] The input unit 15 includes input devices such as an input key, a pointing device, a microphone, a camera, etc. The input unit 15 acquires information input by a user of the information processing device 10, and notifies the processor 11 of the acquired information.

[0026] The output unit 16 includes output devices such as an LCD (Liquid Crystal Display) and a speaker. The output unit 16 may be configured as a touch screen integrally formed with a pointing device constituting the input unit 15. The output unit 16 presents various information to the user according to instructions from the processor 11.

[0027] FIG. 3 shows the functional configuration of the simulation device 100 and the display terminal 200 of this embodiment.

[0028] 3 functionally includes a three-dimensional model management unit 101 that manages three-dimensional models, a widget management unit 102 that manages widgets, a three-dimensional model display unit 103 that displays the three-dimensional models, a request receiving unit 104 that receives a request for the facility equipment 300, a function identification unit 105 that identifies a function related to the facility equipment 300 based on the request, a position identification unit 106 that identifies a position in the three-dimensional model where information of the identified function is to be placed, an analysis unit 107 that performs an analysis for detecting or predicting a failure of the facility equipment 300, a response sending unit 108 that sends a response to the request from the display terminal 200, and a widget function unit 109 that is a function indicated by a widget. The three-dimensional model management unit 101, the widget management unit 102, the three-dimensional model display unit 103, the request receiving unit 104, the function identification unit 105, the position identification unit 106, the analysis unit 107, and the response sending unit 108 are functions that the simulation application 100-1 has. Moreover, the widget function unit 109 is a function that the engineering tool 100-2 has.

[0029] The three-dimensional model management unit 101 has a three-dimensional model that reproduces the real space 400 in which the facility equipment 300 is placed, collects data from the facility equipment 300, and reflects the collected data in the three-dimensional model. By reflecting the data in the three-dimensional model, the operating state of the facility equipment 300 in the real space 400 can be instantly reproduced on the three-dimensional model. The three-dimensional model management unit 101 is realized by the processor 11, the auxiliary storage unit 13, and the communication unit 14.

[0030] The widget management unit 102 imports a widget function, which is a function of the engineering tool 100-2, into the simulation application 100-1 and manages the widget function so that it can be used on the three-dimensional model of the simulation application 100-1. The widget management unit 102 is realized by the processor 11, the auxiliary storage unit 13, and the input unit 15.

[0031] A widget is a graphical user interface that is displayed on a three-dimensional model. A widget function is a function that the engineering tool 100-2 has and is capable of displaying a widget on a three-dimensional model.

[0032] The widget management unit 102 receives, from the user, designation of one or more engineering tools having widget functions to be imported. Upon receiving the designation of the engineering tools, the widget management unit 102 starts importing the widget functions of the designated engineering tools and manages the imported widget functions so that they can be used in the simulation application 100-1.

[0033] For example, when the widget management unit 102 receives a designation of the engineering tool 100-2 from the user, the widget management unit 102 imports the widget function of the widget function unit 109 included in the engineering tool 100-2 and manages the imported widget function.

[0034] Note that importing of widget functions may be started automatically instead of starting in response to a user's designation of an engineering tool. Specifically, the simulation application 100-1 searches for an engineering tool installed in the simulation device 100, and when an engineering tool is found, imports the widget function of the engineering tool. When a plurality of engineering tools are installed in the simulation device 100, the search for the engineering tool and the import of the widget function are repeated. Also, the widget management unit 102 may display a list of importable widget functions on a screen and allow the user to select a widget function that he or she wishes to import.

[0035] The three-dimensional model display unit 103 displays a three-dimensional model that reproduces the real space 400 in which the equipment 300 is placed. Furthermore, the three-dimensional model display unit 103 updates the display of the three-dimensional model based on an operation of the first user. The three-dimensional model display unit 103 is realized by the processor 11, the communication unit 14, the input unit 15, and the output unit 16. The three-dimensional model display unit 103 is an example of a three-dimensional model display means.

[0036] For example, when the simulation application 100-1 is started, the three-dimensional model display unit 103 displays the three-dimensional model managed by the three-dimensional model management unit 101 on the screen of the simulation device 100.

[0037] 4 shows a state in which a three-dimensional model is displayed on the screen of simulation device 100. Simulation application 100-1 is displayed on the screen of simulation device 100, and the three-dimensional model is displayed in window 100-11 of simulation application 100-1, and further, a widget 601, which will be described later, is displayed in window 100-11. Based on an operation on simulation application 100-1 by first user 701, the display of the three-dimensional model in window 100-11 is updated.

[0038] The request receiving unit 104 receives a request for the facility equipment 300 included in the three-dimensional model. The request receiving unit 104 is realized by the processor 11, the communication unit 14, and the input unit 15. The request receiving unit 104 is an example of a request receiving means.

[0039] A request to the facility device 300 is, for example, confirmation of information or a state of the facility device 300, a setting or an operation of the facility device 300, or the like.

[0040] For example, when a first user 701 of the simulation device 100 inputs a voice message saying, “I would like to diagnose a geared motor,” to the simulation application 100-1, as shown in FIG. 4, the request receiving unit 104 receives the request saying, “I would like to diagnose a geared motor.”

[0041] The function identifying unit 105 interprets the request received by the request receiving unit 104 using an interactive AI function, and identifies a function related to the facility device 300 that satisfies the request. The function identifying unit 105 is realized by the processor 11 and the output unit 16. The function identifying unit 105 is an example of a function identifying means.

[0042] The function related to the facility device 300 is, for example, a function possessed by the engineering tool 100-2 that is communicatively connected to the facility device 300 arranged in the real space 400, that is, a widget function managed by the widget management unit 102.

[0043] Specifically, the function identification unit 105 acquires a list of widget functions imported from the engineering tool 100-2 from the widget management unit 102. Next, the function identification unit 105 interprets the request accepted by the request acceptance unit 104 using an interactive AI function, and identifies a function that satisfies the request from the functions included in the acquired list of widget functions.

[0044] For example, for a request such as "I would like to diagnose a geared motor," the function identification unit 105 uses the interactive AI function to interpret the request and identifies a widget function that can diagnose a geared motor from the list. For example, the function identification unit 105 identifies a widget function that performs "estimate gear backlash" as a widget function that satisfies the request such as "I would like to diagnose a geared motor."

[0045] The position specifying unit 106 specifies the position, size, and orientation of the information on the function specified by the function specifying unit 105 in the three-dimensional model. The position specifying unit 106 is realized by the processor 11. The position specifying unit 106 is an example of a position specifying means.

[0046] Specifically, the position identification unit 106 uses a trained model that infers the position, size, and orientation of a widget of a widget function to be placed on a three-dimensional model based on the widget function and an area in which the three-dimensional model is displayed in a window. This trained model is a model generated by learning, for example, information on the position, size, and orientation in which a user has previously placed a widget on a three-dimensional model, and information on the area in which the three-dimensional model in which the user has placed the widget is displayed in a window, as training data.

[0047] For example, the position identification unit 106 identifies the position, size and orientation of the “Gear Backlash Estimation” widget to be placed on the three-dimensional model using a trained model that infers the position, size and orientation of the widget to be placed on the three-dimensional model.

[0048] The three-dimensional model display unit 103 displays the information on the function specified by the function specifying unit 105 in the three-dimensional model, arranging it at the position, size and orientation specified by the position specifying unit 106.

[0049] 4, the three-dimensional model display unit 103 places and displays a widget 601 for "Estimating gear backlash" based on the position, size, and orientation identified by the position identification unit 106. Then, the function identification unit 105 refers to the position where the widget is placed, generates a response such as "A widget for estimating gear backlash has been displayed next to device A," and outputs the generated response by voice.

[0050] In the above, a case has been described in which a widget is added to a three-dimensional model and displayed, but a widget may also be deleted or updated based on a request from a first user. For example, when the request received by the request receiving unit 104 is a request for deleting or updating a widget, the function identifying unit 105 identifies a widget function that satisfies the request from among widget functions of widgets displayed in the three-dimensional model, and the three-dimensional model display unit 103 deletes or updates the widget of the identified widget function and displays it.

[0051] Here, if the function identification unit 105 cannot identify a function based on the request accepted by the request accepting unit 104, the function identification unit 105 uses an interactive AI function to generate questions to narrow down the functions to be identified from among the functions related to the equipment 300 placed in the real space 400 reproduced by the three-dimensional model, based on the request accepted by the request accepting unit 104.

[0052] For example, as shown in FIG. 5, when the request receiving unit 104 receives a request from the first user 701 by voice input, "I would like to adjust the servo used in the conveyor," the function identification unit 105 interprets the content of the request received by the request receiving unit 104 as servo adjustment using the interactive AI function. However, since the list includes multiple available widget functions that adjust the servo, the function identification unit 105 cannot identify a widget function that satisfies the request. In this case, the function identification unit 105 uses the interactive AI function to generate a question, "What adjustment function do you want to use?" and further refers to the list of widget functions to generate an answer, "For the servo used in the conveyor, functions such as one-touch adjustment function and manual adjustment function can be used," and outputs the question and answer by voice.

[0053] When the first user 701 receives the output question and answer and inputs, for example, a voice message saying, "I would like to use the one-touch adjustment function," to the simulation application 100-1, the request receiving unit 104 receives the request saying, "I would like to use the one-touch adjustment function." Then, the function identifying unit 105 identifies the widget function that satisfies the request as the one-touch adjustment function.

[0054] The three-dimensional model display unit 103 then arranges and displays the widget 602 having the widget function performing the one-touch adjustment function, as shown in Fig. 5, based on the position, size, and orientation identified by the position identification unit 106. The function identification unit 105 then references the position where the widget is arranged, generates a response such as "The widget having the one-touch adjustment function has been displayed above the conveyor", and outputs the generated response by voice.

[0055] In addition, when the function identification unit 105 cannot identify a function based on the request accepted by the request accepting unit 104, the function identification unit 105 may identify a function related to the equipment 300 included in the three-dimensional model based on the result of the analysis by the analysis unit 107 and the request accepted by the request accepting unit 104.

[0056] For example, when the request receiving unit 104 receives a request such as "Please tell me the reason why device A stopped," the function identifying unit 105 uses the interactive AI function to interpret the content of the request received by the request receiving unit 104 as identification of the reason why device A stopped. If the list contains multiple widget functions related to identifying the reason why device A stopped, the function identifying unit 105 determines that it has not been able to identify a widget function that satisfies the request. In this case, the function identifying unit 105 causes the analysis unit 107 to start analysis.

[0057] The analysis unit 107 performs analysis to detect a failure of the facility equipment 300 based on the data collected from the facility equipment 300. Furthermore, the analysis unit 107 performs analysis such as estimating the degree of deterioration or wear of the facility equipment 300 and predicting a failure based on the data collected from the facility equipment 300. The analysis unit 107 is realized by the processor 11. The analysis unit 107 is an example of an analysis means.

[0058] For example, the analysis unit 107 analyzes data of the facility equipment 300 that is used in the device A, and obtains an analysis result that an alarm has occurred in the servo used in the device A. The analysis unit 107 may analyze the data collected from the facility equipment 300 using an analytical AI function that detects or analyzes the cause of a failure in the facility equipment 300. It is assumed that the analytical AI function is performed after selection of an AI model and adjustment of parameters have been performed in advance.

[0059] The function identification unit 105 identifies a widget function that executes "display servo alarm information" from a list of widget functions based on the analysis result that an alarm has occurred in the servo and the request "Please tell me the cause of the stop of device A." The three-dimensional model display unit 103 displays a widget 603 that displays servo alarm information at the position, size, and orientation identified by the position identification unit 106, as shown in FIG. 6. Then, the function identification unit 105 refers to the position where the widget is arranged, generates a response that "An alarm has occurred in the servo of device A. A widget for the alarm information has been displayed next to device A," and outputs the generated response by voice. The first user 701 can recognize that the cause is a drop in the battery voltage of the servo by checking the widget 603.

[0060] Furthermore, the three-dimensional model display unit 103 displays the data of the facility device 300 collected by the engineering tool 100-2 by immediately reflecting the data on the function specified by the function specifying unit 105.

[0061] Specifically, when a widget displayed on the screen includes data that requires continuous and immediate display update, such as a motor rotation speed, the three-dimensional model display unit 103 repeatedly executes an update process for updating the display of the widget at a predetermined interval. First, the three-dimensional model display unit 103 requests the widget management unit 102 to update data acquired by a widget function of the widget displayed on the screen. The widget management unit 102 requests the engineering tool 100-2 to update data acquired by the widget function that received the update request. When the widget function unit 109 of the engineering tool 100-2 receives the data update request, the widget function unit 109 communicates with the facility equipment 300 that has data related to the request, and acquires the data from the facility equipment 300. When the widget function unit 109 acquires the data from the facility equipment 300, the widget function unit 109 sends the collected data to the simulation application 100-1 as a response to the update request. When the widget management unit 102 receives a response from the engineering tool 100-2, the widget management unit 102 updates the data of the widget function, and the three-dimensional model display unit 103 displays a widget including the updated data.

[0062] For example, the widget 604 displayed in the window 100-11 in FIG. 7 is a widget of a widget function that acquires the rotation speed of a motor, and the three-dimensional model display unit 103 displays the value of the rotation speed of the motor in a field 604-1. The three-dimensional model display unit 103 also requests the widget management unit 102 to update the value of the rotation speed of the motor acquired by the widget function of the widget 604. The widget management unit 102 requests the engineering tool 100-2 to update the value of the rotation speed of the motor acquired by the widget function of the widget 604. The widget function unit 109 of the engineering tool 100-2 communicates with a motor control device arranged in the real space 400 to acquire the value of the rotation speed of the motor, and sends information on the acquired value to the simulation application 100-1 as a response to the update request. When the widget management unit 102 receives a response from the engineering tool 100-2, it updates the data of the widget function that acquires the rotation speed of the motor, and the three-dimensional model display unit 103 displays the widget 604 including the updated data. In this way, the motor rotation speed value displayed in column 604-1 is continuously and instantly updated to "100 [rpm]", "101 [rpm]", "102 [rpm]", . . .

[0063] Depending on the data to be displayed in the widget, communication from the engineering tool 100-2 to the facility device 300 may be performed multiple times based on a predefined sequence. When acquiring data from the facility device 300, data conversion, data division, etc. may be performed. For example, hexadecimal may be converted to decimal, or acquired data "0xFF00" may be divided into "0xFF" and "0x00". If the widget function is an offline function that does not involve communication with the facility device 300, such as a parameter converter function, the engineering tool 100-2 does not communicate with the facility device 300.

[0064] Furthermore, the information on the function identified by the function identification unit 105 includes a control that enables the facility device 300 to be set or operated.

[0065] A control is something that enables settings or operations to be made to the facility device 300, and is represented by a button, edit box, spreadsheet, etc. included in a widget.

[0066] 8 shows a state in which a widget 605 including a control is displayed in a window 100-11. The widget 605 is a widget with a widget function that can set the notch frequency of the notch filter of device B reproduced in a three-dimensional model. The widget 605 includes an edit box 605-1 that is a control for inputting a character string, and a button 605-2 that is a control associated with a process. When the button 605-2 is selected, the notch frequency of device B is set to the value input in the edit box 605-1.

[0067] When the three-dimensional model display unit 103 receives an operation for a control for inputting a character string, it displays the input character string in a widget control in the three-dimensional model. When the three-dimensional model display unit 103 receives an operation for a control other than a control for inputting a character string, i.e., a control with which a process is associated, it requests the widget management unit 102 to execute the process associated with the control. When the widget management unit 102 receives the request, it requests the engineering tool 100-2 to execute the process associated with the control. When the widget function unit 109 of the engineering tool 100-2 receives the request, it executes the process associated with the control.

[0068] For example, when the first user 701 operates the mouse 1002 to select the edit box 605-1 and inputs a value of "1000" using the keyboard 1001, the three-dimensional model display unit 103 determines that an operation on a control for inputting a character string has been accepted, and displays "1000" in the edit box 605-1. Next, when the first user 701 operates the mouse 1002 to click the button 605-2, the three-dimensional model display unit 103 determines that an operation on a control associated with a process has been accepted, and requests the widget management unit 102 to set the notch frequency of the notch filter of device B to "1000" [Hz]. The widget management unit 102 requests the engineering tool 100-2 to set the notch frequency of the notch filter of device B to "1000" [Hz], and the widget function of the engineering tool 100-2 sets the notch frequency of the notch filter of device B to "1000" [Hz].

[0069] Furthermore, the request receiving unit 104 receives a request for the facility equipment 300 arranged in the real space 400 from the display terminal 200 capable of displaying an image of the real space 400 as a request for the facility equipment 300 included in the three-dimensional model.

[0070] For example, when the second user 702 of the display terminal 200 inputs a voice message such as "I would like to diagnose the geared motor" to the AR display application 200-1 of the display terminal 200 as shown in Fig. 9, the display terminal 200 transmits a request of the content indicated by the input voice to the simulation device 100. When the request receiving unit 104 of the simulation device 100 receives the request of the content "I would like to diagnose the geared motor" from the display terminal 200, the request receiving unit 104 receives the received request as a request for the equipment 300 included in the three-dimensional model.

[0071] In other words, when the request receiving unit 104 receives a request from the display terminal 200, the function identification unit 105 identifies a function related to the equipment 300 that satisfies the request, and the position identification unit 106 identifies the position, size, and orientation in which to place the information of the identified function in the three-dimensional model, just as when the request receiving unit 104 receives a request for the simulation application 100-1.

[0072] When the request receiving unit 104 receives a request for the equipment 300 arranged in the real space 400 from the display terminal 200, the response sending unit 108 sends a response including information on the function identified by the function identifying unit 105 and information on the position, size and orientation identified by the position identifying unit 106 to the display terminal 200. The response sending unit 108 is realized by the processor 11 and the communication unit 14. The response sending unit 108 is an example of a response sending means.

[0073] For example, in response to a request such as “I would like to diagnose a geared motor,” the response sending unit 108 sends to the display terminal 200 a response including information identifying the widget function of “gear backlash estimation,” information on the position, size, and orientation of the widget, and answer information generated by the function identification unit 105 such as “a widget for estimating gear backlash has been displayed next to device A.”

[0074] The widget function unit 109 is one or more functions that the engineering tool 100-2 has. The widget function of the widget function unit 109 is provided to the simulation application 100-1 as a usable module such as a dynamic link library (DLL) or a service such as an application programming interface (API). This allows the simulation application 100-1 to use some or all of the functions of the engineering tool 100-2.

[0075] The display terminal 200 in Figure 3 functionally comprises a request sending unit 201 that sends a request to the simulation device 100, a response receiving unit 202 that receives a response from the simulation device 100, and an AR display unit 203 that overlays information on an image of the real space 400 and displays it.

[0076] The request sending unit 201 sends a request for the facility equipment 300 arranged in the real space 400 to the simulation device 100. The request sending unit 201 is realized by the processor 11, the communication unit 14, and the input unit 15. The request sending unit 201 is an example of a request sending means.

[0077] For example, when the second user 702 of the display terminal 200 inputs the voice, "I would like to diagnose the geared motor," to the AR display application 200-1 of the display terminal 200, as shown in FIG. 9, the request sending unit 201 transmits a request with the content indicated by the input voice to the simulation device 100.

[0078] The response receiving unit 202 receives a response from the simulation device 100. The response receiving unit 202 is realized by the processor 11 and the communication unit 14. The response receiving unit 202 is an example of a response receiving means.

[0079] For example, the response receiving unit 202 receives from the simulation device 100 a response that includes information identifying the widget function of “gear backlash estimation,” information on the widget’s position, size, and orientation, and answer information that reads, “A widget for estimating gear backlash has been displayed next to device A.”

[0080] The AR display unit 203 displays information on a function included in the response received by the response receiving unit 202, superimposed on an image of the real space 400, based on position information included in the response received by the response receiving unit 202. That is, the AR display unit 203 of the display terminal 200 displays information on a function identified by the function identifying unit 105 of the simulation device 100, superimposed on an image of the real space 400, based on information on a position, size, and orientation identified by the position identifying unit 106 of the simulation device 100. The AR display unit 203 is realized by the processor 11 and the output unit 16. The AR display unit 203 is an example of an AR display means.

[0081] 9 shows a state in which an image of the real space 400 is displayed on the screen of the display terminal 200. The AR display application 200-1 is displayed on the screen of the display terminal 200, the image of the real space 400 is displayed in a window 200-11 of the AR display application 200-1, and further, a widget 606, which will be described later, is displayed in the window 200-11. Based on an operation on the AR display application 200-1 by the second user 702, the display of the image of the real space 400 in the window 200-11 is updated.

[0082] For example, the AR display unit 203 identifies the widget to be displayed in the window 200-11 as a "gear backlash estimation" widget based on the received information identifying the widget function. The AR display unit 203 also identifies the position, size, and orientation in which the "gear backlash estimation" widget is displayed in the image of the real space 400 based on the received information on the position, size, and orientation of the widget. As shown in FIG. 9, the AR display unit 203 places and displays the "gear backlash estimation" widget 606 based on the identified position, size, and orientation. Then, the AR display unit 203 outputs a response by voice, "The gear backlash estimation widget has been displayed next to device A."

[0083] In the above description, a widget is added to and displayed in the image of the real space 400, but the widget may also be deleted or updated based on a request from the second user. For example, when the request received by the request receiving unit 104 is a request to delete or update a widget, the function identifying unit 105 identifies a widget function that satisfies the request from among the widget functions of the widgets displayed in the three-dimensional model, and the AR display unit 203 deletes or updates the widget of the identified widget function and displays it.

[0084] Furthermore, when the request receiving unit 104 cannot identify a function based on the request received from the display terminal 200, the function identifying unit 105 uses the interactive AI function to generate a question for narrowing down functions to be identified from among functions related to the equipment 300 placed in the real space 400 reproduced by the three-dimensional model, based on the request received by the request receiving unit 104. Then, the response sending unit 108 sends a response including the generated question to the display terminal 200.

[0085] Furthermore, if the function identifying unit 105 cannot identify a function based on the request accepted by the request accepting unit 104 from the display terminal 200, the analysis unit 107 may execute an analysis, and identify a function related to the facility device 300 based on the result of the analysis by the analysis unit 107 and the request accepted by the request accepting unit 104. Then, the response sending unit 108 sends a response including information on the identified function to the display terminal 200.

[0086] Furthermore, the AR display unit 203 may display data of the facility device 300 collected by the engineering tool 100-2 of the simulation device 100 by immediately reflecting the data in the information of the function identified by the function identification unit 105. Specifically, when data such as a motor rotation speed that requires continuous and immediate display update is included in a widget displayed on the screen, the AR display unit 203 requests the simulation device 100 to perform a data update process repeatedly at regular intervals. The simulation device 100 transmits an update request to the engineering tool 100-2 and receives a response to the request, i.e., a response including the updated data, from the engineering tool 100-2, in the same manner as when updating data in a three-dimensional model. Then, when the widget management unit 102 receives the response from the engineering tool 100-2, it updates the data of the widget function, and the response sending unit 108 sends a response including the updated data to the display terminal 200. The AR display unit 203 displays the widget including the updated data.

[0087] Furthermore, the information on the function identified by the function identifying unit 105, which the AR display unit 203 displays in the image of the real space 400, may include a control that enables settings or operations on the facility device 300.

[0088] 10 shows a state in which a widget 607 including a control is displayed in a window 200-11. The widget 607 is a widget with a widget function that can set the notch frequency of the notch filter of device B placed in real space 400. The widget 607 includes an edit box 607-1 that is a control for inputting a character string, and a button 607-2 that is a control associated with a process. When the button 607-2 is selected, the notch frequency of device B is set to the value input in the edit box 607-1.

[0089] When the AR display unit 203 receives an operation on a control for inputting a character string, the AR display unit 203 displays the input character string in the widget control in the image of the real space 400. When the AR display unit 203 receives an operation on a control associated with a process, the request sending unit 201 sends a request to execute the process associated with the control to the simulation device 100. When the request receiving unit 104 of the simulation device 100 receives the request to execute the process associated with the control, the request receiving unit 104 requests the widget management unit 102 to execute the process associated with the control. When the widget management unit 102 receives the request, the widget management unit 102 requests the engineering tool 100-2 to execute the process associated with the control. When the widget function unit 109 of the engineering tool 100 receives the request, the widget function unit 109 executes the process associated with the control.

[0090] For example, when the second user 702 selects the edit box 607-1 by gesture and inputs a value of “1000” by voice, the AR display unit 203 determines that an operation on a control for inputting a character string has been accepted, and displays “1000” in the edit box 607-1. Next, when the second user 702 clicks the button 607-2 by gesture, the AR display unit 203 determines that an operation on a control associated with a process has been accepted, and the request sending unit 201 sends a request to the simulation device 100 to set the notch frequency of the notch filter of the device B to “1000” [Hz]. When the request receiving unit 104 of the simulation device 100 receives a request from the display terminal 200 to set the notch frequency of the notch filter of the device B to “1000” [Hz], the request receiving unit 104 requests the widget management unit 102 to set the notch frequency of the notch filter of the device B to “1000” [Hz]. The widget management unit 102 requests the engineering tool 100-2 to set the notch frequency of the notch filter of device B to “1000” [Hz], and the widget function of the engineering tool 100-2 sets the notch frequency of the notch filter of device B to “1000” [Hz].

[0091] Next, a widget display process that the simulation device 100 according to the present embodiment executes in response to a request made to the simulation device 100 will be described with reference to the flowchart of Fig. 11. The display process of Fig. 11 is executed, for example, when the request receiving unit 104 receives a request from a first user of the simulation device 100. Note that a plurality of widgets may be displayed on the simulation device 100 by executing the display process of Fig. 11 every time a request is received.

[0092] The function identification unit 105 acquires a list of widget functions imported from the engineering tool 100-2 from the widget management unit 102 (step S101). Next, the function identification unit 105 interprets the request accepted by the request acceptance unit 104 using an interactive AI function, and identifies a function that satisfies the request from the functions included in the acquired list of widget functions (step S102). Then, the function identification unit 105 judges whether the function identification is successful or not (step S103). If the function identification unit 105 judges that the function identification is successful (step S103; YES), the position identification unit 106 specifies the position, size, and orientation in which the information of the function identified by the function identification unit 105 is arranged in the three-dimensional model (step S104). The three-dimensional model display unit 103 arranges and displays the information of the function identified by the function identification unit 105 in the three-dimensional model at the position, size, and orientation specified by the position identification unit 106 (step S105). Furthermore, the function identifying unit 105 generates a response to the request and outputs the generated response (step S106). On the other hand, if the function identifying unit 105 determines that the function identification has not been successful (step S103; NO), the process proceeds to step S107.

[0093] For example, when the request receiving unit 104 receives a request by voice from the first user 701, "I want to diagnose a geared motor," as shown in FIG. 4, the function identifying unit 105 interprets the content of the request using the interactive AI function and identifies a widget function that performs "gear backlash estimation" from the list as a widget function that satisfies the request, "I want to diagnose a geared motor." The position identifying unit 106 identifies a position, size, and orientation in which the widget, "gear backlash estimation," is placed in the three-dimensional model. Then, the three-dimensional model display unit 103, as shown in FIG. 4, places and displays the widget, "gear backlash estimation," based on the position, size, and orientation identified by the position identifying unit 106. The function identifying unit 105 also generates a response, "The widget that estimates gear backlash has been displayed next to the device A," by referring to the position where the widget is placed, and outputs the generated response by voice. On the other hand, if the request receiving unit 104 receives a request such as "I would like to adjust the servo used in the conveyor" or "Please tell me the cause of the stop of device A" and determines that the function identification unit 105 cannot identify the function, the process proceeds to step S107.

[0094] The analysis unit 107 performs an analysis for detecting a failure of the facility equipment 300 based on the data collected from the facility equipment 300 (step S107). The function identification unit 105 identifies a function related to the facility equipment 300 included in the three-dimensional model based on the result of the analysis by the analysis unit 107 and the request accepted by the request acceptance unit 104 (step S108). Then, the function identification unit 105 judges whether the function identification is successful or not (step S109). If the function identification unit 105 judges that the function identification is successful (step S109; YES), the process proceeds to step S104. On the other hand, if the function identification unit 105 judges that the function identification is not successful (step S109; NO), the function identification unit 105 generates and outputs a question for narrowing down the functions to be identified from the functions related to the facility equipment 300 arranged in the real space 400 reproduced by the three-dimensional model based on the request accepted by the request acceptance unit 104 using an interactive AI function (step S110).

[0095] For example, when the request receiving unit 104 receives a request "Please tell me the cause of the stop of device A," the analysis unit 107 analyzes the data of the facility equipment used in device A among the data collected from the facility equipment 300, and obtains an analysis result that an alarm has occurred in the servo used in device A. Then, based on the analysis result that an alarm has occurred in the servo and the request "Please tell me the cause of the stop of device A," the analysis unit 107 identifies a widget function that executes "display of servo alarm information" from the list of widget functions. On the other hand, when the request receiving unit 104 receives a request "I would like to adjust the servo used in the conveyor," the function identification unit 105 cannot identify a function even using the analysis result. In this case, the function identification unit 105 uses the interactive AI function to generate a question "What kind of adjustment function do you want to use?" and an answer "Functions such as one-touch adjustment function and manual adjustment function can be used for the servo used in the conveyor," and outputs the question and the answer.

[0096] Next, a widget update process executed by the simulation device 100 according to the present embodiment will be described with reference to the flowchart of Fig. 12. The update process of Fig. 12 is repeatedly executed at predetermined intervals, for example, when data that requires continuous and immediate display update is included in a widget displayed on the screen.

[0097] The three-dimensional model display unit 103 requests the widget management unit 102 to update data acquired by the widget function of the widget displayed on the screen (step S201). The widget management unit 102 requests the engineering tool 100-2 to update data acquired by the widget function that received the update request (step S202). When the widget function unit 109 of the engineering tool 100-2 receives the data update request, it communicates with the facility equipment 300 that has the data related to the request, and acquires the data from the facility equipment 300 (step S203). When the widget function unit 109 acquires the data from the facility equipment 300, it sends the collected data to the simulation application 100-1 as a response to the update request (step S204). When the widget management unit 102 receives a response from the engineering tool 100-2, it updates the data of the widget function (step S205). The three-dimensional model display unit 103 displays the widget including the updated data (step S206).

[0098] For example, the three-dimensional model display unit 103 requests the widget management unit 102 to update the value of the rotation speed of the motor acquired by the widget function of the widget 604 in Fig. 7. The widget management unit 102 requests the engineering tool 100-2 to update the value of the rotation speed of the motor acquired by the widget function of the widget 604. The widget function unit 109 of the engineering tool 100-2 communicates with a motor control device arranged in the real space 400 to acquire the value of the rotation speed of the motor, and sends information on the acquired value to the simulation application 100-1 as a response to the update request. When the widget management unit 102 receives the response from the engineering tool 100-2, it updates the data of the widget function that acquires the rotation speed of the motor, and the three-dimensional model display unit 103 displays a widget including the updated data.

[0099] Next, a control process using a widget executed by the simulation device 100 according to the present embodiment will be described with reference to the flowchart of Fig. 13. The control process of Fig. 13 is executed, for example, when a widget displayed on the screen includes a control and an operation on the control is received.

[0100] The three-dimensional model display unit 103 judges whether or not an operation on a control for inputting a character string has been accepted (step S301). When the three-dimensional model display unit 103 judges that an operation on a control for inputting a character string has been accepted (step S301; YES), the three-dimensional model display unit 103 displays the input character string in the control of the widget in the three-dimensional model (step S302). On the other hand, when the three-dimensional model display unit 103 judges that an operation on a control for inputting a character string has not been accepted (step S301; NO), the three-dimensional model display unit 103 requests the widget management unit 102 to execute a process associated with the control (step S303). When the widget management unit 102 accepts the request, the widget management unit 102 requests the engineering tool 100-2 to execute a process associated with the control (step S304). Then, when the widget function unit 109 of the engineering tool 100-2 accepts the request, the widget function unit 109 executes the process associated with the control (step S305).

[0101] 8 using the mouse 1002 and inputs a value of "1000" using the keyboard 1001, the three-dimensional model display unit 103 determines that an operation on a control for inputting a character string has been accepted, and displays "1000" in the edit box 605-1. Next, when the first user 701 operates the mouse 1002 to click the button 605-2, the three-dimensional model display unit 103 determines that an operation on a control associated with the process has been accepted, and requests the widget management unit 102 to set the notch frequency of the notch filter of device B to "1000" [Hz]. The widget management unit 102 requests the engineering tool 100-2 to set the notch frequency of the notch filter of device B to “1000” [Hz], and the widget function of the engineering tool 100-2 sets the notch frequency of the notch filter of device B to “1000” [Hz].

[0102] Next, a widget display process executed by the simulation device 100 according to this embodiment in response to a request received from the display terminal 200 will be described with reference to the flowchart in Fig. 14. The display process in Fig. 14 is executed, for example, when the request receiving unit 104 receives a request from a second user of the display terminal 200. Note that the display process in Fig. 14 may be executed every time a request is received, so that a plurality of widgets may be displayed on the display terminal 200. The processes in steps S401 to S404 and steps S407 to S409 in Fig. 14 are the same as those in steps S101 to S104 and steps S107 to S109 in Fig. 11.

[0103] In step S404, when the position, size, and orientation to be placed in the three-dimensional model are specified, the function specifying unit 105 generates a response to the request using the interactive AI function (step S405). Then, the response sending unit 108 transmits a response including information on the function specified by the function specifying unit 105, information on the position, size, and orientation specified by the position specifying unit 106, and information on the generated response to the display terminal 200 (step S406).

[0104] For example, the function identification unit 105 generates a response stating, "A widget for estimating gear backlash has been displayed next to device A.", and the response sending unit 108 sends a response to the display terminal 200, which includes information identifying the widget function of "gear backlash estimation," information on the position, size, and orientation of the widget, and information on the response stating, "A widget for estimating gear backlash has been displayed next to device A."

[0105] In step S409, if the function identification unit 105 determines that the function identification was not successful (step S409; NO), the function identification unit 105 uses the interactive AI function to generate a question for narrowing down the functions to be identified from among the functions related to the equipment 300 placed in the real space 400 reproduced by the three-dimensional model, and the response sending unit 108 transmits a response including the generated question to the display terminal 200 (step S410).

[0106] For example, the function identification unit 105 uses the interactive AI function to generate a question such as "What adjustment function do you want to use?" and a response such as "For servos used in conveyors, functions such as one-touch adjustment function and manual adjustment function are available." Then, the response sending unit 108 transmits a response including the generated question and answer information to the display terminal 200.

[0107] According to this embodiment, the simulation device analyzes requests input by a user using an interactive AI function, so that even if the request is written in natural language that contains ambiguity, information on the functions of appropriate equipment that meets the user's request can be displayed in a three-dimensional model.

[0108] Furthermore, according to this embodiment, the interactive AI function of the simulation device is used to identify functions that satisfy the request from the user of the display terminal, so that even if the request from the user of the display terminal is made in a natural language that includes ambiguity, appropriate information that satisfies the user's request can be displayed. As a result, information on the appropriate facility equipment functions can be displayed superimposed on an image of the real space on the AR display-enabled terminal used by the on-site worker.

[0109] Furthermore, according to this embodiment, even if the requirements from the user of the simulation device or the display terminal are vague and it is difficult to identify a function that satisfies the requirements, a question for identifying a function can be generated using the dialogue AI function, and the user can be prompted to present further requirements. This makes it possible to identify an appropriate function that satisfies the requirements.

[0110] Furthermore, according to this embodiment, even if the requirements from the user of the simulation device or the display terminal are vague and it is difficult to identify a function that satisfies the requirements, by using the dialogue AI function and the analysis function that detects failures, even a user without specialized knowledge can use an advanced analysis function such as the analysis AI function. This makes it possible to easily detect the cause of a failure in equipment or a production line in which the equipment is installed, and to present information on the cause of the failure.

[0111] According to the present embodiment, the functions of an engineering tool can be easily displayed as widgets on a simulation device or a display terminal, allowing the functions of the engineering tool to be easily used by operating the widgets without operating an engineering tool that requires complex pre-settings.

[0112] (Modification) Although the embodiment of the present disclosure has been described above, various modifications and applications are possible in implementing the present disclosure.

[0113] In the above embodiment, the relationship between the simulation system 1 and the facility equipment 300 shown in Fig. 1 is an example, and is not limited thereto. For example, there may be a plurality of simulation devices 100 or display terminals 200. Furthermore, the simulation system 1 may be capable of communicating with the facility equipment 300 arranged in a plurality of different real spaces 400.

[0114] In the above embodiment, the engineering tool 100-2 is installed in the simulation device 100, but the present invention is not limited to this. For example, the engineering tool may be installed in another device that is communicatively connected to the simulation device 100.

[0115] In the above embodiment, the display terminal 200 is a device capable of AR display in which a virtual object is superimposed on an image of the real space 400, but the display terminal 200 is not limited to this. For example, the display terminal 200 may be a device capable of MR (Mixed Reality) display using a translucent head-up display. In the case of the display terminal 200 capable of MR display, the input unit 15 of the display terminal 200 is a positioning device such as a three-dimensional scanner, and the output unit 16 is a translucent image output device such as a head-up display, and the positioning device specifies the position and orientation of the display terminal 200, and the virtual object is synthesized in the real space 400 displayed on the translucent screen. The display terminal 200 may be a glasses type, a mobile terminal type, or the like, and may have any shape.

[0116] In the above embodiment, requests to the simulation apparatus 100 and the display terminal 200 are made by voice input, but this is not limited to this. Requests may be made by text input using the keyboard 1001 of the simulation apparatus 100 or a keyboard externally attached to the display terminal 200.

[0117] In the above embodiment, an example in which a widget is displayed on the simulation device 100 based on a request from the first user 701 of the simulation device 100 and an example in which a widget is displayed on the display terminal 200 based on a request from the second user 702 of the display terminal 200 have been described, but the present invention is not limited to this. For example, a widget may be displayed on the simulation device 100 and the display terminal 200 simultaneously based on a request received from the first user 701 of the simulation device 100. Also, a widget may be displayed on the simulation device 100 and the display terminal 200 simultaneously based on a request received from the second user 702 of the display terminal 200. Also, the simulation device 100 may display a widget on the simulation device 100 and the display terminal 200 based on a request from a user who uses a device different from the simulation device 100 and the display terminal 200.

[0118] In the above embodiment, the simulation device 100 has an interactive AI function, but the simulation device 100 does not need to have the interactive AI function. For example, the simulation device 100 may request the engineering tool 100-2 to update data to be displayed in a widget function identified without using the interactive AI function. Then, the simulation device 100 or the display terminal 200 may display a widget including the updated data.

[0119] In the above embodiment, the flowcharts of the widget display process executed by the simulation device 100 are shown in Fig. 11 and Fig. 14, but the present invention is not limited to this. For example, the process of steps S107 to S109 in Fig. 11 may be omitted, and the process of steps S103, S107 to S110 may be omitted. Furthermore, the process of steps S407 to S409 in Fig. 14 may be omitted, and the process of steps S403, S407 to S410 may be omitted.

[0120] In the above embodiment, the simulation system 1 is applied to an FA system installed in a factory, but the system to which the simulation system 1 is applied is not limited to this. For example, the simulation system 1 can be applied to a power system, a water treatment system, a railway system, a building facility management system, an air conditioning equipment management system, etc.

[0121] In addition, by applying an operating program that specifies the operation of the simulation device 100 according to the above embodiment to an existing personal computer or information terminal device, it is also possible to cause the personal computer or information terminal device to function as the simulation device 100 according to the embodiment.

[0122] Furthermore, the method of distribution of such a program is arbitrary; for example, the program may be stored on a computer-readable recording medium such as a CD-ROM (Compact Disk Read-Only Memory), a DVD (Digital Versatile Disk), or a memory card and distributed, or the program may be distributed via a communications network such as the Internet.

[0123] Various embodiments and modifications of the present disclosure are possible without departing from the broad spirit and scope of the present disclosure. The above-described embodiments are for explaining the present disclosure and do not limit the scope of the present disclosure. In other words, the scope of the present disclosure is indicated by the claims, not the embodiments. Various modifications made within the scope of the claims and the scope of the disclosure equivalent thereto are considered to be within the scope of the present disclosure. [Industrial Applicability]

[0124] According to the present disclosure, it is possible to provide a simulation program, a simulation device, a simulation system, and a display method capable of displaying appropriate information that meets user requirements in a three-dimensional model that reproduces real space. [Explanation of symbols]

[0125] 1 Simulation system, 10 Information processing device, 11 Processor, 12 Main memory unit, 13 Auxiliary memory unit, 14 Communication unit, 15 Input unit, 16 Output unit, 17 Bus, 100 Simulation device, 100-1 Simulation application, 100-2 Engineering tool, 100-11 Window, 101 Three-dimensional model management unit, 102 Widget management unit, 103 Three-dimensional model display unit, 104 Request reception unit, 105 Function identification unit, 106 Position identification unit, 107 Analysis unit, 108 Response sending unit, 109 Widget function unit, 200 Display terminal, 200-1 AR display application, 200-11 Window, 201 Request sending unit, 202 Response reception unit, 203 AR display unit, 300 Equipment, 400 Real space, 500 Communication network, 601 to 607 Widget, 604-1 Columns, 605-1, 607-1 edit boxes, 605-2, 607-2 buttons, 701 first user, 702 second user, 1001 keyboard, 1002 mouse.

Claims

1. Computers, A three-dimensional model display means that displays a three-dimensional model that reproduces the real space in which equipment and facilities are arranged. Request receiving means for receiving requests for equipment and devices included in the three-dimensional model, A function identification means that interprets the request received by the request receiving means using an interactive AI function to identify a function relating to the equipment that satisfies the request. Based on the data collected from the aforementioned equipment, it functions as an analysis means for detecting failures in the aforementioned equipment. The three-dimensional model display means displays information about the function identified by the function identification means in the three-dimensional model. If the function identification means cannot identify the function based on the request received by the request receiving means, it identifies the function relating to the equipment included in the three-dimensional model based on the results of the analysis by the analysis means and the request received by the request receiving means. Simulation program.

2. The aforementioned computer, The information of the function identified by the function identification means is further used as a position identification means to identify the position, size, and orientation of the three-dimensional model. The three-dimensional model display means displays the information of the function identified by the function identification means in the three-dimensional model, placing it at the position, size, and orientation identified by the position identification means. The simulation program according to claim 1.

3. The request receiving means receives requests for equipment located in the real space, received from a display terminal capable of displaying images of the real space and connected to the computer, as requests for equipment included in the three-dimensional model. The aforementioned computer, When the request receiving means receives a request from the display terminal for equipment located in the real space, it is configured to function as a response sending means that sends a response to the display terminal that includes information about the function identified by the function identifying means and information about the location, size, and orientation identified by the location identifying means. The display terminal overlays the information of the function included in the response onto the image of the real space, based on the position, size, and orientation information included in the response. The simulation program according to claim 2.

4. If the function identification means cannot identify the function based on the request received by the request receiving means, it uses the conversational AI function to generate questions to narrow down the function to be identified from among the functions of the equipment placed in the real space reproduced by the three-dimensional model, based on the request received by the request receiving means. A simulation program according to any one of claims 1 to 3.

5. The information of the function identified by the function identification means includes controls that enable settings or operations on the equipment. A simulation program according to any one of claims 1 to 3.

6. The functions relating to the aforementioned equipment are those of an engineering tool that is communicatively connected to the equipment located in the physical space. A simulation program according to any one of claims 1 to 3.

7. The three-dimensional model display means displays the data of the equipment collected by the engineering tool, immediately reflecting it in the information of the function identified by the function identification means. The simulation program according to claim 6.

8. A three-dimensional model display means that displays a three-dimensional model that reproduces the real space in which the equipment is placed, A request receiving means for receiving requests for equipment and devices included in the three-dimensional model, A function identification means that identifies a function relating to the equipment that satisfies the request by interpreting the request received by the request receiving means using an interactive AI function, The system includes an analysis means that performs analysis for detecting malfunctions of the equipment based on data collected from the equipment, The three-dimensional model display means displays information about the function identified by the function identification means in the three-dimensional model. If the function identification means cannot identify the function based on the request received by the request receiving means, it identifies the function relating to the equipment included in the three-dimensional model based on the results of the analysis by the analysis means and the request received by the request receiving means. Simulation device.

9. A simulation system comprising a simulation device that reproduces the real-world space in which equipment is placed in a digital space, and a display terminal capable of displaying images of the real-world space, which are connected in a communication manner, The aforementioned simulation device is A three-dimensional model display means that displays a three-dimensional model that reproduces the real space in which the aforementioned equipment is placed, A request receiving means for receiving requests for equipment and devices included in the three-dimensional model, A function identification means that identifies a function relating to the equipment that satisfies the request by interpreting the request received by the request receiving means using an interactive AI function, A positioning means for determining the position, size, and orientation of the function information identified by the function identification means in the three-dimensional model, The system includes an analysis means that performs analysis for detecting malfunctions of the equipment based on data collected from the equipment, The three-dimensional model display means displays the information of the function identified by the function identification means in the three-dimensional model, arranging it at the position, size, and orientation identified by the position identification means. If the function identification means cannot identify the function based on the request received by the request receiving means, it identifies the function of the equipment included in the three-dimensional model based on the results of the analysis by the analysis means and the request received by the request receiving means. The request receiving means receives requests for equipment located in the real space from the display terminal as requests for equipment included in the three-dimensional model. When the request receiving means receives a request from the display terminal for equipment located in the real space, the response sending means further provides the display terminal with a response that includes information about the function identified by the function identifying means and information about the location, size, and orientation identified by the location identifying means. The aforementioned display terminal is The simulation device includes a request sending means for sending requests to equipment and devices placed in the real space, The simulation device includes a response receiving means for receiving the response, The system includes an AR display means that displays information about the functions included in the response received by the response receiving means, superimposed on the image of the real space based on the position, size, and orientation information included in the response received by the response receiving means. Simulation system.

10. A display method performed by a simulation device, The aforementioned simulation device A three-dimensional model is displayed that reproduces the real-world space where the equipment and facilities are placed. We accept requests for equipment and machinery included in the aforementioned three-dimensional model. By interpreting the received request using an interactive AI function, the function of the equipment that satisfies the request is identified. Based on the data collected from the aforementioned equipment, an analysis is performed to detect malfunctions in the aforementioned equipment. If the function cannot be identified based on the received request, the function of the equipment included in the three-dimensional model is identified based on the results of the analysis and the received request. In the aforementioned three-dimensional model, information on the identified function is displayed. Display method.

Images