Agent system and method for generating technical management information for agent system

Abstract

Collect tacit knowledge from experts. [Solution] The agent system 2 is characterized by comprising a video playback unit 26 that displays a video of the site on a display 41 and presents it to the user 4, a questioning unit 27 that asks the agent using a large-scale language model 200 questions of the user 4 about how to respond to each situation in the manufacturing site 1, and a data collection unit 28 that collects dialogue between the agent and the user 4.

Description

[Technical Field]

[0001] The present invention relates to an agent system and a method for generating technical management information for an agent system. [Background technology]

[0002] The retirement of skilled engineers due to the aging of the population in recent years has become a major issue. Before these skilled engineers retire, their skills must be passed on, but in this process, it is important to draw out their invisible tacit knowledge. If the tacit knowledge of skilled engineers can be fully extracted, it will be possible to build specialized large-scale language models (LLMs) with knowledge equivalent to that of experts, as well as highly accurate chatbots.

[0003] Patent Document 1 describes an invention of a chatbot system that trains communication skills in specific situations. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2023-171705 Summary of the Invention [Problem to be solved by the invention]

[0005] The construction method for specialized LLMs is clear, but the information to be learned must include tacit knowledge. In the first place, experts are not aware that their own knowledge is tacit knowledge. Experts often use tacit knowledge in their actions as a matter of course, or when faced with an emergency. However, it is difficult to create all emergency situations in reality. Furthermore, experts are often not good at explaining technology to others in an easy-to-understand manner or documenting it.

[0006] Therefore, an object of the present invention is to collect tacit knowledge from experts. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems, the agent system of the present invention comprises a site image display unit that displays a reproduced site image to a user, a questioning unit that causes an agent using a large-scale language model to ask the user questions about how to respond to each situation at the site, and a data collection unit that collects dialogue between the agent and the user. When the questioning unit detects an utterance from the user, the questioning unit starts a dialogue with the user based on the utterance. It is characterized by: The method for generating technical management information for an agent system of the present invention includes the steps of: a site image display unit of the agent system displays an image of a reproduced site to a user; a questioning unit of the agent system having an agent using a large-scale language model ask the user questions about how to respond to each situation at the site; a data collection unit of the agent system collects dialogue between the agent and the user; and a site technical data generation unit of the agent system generates information related to site technical management from the dialogue collected by the data collection unit. a step of starting a dialogue with the user based on the utterance when the questioning unit detects the utterance of the user; The present invention is characterized by comprising: Other means will be described in the detailed description of the invention. [Effects of the Invention]

[0008] According to the present invention, it is possible to collect tacit knowledge from experts. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a configuration diagram of an agent system according to a first embodiment. [Figure 2] FIG. 10 is a diagram showing maintenance record information. [Figure 3] FIG. 2 is a diagram illustrating some functional units of the agent system. [Figure 4] FIG. 10 is a diagram illustrating prompts used by the data preprocessing unit. [Figure 5] FIG. 10 is a diagram showing facility coordinate failure occurrence information. [Figure 6] FIG. 10 is a diagram showing a prompt used by the question content generator. [Figure 7] FIG. 10 is a diagram showing question information. [Figure 8] FIG. 2 is a diagram illustrating some functional units of the agent system. [Figure 9A] FIG. 10 is a diagram illustrating a dialogue screen between an agent and a user. [Figure 9B] FIG. 10 is a diagram illustrating a dialogue screen between an agent and a user. [Figure 9C] FIG. 10 is a diagram illustrating a dialogue screen between an agent and a user. [Figure 10] 10 is a flowchart of a process of a video playback unit. [Figure 11] 10 is a flowchart of a process performed by a questioning unit. [Figure 12] FIG. 10 is a diagram showing dialogue information. [Figure 13] FIG. 10 illustrates prompts used by the data collector. [Figure 14] FIG. 10 is a diagram showing question and answer data. [Figure 15] FIG. 10 is a configuration diagram of an agent system according to a second embodiment. [Figure 16] FIG. 2 is a diagram illustrating some functional units of the agent system. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this system, a generation AI generates videos showing situations such as breakdowns and accidents, and then synthesizes them into a walkthrough video of the site. The user then converses with the agent while watching the video of the virtual manufacturing site. In this way, the system of this embodiment collects on-site technical data, including tacit knowledge, from the user.

[0011] In other words, the system of this embodiment displays various situations, including emergencies, to the user through video, and then has an agent ask questions about how to respond to each situation, thereby eliciting implicit knowledge from the user in all situations. The agent that asks the user questions is assumed to be the grandchild of an elderly engineer, who is likely to find meaning in life.

[0012] According to a Cabinet Office survey of four countries, including Japan, on when people feel a sense of purpose in life, the answer that was given most frequently in all countries was "spending time with family, such as children and grandchildren." In Japan, 55.3% of people responded that they feel a sense of purpose in life when spending time with family, such as children and grandchildren. Therefore, the agent system of this embodiment uses "grandchildren" as the agent that asks questions to the user.

[0013] FIG. 1 is a configuration diagram of an agent system 2 according to the first embodiment. The agent system 2 includes a large-scale language model 200, a data preprocessing unit 21 for carrying out advance preparations, an emergency situation generation unit 22, a mapping information generation unit 23, and a question content generation unit 25.

[0014] The data preprocessing unit 21 converts the maintenance record information 31 and the equipment installation location information 32 of the equipment maintenance management system 3 into Markdown format by processing using the large-scale language model 200 or rule-based processing. The equipment maintenance management system 3 is a system for supporting the maintenance management of each piece of equipment installed in a factory, plant, etc. Maintenance record information 31 is information that records the maintenance of each piece of equipment installed in a factory, plant, etc. Equipment installation location information 32 is information that stores the installation location of each piece of equipment installed in a factory, plant, etc.

[0015] The emergency situation generation unit 22 uses an image generation AI to generate an image showing an emergency situation based on the maintenance record information 31 converted into Markdown format. The maintenance record information 31 includes the situation at the time of the failure, etc. For example, the emergency situation generation unit 22 reads an image of factory equipment and instructs the image generation AI to issue a prompt that causes an event such as a water leak or smoke to occur. This generates a video in which the event described in the prompt occurs.

[0016] Specifically, the emergency situation generation unit 22 inputs an image of a tank in a factory and instructs the image generation AI with the prompt, "Please make water leak from this tank." This causes the image generation AI to generate a video of water leaking from a tank in a factory. The emergency situation generation unit 22 repeats this process for each piece of equipment to obtain a video of an emergency at each piece of equipment.

[0017] The mapping information generation unit 23 maps corresponding equipment to the location information recorded as metadata of the site walk-through video 111, based on the information obtained by converting the maintenance record information 31 into Markdown format. This site walk-through video 111 was captured at the manufacturing site 1 by the 360-degree camera 11 along with the location information. Note that the mapping information generation unit 23 may map corresponding equipment based on the time information of the site walk-through video 111 on a rule-based basis, and is not limited to this.

[0018] The question content generation unit 25 generates a question for the user 4 based on the maintenance record information converted into Markdown format by the data preprocessing unit 21. The question content generation unit 25 generates a question in a tone that reproduces a "question from a grandchild" in response to a prompt instruction to the large-scale language model 200.

[0019] The video composition unit 24 uses a machine learning model to compose the site walk-through video 111, which has been previously captured by the 360-degree camera 11, with the emergency site video 221 based on the time information and facility mapping information of the site walk-through video 111. In this way, the video composition unit 24 creates a video of the site space where the emergency occurred.

[0020] The agent system 2 further includes a video playback unit 26, a questioning unit 27, a data collection unit 28, a field technical data generation unit 29, and field technical data 291, which are used during use.

[0021] The video playback unit 26 plays the video on the display 41, thereby displaying a video that recreates the scene space to the user 4. The video playback unit 26 functions as a scene video display unit that displays the video that recreates the scene to the user 4. The video playback unit 26 displays a video that combines a scene video of the emergency with a scene video that was shot in advance on the display 41, which is a display device. Then, when the video displays the emergency equipment, the video playback unit 26 causes the questioning unit 27 to start asking the user 4 questions about the content of the emergency and how to respond.

[0022] The user 4 visually views the video of the site space on the display 41. When a specific piece of equipment appears on the video of the site space, a question prepared in advance is output in a synthesized voice from the smartphone 42. The user 4 answers this question by voice.

[0023] When the video playback unit 26 displays a specific facility on the display 41, the questioning unit 27 asks the user 4 a question generated in advance by the question content generation unit 25 using synthesized voice and text display. The user 4 responds to this question by voice. The questions and answers are one question and one answer. The questioning unit 27 continues the dialogue by repeatedly asking probing questions such as "Why?" for each answer. From the start of this dialogue until its end, the video playback unit 26 pauses the playback of the video. This prevents the video playback unit 26 from ending the playback of the facility information of the target facility during the dialogue and from playing back the next facility information of the target facility. In addition, the sound of the site walkthrough video 111 can be muted during the dialogue, so as not to interrupt the dialogue between the agent and the user 4.

[0024] The questioning unit 27 causes an agent using the large-scale language model 200 to ask the user 4 questions, using synthesized voice and text, about how to respond to various situations that arise in each piece of equipment at the manufacturing site 1.

[0025] Furthermore, the questioning unit 27 can accept utterances made by the user 4 without any prior questioning. The questioning unit 27 accepts a specific keyword such as "Hey, you know" as a trigger for utterances by the user 4. In other words, when the questioning unit 27 detects an utterance from the user 4, it starts a dialogue with the user 4 based on this utterance. This enables the questioning unit 27 to extract tacit knowledge based on video in normal situations.

[0026] When the answer of user 4 is repeated a predetermined number of times, or when a predetermined keyword is detected from the answer of user 4, questioning unit 27 summarizes the conversation up to that point in synthesized voice and text to user 4, and ends the dialogue with user 4. This dialogue makes it possible to appropriately collect the tacit knowledge possessed by user 4.

[0027] The questioning unit 27 further records the conversation between the user 4 and the agent, and converts the user 4's response into text information to generate the next, more in-depth question. Note that the conversation between the user 4 and the agent may be recorded by the questioning unit 27 or by the smartphone 42.

[0028] The data collection unit 28 collects video data of the portions played by the video playback unit 26 during the dialogue between the agent and the user 4, and text information of the dialogue between the agent and the user 4 collected by the questioning unit 27. By collecting video data of the portions played by the video playback unit 26 during the dialogue between the agent and the user 4 by the data collection unit 28, it is possible to check the portions in which the user 4 explains using pronouns, etc. after the fact in the video.

[0029] The field technical data generation unit 29 generates and manages field technical data 291 related to field technical management from the video data and dialogue text information using the large-scale language model 200. The field technical data 291 is question-and-answer data in which questions related to equipment failures and their answers are associated with each other. This field technical data 291 reflects the tacit knowledge of the user 4. The field technical data generation unit 29 may also generate and manage the field technical data 291 related to field technical management on a rule-based basis from the video data and dialogue text information. The field technical data generation unit 29 may store, as the field technical data 291, a video clip containing a still image of the equipment related to the question in addition to a combination of a question and its answer. This makes it possible to check the content of the field technical data 291 later using video.

[0030] FIG. 2 is a diagram showing the maintenance record information 31. As shown in FIG. The maintenance record information 31 includes an ID column, a facility name column, an installation location column, a failure content column, a failure cause column, and a response content column.

[0031] The ID column stores identification information for the maintenance record. The equipment name column stores the name of each piece of equipment installed at the manufacturing site 1. The installation location field stores the location of each piece of equipment installed at the manufacturing site 1. The fault content column stores the details of a fault that occurs in each piece of equipment installed at the manufacturing site 1.

[0032] The fault cause column stores the cause of a fault that occurs in each piece of equipment installed at the manufacturing site 1. The response content column stores the content of a response to a fault that occurs in each piece of equipment installed at the manufacturing site 1.

[0033] FIG. 3 is a diagram illustrating some of the functional units of the agent system 2. As shown in FIG. Maintenance record information 31 and equipment installation location information 32 are input to the data pre-processing unit 21 of the agent system 2, and maintenance record Markdown information 211 and equipment installation location Markdown information 212 are output. The data pre-processing unit 21 may perform processing using a large-scale language model 200 or rule-based processing, and is not limited to this.

[0034] The maintenance record markdown information 211 and the equipment installation location markdown information 212 are input to the mapping information generation unit 23. The mapping information generation unit 23 outputs equipment coordinate failure occurrence information 231. The mapping information generation unit 23 may perform processing using either the large-scale language model 200 or rule-based processing, and is not limited thereto.

[0035] The maintenance record Markdown information 211 is further input to the question content generation unit 25. Then, the question content generation unit 25 generates question information 251. The question content generation unit 25 may perform processing using either the large-scale language model 200 or rule-based processing, and is not limited thereto.

[0036] 4 is a diagram showing a prompt 230 used by the mapping information generating unit 23. The prompt 230 is transcribed below in text form.

[0037] # Task Please generate coordinate information for each facility under the following constraints. # Constraints Generate in 3D -Include equipment in case of failure

[0038] The mapping information generation unit 23 outputs equipment coordinate failure occurrence information 231 based on the maintenance record Markdown information 211 and the equipment installation location Markdown information 212. The mapping information generation unit 23 generates the equipment coordinate failure occurrence information 231, which is mapping information, by providing a prompt 230 to the large-scale language model 200. Note that the present invention is not limited to this, and the mapping information generation unit 23 may also generate the equipment coordinate failure occurrence information 231, which is mapping information, on a rule basis, and is not limited thereto.

[0039] FIG. 5 is a diagram showing the facility coordinate failure occurrence information 231. As shown in FIG. The facility coordinate failure occurrence information 231 includes a facility name column, a coordinate column, and a failure occurrence status column. The equipment name column stores the name of each piece of equipment. The coordinate column stores the three-dimensional coordinates of each piece of equipment. The failure occurrence status column stores text describing the status of failures that may occur in each piece of equipment.

[0040] FIG. 6 is a diagram showing a prompt 250 used by the question content generator 25. As shown in FIG. Prompt 250 is transcribed below.

[0041] # Task Asking 5 why questions about conservation records with the following constraints Please # Constraints -Speaks in a tone similar to a grandchild asking a question to his grandfather Ask questions from a child's perspective Ask questions about each facility The timing to ask questions is when you reach a specific facility. -Continuing to ask questions about answers · Don't ask a question on the third answer, just respond

[0042] The question content generator 25 generates question information 251 by providing the prompt 250 of Fig. 6 to the large-scale language model 200. This question information 251 is shown in Fig. 7, which will be described later.

[0043] In addition, in consideration of the case where the expert user 4 is female, it is advisable to have the user input the gender in advance. If the expert user 4 is female, the prompt 250 is modified to include a constraint such as "speak in a tone like a grandchild asking a question to her grandmother." This allows the expert user 4 to ask a question that feels natural even if she is female.

[0044] 7 is a diagram showing question information 251. The question information 251 contains questions that the agent will ask the user 4 when the user arrives at each facility. The text of the question information 251 is transcribed below.

[0045] When you reach the conveyor belt: "Grandpa, why is the conveyor belt smoking? Is it because the motor is too hot?" When the cooling tank is reached: "Grandpa, why was water dripping from the cooling tank? Is the tank old?" When you reach the press: "Grandpa, why was the press rattling? Why do bearings wear out?" When you reach the boiler: "Grandpa, why was steam coming out of the boiler? Why do gaskets deteriorate?" When you reach the air compressor: "Grandpa, why was there a strange noise and smoke coming from the air compressor? How do the internal parts break down?"

[0046] In this way, by having an agent modeled after a grandchild, who is likely to give elderly engineers a sense of purpose in life, ask questions, the elderly engineers are more likely to feel a sense of purpose in life and are more likely to gather tacit knowledge than if they were asked questions by a more dry agent.

[0047] FIG. 8 is a diagram illustrating some of the functional units of the agent system 2. The emergency situation generation unit 22 generates an emergency scene video 221 by providing the normal situation scene video 12 and an abnormality occurrence prompt 220 to the large-scale language model 200 . When the video composition unit 24 receives the site walkthrough video 111, the emergency site video 221, and the equipment coordinate failure occurrence information 231, the video composition unit 24 combines the site walkthrough video 111 and the emergency site video 221 to generate a composite video 241, and also generates failure occurrence meta information 242. The failure occurrence meta information 242 is meta information indicating which part of the composite video 241 contains a scene of equipment in which a failure has occurred. Note that the failure occurrence meta information 242 may be stored in each frame of the composite video 241, and is not limited thereto.

[0048] The video playback unit 26 plays the composite video 241 on the display 41, and also refers to the failure occurrence meta information 242, and when a scene of the equipment in which a failure has occurred is played back, notifies the question unit 27 of that fact. When the question unit 27 receives a notification of which facility the fault scene is related to based on the question information 251, the question unit 27 starts questioning the user 4 via the smartphone 42.

[0049] 9A to 9C are diagrams illustrating a screen 421 displaying a dialogue between an agent and a user 4. FIG. 9A, an agent icon 51, a speech bubble 52a indicating the agent's question, a user icon 53, a speech bubble 54a indicating the answer of user 4, and a microphone icon 55 are displayed on screen 421 of smartphone 42. This microphone icon 55 is an icon indicating that the voice of user 4 is being recorded.

[0050] Agent icon 51 is an icon showing an agent simulating the grandchild of user 4. Questioning unit 27 uses an application installed on smartphone 42 to output the agent's question in synthetic voice, and also displays speech bubble 52a displaying the question in text on screen 421. This allows user 4 to understand the question by checking speech bubble 52a even if he or she misses the synthetic voice spoken by the agent.

[0051] The user 4 speaks a response to the agent's question. The smartphone 42 records the content of this speech and outputs it to the query unit 27. The query unit 27 converts the recorded data into a response text using the large-scale language model 200 and displays the response text in the speech bubble 54c. This allows the user 4 to visually confirm whether the agent has correctly heard his or her response. Furthermore, the questioning unit 27 uses the large-scale language model 200 to generate a second question that delves deeper into this answer.

[0052] Screen 421 of smartphone 42 in Figure 9B displays agent icon 51, speech bubble 52b indicating the agent's second question, user icon 53, speech bubble 54b indicating user 4's second answer, and microphone icon 55.

[0053] Questioning unit 27 causes an application installed on smartphone 42 to output the agent's second question in a synthesized voice, and displays on screen 421 a speech bubble 52b displaying the second question in text form.

[0054] The user 4 utters a second answer to the agent's second question. The smartphone 42 records this speech and outputs it to the questioning unit 27. The questioning unit 27 converts the recorded data into a second answer text using the large-scale language model 200 and displays this second answer text in the speech bubble 54b. Furthermore, the questioning unit 27 uses the large-scale language model 200 to generate a third question that delves deeper into the second answer.

[0055] Screen 421 of smartphone 42 in Figure 9C displays agent icon 51, speech bubble 52c indicating the agent's third question, user icon 53, speech bubble 54c indicating user 4's third answer, speech bubble 52d indicating the agent's summary of the conversation, and microphone icon 55.

[0056] The questioning unit 27 uses an application installed on the smartphone 42 to output the agent's question in a synthesized voice, and also displays a balloon 52c on the screen 421 displaying the question in text form.

[0057] User 4 utters a third answer to the agent's third question. Smartphone 42 records this utterance and outputs it to question unit 27. Question unit 27 converts this recorded data into a third answer text using large-scale language model 200 and displays this third answer text in speech bubble 54c. Furthermore, question unit 27 generates text summarizing the dialogue up to that point using large-scale language model 200 and displays this summary text in speech bubble 54d.

[0058] FIG. 10 is a flowchart of the processing of the video playback unit 26. First, the video playback unit 26 starts playing the video (step S10). Then, the video playback unit 26 determines whether the question unit 27 has detected the start of a question by the user 4 (step S11). If the start of a question by the user 4 has been detected (Yes), the process proceeds to step S14. If the start of a question by the user 4 has not been detected (No), the process proceeds to step S12.

[0059] In step S12, the video playback unit 26 determines whether the scene being played back is a failure location. If the scene being played back is not a failure location (No), the process returns to step S11. If the scene being played back is a failure location (Yes), the process proceeds to step S13.

[0060] In step S13, the video playback unit 26 instructs the questioning unit 27 to start asking questions about the location of the failure. Then, the video playback unit 26 pauses the video that is being played back (step S14).

[0061] The video playback unit 26 determines whether the interrogator 27 has finished the dialogue (step S15). If the interrogator 27 has not finished the dialogue (No), the process returns to step S15. If the interrogator 27 has finished the dialogue (Yes), the process returns to step S10.

[0062] 11 is a flowchart of the processing of the interrogator 27. This flowchart will be explained with reference to FIG. First, the questioning unit 27 determines whether or not a conversation spontaneously uttered by the user 4 has been accepted from the voice recorded by the microphone of the smartphone 42 (step S20). If a conversation uttered by the user 4 has been accepted (Yes), the process proceeds to step S23 to generate an in-depth question, and then to step S24. If a question uttered by the user 4 has not been accepted (No), the process proceeds to step S21.

[0063] In step S21, the questioning unit 27 determines whether or not it has received an instruction to start asking questions about the failure location from the video playback unit 26. If it has not received an instruction to start asking questions about the failure location (No), the process returns to step S20. If it has received an instruction to start asking questions about the failure location (Yes), the process proceeds to step S22.

[0064] In step S22, the questioning unit 27 starts asking questions from the agent, and in step S24, the questioning unit 27 records the answers of the user 4. Next, the questioning unit 27 determines whether or not a condition for ending the dialogue with the user 4 has been met (step S25). The condition for ending the dialogue is that the user 4 has answered a predetermined number of times, that the answers of the user 4 contain a specific keyword, or that the user 4 has remained silent for a predetermined period of time.

[0065] If the interrogator 27 has ended the dialogue (Yes), the process proceeds to step S28. If the condition for ending the dialogue is not met (No), the process proceeds to step S26.

[0066] In step S26, questioning unit 27 generates an in-depth question. Then, questioning unit 27 outputs the question from the agent in synthesized voice and text via smartphone 42 (step S27), and the process returns to step S24.

[0067] In step S28, questioning unit 27 generates a summary of the answers. Then, questioning unit 27 outputs the summary of the answers from the agent in synthesized voice and text via smartphone 42 (step S29), and the process returns to step S20.

[0068] FIG. 12 is a diagram showing the dialogue information 271. This dialogue information 271 is a summary of the dialogue between the user 4 and the agent by the questioning unit 27. An example of the dialogue information 271 is transcribed below as text.

[0069] The purpose of checking the motor's operation is to make sure that it is working properly, because if it doesn't work, the whole machine won't work properly. Belts wear and deteriorate over time and need to be replaced, as old belts can break and cause serious problems. Sensor calibration is the process of adjusting the sensor so that it can measure accurately. If you don't do this, you will proceed with work based on incorrect data, so it's very important. If the cooling fan has stopped working, it could be due to a malfunction or a power supply problem. If the cooling fan doesn't work, the machine may get too hot and break down, so it's important to check it as soon as possible. Inspecting electrical wiring means making sure that the wires are connected correctly and that they're not damaged. Failure to do so could result in a short circuit or fire, so it's important to inspect them thoroughly. The reason the next inspection is in three months is because by checking regularly, we can deal with any problems before they become serious. Long-term inspections are necessary to make the machine last longer. Strengthening inspections of the cooling fans means checking them more frequently, as they are a particularly important part, and if the cooling is not working properly, it will affect the whole machine. When educating workers, we teach them the knowledge to work safely and how to use machines correctly. If they learn these things thoroughly, accidents can be prevented and work can proceed smoothly.

[0070] FIG. 13 is a diagram illustrating a prompt 280 used by the data collection unit 28. Prompt 280 is transcribed below.

[0071] # Task Please create Q&A data with the following constraints. # Constraints -Text files as answers to questions -Speak in a mechanical tone

[0072] The data collection unit 28 generates field technical data 291 by providing prompts 280 to the large-scale language model 200 .

[0073] FIG. 14 is a diagram showing the field technical data 291. The on-site technical data 291 includes a question column and an answer column. The question column stores questions about fault information for each piece of equipment. The answer column stores information summarizing the answers from the user 4 to each question. By training a machine learning model on this on-site technical data 291, it is possible to realize a chatbot that reflects the tacit knowledge of the user 4. Additionally, the on-site technical data 291 can be used to create FAQs (Frequency Asked Questions) or as materials for technical education.

[0074] FIG. 15 is a configuration diagram of an agent system 2A according to the second embodiment. Unlike the first embodiment, the agent system 2A according to the second embodiment includes a model deployment unit 20 and a local reproduction space display unit 26A. In the second embodiment, instead of displaying a moving image to the user 4 on a display 41, an image of a virtual space is displayed to the user 4 through virtual reality goggles 43.

[0075] The model development unit 20 develops the video information synthesized by the video synthesis unit 24 into a virtual space model. Then, the site reproduction space display unit 26A displays an image of the site reproduction space on the virtual reality goggles 43 based on this virtual space model. The site reproduction space display unit 26A functions as a site image display unit that displays an image of the site reproduction to the user 4. The user 4 wears virtual reality goggles 43 and experiences a walk-through of the virtual space. When the user 4 reaches the emergency facility in the virtual space, the scene reproduction space display unit 26A causes the questioning unit 27 to start asking questions to the user 4.

[0076] FIG. 16 is a diagram illustrating some of the functional units of the agent system 2A. The emergency situation generation unit 22 generates an emergency scene video 221 by providing the normal situation scene video 12 and an abnormality occurrence prompt 220 to the large-scale language model 200 .

[0077] When the video composition unit 24 receives the site walkthrough video 111, the emergency site video 221, and the equipment coordinate failure occurrence information 231, it combines the site walkthrough video 111 and the emergency site video 221 to generate a composite video 241, and also generates failure occurrence meta information 242. The failure occurrence meta information 242 is meta information indicating which part of the composite video 241 contains a scene of equipment in which a failure has occurred.

[0078] The model development unit 20 develops the composite video 241 and the failure occurrence meta information 242 into a virtual space model. Then, the site reproduction space display unit 26A displays an image of the virtual space that reproduces the site on the virtual reality goggles 43, based on the virtual space model into which the composite video 241 and the failure occurrence meta information 242 have been developed.

[0079] The site reproduction space display unit 26A further refers to the failure occurrence meta information 242 deployed in the virtual space. When the user 4 reaches the facility in the virtual space where a failure has occurred, the site reproduction space display unit 26A notifies the question unit 27 of that fact.

[0080] When the interrogator 27 receives a notification of which facility the faulty part has been reached based on the interrogator information 251, the interrogator 27 starts interrogating the user 4 using the smartphone 42.

[0081] The configuration and effects of the present invention will be described below.

[0082] [1] a site video display unit (video playback unit 26, site reproduction space display unit 26A) that displays a video of the site reproduction to the user (4); a questioning unit (27) that causes an agent using a large-scale language model (200) to ask the user (4) questions about how to respond to each situation in the field; a data collection unit (28) that collects conversations between the agent and the user (4); An agent system comprising:

[0083] This makes it possible to collect tacit knowledge from users who are experts.

[0084] [2] a field technical data generation unit (29) that generates field technical data (291) related to field technical management from the conversations collected by the data collection unit (28); 2. The agent system according to claim 1, further comprising:

[0085] This allows conversations collected from experts to be compiled as tacit knowledge.

[0086] [3] The on-site technical data generation unit (29) generates question and answer data related to on-site technical management. 3. The agent system according to claim 2, wherein:

[0087] This allows conversations collected from experts to be compiled as question and answer data.

[0088] [4] The data collection unit (28) further collects the video displayed by the on-site video display unit (video playback unit 26, on-site reproduction space display unit 26A) during the conversation in association with the conversation. 2. The agent system according to claim 1, wherein:

[0089] This allows video to supplement content that is difficult to understand from the conversation of an expert alone.

[0090] [5] When the questioning unit (27) detects an utterance from the user (4), it starts a dialogue with the user (4) based on the utterance. 2. The agent system according to claim 1, wherein:

[0091] This allows users' utterances to be collected as tacit knowledge.

[0092] [6] The questioning unit (27) ends the dialogue with the user (4) when the answer of the user (4) is repeated a predetermined number of times. 6. The agent system according to claim 5, wherein:

[0093] This allows for appropriate digging into user responses and appropriate collection of tacit knowledge.

[0094] [7] The questioning unit (27) ends the dialogue with the user (4) when a predetermined keyword is detected from the answer of the user (4). 6. The agent system according to claim 5, wherein:

[0095] This makes it possible to properly detect when the user has finished answering.

[0096] [8] The on-site video display unit (video playback unit 26) displays a video on the display device (display 41) that is a composite of a pre-recorded on-site video (on-site walk-through video 111) and an on-site video of an emergency. 2. The agent system according to claim 1, wherein:

[0097] This means:

[0098] [9] The on-site video display unit (video playback unit 26) causes the questioning unit (27) to start asking questions to the user (4) when the video displays the emergency equipment. 9. The agent system according to claim 8.

[0099] This allows the agent to start asking questions about emergency facilities at the appropriate time.

[0100]

[10] The on-site video display unit (video playback unit 26) pauses playback of the video while the questioning unit (27) is having a conversation with the user (4). 10. The agent system according to claim 9.

[0101] This prevents the video playback of the target equipment from ending or the video playback of the next target equipment from starting during a conversation between the user and the agent. Also, because stopping the video playback mutes the audio, the conversation between the user and the agent is not interrupted.

[0102]

[11] The on-site image display unit (on-site reproduction space display unit 26A) displays an image of a virtual space that reproduces the on-site situation on the virtual reality goggles (43) based on a virtual space model in which on-site information in normal times and on-site information in an emergency are developed. 2. The agent system according to claim 1, wherein:

[0103] This allows the user to interact with the agent in a more realistic manner based on the images in the virtual space.

[0104]

[12] The on-site image display unit (on-site reproduction space display unit 26A) causes the questioning unit (27) to start asking questions to the user (4) when the user reaches the emergency facility in the virtual space. 12. The agent system according to claim 11.

[0105] This allows the agent to start asking questions to the user at the appropriate time.

[0106]

[13] The agent speaks in a childlike tone. 12. The agent system according to claim 11.

[0107] This allows users to experience "spending time with grandchildren and others," which is a time when they feel a sense of purpose in life, and allows them to talk to agents without stress.

[0108]

[14] The agent converses with the user as the user's grandchild. 12. The agent system according to claim 11.

[0109] This allows users to experience "spending time with grandchildren and others," which is a time when they feel a sense of purpose in life, and allows them to talk to agents without stress.

[0110]

[15] a step in which a site video display unit (video playback unit 26, site reproduction space display unit 26A) of the agent system (2, 2A) displays a video of the site reproduction to a user (4); a step in which a questioning unit (27) of the agent system (2, 2A) asks an agent using a large-scale language model (200) questions of the user (4) about how to respond to each situation in the field; a step in which a data collection unit (28) of the agent system (2, 2A) collects conversations between the agent and the user (4); a step in which a field technical data generating unit (29) of the agent system (2, 2A) generates information (field technical data 291) relating to field technical management from the dialogue collected by the data collecting unit (28); A method for generating technical management information for an agent system, comprising:

[0111] This makes it possible to collect tacit knowledge from experts.

[0112] <<Variation>> The present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those including all of the described configurations. It is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is also possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0113] The above-described configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware such as an integrated circuit. The above-described configurations, functions, etc. may be realized by software by a processor interpreting and executing a program that realizes each function. Information such as the programs, tables, and files that realize each function can be stored in a storage device such as a memory, a hard disk, or an SSD (Solid State Drive), or on a storage medium such as a flash memory card or a DVD (Digital Versatile Disk).

[0114] In each embodiment, the control lines and information lines shown are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it can be considered that almost all components are interconnected. As modified examples of the present invention, for example, the following (a) to (d) are available.

[0115] (a) The agent with which the agent system interacts with the user may be of any age and gender that the user finds meaningful, and is not limited to a "grandchild." (b) The terminal through which the agent interacts with the user is not limited to a combination of a display and a smartphone, or a combination of virtual reality goggles and a smartphone, but may be any combination of a display device and a microphone. (c) The images that the agent system displays to the user are not limited to videos taken with a 360-degree camera, but may be videos taken with any type of imaging device. (d) The system targeted by the agent system is not limited to a facility maintenance management system, but may be applied to any system, such as a production management system or a quality control system. [Explanation of symbols]

[0116] 2 Agent System 200 large-scale language models 21 Data preprocessing section 22 Emergency Situation Generation Department 23 Mapping information generation unit 25 Question content generation section 3. Equipment maintenance management system 31 Maintenance Record Information 32 Facility location information 1 Manufacturing site 111 On-site walk-through video 4 User 24 Video composition section 221 Emergency Scene Video 26 Video playback unit (on-site video display unit) 27 Questions 28 Data Collection Department 29 Field Technical Data Generation Department 291 Field Technical Data 41 Display 42 Smartphone 211 Maintenance Record Markdown Information 212 Equipment Location Markdown Information 231 Facility coordinate failure information 251 Question Information 230 prompt 250 prompts 12 Normal operation video 220 Abnormality Prompt 241 composite videos 242 Fault occurrence meta information 421 screens 51 Agent Icon 53 User Icon 55 Microphone Icon 52a~52d Speech bubbles 54a~54d Speech bubbles 271 Dialogue Information 280 prompt 2A Agent System 20 Model Development Section 26A On-site reproduction space display unit (on-site video display unit) 43 Virtual Reality Goggles

Claims

1. a site image display unit that displays a reproduced image of the site to a user; a questioning unit that causes an agent using a large-scale language model to ask the user how to respond to each situation in the field; a data collection unit that collects conversations between the agent and the user; Equipped with when detecting an utterance from the user, the questioning unit starts a dialogue with the user based on the utterance. An agent system characterized by:

2. a field technical data generation unit that generates field technical data related to field technical management from the conversations collected by the data collection unit; 2. The agent system according to claim 1, further comprising:

3. the on-site technical data generation unit generates question and answer data related to on-site technical management; 3. The agent system according to claim 2.

4. The data collection unit further collects the video displayed by the on-site video display unit during the conversation in association with the conversation.

2. The agent system according to claim 1.

5. the questioning unit terminates the dialogue with the user when the answer of the user is repeated a predetermined number of times.

2. The agent system according to claim 1.

6. the questioning unit terminates the dialogue with the user when a predetermined keyword is detected from the answer of the user.

2. The agent system according to claim 1.

7. a site image display unit that displays a reproduced image of the site to a user; a questioning unit that causes an agent using a large-scale language model to ask the user how to respond to each situation in the field; a data collection unit that collects conversations between the agent and the user; Equipped with The on-site video display unit displays a video on a display device in which a video of the on-site in an emergency is combined with a video of the on-site that has been shot in advance, and when the video displays the equipment in the emergency, causes the question unit to start asking questions to the user, and pauses playback of the video while the question unit is interacting with the user. An agent system characterized by:

8. A site image display unit that displays to a user, using virtual reality goggles, an image of a virtual space that recreates the site based on a virtual space model in which site information under normal circumstances and site information under emergency circumstances are developed; a questioning unit that causes an agent using a large-scale language model to ask the user how to respond to each situation in the field; a data collection unit that collects conversations between the agent and the user; Equipped with An agent system characterized by:

9. the on-site image display unit causes the questioning unit to start asking questions to the user when the on-site image display unit reaches the emergency facility in the virtual space; 9. The agent system according to claim 8.

10. The agent speaks in a childlike tone.

9. The agent system according to claim 8.

11. The agent converses with the user as the user's grandchild.

9. The agent system according to claim 8.

12. a step in which a site image display unit of the agent system displays a reproduced image of the site to a user; a step in which a questioning unit of the agent system causes an agent using a large-scale language model to ask the user how to respond to each situation in the field; a step in which a data collection unit of the agent system collects conversations between the agent and the user; a step in which a field technical data generation unit of the agent system generates information related to field technical management from the dialogue collected by the data collection unit; a step of starting a dialogue with the user based on the utterance when the questioning unit detects the utterance of the user; A method for generating technical management information for an agent system, comprising:

13. a step in which a site image display unit of the agent system displays a reproduced image of the site to a user; a step in which a questioning unit of the agent system causes an agent using a large-scale language model to ask the user how to respond to each situation in the field; a step in which a data collection unit of the agent system collects conversations between the agent and the user; a step in which a field technical data generation unit of the agent system generates information related to field technical management from the dialogue collected by the data collection unit; a step of displaying a video on a display device by the on-site video display unit, the video being obtained by combining a pre-recorded on-site video with an on-site video of an emergency; a step of causing the questioning unit to start asking questions to the user when the video displays the facility in an emergency; pausing the playback of the video while the questioning unit is interacting with the user; A method for generating technical management information for an agent system, comprising:

14. a step in which an on-site image display unit of the agent system displays an image of a virtual space that recreates the on-site based on a virtual space model in which on-site information under normal circumstances and on-site information under emergency circumstances is developed, to a user using virtual reality goggles; a step in which a questioning unit of the agent system causes an agent using a large-scale language model to ask the user how to respond to each situation in the field; a step in which a data collection unit of the agent system collects conversations between the agent and the user; a step in which a field technical data generation unit of the agent system generates information related to field technical management from the dialogue collected by the data collection unit; A method for generating technical management information for an agent system, comprising:

15. A step of causing the questioning unit to start asking questions to the user when the on-site image display unit reaches an emergency facility in the virtual space; 15. The method for generating technical management information for an agent system according to claim 14, further comprising:

Images