Work assisting device, work assisting system and work assisting method

Abstract

To further easily enable a highly practicable work assist as for a work relating to at least a plant or an infrastructure.SOLUTION: A work assisting device includes a control unit (103) which obtains and accumulates, from a worker terminal (10) utilized by a work involved person for a work at a work site, know-how information on the work relating to at least a plant (P1) or an infrastructure, generates, when inquiry information relating to the work is received, a question that requests a generative AI utilizing a large-scale language model to generate a reply using the know-how information based on the inquiry information, generates presentation information to the worker terminal (10) that is an inquiry originator based on the reply from the generative AI to the question, and transmits the presentation information to the worker terminal (10).SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to a work assistance device, a work assistance system, and a work assistance method. [Background technology]

[0002] Conventionally, in various plants such as factories, power plants, substations, and oil refineries, and in various infrastructures such as roads, gas, electricity, and water and sewerage systems, maintenance work such as inspection, repair, operation, and management of various facilities and equipment, road surfaces, wall surfaces, electrical wires, and pipes is essential.

[0003] In such maintenance work, it is essential to pass on the knowledge and rules of experience of experienced technicians. However, the reality is that this is difficult to achieve due to factors such as the aging of technology resulting from the declining birthrate and aging population, as well as changes in the industrial structure.

[0004] In response to this situation, for example, in water treatment facilities, a technology has been proposed in which the know-how information possessed by skilled engineers is compiled into a database, and desired know-how information can be extracted by inputting keywords for the target equipment, events, etc. (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2012-118830 Summary of the Invention [Problem to be solved by the invention]

[0006] However, there is room for further improvement in the above-mentioned conventional techniques in terms of providing simpler and more practical work support at least for work related to plants or infrastructure.

[0007] For example, when using the above-mentioned conventional technology, in order to extract the desired know-how information, it is necessary to input keywords in text that specifically describe what kind of event it is, what the cause is, and what method to take, based on the so-called 5W1H (When, Where, Who, What, Why, How) premise. In other words, the above-mentioned conventional technology has the problem of being cumbersome and time-consuming.

[0008] Furthermore, it is believed that workers often have questions about various phenomena they see at the actual work site, but the above-mentioned conventional technology does not mention whether or not keywords can be input at the work site. Therefore, when using the above-mentioned conventional technology, there is a risk that workers will not be able to immediately resolve questions that arise at the work site.

[0009] An object of the present invention is to provide a work support device, a work support system, and a work support method that can provide simpler and more practical work support for at least work related to a plant or infrastructure. [Means for solving the problem]

[0010] A work support device according to one aspect includes a control unit that acquires and stores know-how information about work related to at least a plant or infrastructure from a terminal device used at a work site by a worker performing the work, and when inquiry information about the work is received, generates a question based on the inquiry information, requesting a generation AI using a large-scale language model to generate an answer using the know-how information, generates presentation information for the terminal device that is the source of the inquiry based on the answer from the generation AI to the question, and transmits the presentation information to the terminal device.

[0011] According to one aspect, the work support system includes a server device that manages work support tasks for work related to at least a plant or infrastructure, and a terminal device used by a worker performing the work at a work site. The server device acquires and stores know-how information for the work from the terminal device, and upon receiving inquiry information related to the work, generates a question based on the inquiry information, requesting a generation AI using a large-scale language model to generate an answer using the know-how information. The server device generates presentation information for the terminal device that originated the inquiry based on the answer from the generation AI to the question, and transmits the presentation information to the terminal device. In a first mode, the terminal device transmits the know-how information to the server device. In a second mode, the terminal device transmits the inquiry information to the server device and receives the presentation information in response to the inquiry information from the server device and presents it to the worker.

[0012] In one aspect of the work support method, a computer acquires and stores know-how information about work related to at least a plant or infrastructure from a terminal device used at a work site by a worker performing the work, and when inquiry information about the work is received, generates a question based on the inquiry information, requesting a generation AI using a large-scale language model to generate an answer using the know-how information, generates presentation information for the terminal device that originated the inquiry based on the answer from the generation AI to the question, and transmits the presentation information to the terminal device. [Effects of the Invention]

[0013] According to one embodiment, it is possible to provide a work support device, a work support system, and a work support method that can provide simpler and more practical work support for at least work related to a plant or infrastructure. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram illustrating an outline of a work support method according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of the configuration of a worker terminal according to the embodiment. [Figure 3] FIG. 10 is a diagram illustrating an example of a display on a worker terminal. [Figure 4] FIG. 10 is a diagram showing an example of technical information, inquiry information, and a response from a generation AI. [Figure 5] FIG. 10 is a diagram illustrating an example of presentation information. [Figure 6] FIG. 2 is a block diagram illustrating an example of the configuration of a worker terminal according to the embodiment. [Figure 7] FIG. 2 is a block diagram illustrating an example of the configuration of a server device according to the embodiment. [Figure 8] FIG. 10 is a diagram illustrating an example of registration of technical information. [Figure 9] FIG. 2 is a diagram showing a processing sequence executed by the work support system according to the embodiment. [Figure 10] FIG. 10 is a diagram (part 1) showing another example of technical information, inquiry information, and the response of the generation AI. [Figure 11] FIG. 2 is a diagram (part 2) showing another example of technical information, inquiry information, and the response of the generation AI. [Figure 12] FIG. 2 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the server device according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, embodiments of a work assistance device, a work assistance system, and a work assistance method disclosed in the present application will be described in detail with reference to the accompanying drawings. Note that the present invention is not limited to these embodiments. Furthermore, the same elements are given the same reference numerals, redundant explanations are omitted as appropriate, and the respective embodiments can be combined as appropriate within a range that does not cause inconsistency.

[0016] In the following, when it is necessary to distinguish between multiple identical elements, the symbol indicating the element may be followed by a number in the form "-n" (n is a natural number). When no particular distinction is necessary, this numbering will not be used.

[0017] In the following description, the work support system according to the embodiment is assumed to be a work support system 1 (see FIG. 1) that supports maintenance work in a plant P1 (see FIG. 1). The work support device according to the embodiment is assumed to be a server device 100 (see FIG. 1). The server device 100 is a device that manages all work support tasks in the work support system 1.

[0018] In the work support system 1, the server device 100 registers technical information from experienced engineers in a technical information database (DB) 102b (see FIG. 7). Technical information may also be interpreted as "know-how information." The server device 100 also receives inquiry information from worker terminals 10 used by workers at the work site. The server device 100 then generates a question for a generating AI (artificial intelligence) based on the received inquiry information. At this time, the server device 100 generates the question so that the generating AI can provide an answer based on various technical information registered in the technical information DB 102b. The server device 100 also generates presentation information for the worker who made the inquiry based on the answer from the generating AI, and displays the information on the worker terminal 10.

[0019] [Outline of the work assistance method according to this embodiment] An overview of the work support method according to this embodiment will be described with reference to Fig. 1. Fig. 1 is an explanatory diagram illustrating the overview of the work support method according to this embodiment.

[0020] As shown in FIG. 1, a work support system 1 according to the embodiment is a system in a plant P1. The work support system 1 includes a plurality of worker terminals 10, a server device 100, and a generation AI server 200. The worker terminals 10 and the server device 100 are arranged to be able to communicate with each other. The server device 100 is also arranged to be able to communicate with the generation AI server 200.

[0021] The worker terminal 10 is a portable terminal device that is used at the work site by a worker (hereinafter referred to as "worker U" as appropriate) performing maintenance work in the plant P1.

[0022] The worker terminal 10 is provided as, for example, a wearable device that can be worn by the worker U. In this embodiment, an example is given in which the worker terminal 10 is a wearable device and an AR (Augmented Reality) terminal. Note that the worker terminal 10 may also be realized by a mobile computer such as a tablet or a smartphone.

[0023] 2 is a diagram illustrating an example of the configuration of the worker terminal 10 according to the embodiment. As illustrated in FIG. 2, the worker terminal 10 is provided as AR glasses that can be worn on the head of a worker U. The worker terminal 10 includes a camera 13, a microphone 14, a display 15, and a speaker 16.

[0024] The camera 13 is provided so as to be able to capture an image of the real space in front of the worker U. The microphone 14 is provided so as to be able to collect the voice of the worker U and sounds at the work site. The display 15 is a display device that superimposes virtual information that does not actually exist in the real space in front of the worker U and presents it to the worker U.

[0025] The display 15 may be an optical see-through type or a video see-through type. In this embodiment, the display 15 is a video see-through type that displays virtual information superimposed on an image (including a moving image and a still image) captured by the camera 13. The speaker 16 presents audio information to the worker U.

[0026] Returning to the explanation of Fig. 1, in this embodiment, as shown in Fig. 1, it is assumed that worker U-1 at work site A uses worker terminal 10-1, and worker U-2 at work site B uses worker terminal 10-2. It is also assumed that worker U-1 at work site A is a skilled technician, and worker U-2 at work site B is an unskilled worker.

[0027] In the work support system 1 according to the embodiment, the server device 100 first acquires and stores technical information of skilled engineers (step S1). Here, FIG. 3 is a diagram showing an example of a display on the worker terminal 10.

[0028] 3, the worker terminal 10 has a tutor mode and a Q&A mode. These modes can be switched by the worker U operating a switching button B1. The operation of each operation component such as the switching button B1 can be performed via the operation unit 11, which will be described later.

[0029] Tutor mode is a mode in which technical information provided by a skilled technician can be recorded. For example, if worker U-1 selects this mode and operates the record button B2, the worker U-1 can record the work he or she is doing.

[0030] At this time, the image captured by the camera 13 is displayed in the area R1 and is also recorded in the storage unit 18 (see FIG. 6), which will be described later. In addition, the voice uttered by the worker U-1 and the sounds generated during the work are collected by the microphone 14 and are also recorded in the storage unit 18.

[0031] Furthermore, by using the drawing tool B3, the worker U-1 can draw, for example, a noteworthy portion of the image displayed in the region R1. FIG. 3 shows a schematic representation of the piping that is the target of the work, displayed in the region R1. The worker U-1 can use the drawing tool B3 to draw any desired image, such as by circling the portion of the piping that corresponds to corrosion (see the closed curve D1 in the figure) or by drawing an arrow. This drawing information can also be included in the image and recorded.

[0032] Furthermore, images, sounds, and the like recorded in the tutor mode are transmitted from the worker terminal 10 to the server device 100 as technical information.

[0033] The Q&A mode is also explained below. The Q&A mode is a mode that allows various information to be recorded as inquiry information when, for example, an unskilled worker has a question while working on-site.

[0034] For example, worker U-2 can select this mode and operate the record button B2 to record various information that worker U-2 wants to inquire about. That is, worker U-2 can record images captured by camera 13, audio collected by microphone 14, drawing information drawn using drawing tool B3, etc.

[0035] Moreover, various information recorded in the Q&A mode is transmitted from the worker terminal 10 to the server device 100 as inquiry information.

[0036] Returning to the explanation of Figure 1, in this way, the server device 100 first acquires the technical information recorded in tutor mode and registers and processes it in the technical information DB 102b (step S2). The captured image is registered in the image DB. The collected sound is registered in the sound DB. The sound is converted into text as appropriate and registered in the text DB. The text registered in the text DB is used by the server device 100, for example, when the interface for the generation AI described below is text only.

[0037] The text registered in the text DB is used by the server device 100 to generate a dictionary DB that converts, for example, industry jargon or dialects into standard terms. The server device 100 also links the text to corresponding images, sounds, text, etc. in the technical information DB 102b and assigns tag information.

[0038] Next, the server device 100 acquires inquiry information from, for example, worker U-2 who is working at work site B (step S3). The inquiry information is recorded in the worker terminal 10-2 as described with reference to FIG. 3 and transmitted to the server device 100.

[0039] Then, the server device 100 generates a question to be sent to the generation AI (generation AI server 200) based on the inquiry information (step S4).

[0040] The generation AI server 200 is a server device that functions as a generation AI. The generation AI server 200 is realized, for example, as a private cloud. The generation AI server 200 has a generation AI model (not shown). The generation AI server 200 loads and operates this generation AI model as part of a program, thereby functioning as a generation AI that generates answers to questions from the server device 100 and transmits them to the server device 100.

[0041] A generative AI model is a multimodal large-scale language model that can accept multiple types of input as modalities, such as text, images, and audio. Examples of generative AI models include transfer-based models and RNN (Recurrent Neural Network)-based models.

[0042] Examples of transfer-based models include, but are not limited to, GPT (Generative Pre-trained Transformer) and BARD (Bidirectional Auto Regressive Dialogues). Examples of RNN-based models include, but are not limited to, RWKV (Receptance Weighted Key Value). Note that the generative AI model may be, for example, a unimodal large-scale language model that can only accept text.

[0043] In this embodiment, the generation AI model of the generation AI server 200 is assumed to be individualized (which may be read as "fine-tuned") according to the work support tasks in the plant P1.

[0044] The server device 100 generates questions appropriately according to the interface to the generation AI server 200. At this time, the server device 100 generates questions for the generation AI server 200 so as to generate answers using each piece of data registered in the technical information DB 102b.

[0045] The server device 100 then exchanges questions and answers with the generation AI server 200 (step S5). The server device 100 also generates presentation information based on the answer obtained from the generation AI server 200 (step S6). The server device 100 then transmits the presentation information in response to the inquiry information to the worker terminal 10-2 (step S7).

[0046] Here, we will show examples of technical information, inquiry information, and the response of the generation AI, as well as examples of presentation information generated in response to the response. Figure 4 is a diagram showing examples of technical information, inquiry information, and the response of the generation AI. Figure 5 is a diagram showing an example of presentation information.

[0047] 4, it is assumed that voices and images of maintenance work on a pipe in plant P1 are provided by worker U-1 as technical information from work site A. This technical information is registered in technical information DB 102b by server device 100 as described above.

[0048] Then, suppose that an inquiry about corrosion on a certain pipe is received as inquiry information from work site B, which performs maintenance work on pipes similar to work site A. In this case, the worker terminal 10-2 at work site B transmits to the server device 100, for example, a voice message such as "There is something that looks like corrosion on the pipes of the plant. Is this corrosion?" and an image (including the drawing information described above) of what appears to be corrosion on the pipes at work site B.

[0049] The server device 100 then generates a question for the generation AI server 200 based on this query information. If the generation AI model is a multimodal large-scale language model, the server device 100 generates a question using the voice and images included in the query information. If the generation AI model is a unimodal large-scale language model that accepts only text, the server device 100 generates a question by, for example, appropriately converting the voice, etc. included in the query information into text.

[0050] The generation AI server 200 generates answers to questions using each piece of data registered in the technical information DB 102b. In the example of Figure 4, the generation AI server 200 generates answers using technical information recorded at work site A, which is technical information about piping similar to that at work site B.

[0051] Then, based on this response, the server device 100 generates presentation information corresponding to the worker terminal 10-2 that is the source of the inquiry. As shown in Fig. 5, the server device 100 generates the presentation information such that, for example, an area R2 presenting at least a part of the response of the generation AI is superimposed on an area R1 of the display 15. The server device 100 also generates the presentation information such that, for example, an area R3 presenting an image of corrosion at work site A as a reference image is similarly superimposed on the area R1.

[0052] While FIG. 5 shows an example in which at least a portion of the response of the generating AI is displayed in area R2, it may also be output as audio from speaker 16. Worker U-2 can proceed with work at work site B while receiving such presented information in response to his or her inquiry. Furthermore, even when worker U-1 is absent from work site B, worker U-2 can essentially receive technical knowledge from worker U-1.

[0053] As described above, in the work support method according to the embodiment, the server device 100 acquires and stores technical information about work related to at least the plant P1 or infrastructure from the worker terminal 10 used by the worker U at the work site. When the server device 100 receives inquiry information about the work, it generates a question based on the inquiry information, requesting a generation AI using a large-scale language model to generate an answer using the technical information. The server device 100 also generates presentation information for the worker terminal 10 that is the source of the inquiry, based on the answer from the generation AI to the question. The server device 100 also transmits the presentation information to the worker terminal 10. This makes it possible to provide simpler and more practical work support for at least work related to the plant P1 or infrastructure.

[0054] An example of the configuration of the work assistance system 1 to which the work assistance method according to the embodiment is applied will be described in more detail below.

[0055] [Configuration example of worker terminal 10] Fig. 6 is a block diagram showing an example of the configuration of the worker terminal 10 according to the embodiment. Note that Fig. 5 and Fig. 6 shown later show functional blocks of only the components necessary for explaining the embodiment, and descriptions of general components are omitted.

[0056] In addition, in the description using FIGS. 5 and 6, the description of components that have already been described will be appropriately simplified or omitted.

[0057] As shown in FIG. 5, the worker terminal 10 includes an operation unit 11, a sensor unit 12, a camera 13, a microphone 14, a display 15, a speaker 16, a communication unit 17, a memory unit 18, and a control unit 19.

[0058] The operation unit 11 is realized as, for example, a controller for operating AR glasses. The operation unit 11 accepts operations from the worker U on the worker terminal 10. For example, the operation unit 11 accepts operations on the aforementioned switching button B1, recording button B2, and drawing tool B3.

[0059] The sensor unit 12 is a group of various sensors mounted on the worker terminal 10. The sensor unit 12 includes, for example, a GPS (Global Positioning System) sensor, a G sensor, an angular velocity sensor, etc. The sensor data from the sensor unit 12 can be appropriately included as supplementary information in, for example, technical information or inquiry information.

[0060] The camera 13, microphone 14, display 15 and speaker 16 have already been explained, so their explanation will be omitted here.

[0061] The communication unit 17 is realized by a network adapter, etc. The communication unit 17 connects the worker terminal 10 and the server device 100 so that they can communicate with each other via wireless communication and / or wired communication.

[0062] The storage unit 18 is realized by a storage device such as a RAM (Random Access Memory), a flash memory, or an HDD (Hard Disk Drive). The storage unit 18 may be realized by a memory card such as an SD card. The storage unit 18 stores a program according to the embodiment executed by the control unit 19. The storage unit 18 also stores various types of information used in the information processing executed by the control unit 19.

[0063] 6, the storage unit 18 stores recording information 18a and output control information 18b. The recording information 18a stores images, sounds, etc. that are recorded by operating the above-mentioned record button B2. The output control information 18b is information that includes various parameters used when the output control unit 19f (described later) outputs presentation information.

[0064] The control unit 19 corresponds to a so-called processor and is realized by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphical Processing Unit), or the like.

[0065] The control unit 19 reads the program according to the embodiment stored in the storage unit 18 and executes it using the RAM as a work area. The control unit 19 can also be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0066] The control unit 19 has a mode setting unit 19a, a recording unit 19b, a technical information generation unit 19c, an inquiry information generation unit 19d, an acquisition unit 19e, and an output control unit 19f, and realizes or executes the functions and actions of information processing described below.

[0067] The internal configuration of the control unit 19 is not limited to the configuration shown in Fig. 6, and may be any other configuration as long as it is capable of executing the information processing described below. Furthermore, the connection relationship between the processing units included in the control unit 19 is not limited to the connection relationship shown in Fig. 6, and may be any other connection relationship.

[0068] The mode setting unit 19a switches the worker terminal 10 to the tutor mode or the Q&A mode based on the operation of the switching button B1. The recording unit 19b records images, sounds, etc. of the technical information or the inquiry information based on the operation of the recording button B2.

[0069] In the tutor mode, the technical information generation unit 19c generates technical information to be transmitted to the server device 100 based on the recorded information 18a. The technical information generation unit 19c also transmits the generated technical information to the server device 100 via the communication unit 17.

[0070] In the Q&A mode, the inquiry information generation unit 19d generates inquiry information to be transmitted to the server device 100 based on the recorded information 18a. The inquiry information generation unit 19d also transmits the generated inquiry information to the server device 100 via the communication unit 17.

[0071] The acquisition unit 19e acquires the presentation information generated in the server device 100 via the communication unit 17. The output control unit 19f performs output control to present the presentation information on the display 15 and / or the speaker 16, based on the presentation information acquired by the acquisition unit 19e and the output control information 18b.

[0072] [Configuration example of server device 100] Next, a configuration example of the server device 100 according to the embodiment will be described. Fig. 7 is a block diagram showing a configuration example of the server device 100 according to the embodiment.

[0073] 7, the server device 100 includes a communication unit 101, a storage unit 102, and a control unit 103. The communication unit 101 is realized by a network adapter or the like. The communication unit 101 communicatively connects the server device 100 to the worker terminal 10 and the generation AI server 200 via wireless and / or wired communication.

[0074] The storage unit 102 is realized by a storage device such as a RAM, a flash memory, or an HDD. The storage unit 102 stores a program according to the embodiment executed by the control unit 103. The storage unit 102 also stores various types of information used in the information processing executed by the control unit 103.

[0075] In the example of FIG. 7, the storage unit 102 stores a voice recognition model 102a, a technical information DB 102b, generated AI interface information 102c, and presentation information generation information 102d.

[0076] The voice recognition model 102a is a learning model used for voice recognition of the voice uttered by the worker U included in the technical information or inquiry information from the worker terminal 10. The voice recognition model 102a is, for example, a Hidden Markov Model (HMM) or a Deep Neural Network (DNN) model. The voice recognition model 102a may be a combined model of the HMM and the DNN model. The voice recognition model 102a is used to convert the voice of the worker U into text and to generate a dictionary DB.

[0077] As already mentioned, the technical information DB 102b is a database in which technical information transmitted from the worker terminal 10 in the tutor mode is registered. The technical information DB 102b includes an image DB, a voice DB, a text DB, and a dictionary DB.

[0078] Fig. 8 is a diagram showing an example of registration of technical information. In the technical information DB 102b, image data in the image DB, audio data in the audio DB, text data in the text DB, etc. are linked together through registration processing by the registration processing unit 103b, which will be described later, as shown in Fig. 8. At this time, sensor data from the sensor unit 12, etc., described above, are also linked together as supplementary information.

[0079] Furthermore, as shown in Fig. 8, in the registration process by the registration processing unit 103b, various tag data are assigned to the linked data groups. For example, in the example already shown in Fig. 3, tag data indicating meta information of technical information such as "piping," "corrosion," and "work site A" are assigned. The generation AI server 200 can also use such tag data as label information when generating answers to questions.

[0080] Returning to the explanation of Figure 7, the generated AI interface information 102c is information about the interface through which the server device 100 exchanges questions and answers with the generated AI server 200. The generated AI interface information 102c includes, for example, an API (Application Programming Interface) for the paired generated AI.

[0081] The presentation information generation information 102d is information including various parameters used when generating presentation information for the worker terminal 10. The presentation information generation information 102d includes, for example, information regarding the coordinate system of the AR display space on the display 15 of the worker terminal 10.

[0082] The control unit 103 corresponds to a so-called processor and is realized by a CPU, an MPU, a GPU, or the like.

[0083] The control unit 103 reads the program according to the embodiment stored in the storage unit 102 and executes it using the RAM as a work area. The control unit 103 can also be realized by an integrated circuit such as an ASIC or an FPGA.

[0084] The control unit 103 has an acquisition unit 103a, a registration processing unit 103b, a reception unit 103c, a pair generation AI processing unit 103d, and a presentation information generation unit 103e, and realizes or executes the functions and actions of information processing described below.

[0085] The internal configuration of the control unit 103 is not limited to the configuration shown in Fig. 7, and may be any other configuration as long as it is capable of executing the information processing described below. Furthermore, the connection relationship between the processing units included in the control unit 103 is not limited to the connection relationship shown in Fig. 7, and may be any other connection relationship.

[0086] The acquisition unit 103a acquires technical information or inquiry information from the worker terminal 10 via the communication unit 101. The registration processing unit 103b executes the above-mentioned registration process of appropriately processing the technical information acquired by the acquisition unit 103a and registering it in the technical information DB 102b.

[0087] The receiving unit 103c receives the inquiry information acquired by the acquiring unit 103a, and causes the pair generation AI processing unit 103d to execute pair generation AI processing according to the received inquiry information.

[0088] The pair generation AI processing unit 103d executes pair generation AI processing based on the inquiry information received by the receiving unit 103c. Specifically, the pair generation AI processing unit 103d generates a question to be sent to the generation AI server 200 based on the inquiry information and the generation AI interface information 102c. At this time, the pair generation AI processing unit 103d converts the spoken voice of the worker U included in the inquiry information into text using the speech recognition model 102a, if necessary.

[0089] The pair generation AI processing unit 103d also transmits the generated question to the generation AI server 200 via the communication unit 101. The pair generation AI processing unit 103d also obtains the answer to the question from the generation AI server 200 via the communication unit 101.

[0090] The presentation information generation unit 103e generates presentation information to be transmitted to the worker terminal 10 based on the response from the generation AI server 200 acquired by the pair generation AI processing unit 103d and the presentation information generation information 102d. The presentation information generation unit 103e also transmits the generated presentation information via the communication unit 101 to the worker terminal 10 that is the source of the inquiry.

[0091] [Processing procedure executed by work support system 1] Next, the processing procedure executed by the work support system 1 will be described with reference to Fig. 9. Fig. 9 is a diagram showing a processing sequence executed by the work support system 1 according to the embodiment.

[0092] 9, it is assumed that the worker terminal 10 is started in tutor mode (step S101). In this case, the worker terminal 10 records the state of work of a worker U who uses the worker terminal 10, and generates technical information (step S102).

[0093] Then, the worker terminal 10 transmits the generated technical information to the server device 100 (step S103). The server device 100 registers the technical information received from the worker terminal 10 (step S104). By repeating steps S101 to S104, know-how information from experienced engineers is accumulated in the technical information DB 102b.

[0094] Meanwhile, it is assumed that the worker terminal 10 is started in the Q&A mode (step S105). In this case, the worker terminal 10 generates inquiry information based on images, voices, etc. recorded in response to operations by the worker U who uses the worker terminal 10 (step S106).

[0095] Then, the worker terminal 10 transmits the generated inquiry information to the server device 100 (step S107). The server device 100 generates a question for the generation AI based on the inquiry information received from the worker terminal 10 (step S108). Then, the server device 100 transmits the generated question to the generation AI, i.e., the generation AI server 200 (step S109).

[0096] Upon receiving this question, the generation AI server 200 generates an answer corresponding to the question using the generation AI model (step S110) and transmits it to the server device 100 (step S111). The server device 100 generates presentation information based on the answer received from the generation AI server 200 (step S112) and transmits it to the worker terminal 10 (step S113).

[0097] Then, the worker terminal 10 presents the presentation information received from the server device 100 (step S114).

[0098] [Other examples of technical information, inquiry information, and AI-generated answers] While an example of technical information, inquiry information, and a response from the generation AI was explained using Figure 4, other examples will be explained using Figures 10 and 11. Figure 10 is a diagram (part 1) showing another example of technical information, inquiry information, and a response from the generation AI. Also, Figure 11 is a diagram (part 2) showing another example of technical information, inquiry information, and a response from the generation AI.

[0099] Fig. 10 shows an example of pipe maintenance work using the hammering sound method. In this case, as shown in Fig. 10, technical information from work site A is provided from worker terminal 10-1 in the form of, for example, tips on how to hammer a pipe, sounds that are generated when actually hammering, echoes of the generated sounds, and audio and images including explanations of what can be learned from the echoes.

[0100] The server device 100 registers this technical information in the technical information DB 102b. At this time, the server device 100 may perform image analysis to identify the actual part of the pipe that will be struck (i.e., the point of application) and the position and posture of the hammer, and may also register images and text that are the results of the analysis. This further strengthens the technical information and increases the likelihood of obtaining a highly accurate answer from the generation AI.

[0101] The server device 100 may also perform frequency analysis of the sound of actually striking a pipe and the resulting reverberation, and register the analysis results, such as a spectrogram image. This method can also further strengthen the technical information.

[0102] Then, when an inquiry such as "Do you have any tips?" is received from work site B, which is also performing pipe maintenance work using the hammering method at work site A, server device 100 generates a question for generation AI server 200 based on this inquiry information.

[0103] The generation AI server 200 generates answers to questions using each piece of data registered in the technical information DB 102b. In the example of Fig. 10, the generation AI server 200 generates answers using technical information from work site A, where pipe maintenance work was performed using the same hammering sound method as work site B.

[0104] At this time, the generation AI server 200 generates a response that clearly indicates the part corresponding to, for example, "around here," as shown in Figure 10. This is possible because the server device 100 registers images, etc., that are the analysis results of image analysis that identify the part of the pipe that will actually be struck and the position and posture of the hammer, as described above.

[0105] Next, Fig. 11 shows an example of receiving an inquiry from worker U-2 who will be carrying out maintenance work on piping at a small work site B. In this case, as shown in Fig. 11, for example, voice and images relating to tools used when carrying out maintenance work on piping at a similarly small work site A are useful technical information from work site A.

[0106] In this case, if there is an inquiry from work site B, such as "What tools would be useful to have?", the server device 100 generates a question for the generation AI server 200 based on this inquiry information. Note that the inquiry information from work site B at this time does not necessarily have to be made from work site B.

[0107] The generation AI server 200 generates an answer to the question using each piece of data registered in the technical information DB 102b. In the example of Fig. 11, the generation AI server 200 uses technical information about a small work site A similar to work site B to generate an answer suggesting that a fiberscope, a tiny drone, etc. would be useful.

[0108] This means that even before actually going to work site B, worker U-2 who is scheduled to work at work site B can receive useful answers and presented information based on them via the generation AI based on the technical information registered in the technical information DB 102b.

[0109] [Variations] In the above-described embodiment, the worker terminal 10 can be switched to the tutor mode by operating the switching button B1, but it may be configured so that only predetermined qualified personnel or certified engineers can switch to the tutor mode.

[0110] In this case, the work assistance system 1 may have authentication information of qualified personnel or certified engineers, and the worker terminal 10 may be configured to switch to tutor mode only when the worker U using the worker terminal 10 at startup is authenticated by this authentication information. This increases the reliability of the technical information stored in the technical information DB 102b. This also enables the generation AI to generate highly reliable answers.

[0111] Furthermore, in the above-described embodiment, the main example has been technology transfer related to maintenance work in plant P1, but in addition to technology transfer, this embodiment can also be utilized to overcome issues at work sites, eliminate near misses, improve work efficiency, and the like.

[0112] [effect] As described above, the server device 100 (corresponding to an example of a "work support device") according to the embodiment includes the control unit 103. The control unit 103 acquires and stores know-how information about work related to at least the plant P1 or infrastructure from the worker terminal 10 used by the worker U at the work site, and when inquiry information about the work is received, generates a question based on the inquiry information, requesting a generation AI using a large-scale language model to generate an answer using the know-how information, generates presentation information for the worker terminal 10 that is the source of the inquiry based on the answer from the generation AI to the question, and transmits the presentation information to the worker terminal 10. This makes it possible to provide simpler and more practical work support for at least work related to the plant P1 or infrastructure.

[0113] Furthermore, the know-how information and the inquiry information include at least images, audio, and text converted from the audio related to the work recorded by the worker terminal 10, and the control unit 103 generates the question using at least one of the images, audio, and text in accordance with a modality acceptable to the large-scale language model. This allows the generation AI to generate an appropriate question in accordance with the large-scale language model used.

[0114] Furthermore, the large-scale language model is a multimodal large-scale language model, which allows questions to be generated using multiple types of data from the images, audio, and text, enabling the AI ​​to provide highly accurate answers to the questions.

[0115] Furthermore, the control unit 103 performs image analysis on the image included in the know-how information and adds the results of this image analysis to the know-how information. This further strengthens the know-how information, increasing the likelihood of obtaining a highly accurate answer from the generation AI.

[0116] The results of the image analysis also include information about the position, posture, and point of application of the tool used by the worker U in the above work. This makes it possible to obtain answers that clearly indicate, for example, the position of the point of application when hitting a pipe with a hammer, as well as the position and posture of the hammer.

[0117] Furthermore, the control unit 103 performs frequency analysis on the voice included in the know-how information and adds the results of this frequency analysis to the know-how information. This allows the know-how information to be further reinforced with respect to the voice, increasing the possibility of obtaining a highly accurate answer from the generation AI.

[0118] The results of the frequency analysis also include a spectrogram image, which increases the likelihood of obtaining a highly accurate answer regarding the sound using a spectrogram image.

[0119] The worker terminal 10 is an AR terminal, which enables work support using AR technology.

[0120] Furthermore, the control unit 103 generates the presentation information so that the presentation information is superimposed on the real space in the AR display space of the AR terminal, thereby enabling the presentation of highly convenient presentation information using AR technology.

[0121] The know-how information also includes technical information about the work provided by a skilled technician, the worker U. This allows the knowledge and rules of thumb of the skilled technician to be passed on to the next generation.

[0122] [Other embodiments] Although the embodiments of the present invention have been described above, the present invention may be embodied in various different forms other than the above-described embodiments.

[0123] [system] The information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified.

[0124] Furthermore, the components of each device shown in the figure are functional concepts and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of each device is not limited to that shown. In other words, all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0125] Furthermore, all or any part of the processing functions performed by each device may be realized by a CPU and a program analyzed and executed by the CPU, or may be realized as hardware using wired logic.

[0126] [Hardware] The worker terminal 10 and the server device 100 according to the above-described embodiment are realized by, for example, a computer 1000 configured as shown in Fig. 12. The following description will be given taking the server device 100 as an example. Fig. 12 is a hardware configuration diagram showing an example of the computer 1000 that realizes the functions of the server device 100 according to the embodiment.

[0127] 12, the computer 1000 includes a communication device 1000a, a secondary storage device 1000b, a memory 1000c, and a processor 1000d. The components shown in FIG. 12 are interconnected by a bus or the like.

[0128] The communication device 1000a is a network interface card (NIC) or the like, and communicates with other devices. The secondary storage device 1000b is realized by a flash memory, a hard disk drive, or the like, and stores programs and databases that operate the functions shown in FIG.

[0129] The processor 1000d reads out a program that executes the same processes as the respective processing units shown in FIG. 7 from the secondary storage device 1000b, etc., and expands it into the memory 1000c, thereby operating a thread that executes each function described in FIG. 7, etc. For example, this thread executes the same functions as the respective processing units possessed by the server device 100. Specifically, the processor 1000d reads out a program having the same functions as the acquisition unit 103a, the registration processing unit 103b, the reception unit 103c, the pair generation AI processing unit 103d, and the presentation information generation unit 103e from the secondary storage device 1000b, etc. Then, the processor 1000d executes a thread that executes the same processes as the acquisition unit 103a, the registration processing unit 103b, the reception unit 103c, the pair generation AI processing unit 103d, the presentation information generation unit 103e, etc.

[0130] In this way, the computer 1000 operates as an information processing device that executes various processing methods by reading and executing the program. The computer 1000 can also realize functions similar to those of the above-described embodiments by reading the program from a recording medium using a medium reading device and executing the read program. Note that the program referred to here is not limited to being executed solely by the computer 1000. For example, the present invention can be similarly applied to cases where a computer or server having a different hardware configuration executes the program, or where these execute the program in cooperation with each other.

[0131] This program can be distributed via a network such as the Internet. This program can also be recorded on a computer-readable recording medium such as a hard disk drive (HDD), a flexible disk (FD), a CD-ROM, a magneto-optical disk (MO), or a digital versatile disk (DVD), and can be executed by being read from the recording medium by a computer. A recording medium on which such a program is recorded is also an aspect of the present disclosure.

[0132] [others] Some examples of combinations of the disclosed technical features are listed below.

[0133] (1) Acquire and store know-how information on at least plant or infrastructure work from terminal devices used by workers at the work site, When receiving inquiry information regarding the work, a question is generated based on the inquiry information, requesting a generation AI using a large-scale language model to generate an answer using the know-how information; generating presentation information for the terminal device that is the source of the inquiry based on the answer from the generation AI to the question; a control unit that transmits the presentation information to the terminal device; A work support device comprising: (2) the know-how information and the inquiry information include at least images, voice, and text converted from the voice, related to the work, recorded by the terminal device; The control unit generating the query using at least one of the image, the audio, and the text in accordance with a modality acceptable to the large-scale language model; (1) A work support device as described above. (3) the large-scale language model is a multimodal large-scale language model; (2) A work support device according to the present invention. (4) The control unit performing an image analysis on the image included in the know-how information, and further adding the result of the image analysis to the know-how information; A work support device according to (2) or (3). (5) The results of the image analysis include information regarding the position, posture, and point of application of a tool used by the worker in the task. (4) A work support device according to the present invention. (6) The control unit performing a frequency analysis on the voice included in the know-how information, and adding the result of the frequency analysis to the know-how information; The work support device according to any one of (2) to (5). (7) The result of the frequency analysis includes a spectrogram image. (6) A work support device according to the present invention. (8) The terminal device is an AR (Augmented Reality) terminal. A work support device according to any one of (1) to (7). (9) The control unit generating the presentation information so that the presentation information is superimposed on a real space in an AR display space of the AR terminal; (8) A work support device according to the present invention. (10) The know-how information includes technical information related to the work provided by the worker who is a skilled engineer. A work support device according to any one of (1) to (8). (11) The system includes a server device that manages work support services for work related to at least a plant or infrastructure, and a terminal device that is used at the work site by workers engaged in the work, The server device Acquiring know-how information about the work from the terminal device and storing it; When receiving inquiry information regarding the work, a question is generated based on the inquiry information, requesting a generation AI using a large-scale language model to generate an answer using the know-how information; generating presentation information for the terminal device that is the source of the inquiry based on the answer from the generation AI to the question; transmitting the presentation information to the terminal device; The terminal device When the first mode is selected, the know-how information is transmitted to the server device; When in the second mode, the inquiry information is transmitted to the server device, and the presentation information in response to the inquiry information is received from the server device and presented to the worker. Work support system. (12) The computer Acquire and store know-how information on at least plant or infrastructure work from terminal devices used by workers at the work site, When receiving inquiry information regarding the work, a question is generated based on the inquiry information, requesting a generation AI using a large-scale language model to generate an answer using the know-how information; generating presentation information for the terminal device that is the source of the inquiry based on the answer from the generation AI to the question; transmitting the presentation information to the terminal device; A work support method for performing a process. (13) On the computer, Acquire and store know-how information on at least plant or infrastructure work from terminal devices used by workers at the work site, When receiving inquiry information regarding the work, a question is generated based on the inquiry information, requesting a generation AI using a large-scale language model to generate an answer using the know-how information; generating presentation information for the terminal device that is the source of the inquiry based on the answer from the generation AI to the question; transmitting the presentation information to the terminal device; A program that executes a process. (14) A computer-readable recording medium on which a program is recorded, The program On the computer, Acquire and store know-how information on at least plant or infrastructure work from terminal devices used by workers at the work site, When receiving inquiry information regarding the work, a question is generated based on the inquiry information, requesting a generation AI using a large-scale language model to generate an answer using the know-how information; generating presentation information for the terminal device that is the source of the inquiry based on the answer from the generation AI to the question; transmitting the presentation information to the terminal device; A computer-readable recording medium that causes processing to be performed. [Explanation of symbols]

[0134] 1. Work support system 10 Worker terminal 11 Control section 12 Sensor section 13 Camera 14. Mike 15 Display 16 speakers 17 Communications Department 18 Memory section 18a Record Information 18b Output control information 19 Control Unit 19a Mode setting section 19b Recording section 19c Technical information generation department 19d Inquiry information generation unit 19e Acquisition Department 19f Output control section 100 Server device 101 Communications Department 102 Storage section 102a Speech Recognition Model 102c Generated AI interface information 102d Presentation information generation information 103 Control Unit 103a Acquisition Department 103b Registration processing unit 103c Reception 103d Pair generation AI processing unit 103e Presentation information generation section 200 Generation AI Server U Worker

Claims

1. Acquire and store know-how information on at least plant or infrastructure work from terminal devices used by workers at the work site, When receiving inquiry information regarding the work, a question is generated based on the inquiry information, requesting a generation AI using a large-scale language model to generate an answer using the know-how information; generating presentation information for the terminal device that is the source of the inquiry based on the answer from the generation AI to the question; a control unit that transmits the presentation information to the terminal device; A work support device comprising:

2. the know-how information and the inquiry information include at least images, voice, and text converted from the voice, related to the work, recorded by the terminal device; The control unit generating the query using at least one of the image, the audio, and the text in accordance with a modality acceptable to the large-scale language model; The work support device according to claim 1 .

3. the large-scale language model is a multimodal large-scale language model; The work support device according to claim 2 .

4. The control unit performing an image analysis on the image included in the know-how information, and further adding the result of the image analysis to the know-how information; The work support device according to claim 2 .

5. The results of the image analysis include information regarding the position, posture, and point of application of a tool used by the worker in the task. The work support device according to claim 4.

6. The control unit performing a frequency analysis on the voice included in the know-how information, and adding the result of the frequency analysis to the know-how information; The work support device according to claim 2 .

7. The result of the frequency analysis includes a spectrogram image. The work support device according to claim 6.

8. The terminal device is an AR (Augmented Reality) terminal. The work support device according to claim 1 .

9. The control unit generating the presentation information so that the presentation information is superimposed on a real space in an AR display space of the AR terminal; The work support device according to claim 8.

10. The know-how information includes technical information related to the work provided by the worker who is a skilled engineer. The work support device according to any one of claims 1 to 9.

11. The system includes a server device that manages work support services for work related to at least a plant or infrastructure, and a terminal device that is used at the work site by workers engaged in the work, The server device Acquiring know-how information about the work from the terminal device and storing it; When receiving inquiry information regarding the work, a question is generated based on the inquiry information, requesting a generation AI using a large-scale language model to generate an answer using the know-how information; generating presentation information for the terminal device that is the source of the inquiry based on the answer from the generation AI to the question; transmitting the presentation information to the terminal device; The terminal device When the first mode is selected, the know-how information is transmitted to the server device; When in the second mode, the inquiry information is transmitted to the server device, and the presentation information in response to the inquiry information is received from the server device and presented to the worker. Work support system.

12. The computer Acquire and store know-how information on at least plant or infrastructure work from terminal devices used by workers at the work site, When receiving inquiry information regarding the work, a question is generated based on the inquiry information, requesting a generation AI using a large-scale language model to generate an answer using the know-how information; generating presentation information for the terminal device that is the source of the inquiry based on the answer from the generation AI to the question; transmitting the presentation information to the terminal device; A work support method for performing a process.

Images