Work assistance device, work assistance system, and work assistance method

Abstract

The present invention relates to a work assistance device, a work assistance system, and a work assistance method, which perform more convenient and more practical work assistance with respect to at least work relating to a workshop or an infrastructure. The work support device is provided with a control unit that acquires and accumulates skill information of at least work relating to a workshop or an infrastructure from a terminal device used by a worker of the work at a work site, and when advisory information relating to the work is accepted, controls the work support device to support the work on the basis of the advisory information. A query requested so as to generate an answer using the skill information to a generated AI using a large-scale language model is generated, presentation information to a terminal device serving as a consultation source is generated on the basis of the answer from the generated AI to the query, and the presentation information is transmitted to the terminal device.

Description

Technical Field

[0001] The present invention relates to a work assisting device, a work assisting system and a work assisting method. Background Art

[0002] Currently, maintenance work such as inspection, repair, operation, and management of various equipment, devices, road surfaces, walls, wires, and piping is essential in various workshops such as factories, power plants, substations, and essential oil facilities, as well as various infrastructure such as roads, gas, electricity, and water supply systems.

[0003] In such maintenance work, the inheritance of the knowledge and experience of skilled technicians is indispensable. However, in reality, this inheritance is difficult due to the aging of technology accompanied by a declining birthrate and an aging population, and changes in the industrial structure.

[0004] In response to this reality, a technology has been proposed that databases the skill information of skilled workers in, for example, water treatment facilities, and extracts desired skill information by inputting keywords such as target equipment and phenomena (see, for example, Patent Document 1).

[0005] Patent Document 1: Japanese Patent Application Laid-Open No. 2012-118830 Summary of the Invention

[0006] However, the above-mentioned conventional technologies still have room for further improvement in terms of providing simpler and more practical work assistance, at least with respect to work related to workshops or infrastructure.

[0007] For example, using the above-mentioned prior art, extracting desired skill information requires entering keywords into the text, based on the so-called 5W1H (When, Where, Who, What, Why, How) criteria, to determine which method to use for which phenomenon. This prior art is complex and labor-intensive.

[0008] Furthermore, it is conceivable that workers often have questions about various phenomena they observe on-site, but the above-mentioned prior art does not mention whether keyword input is possible at the work site. Therefore, when using the above-mentioned prior art, workers may not be able to immediately resolve questions that arise at the work site.

[0009] An object of the present invention is to provide a work assisting device, a work assisting system, and a work assisting method that can provide simpler and more practical work assisting at least with respect to work related to a workshop or infrastructure.

[0010] One aspect involves a work assisting device having a control unit that obtains and accumulates at least skill information of work related to a workshop or infrastructure from a terminal device used by a worker performing the work at a work site, and when receiving consultation information related to the work, generates a query based on the consultation information in a manner that requests an answer using the skill information to be generated for a generative AI using a large-scale language model, generates prompt information to the terminal device serving as a consultation source based on the answer from the generative AI to the query, and sends the prompt information to the terminal device.

[0011] One aspect of a work support system includes: a server device that manages at least work support services for work related to a workshop or infrastructure; and a terminal device used by a worker performing the work at a work site. The server device acquires and stores skill information for the work from the terminal device. Upon receiving consultation information related to the work, the server device generates, based on the consultation information, a query requesting a generative AI using a large-scale language model to generate an answer utilizing the skill information. Based on the answer to the query from the generative AI, the server device generates prompt information for the terminal device, serving as the source of the consultation, and transmits the prompt information to the terminal device. The terminal device transmits the skill information to the server device in a first mode and the consultation information to the server device in a second mode. Furthermore, the server device receives the prompt information in response to the consultation information from the server device and presents it to the worker.

[0012] One aspect involves a job assistance method that causes a computer to perform the following processing: at least skill information for a job related to a workshop or infrastructure is obtained and accumulated from a terminal device used by a worker performing the job at a job site; when consultation information related to the job is received, based on the consultation information, a query is generated in a manner that requests an answer using the skill information to be generated for a generative AI using a large-scale language model; based on the answer from the generative AI to the query, prompt information is generated to the terminal device serving as the consultation source, and the prompt information is sent to the terminal device.

[0013] Effects of the Invention

[0014] According to one embodiment, a work assisting device, a work assisting system, and a work assisting method can be provided that can provide simpler and more practical work assisting at least with respect to work related to a workshop or infrastructure. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1This is a diagram schematically illustrating a work assisting method according to an embodiment.

[0016] Figure 2 It is a diagram showing a configuration example of a worker terminal according to an embodiment.

[0017] Figure 3 It is a diagram showing an example of display on the operator terminal.

[0018] Figure 4 This diagram shows an example of technical information, consultation information, and answers generated by AI.

[0019] Figure 5 This is a diagram showing an example of prompt information.

[0020] Figure 6 This is a block diagram showing a configuration example of a worker terminal according to an embodiment.

[0021] Figure 7 This is a block diagram showing a configuration example of a server device according to an embodiment.

[0022] Figure 8 This is a diagram showing an example of registration of technical information.

[0023] Figure 9 This is a diagram showing a processing procedure executed by the work support system according to the embodiment.

[0024] Figure 10 This is a diagram (part 1) showing another example of technical information, consultation information, and answers generated by AI.

[0025] Figure 11 This is a diagram (part 2) showing another example of technical information, consultation information, and answers generated by AI.

[0026] Figure 12 This is a hardware configuration diagram showing an example of a computer that realizes the functions of the server device according to the embodiment. DETAILED DESCRIPTION

[0027] Below, embodiments of the work assist device, work assist system, and work assist method disclosed in this application are described in detail based on the accompanying drawings. The present invention is not limited to these embodiments. Identical elements may be denoted by the same reference numerals, and duplicate descriptions may be omitted as appropriate. The various embodiments may be combined as appropriate within the scope of non-contradiction.

[0028] In the following, when it is necessary to distinguish a plurality of identical elements, a reference number may be appended after the reference number representing the element in the form of "-n" (n is a natural number).

[0029] In the following, the work support system according to the embodiment is assumed to be the support workshop P1 (see Figure 1 ) maintenance work support system 1 (refer to Figure 1 ). In addition, the work assisting device according to the embodiment is set as the server device 100 (refer to Figure 1 The server device 100 is a device that manages all work support services in the work support system 1 .

[0030] Regarding the work support system 1, the server device 100 sends a technical information DB (Database) 102b (see Figure 7 ) registers technical information from skilled technicians. Technical information can be renamed as "skill information". In addition, the server device 100 accepts consulting information from the operator terminal 10 used by the staff at the work site. In addition, the server device 100 generates an inquiry to the generated AI (Artificial Intelligence) based on the accepted consulting information. At this time, the server device 100 generates the inquiry in a manner that can obtain an answer based on various technical information registered to the technical information DB 102b from the generated AI. In addition, the server device 100 generates prompt information for the staff who makes the inquiry based on the answer from the generated AI, and prompts it to the operator terminal 10.

[0031] [Overview of the Work Assistance Method According to the Present Embodiment]

[0032] use Figure 1 An overview of the work assistance method according to this embodiment will be described. Figure 1 This is a diagram schematically illustrating a work assisting method according to an embodiment.

[0033] like Figure 1 As shown, the work support system 1 according to the embodiment is a system for workshop P1. The work support system 1 includes multiple worker terminals 10, a server device 100, and an AI generation server 200. The worker terminals 10 and the server device 100 are configured to communicate with each other. Furthermore, the server device 100 is configured to communicate with the AI ​​generation server 200.

[0034] The operator terminal 10 is a mobile terminal device used at a work site by a worker who performs maintenance work in the workshop P1 (hereinafter referred to as a “operator U” as appropriate).

[0035] The operator terminal 10 is provided as a wearable device, for example, so that it can be worn by the operator U. In this embodiment, the operator terminal 10 is such a wearable device, but an AR (Augmented Reality) terminal can also be used as an example. Furthermore, the operator terminal 10 can be implemented as a mobile computer such as a tablet computer or a smartphone.

[0036] Figure 2 1 is a diagram showing a configuration example of the operator terminal 10 according to the embodiment. Figure 2 As shown, the operator terminal 10 is provided as AR glasses that can be worn on the head of the operator U. The operator terminal 10 includes a camera 13 , a microphone 14 , a display 15 , and a speaker 16 .

[0037] The camera 13 is provided to capture the real space in front of the operator U. The microphone 14 is provided to collect the voice of the operator U and sounds from the scene. The display 15 is a display device that superimposes virtual information that does not originally exist in the real space on the real space in front of the operator U and presents it to the operator U.

[0038] The display 15 may be an optical see-through type or a video see-through type. In this embodiment, the display 15 is a video see-through type that displays virtual information superimposed on an image (including animation and still images) captured by the camera 13. The speaker 16 provides voice information to the operator U.

[0039] Back to Figure 1 In addition, if Figure 1 As shown, in this embodiment, worker U-1 at work site A uses worker terminal 10-1, and worker U-2 at work site B uses worker terminal 10-2. Worker U-1 at work site A is a skilled worker, and worker U-2 at work site B is an unskilled worker.

[0040] Furthermore, in the work support system 1 according to the embodiment, first, the server device 100 acquires and accumulates technical information of skilled workers (step S1). Figure 3 It is a diagram showing an example of display on the monitor of the operator terminal 10 .

[0041] like Figure 3 As shown, the operator terminal 10 has a tutorial mode and a Q&A mode. These modes can be switched by the operator U operating a switch button B1. The switch button B1 and other operating members can be operated via the operation unit 11 described later.

[0042] The tutorial mode is a mode in which technical information provided by an experienced technician can be recorded. For example, when the worker U-1 selects this mode and operates the record button B2, the worker U-1 can record the status of the work performed.

[0043] At this time, the image captured by the camera 13 is displayed in the region R1 and stored in the storage unit 18 (see Figure 6 ) is recorded. In addition, the microphone 14 collects the voice uttered by the operator U-1 and the sound produced during the work and records them in the storage unit 18.

[0044] In addition, the operator U-1 can use the drawing tool B3 to draw, for example, a portion of the image displayed in the region R1 that requires attention. Figure 3 The figure schematically shows a pipe being worked on, shown in region R1. The operator U-1 can draw any desired pattern using a drawing tool B3, for example, by enclosing the corroded portion of the pipe with a circle (see closed curve D1 in the figure) or drawing an arrow. This drawing information can also be recorded in the image.

[0045] Furthermore, images, voices, and the like recorded in the tutoring mode are transmitted from the operator terminal 10 to the server device 100 as technical information.

[0046] The Q&A mode is also described in advance. The Q&A mode is a mode in which, for example, when an unskilled worker has questions during on-site work, various information can be recorded as inquiry information.

[0047] For example, by selecting this mode and pressing the record button B2, the operator U-2 can record various information that the operator U-2 wishes to consult. Specifically, the operator U-2 can record images captured by the camera 13, voices collected by the microphone 14, and drawing information drawn using the drawing tool B3.

[0048] Furthermore, various information recorded in the Q&A mode is transmitted from the operator terminal 10 to the server device 100 as inquiry information.

[0049] Back to Figure 1 Thus, first, the server device 100 obtains the technical information recorded in the coaching mode and registers / processes it in the technical information DB 102b (step S2). The captured image is registered in the image DB. The collected voice is registered in the voice DB. In addition, the voice is appropriately converted into text and registered in the text DB. For example, in the case where the interface for generating AI described later is only text, the server device 100 uses the text registered in the text DB.

[0050] Furthermore, the server 100 utilizes texts registered in the text DB to generate a dictionary DB for converting business terms and dialects into standard terms. Furthermore, the server 100 associates corresponding images, voices, and texts and assigns label information to the technical information DB 102b.

[0051] Next, the server device 100 obtains, for example, consultation information of the worker U-2 who is a worker at the work site B (step S3). Figure 3 As described above, the consultation information is recorded in the operator terminal 10 - 2 and transmitted to the server device 100 .

[0052] Then, the server device 100 generates a query to the generation AI (generation AI server 200 ) based on the inquiry information (step S4 ).

[0053] Generative AI server 200 is a server device that functions as a generative AI. Generative AI server 200 is implemented, for example, as a private cloud. Generative AI server 200 includes a generative AI model (not shown). Generative AI server 200 reads and executes this generative AI model as part of a program, thereby functioning as generative AI that generates responses to inquiries from server device 100 and transmits them to server device 100.

[0054] Generative AI models are, for example, multimodal large-scale language models that can accept multiple inputs, such as text, images, and speech. Examples of generative AI models include transfer-based models and RNN (Recurrent Neural Network)-based models.

[0055] Examples of transfer-based models include, but are not limited to, GPT (Generative Pre-trained Transformer) and BARD (Bidirectional Auto Regressive Dialogues). Examples of RNN-based models include, but are not limited to, RWKV (Receptance Weighted Key Value). Furthermore, a generative AI model can be, for example, a unimodal large-scale language model that can only accept text.

[0056] In addition, in the present embodiment, the generation AI model of the generation AI server 200 is individualized (which may be referred to as “fine-tuning”) according to the work support task of the plant P1.

[0057] The server device 100 appropriately generates a query according to the interface for the generated AI server 200. At this time, the server device 100 generates a query for the generated AI server 200 so as to generate an answer using each data registered in the technical information DB 102b.

[0058] The server device 100 then exchanges inquiries and responses with the AI ​​generation server 200 (step S5). Furthermore, the server device 100 generates prompt information based on the responses received from the AI ​​generation server 200 (step S6). Furthermore, the server device 100 transmits the prompt information in response to the inquiry to the operator terminal 10-2 (step S7).

[0059] Here, an example of technical information, inquiry information, answers generated by AI, and prompt information generated based on the answers is listed in advance. Figure 4 This is a diagram showing an example of technical information, consultation information, and the answer generated by AI. Figure 5 This is a diagram showing an example of prompt information.

[0060] like Figure 4 As shown, worker U-1 provides voice and images during maintenance work on a certain pipe in workshop P1 as technical information from work site A. As described above, the server device 100 registers this technical information in the technical information DB 102b.

[0061] Furthermore, as inquiry information from work site B, which is also performing maintenance work on a pipe like work site A, there is an inquiry regarding corrosion on a particular pipe. In this case, the operator terminal 10-2 at work site B transmits a voice message such as "There is corrosion on the pipe in the workshop. Is this corrosion?" and an image (including the aforementioned image information) that appears to show corrosion on the pipe at work site B to the server device 100.

[0062] Server device 100 then generates a query for generation AI server 200 based on the consultation information. If the generation AI model is a multimodal large-scale language model, server device 100 generates the query using the audio and images included in the consultation information. If the generation AI model is a unimodal large-scale language model that only accepts text, server device 100 generates the query by, for example, appropriately converting the audio and images included in the consultation information into text.

[0063] The AI ​​generation server 200 generates an answer to the inquiry using each data registered in the technical information DB 102b. Figure 4In the example of , the generation AI server 200 generates an answer using, for example, technical information recorded at the work site A, which is technical information about the same piping as that at the work site B.

[0064] Then, the server device 100 generates prompt information corresponding to the operator terminal 10-2 as the inquiry source based on the answer. Figure 5 As shown, for example, server device 100 generates presentation information such that region R2, which displays at least a portion of the AI-generated answer, is displayed superimposed on region R1 of display 15. Furthermore, server device 100 generates presentation information such that region R3, which displays an image of corrosion at work site A as a reference image, is displayed superimposed on region R1.

[0065] In addition, Figure 5 While an example is shown in which at least a portion of the AI-generated answer is displayed in region R2, it can also be output as voice from speaker 16. Worker U-2 can receive this prompt information regarding their own consultation while performing work at work site B. Furthermore, at work site B, where worker U-1 is not present, worker U-2 can effectively inherit skills from worker U-1.

[0066] Thus, in the operation assistance method involved in the embodiment, the server device 100 obtains and accumulates technical information of at least the operation related to the workshop P1 or the infrastructure from the operator terminal 10 used by the operator U at the operation site. In addition, when the server device 100 receives consultation information related to the operation, it generates a query based on the consultation information in a manner that generates an answer using the technical information for the generated AI using the large-scale language model. In addition, the server device 100 generates prompt information to the operator terminal 10 that is the consultation source based on the answer from the generated AI to the query. In addition, the server device 100 sends the prompt information to the operator terminal 10. As a result, at least with respect to the operation related to the workshop P1 or the infrastructure, it is possible to perform simpler and more practical operation assistance.

[0067] Next, a configuration example of the work assistance system 1 to which the work assistance method according to the embodiment is applied will be described in more detail.

[0068] [Configuration Example of the Worker Terminal 10]

[0069] Figure 6 1 is a block diagram showing a configuration example of the operator terminal 10 according to the embodiment. Figure 5 and as shown hereafter Figure 6 In the figure, only the components necessary for explaining the present embodiment are shown as functional blocks, and description of general components is omitted.

[0070] In addition, when using Figure 5 and Figure 6 In the description of the present invention, the description of the structural elements that have already been described is appropriately simplified or omitted.

[0071] like Figure 5 As shown, the operator terminal 10 includes an operation unit 11 , a sensor unit 12 , a camera 13 , a microphone 14 , a display 15 , a speaker 16 , a communication unit 17 , a storage unit 18 , and a control unit 19 .

[0072] The operation unit 11 is realized as a controller for operating AR glasses, for example. The operation unit 11 receives operations from the operator U on the operator terminal 10. For example, the operation unit 11 receives operations on the aforementioned switch button B1, record button B2, and drawing tool B3.

[0073] The sensor unit 12 is a group of various sensors mounted on the operator terminal 10. The sensor unit 12 includes, for example, a GPS (Global Positioning System) sensor, a G sensor, an angular velocity sensor, etc. The sensor data generated by the sensor unit 12 may appropriately include, for example, technical information and consulting information as supplementary information.

[0074] The camera 13 , the microphone 14 , the display 15 , and the speaker 16 have already been described, and thus their description is omitted here.

[0075] The communication unit 17 is implemented by a network adapter, etc. The communication unit 17 connects the operator terminal 10 and the server device 100 to be communicable through wireless communication and / or wired communication.

[0076] The storage unit 18 is implemented by a storage device such as RAM (Random Access Memory), flash memory, or a hard disk drive (HDD). The storage unit 18 can be implemented by a memory card such as an SD card. The storage unit 18 stores programs related to the embodiments executed by the control unit 19. Furthermore, the storage unit 18 stores various information used in the information processing performed by the control unit 19.

[0077] exist Figure 6 In the example shown in FIG. 1 , the storage unit 18 stores recording information 18a and output control information 18b. The recording information 18a stores images, sounds, and the like recorded by operating the recording button B2. The output control information 18b includes various parameters used by the output control unit 19f, described later, when outputting presentation information.

[0078] The control unit 19 corresponds to a so-called processor and is implemented by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphical Processing Unit), or the like.

[0079] The control unit 19 reads the program according to the embodiment stored in the storage unit 18 and executes it using the RAM as a work area. Alternatively, the control unit 19 may be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0080] The control unit 19 includes a mode setting unit 19a, a recording unit 19b, a technical information generating unit 19c, a consultation information generating unit 19d, an acquiring unit 19e, and an output control unit 19f, and realizes or executes the functions and effects of information processing described below.

[0081] In addition, the internal structure of the control unit 19 is not limited to Figure 6 The structure shown in FIG. 1 may be another structure if it is a structure capable of executing the information processing described below. In addition, the connection relationship between the various processing units of the control unit 19 is not limited to Figure 6 The connection relationship shown can also be other connection relationships.

[0082] The mode setting unit 19a switches the operator terminal 10 to the coaching mode or the Q&A mode based on the operation of the switch button B1. The recording unit 19b records images, sounds, etc. in the technical information or consultation information based on the operation of the record button B2.

[0083] In the tutorial mode, the technical information generating unit 19c generates technical information to be transmitted to the server device 100 based on the log information 18a. The technical information generating unit 19c also transmits the generated technical information to the server device 100 via the communication unit 17.

[0084] In the Q&A mode, the consultation information generating unit 19d generates consultation information to be transmitted to the server device 100 based on the log information 18a. The consultation information generating unit 19d also transmits the generated consultation information to the server device 100 via the communication unit 17.

[0085] The acquisition unit 19e acquires the presentation information generated in the server device 100 via the communication unit 17. The output control unit 19f performs output control to display the presentation information to the display 15 and / or the speaker 16 based on the presentation information acquired by the acquisition unit 19e and the output control information 18b.

[0086] [Configuration Example of Server Device 100]

[0087] Next, a configuration example of the server device 100 according to the embodiment will be described. Figure 7 This is a block diagram showing a configuration example of the server device 100 according to the embodiment.

[0088] like Figure 7 As shown, the server device 100 includes a communication unit 101, a storage unit 102, and a control unit 103. The communication unit 101 is implemented by a network adapter, etc. The communication unit 101 connects the server device 100, the operator terminal 10, and the generated AI server 200 to enable communication through wireless communication and / or wired communication.

[0089] The storage unit 102 is implemented by a storage device such as RAM, flash memory, or HDD. The storage unit 102 stores programs according to the embodiment executed by the control unit 103. The storage unit 102 also stores various information used in information processing executed by the control unit 103.

[0090] exist Figure 7 In the example of , the storage unit 102 stores a speech recognition model 102a, a technical information DB 102b, generated AI interface information 102c, and prompt information generation information 102d.

[0091] The speech recognition model 102a is a learning model used to recognize speech uttered by the operator U, contained in the technical information or consultation information from the operator terminal 10. Examples of the speech recognition model 102a include a Hidden Markov Model (HMM) or a Deep Neural Network (DNN) model. Alternatively, the speech recognition model 102a may be a composite model of an HMM and a DNN model. The speech recognition model 102a is used to convert the speech of the operator U into text and to generate a dictionary database.

[0092] As described above, the skill information DB 102b is a database that registers the skill information transmitted from the worker terminal 10 in the tutoring mode. The skill information DB 102b includes an image DB, a voice DB, a text DB, and a dictionary DB.

[0093] Here, Figure 81 is a diagram showing an example of registration of technical information. The technical information DB 102b is registered by the registration processing unit 103b described later. Figure 8 As shown, the image data of the image DB, the voice data of the voice DB, the text data of the text DB, etc. are associated with each other. At this time, the sensor data etc. based on the sensor unit 12 are also associated as supplementary information.

[0094] In addition, if Figure 8 As shown, in the registration process by the registration processing unit 103b, various tag data are given to the data group that has been associated in this way. Figure 3 In the example shown, tag data representing meta-information of technical information such as "piping," "corrosion," and "work site A" are added. The AI ​​generation server 200 can also use this tag data as tag information when generating a response to a query.

[0095] Back to Figure 7 The generated AI interface information 102c is information related to the interface for the server device 100 and the generated AI server 200 to exchange inquiries and answers. The generated AI interface information 102c includes, for example, an API (Application Programming Interface) for generated AI.

[0096] The presentation information generation information 102d is information including various parameters used when generating presentation information for the operator terminal 10. The presentation information generation information 102d includes information on the coordinate system of the AR display space of the display 15 of the operator terminal 10, for example.

[0097] The control unit 103 corresponds to a so-called processor and is implemented by a CPU, an MPU, a GPU, or the like.

[0098] The control unit 103 reads the program according to the embodiment stored in the storage unit 102 and executes the program using the RAM as a work area. Alternatively, the control unit 103 may be implemented by an integrated circuit such as an ASIC or an FPGA.

[0099] The control unit 103 includes an acquisition unit 103a, a registration processing unit 103b, a reception unit 103c, a pair generation AI processing unit 103d, and a presentation information generation unit 103e, and has functions and roles for realizing or executing information processing described below.

[0100] In addition, the internal structure of the control unit 103 is not limited to Figure 7 The structure shown in FIG. 1 may be another structure if it is a structure capable of executing the information processing described below. In addition, the connection relationship between the various processing units of the control unit 103 is not limited to Figure 7 The connection relationship shown can also be other connection relationships.

[0101] The acquisition unit 103a acquires technical information or inquiry information from the operator terminal 10 via the communication unit 101. The registration processing unit 103b appropriately processes the technical information acquired by the acquisition unit 103a and performs the aforementioned registration process of registering the technical information in the technical information DB 102b.

[0102] The receiving unit 103c receives the consultation information received by the acquiring unit 103a and causes the generation AI processing unit 103d to execute generation AI processing (ie, processing for the generation AI server 200) according to the received consultation information.

[0103] The generated AI processing unit 103d performs generated AI processing based on the consultation information received by the receiving unit 103c. Specifically, the generated AI processing unit 103d generates a query to be sent to the generated AI server 200 based on the consultation information and the generated AI interface information 102c. If necessary, the generated AI processing unit 103d utilizes the speech recognition model 102a to textualize the speech uttered by the operator U contained in the consultation information.

[0104] Furthermore, the generated AI processing unit 103d transmits the generated query to the generated AI server 200 via the communication unit 101. Furthermore, the generated AI processing unit 103d obtains an answer to the query from the generated AI server 200 via the communication unit 101.

[0105] The presentation information generator 103e generates presentation information to be sent to the operator terminal 10 based on the response from the generation AI server 200 received by the generation AI processor 103d and the presentation information generation information 102d. Furthermore, the presentation information generator 103e transmits the generated presentation information to the operator terminal 10, which is the source of the inquiry, via the communication unit 101.

[0106] [Processing Flow Executed by the Work Support System 1]

[0107] Next, use Figure 9 The flow of processing executed by the work support system 1 will be described. Figure 9 It is a diagram showing a processing procedure executed by the work support system 1 according to the embodiment.

[0108] like Figure 9 As shown, it is assumed that the operator terminal 10 is activated in the tutorial mode (step S101 ). In this case, the operator terminal 10 records the work status of the operator U using the operator terminal 10 and generates technical information (step S102 ).

[0109] The operator terminal 10 then transmits the generated technical information to the server 100 (step S103). The server 100 registers the technical information received from the operator terminal 10 (step S104). Steps S101 to S104 are repeated to accumulate the skilled worker's skill information in the technical information DB 102b.

[0110] On the other hand, the operator terminal 10 is assumed to be activated in the Q&A mode (step S105 ). In this case, the operator terminal 10 generates consultation information based on images, voices, etc. recorded according to the operation of the operator U using the operator terminal 10 (step S106 ).

[0111] The operator terminal 10 then transmits the generated consultation information to the server device 100 (step S107). The server device 100 generates a query to the generation AI based on the consultation information received from the operator terminal 10 (step S108). The server device 100 then transmits the generated query to the generation AI, i.e., the generation AI server 200 (step S109).

[0112] Upon receiving the query, the generative AI server 200 generates an answer corresponding to the query using the generative AI model (step S110) and transmits it to the server device 100 (step S111). The server device 100 generates prompt information based on the answer received from the generative AI server 200 (step S112) and transmits it to the operator terminal 10 (step S113).

[0113] Then, the operator terminal 10 presents the presentation information received from the server device 100 (step S114 ).

[0114] [Technical information, consulting information, and other examples of AI-generated responses]

[0115] In addition, when using Figure 4 The description of the technical information, consultation information and an example of generating AI answers are explained. Figure 10 and Figure 11 Other examples are described. Figure 10 This is a diagram showing another example of technical information, consultation information, and answers generated by AI (part 1). Figure 11 This is a diagram (part 2) showing another example of technical information, consultation information, and answers generated by AI.

[0116] exist Figure 10 An example of a case where a maintenance operation of a pipe is performed using a knocking method is shown in FIG. Figure 10As shown, as technical information from the work site A, the operator terminal 10-1 provides, for example, tips for tapping a pipe, the sound produced when tapping is actually performed, the echo of the sound produced, and explanations understood based on the echo, voice, images, etc.

[0117] The server device 100 registers this technical information with the technical information DB 102b. At this point, the server device 100 can perform image analysis to determine the actual location of the pipe being struck (i.e., the point of application), the position and posture of the hammer, and other information, and also register the image and text generated as analysis results. This allows for further supplementation of technical information, increasing the likelihood of obtaining highly accurate answers from the generated AI.

[0118] In addition, the server device 100 can perform frequency analysis of the actual sound of tapping the pipe and the resulting echo, and register a spectrum image as the analysis result. This method can also further supplement technical information.

[0119] Furthermore, when there is consultation information from work site B, which performs piping maintenance work based on the knocking method similarly to work site A, such as a consultation such as "What are the tips?", the server device 100 generates a query to the AI ​​server 200 based on the consultation information.

[0120] The AI ​​generation server 200 generates an answer to the inquiry using each data registered in the technical information DB 102b. Figure 10 In the example of , the generation AI server 200 generates an answer using technical information of the work site A where the same maintenance work of piping based on the knocking method as that of the work site B is performed.

[0121] At this time, the AI ​​server 200 is generated as follows Figure 10 As shown, an answer is generated that clearly indicates the location corresponding to "near it". This allows the server device 100 to pre-register an image or the like as an analysis result of the image analysis that identifies the location of the pipe actually struck, the position and posture of the hammer, etc.

[0122] Then, in Figure 11 For example, FIG. 1 shows an example in which a consultation is received from an operator U-2 who is performing maintenance work on piping at a narrow work site B. In this case, Figure 11 As shown, for example, voice, images, etc. related to tools and the like during maintenance work on piping at a similarly narrow work site A are useful as technical information from the work site A.

[0123] In this case, if the inquiry information from work site B is, for example, "What tool would be convenient if available?", server device 100 generates a query to generation AI server 200 based on the inquiry information. In this case, the inquiry information from work site B does not necessarily need to be generated from work site B.

[0124] The AI ​​generation server 200 generates an answer to the inquiry using each data registered in the technical information DB 102b. Figure 11 In the example, the AI ​​server 200 generates an answer to the situation that a fiber endoscope, an extremely small drone, etc. can be provided by using technical information of the narrow work site A that is similar to the work site B.

[0125] As can be seen from this, before actually going to the work site B, the worker U-2 scheduled to perform work at the work site B also receives useful answers and prompt information based on the answers by generating AI based on the technical information registered in the technical information DB 102b.

[0126] [Modification]

[0127] Furthermore, in the above-described embodiment, the operator terminal 10 is switched to the coaching mode by operating the switch button B1 . However, only a predetermined qualified person or certified technician can switch to the coaching mode.

[0128] In this case, the work support system 1 contains authentication information for qualified personnel and certified technicians. The operator terminal 10 can switch to the tutoring mode only when the operator U using the operator terminal 10 is authenticated by this authentication information at startup. This improves the reliability of the technical information stored in the technical information DB 102b. Furthermore, the generated AI can generate highly reliable answers.

[0129] In addition, in the above embodiment, the technology inheritance related to the maintenance work of workshop P1 is listed as the main example, but not only technology inheritance is possible, but this embodiment can also be flexibly used to overcome problems at the work site, eliminate near-misses, and improve the efficiency of work.

[0130] [Effect]

[0131] As described above, the server device 100 (equivalent to an example of a "work assistance device") involved in the embodiment includes a control unit 103. The control unit 103 acquires and accumulates skill information of at least work related to the workshop P1 or infrastructure from the operator terminal 10 used by the operator U at the work site. When receiving consultation information related to the above-mentioned work, the control unit 103 generates a query based on the consultation information in a manner that generates an answer using the above-mentioned skill information for the generation AI using a large-scale language model. Based on the above-mentioned answer from the above-mentioned generation AI to the above-mentioned query, prompt information is generated to the operator terminal 10 serving as the consultation source, and the above-mentioned prompt information is sent to the operator terminal 10. As a result, it is possible to perform simpler and more practical work assistance at least with respect to work related to the workshop P1 or infrastructure.

[0132] Furthermore, the skill information and consultation information include at least an image, voice, and text converted from the voice related to the task, recorded by the operator terminal 10. The control unit 103 generates the query based on at least one of the image, voice, and text, corresponding to a modality acceptable to the large-scale language model. This enables the generation of an appropriate query that corresponds to the large-scale language model used to generate the AI.

[0133] Furthermore, the large-scale language model is a multimodal large-scale language model, which enables the generation of multiple types of queries using the images, voices, and texts, and the generation of AI to obtain highly accurate responses corresponding to the queries.

[0134] Furthermore, the control unit 103 performs image analysis on the image included in the skill information and adds the result of the image analysis to the skill information. This further supplements the skill information and increases the likelihood of obtaining a highly accurate answer from the generated AI.

[0135] The image analysis results also include information about the position and posture of the tool used by the operator U during the work, as well as the point of application of the tool. This allows, for example, an answer to clearly indicate the position of the point of application, the position of the hammer, and the posture of the hammer when striking a pipe with the hammer.

[0136] Furthermore, the control unit 103 performs frequency analysis on the speech included in the skill information and adds the results of the frequency analysis to the skill information. This allows for further supplementation of skill information regarding the speech, increasing the likelihood of obtaining highly accurate responses from the generated AI.

[0137] Furthermore, the result of the frequency analysis includes a spectrum image, thereby increasing the possibility of obtaining a highly accurate answer to the speech using the spectrum image.

[0138] Furthermore, the operator terminal 10 is an AR terminal, thereby enabling work assistance using AR technology.

[0139] Furthermore, the control unit 103 generates the presentation information so that the presentation information is superimposed on the real space in the AR display space of the AR terminal. This allows for presentation of highly convenient presentation information using AR technology.

[0140] Furthermore, the skill information includes technical information related to the work provided by the skilled worker U. This allows the skilled worker's knowledge and experience to be inherited.

[0141] [Other embodiments]

[0142] Furthermore, although the embodiment of the present invention has been described so far, the present invention can be implemented in various different forms other than the above-described embodiment.

[0143] [system]

[0144] Information including the processing flow, control flow, specific names, various data, and parameters described above and shown in the drawings can be arbitrarily changed unless otherwise specified.

[0145] Furthermore, the components of the devices shown in the diagrams are functional concepts and do not necessarily need to be physically configured as shown. Specifically, the specific methods of distributing and integrating the devices are not limited to those shown. Specifically, all or part of the components can be functionally or physically distributed / integrated in arbitrary units based on various loads, usage conditions, and the like.

[0146] Furthermore, all or any part of each processing function performed by each device may be realized by a CPU and a program analyzed and executed by the CPU, or may be realized as hardware based on wired logic.

[0147] [hardware]

[0148] The operator terminal 10 and the server device 100 according to the above embodiment are, for example, Figure 12 The structure shown is realized by a computer 1000. The following description will be given using the server device 100 as an example. Figure 12 This is a hardware configuration diagram showing an example of a computer 1000 that realizes the functions of the server device 100 according to the embodiment.

[0149] like Figure 12 As shown, the computer 1000 includes a communication device 1000a, a secondary storage device 1000b, a memory 1000c, and a processor 1000d. Figure 12The components shown are connected to each other by a bus or the like.

[0150] The communication device 1000a is a NIC (Network Interface Card) and communicates with other devices. The secondary storage device 1000b is implemented by a flash memory, HDD, etc. Figure 7 The functions shown are executed in programs and databases.

[0151] Processor 1000d executes Figure 7 The program for the same processing of each processing unit shown is read from the secondary storage device 1000b and expanded in the memory 1000c, so that the execution Figure 7 The threads for each function described in

[103] , ...

[0152] In this manner, the computer 1000 operates as an information processing device that reads and executes programs to perform various processing methods. Furthermore, the computer 1000 can also utilize a media reader to read the program from a recording medium and execute the read program, thereby achieving the same functions as the above-described embodiment. Furthermore, the program described herein is not limited to being executed solely by the computer 1000. For example, the present invention can also be applied to the execution of a program by a computer or server having other hardware structures, or when these computers or servers collaborate to execute the program.

[0153] The program can be distributed via a network such as the Internet. In addition, the program can be recorded on a computer-readable recording medium such as an HDD, a floppy disk (FD), a CD-ROM, an MO (Magneto-Optical disk), or a DVD (Digital Versatile Disc), and executed by the computer by reading from the recording medium. A recording medium having such a program recorded thereon is also one embodiment of the present disclosure.

[0154] [other]

[0155] Several examples of combinations of disclosed technical features are described below.

[0156] (1) A work assisting device, wherein:

[0157] The work support device includes a control unit that acquires and stores skill information of at least a work related to a workshop or infrastructure from a terminal device used by a worker performing the work at a work site.

[0158] When consultation information related to the job is received, a query is generated based on the consultation information, requesting an answer generated by a generative AI using a large-scale language model using the skill information.

[0159] generating, based on the answer from the generation AI to the inquiry, presentation information to the terminal device serving as the inquiry source;

[0160] The prompt information is sent to the terminal device.

[0161] (2) The work assisting device according to (1), wherein:

[0162] The skill information and the consultation information include at least an image, a voice, and a text converted from the voice related to the task recorded by the terminal device.

[0163] The control unit generates the query based on at least any one of the image, the voice, and the text corresponding to a modality acceptable to the large-scale language model.

[0164] (3) The work assisting device according to (2), wherein:

[0165] The large-scale language model is a multimodal large-scale language model.

[0166] (4) The work assisting device according to (2) or (3), wherein:

[0167] The control unit performs image analysis on the image included in the skill information, and further adds the result of the image analysis to the skill information.

[0168] (5) The work assisting device according to (4), wherein:

[0169] The result of the image analysis includes information on the position and posture of a tool used by the worker during the work and the point of application of the tool.

[0170] (6) The work assist device according to any one of (2) to (5), wherein:

[0171] The control unit performs frequency analysis on the voice included in the skill information, and further adds the result of the frequency analysis to the skill information.

[0172] (7) The work assisting device according to (6), wherein:

[0173] The result of the frequency analysis includes a spectrum image.

[0174] (8) The work assisting device according to any one of (1) to (7), wherein:

[0175] The terminal device is an AR (Augmented Reality) terminal.

[0176] (9) The work assisting device according to (8), wherein:

[0177] The control unit generates the presentation information so that the presentation information is superimposed on a real space in an AR display space of the AR terminal.

[0178] (10) The work assisting device according to any one of (1) to (9), wherein:

[0179] The skill information includes technical information related to the work provided by the worker who is a skilled worker.

[0180] (11) A work support system, wherein:

[0181] The work support system includes: a server device that manages at least work support services related to a workshop or infrastructure; and a terminal device that is used by a worker at a work site.

[0182] The server device acquires and stores the skill information of the operation from the terminal device,

[0183] When consultation information related to the job is received, a query is generated based on the consultation information, requesting the generation AI using the large-scale language model to generate an answer using the skill information.

[0184] generating, based on the answer from the generation AI to the inquiry, presentation information to the terminal device serving as the inquiry source;

[0185] Sending the prompt information to the terminal device,

[0186] In the first mode, the terminal device transmits the skill information to the server device.

[0187] In the second mode, the consultation information is transmitted to the server device, and the presentation information corresponding to the consultation information is received from the server device and presented to the staff.

[0188] (12) A method for assisting an operation, wherein:

[0189] The work assisting method causes a computer to execute the following processing, namely,

[0190] Skill information for at least one operation related to a workshop or infrastructure is acquired and stored from a terminal device used by a worker at the operation site.

[0191] When consultation information related to the job is received, a query is generated based on the consultation information, requesting the generation AI using the large-scale language model to generate an answer using the skill information.

[0192] generating, based on the answer from the generation AI to the inquiry, presentation information to the terminal device serving as the inquiry source;

[0193] The prompt information is sent to the terminal device.

[0194] (13) A program, wherein:

[0195] The program causes the computer to execute the following processing, namely,

[0196] Skill information for at least one operation related to a workshop or infrastructure is acquired and stored from a terminal device used by a worker at the operation site.

[0197] When consultation information related to the job is received, a query is generated based on the consultation information, requesting the generation AI using the large-scale language model to generate an answer using the skill information.

[0198] generating, based on the answer from the generation AI to the inquiry, presentation information to the terminal device serving as the inquiry source;

[0199] The prompt information is sent to the terminal device.

[0200] (14) A computer-readable recording medium recording a program, wherein:

[0201] The program causes the computer to execute the following processing, namely,

[0202] Skill information for at least one operation related to a workshop or infrastructure is acquired and stored from a terminal device used by a worker at the operation site.

[0203] When consultation information related to the job is received, a query is generated based on the consultation information, requesting the generation AI using the large-scale language model to generate an answer using the skill information.

[0204] generating, based on the answer from the generation AI to the inquiry, presentation information to the terminal device serving as the inquiry source;

[0205] The prompt information is sent to the terminal device.

[0206] Description of the label

[0207] 1. Operational assistance system

[0208] 10 Operator Terminal

[0209] 11 Operation section

[0210] 12 Sensor unit

[0211] 13 Camera

[0212] 14 Microphone

[0213] 15 Display

[0214] 16 speakers

[0215] 17 Ministry of Communications

[0216] 18 Storage

[0217] 18a Recording Information

[0218] 18b output control information

[0219] 19 Control Department

[0220] 19a Mode setting unit

[0221] 19b Records Department

[0222] 19c Technical Information Generation Department

[0223] 19d Consulting Information Generation Department

[0224] 19e Acquisition Department

[0225] 19f output control unit

[0226] 100 server devices

[0227] 101 Department of Communications

[0228] 102 Storage Department

[0229] 102a speech recognition model

[0230] 102c Generate AI interface information

[0231] 102d prompt information generation information

[0232] 103 Control Department

[0233] 103a Acquisition Department

[0234] 103b Registration Processing Department

[0235] 103c Reception Department

[0236] 103d pair generation AI processing unit

[0237] 103e prompt information generation unit

[0238] 200 Generate AI Server

[0239] U Operator

Claims

1. A work assist device, wherein: The work support device includes a control unit that acquires and stores at least skill information of work related to a workshop or infrastructure from a terminal device used by a worker at a work site. When consultation information related to the job is received, a query is generated based on the consultation information, requesting the generative AI using the large-scale language model to generate an answer using the skill information. generating, based on the answer from the generating AI to the inquiry, presentation information to the terminal device serving as the inquiry source, The prompt information is sent to the terminal device.

2. The work assist device according to claim 1, wherein: The skill information and the consultation information include at least an image, a voice, and a text converted from the voice related to the task recorded by the terminal device. The control unit generates the query based on at least any one of the image, the voice, and the text corresponding to a modality acceptable to the large-scale language model.

3. The work assist device according to claim 2, wherein: The large-scale language model is a multimodal large-scale language model.

4. The work assist device according to claim 2, wherein: The control unit performs image analysis on the image included in the skill information, and further adds the result of the image analysis to the skill information.

5. The work assist device according to claim 4, wherein: The result of the image analysis includes information on the position and posture of a tool used by the worker during the work and the point of application of the tool.

6. The work assist device according to claim 2, wherein: The control unit performs frequency analysis on the voice included in the skill information, and further adds the result of the frequency analysis to the skill information.

7. The work assist device according to claim 6, wherein: The result of the frequency analysis includes a spectrum image.

8. The work assist device according to claim 1, wherein The terminal device is an AR terminal, and AR stands for augmented reality.

9. The work assist device according to claim 8, wherein: The control unit generates the presentation information so that the presentation information is superimposed on a real space in an AR display space of the AR terminal.

10. The work assist device according to any one of claims 1 to 9, wherein: The skill information includes technical information related to the work provided by the worker who is a skilled worker.

11. A work assistance system, wherein: The work support system includes: a server device that manages at least work support services related to a workshop or infrastructure; and a terminal device that is used by a worker at a work site. The server device acquires and stores the skill information of the operation from the terminal device, When consultation information related to the job is received, a query is generated based on the consultation information, requesting the generative AI using the large-scale language model to generate an answer using the skill information. generating, based on the answer from the generation AI to the inquiry, presentation information to the terminal device serving as the inquiry source; Sending the prompt information to the terminal device, In the first mode, the terminal device transmits the skill information to the server device. In the second mode, the consultation information is transmitted to the server device, and the presentation information corresponding to the consultation information is received from the server device and presented to the staff.

12. A method for assisting a job, wherein: The work assistance method causes a computer to execute the following processing: At least the skill information of the work related to the workshop or infrastructure is acquired and stored from the terminal device used by the workers of the work at the work site. When consultation information related to the job is received, a query is generated based on the consultation information, requesting the generative AI using the large-scale language model to generate an answer using the skill information. generating, based on the answer from the generation AI to the inquiry, presentation information to the terminal device serving as the inquiry source; The prompt information is sent to the terminal device.

Images