Information processing device, work content determination method, and work content determination program

Abstract

This streamlines the process of confirming the completion of on-site work, thereby reducing the likelihood of overlooking details. [Solution] The information processing device 10 includes an acquisition unit 11A that acquires work data from a terminal device 20 of a field worker, associating a checklist with predetermined confirmation points for each step of the field work with a captured image of the work content taken for each confirmation point on the checklist; a reading unit 11B that reads work content information representing the work content from the captured image of the work data; a determination unit 11C that uses a generation AI 150 to determine whether or not the field work has been completed for each confirmation point based on the work content information; and a reply unit 11D that returns the determination result to the terminal device 20.

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, a work content determination method, and a work content determination program.

Background Art

[0002] For example, Patent Document 1 describes a maintenance support system that double-checks maintenance work in a remote environment. This maintenance support system displays a procedure screen including the content of steps for ordering maintenance work, an executor checkbox, and a reinspector checkbox on the user terminal of the executor and the user terminal of the reinspector, and shares the executor check result and the reinspector check result between the user terminal of the executor and the user terminal of the reinspector.

[0003] Also, Patent Document 2 describes an inspection apparatus capable of determining the quality of an inspection object. This inspection apparatus uses, as a template image, an image obtained by imaging the state before an inspection object is attached to a correct position with a template for performing image comparison, and first performs comparison with the inspection object.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] By the way, when a field worker confirms that the field work has been completed without problems, since a third party other than the field worker visually confirms it, it takes time and there may be an omission in confirmation due to oversight in manual confirmation.

[0006] This disclosure aims to provide an information processing device, a work content determination method, and a work content determination program that can streamline the confirmation of the completion of on-site work and suppress the occurrence of confirmation omissions. [Means for solving the problem]

[0007] To achieve the above objective, the information processing device relating to this disclosure includes: an acquisition unit that acquires work data from a terminal device of a field worker, which associates a checklist with predetermined confirmation points for each step of the field work process and a captured image of the work content taken for each confirmation point on the checklist; a reading unit that reads work content information representing the work content from the captured image of the work data; a determination unit that uses a pre-trained model that has been machine-learned to determine whether or not the field work has been completed for each confirmation point based on the work content information; and a reply unit that returns the result of the determination to the terminal device. [Effects of the Invention]

[0008] According to this disclosure, it is possible to provide an information processing device, a work content determination method, and a work content determination program that can streamline the confirmation of the completion of on-site work and suppress the occurrence of confirmation omissions. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example of the configuration of the work content determination system according to the embodiment. [Figure 2] This is a block diagram showing an example of the functional configuration of an information processing apparatus according to the embodiment. [Figure 3] This is a front view showing an example of a checklist screen displayed on a terminal device. [Figure 4] This is a front view showing an example of a checklist screen displayed on a terminal device. [Figure 5] This is a front view showing an example of a checklist screen displayed on a terminal device. [Figure 6] This figure shows an example of a prompt message that will be input to the generating AI. [Figure 7] This flowchart shows an example of the processing flow by the application program according to the embodiment. [Figure 8] This flowchart shows an example of the processing flow by the work content determination program according to the embodiment. [Modes for carrying out the invention]

[0010] Hereinafter, an example of an embodiment of this disclosure will be described with reference to the drawings. In each drawing, identical or equivalent components and parts are given the same reference numerals. Also, the dimensional ratios in the drawings are exaggerated for illustrative purposes and may differ from the actual ratios.

[0011] The information processing device according to this embodiment acquires work data from a field worker's terminal device, which associates a checklist with predetermined confirmation points for each step of the field work process and images of the work taken for each confirmation point on the checklist. The device reads work content information representing the work from the images in the work data, and uses a pre-trained machine learning model to determine whether the field work has been completed for each confirmation point based on the work content information, and returns the result of the determination to the terminal device. This makes it possible to streamline the confirmation of the completion of field work and suppress the occurrence of confirmation omissions.

[0012] Figure 1 shows an example of the configuration of the work content determination system 100 according to this embodiment. As shown in Figure 1, the work content determination system 100 according to this embodiment includes an information processing device 10 that performs work content determination at the site and a terminal device 20 used by the site worker. The information processing device 10 and the terminal device 20 are connected to each other via a network N. The network N is, for example, a communication network such as the Internet, LAN (Local Area Network), or WAN (Wide Area Network), and may be wired or wireless. The information processing device 10 is, for example, a general-purpose computer device such as a server computer or a personal computer (PC). The terminal device 20 is, for example, a terminal device such as a smartphone, tablet terminal, or PC. Alternatively, the terminal device 20 may be a dedicated terminal used for work content determination processing at the site (hereinafter referred to as "work content determination processing").

[0013] In this embodiment, the terminal device 20 stores an application program 20A for requesting the information processing device 10 to perform work content determination processing. By executing the application program 20A, the terminal device 20 requests the information processing device 10 to perform work content determination processing. The application program 20A is executed in cooperation with the work content determination program 15A, which will be described later.

[0014] The information processing device 10 according to this embodiment includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, an input / output interface (I / O) 14, a storage unit 15, a display unit 16, an operation unit 17, and a communication unit 18.

[0015] CPU 11, ROM 12, RAM 13, and I / O 14 are each connected via a bus. To I / O 14, each functional unit including a storage unit 15, a display unit 16, an operation unit 17, and a communication unit 18 is connected. These functional units can communicate with each other with the CPU 11 via I / O 14.

[0016] The control unit is constituted by CPU 11, ROM 12, RAM 13, and I / O 14. The control unit may be configured as a sub-control unit that controls some operations of the information processing apparatus 10, or may be configured as part of a main control unit that controls the overall operations of the information processing apparatus 10. For some or all of the blocks of the control unit, for example, an integrated circuit such as LSI (Large Scale Integration) or an IC chip set is used. Individual circuits may be used for the above blocks, or circuits in which some or all are integrated may be used. The above blocks may be provided integrally, or some blocks may be provided separately. Also, in each of the above blocks, a part thereof may be provided separately. For the integration of the control unit, not limited to LSI, a dedicated circuit or a general-purpose processor may be used.

[0017] As the storage unit 15, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), a flash memory, etc. are used. In the storage unit 15, a work content determination program 15A for executing the work content determination process according to this embodiment is stored. Note that this work content determination program 15A may be stored in the ROM 12.

[0018] The work content determination program 15A may be, for example, pre-installed in the information processing device 10. The work content determination program 15A may be stored in a non-volatile storage medium, distributed via the network N, and appropriately installed in the information processing device 10. Examples of non-volatile storage media include CD-ROM (Compact Disc Read Only Memory), magneto-optical disks, HDDs, DVD-ROM (Digital Versatile Disc Read Only Memory), flash memories, memory cards, etc.

[0019] For the display unit 16, for example, a liquid crystal display (LCD: Liquid Crystal Display), an organic EL (Electro Luminescence) display, etc. are used. The display unit 16 may integrally have a touch panel. For the operation unit 17, for example, devices for operation input such as a keyboard and a mouse are provided. The display unit 16 and the operation unit 17 receive various instructions from the user of the information processing device 10. The display unit 16 displays various information such as the result of the process executed according to the instruction received from the user and notifications regarding the process.

[0020] As an example, the communication unit 18 is connected to the network N such as the Internet, LAN, or WAN, and communication with the terminal device 20 via the network N is enabled.

[0021] By the way, as described above, when the field worker confirms that the field work has been completed without problems, since a third party other than the field worker visually confirms it, it takes time and there may be an oversight in the confirmation due to a miss in manual confirmation.

[0022] Therefore, the CPU 11 of the information processing device 10 according to the present embodiment functions as each unit shown in FIG. 2 by writing the work content determination program 15A stored in the ROM 12 or the storage unit 15 into the RAM 13 and executing it.

[0023] Figure 2 is a block diagram showing an example of the functional configuration of the information processing device 10 according to this embodiment. As shown in Figure 2, the CPU 11 of the information processing device 10 according to this embodiment functions as an acquisition unit 11A, a reading unit 11B, a determination unit 11C, a reply unit 11D, and a display control unit 11E.

[0024] The memory unit 15 stores the artificial intelligence (AI) 150. The artificial intelligence 150 is an example of a pre-trained model that has been machine-learned, specifically, ChatGPT®, Gemini®, etc. The artificial intelligence 150 is obtained, for example, by performing deep learning on a neural network. The artificial intelligence 150 receives prompts containing instructions and inference data such as captured images, which are image data. The artificial intelligence 150 determines whether the task has been completed according to the instructions indicated by the prompts using the input inference data, and outputs the determination result. Note that the pre-trained model is not limited to the artificial intelligence 150; for example, a large language model (LLM) may also be applied. This large language model (LLM) is a computer language model composed of a neural network with many parameters, and is trained using supervised or semi-supervised learning with a vast amount of unlabeled text.

[0025] Furthermore, the storage unit 15 stores, for example, a contract database (DB) 151, equipment database 152, know-how database 153, work procedure database 154, checklist database 155, captured image database 156, work judgment database 157, and work record database 158. The contract database 151 stores contract information related to the contract of the business operator performing the on-site work. The equipment database 152 stores equipment information related to the equipment that is the target of the on-site work. The know-how database 153 stores know-how information related to the know-how of on-site work. The work procedure database 154 stores work procedure manuals for on-site work. The checklist database 155 stores checklists for checking whether or not the on-site work has been completed. The captured image database 156 stores captured images of the work content of the on-site work. The work judgment database 157 stores the judgment conditions for checking whether or not the on-site work has been completed. The work record database 158 stores the results of checking whether or not the on-site work has been completed as a work record.

[0026] The acquisition unit 11A acquires work data from the field worker's terminal device 20. This work data is data that associates a checklist, which has predetermined confirmation points for each step of the field work, with images taken of the work content for each confirmation point on the checklist. Here, the confirmation points are defined as points to be checked in the work content, and include, for example, confirmation of the setting value of the target equipment, confirmation of the setting time of the target equipment, confirmation of the IP address of the target equipment, etc. The images taken include, for example, images of the setting value of the target equipment, images of the setting time of the target equipment, images of the IP address of the target equipment, etc., and may be one or more still images or a video. The images taken include, for example, paper media such as receipts, setting printouts, tickets, screens of terminals and control panels, switch status (on / off), wiring status, etc.

[0027] The reading unit 11B reads work content information representing the work content from the captured image of the work data acquired by the acquisition unit 11A. The reading unit 11B is, for example, OCR (Optical Character Recognition). The work content information is information used to determine the work content. Specifically, for example, it reads the setting value of the target device, the setting time, the IP address, the switch status (on / off), etc., as work content information by performing character recognition on the captured image.

[0028] The determination unit 11C uses the generation AI 150 to determine whether the on-site work has been completed at each confirmation point based on the work content information. Specifically, the generation AI 150 determines that the on-site work has been completed if the work content identified by the work content information satisfies predetermined determination conditions, and determines that the on-site work has not been completed if the work content identified by the work content information does not satisfy predetermined determination conditions. In other words, the generation AI 150 obtains the determination conditions for determining whether the on-site work has been completed at each confirmation point from the work determination DB 157, and outputs a determination result (completed or not completed) when work content information is input.

[0029] The reply unit 11D returns the determination result from the determination unit 11C to the terminal device 20 of the field worker. Specifically, if the reply unit 11D determines that the field work is complete, it returns "OK" indicating that the work is complete, and if it determines that the field work is not complete, it returns "NG" indicating that the work is incomplete. When returning "NG", it is desirable to also return the reason for the "NG" decision.

[0030] In other words, as the field worker proceeds to check items in order according to the checklist displayed on the terminal device 20, a button to "request judgment" becomes available when a checkpoint requiring a judgment on the work content is reached. When the field worker presses this button, the work content, checklist, and captured images up to that point are sent to the information processing device 10. The information processing device 10 judges various types of work; for example, it determines whether the settings have been completed correctly. Specifically, it determines whether the setting values ​​have been reflected according to the procedure (for example, whether the equipment setting items are correct, the time is correct, the IP address is correct, the on / off status is correct, and settings that are not the target of the work have not changed before and after the work). The terminal device 20 then receives the judgment result, and if the result is "OK", the subsequent checklist becomes active and the work can be continued. If the judgment result is "NG", the reason for NG is referred to and the check process is repeated.

[0031] Next, with reference to Figures 3 to 5, the checklist screen 21 displayed on the terminal device 20 will be described in detail. The checklist screen 21 is displayed when the application program 20A is executed, and it is a screen for associating the checklist with the captured images. The display control unit 11E controls the display of the checklist screen 21 on the terminal device 20.

[0032] Figures 3 to 5 are front views showing an example of a checklist screen 21 displayed on the terminal device 20. In the checklist screen 21 shown in Figure 3, confirmation points and captured images are associated for each checklist. In the example in Figure 3, for checklist A, captured image P1 is associated with confirmation point (1), and captured image P2 is associated with confirmation point (2). In this case, the "Request Judgment" button is displayed as clickable. When the field worker clicks the "Request Judgment" button, work data including checklist A and the captured images is sent to the information processing device 10. Note that depending on the content of the confirmation points, captured images may not be required, in which case the captured images are not associated.

[0033] Meanwhile, as described above, the information processing device 10 reads work content information from the captured image in response to a judgment request from the terminal device 20, and uses the generating AI 150 to determine whether or not the on-site work has been completed at each confirmation point based on the work content information. The information processing device 10 then returns the judgment result to the terminal device 20.

[0034] The checklist screen 21 shown in Figure 4 shows an example of a screen when the judgment result is "OK". If the judgment result is "OK" for all confirmation points in checklist A, the system proceeds to the subsequent checklist B. On the other hand, the checklist screen 21 shown in Figure 5 shows an example of a screen when some of the judgment results are "NG". In the example in Figure 5, the judgment result for confirmation point (1) is "NG", and the reason for "NG" is also displayed. Based on the displayed reason for "NG", the field worker redoes the field work at confirmation point (1), registers the captured image again, and sends another judgment request to the information processing device 10.

[0035] Figure 6 shows an example of a prompt statement input to the generation AI 150. Figure 6 shows an example of a system prompt statement and an example of a user prompt statement. The system prompt statement and user prompt statement shown in Figure 6 are input to the generation AI 150, and the generation AI 150 outputs a judgment result according to the system prompt statement and user prompt statement. The work content determination by the information processing device 10 is performed by combining the generation AI 150 and the reading unit 11B (OCR). Specifically, prompt statements that will serve as processing instructions are prepared and registered for each pattern necessary for work content determination, and the prompt statements necessary for work content determination are combined and executed.

[0036] Next, the operation of the work content determination system 100 according to this embodiment will be explained with reference to Figures 7 and 8.

[0037] Figure 7 is a flowchart showing an example of the processing flow by the application program 20A according to this embodiment. When a field worker instructs the terminal device 20 to execute processing by the application program 20A, the CPU of the terminal device 20 executes the application program 20A.

[0038] In step S101 of Figure 7, the CPU of the terminal device 20 displays, for example, the checklist screen 21 shown in Figure 3 above, and accepts input of captured images for each checkpoint on the checklist.

[0039] In step S102, when the CPU of the terminal device 20 detects, for example, the pressing of the "Request Judgment" button on the checklist screen 21 shown in Figure 3 above, it sends work data associating the checklist with the captured image to the information processing device 10.

[0040] In step S103, the CPU of the terminal device 20 determines whether or not it has received the result of the work content determination from the information processing device 10. If it determines that it has received the determination result (positive determination), it proceeds to step S104. If it determines that it has not received the determination result (negative determination), it remains in a waiting state in step S103.

[0041] In step S104, the CPU of the terminal device 20 determines whether the work has been completed for each checkpoint, for example, based on the checklist screen 21 shown in Figure 4 or Figure 5 above. If it is determined that the work has been completed for each checkpoint (positive determination), the process proceeds to step S105. If it is determined that the work has not been completed for each checkpoint (negative determination), the process returns to step S101 and is repeated.

[0042] In step S105, the CPU of the terminal device 20 determines whether or not the next checklist exists. If it determines that the next checklist exists (positive determination), the process returns to step S101 and is repeated for the next checklist. If it determines that there is no next checklist (negative determination), the series of processes by this application program 20A is terminated.

[0043] Figure 8 is a flowchart showing an example of the processing flow by the work content determination program 15A according to this embodiment. When the execution of processing by the work content determination program 15A is instructed in cooperation with the application program 20A, the CPU 11 of the information processing device 10 executes the work content determination program 15A, which is stored in the ROM 12 or storage unit 15, by writing it to the RAM 13.

[0044] In step S111 of Figure 8, the CPU 11 acquires work data from the field worker's terminal device 20. As mentioned above, the work data is data that associates a checklist, which has predetermined confirmation points for each step of the field work, with images taken of the work content for each confirmation point on the checklist. The images taken include, for example, paper documents such as receipts, setting printouts, and tickets, screens of terminals and control panels, switch status (on / off), and wiring status.

[0045] In step S112, the CPU 11 reads work content information from the captured image of the work data. As mentioned above, the work content information is information used to determine the work content, and includes, for example, the setting value of the target device, the setting time, the IP address, the switch status (on / off), etc.

[0046] In step S113, the CPU 11, for example, uses the generating AI 150 to determine whether the on-site work has been completed for each checkpoint on the checklist based on the work content information. If it is determined that the on-site work has been completed (positive determination), the process proceeds to step S114; if it is determined that the on-site work has not been completed (negative determination), the process proceeds to step S115.

[0047] In step S114, the CPU 11 registers in the work record DB 158 that the field work has been completed, and then proceeds to step S116.

[0048] Meanwhile, in step S115, the CPU 11 registers in the work record DB 158 that the field work is incomplete, and proceeds to step S116.

[0049] In step S116, the CPU 11 returns the result of the field work determination to the terminal device 20. Specifically, if the CPU 11 determines that the field work is complete, it returns "OK" to indicate that the work is complete; if it determines that the field work is not complete, it returns "NG" to indicate that the work is incomplete. When returning "NG," it is desirable to also return the reason for the "NG" determination.

[0050] In step S117, the CPU 11 determines whether or not it is the final checkpoint. If it determines that it is not the final checkpoint (negative result), the process returns to step S111 and is repeated. If it determines that it is the final checkpoint (positive result), the series of processes by the work content determination program 15A is terminated.

[0051] Thus, according to this embodiment, the completion of on-site work can be confirmed using a pre-trained model without human intervention. Therefore, the completion confirmation of on-site work can be made more efficient, and the occurrence of confirmation errors can be suppressed.

[0052] Furthermore, the information processing that the CPU reads and executes in each of the above embodiments may be executed by various processors other than the CPU. Examples of such processors include PLDs (Programmable Logic Devices) such as FPGAs (Field-Programmable Gate Arrays) whose circuit configuration can be changed after manufacturing, and dedicated electrical circuits that are processors with circuit configurations specifically designed to execute specific processing, such as ASICs (Application Specific Integrated Circuits). In addition, the information processing may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (for example, multiple FPGAs, and a combination of a CPU and an FPGA). More specifically, the hardware structure of these various processors is an electrical circuit that combines circuit elements such as semiconductor elements.

[0053] Furthermore, while the above embodiments describe a configuration in which the information processing program is pre-stored (installed) in ROM or storage, the invention is not limited to this. The program may be provided in a form recorded on a non-transitory recording medium such as a CD-ROM (Compact Disk Read Only Memory), DVD-ROM (Digital Versatile Disk Read Only Memory), or USB (Universal Serial Bus) memory. Alternatively, the program may be provided in a form that is downloaded from an external device via a network.

[0054] The following are additional notes regarding this disclosure.

[0055] (Additional note 1) An acquisition unit acquires work data from a terminal device used by a field worker, which associates a checklist with predetermined checkpoints for each step of the field work process, and images taken of the work content for each checkpoint on the checklist. A reading unit reads work content information representing the work content from the captured image of the work data, A determination unit that uses a pre-trained model to determine whether or not on-site work has been completed at each confirmation point based on the work content information, A reply unit that returns the result of the determination to the terminal device, Equipped with an information processing device. (Additional note 2) The trained model determines that the on-site work is completed when the work content identified by the work content information satisfies predetermined conditions. The information processing device described in Appendix 1. (Additional note 3) The trained model determines that the on-site work is not completed if the work content identified by the work content information does not meet the predetermined conditions. The information processing device described in Appendix 2. (Additional note 4) The system further includes a display control unit that controls the display of a checklist screen on the terminal device for associating the checklist with the captured image. An information processing device as described in any one of the following appendices 1 to 3. (Additional note 5) The checklist screen displays "Task Completed" at the corresponding checkpoint if the result of the judgment is "Task Completed," and displays the reason for the task being incomplete at the corresponding checkpoint if the result of the judgment is "Task Not Completed." The information processing device described in Appendix 4. (Additional note 6) The aforementioned trained model is a generative AI or a large-scale language model. An information processing device as described in any one of the following appendices 1 to 5. (Additional note 7) Work data is acquired from the terminal device of the on-site worker, which associates a checklist with predetermined checkpoints for each step of the on-site work process, and images taken of the work content for each checkpoint on the checklist. From the aforementioned captured image of the work data, read the work content information representing the work content. Using a pre-trained model that has been machine-learned, it is determined whether or not the on-site work has been completed at each checkpoint based on the work content information. The process of returning the result of the above determination to the terminal device is as follows: A method for determining the type of task a computer should perform. (Additional note 8) Work data is acquired from the terminal device of the on-site worker, which associates a checklist with predetermined checkpoints for each step of the on-site work process, and images taken of the work content for each checkpoint on the checklist. From the aforementioned captured image of the work data, read the work content information representing the work content. Using a pre-trained model that has been machine-learned, it is determined whether or not the on-site work has been completed at each checkpoint based on the work content information. The process of returning the result of the above determination to the terminal device is as follows: A program that determines what task to execute on a computer. [Explanation of Symbols]

[0056] 10 Information Processing Devices 11 CPU 11A Acquisition Department 11B Reading Unit 11C Judgment section 11D Reply Section 11E Display Control Unit 12 ROM 13 RAM 14 I / O 15 Storage section 15A Work Content Determination Program 16 Display section 17 Control section 18 Communications Department 20 Terminal devices 20A Application Program 21 Checklist screen 100 Work Content Determination System 150 Generation AI

Claims

1. An acquisition unit acquires work data from a terminal device used by a field worker, which associates a checklist with predetermined checkpoints for each step of the field work process, and images taken of the work content for each checkpoint on the checklist. A reading unit reads work content information representing the work content from the captured image of the work data, A determination unit that uses a pre-trained model to determine whether or not on-site work has been completed at each confirmation point based on the work content information, A reply unit that returns the result of the determination to the terminal device, Equipped with an information processing device.

2. The trained model determines that the on-site work is completed when the work content identified by the work content information satisfies predetermined conditions. The information processing apparatus according to claim 1.

3. The trained model determines that the on-site work is not completed if the work content identified by the work content information does not meet the predetermined conditions. The information processing apparatus according to claim 2.

4. The system further includes a display control unit that controls the display of a checklist screen on the terminal device for associating the checklist with the captured image. The information processing apparatus according to claim 1.

5. The checklist screen displays "Task Completed" at the corresponding checkpoint if the result of the judgment is "Task Completed," and displays the reason for the task being incomplete at the corresponding checkpoint if the result of the judgment is "Task Not Completed." The information processing apparatus according to claim 4.

6. The aforementioned trained model is a generative AI or a large-scale language model. The information processing apparatus according to claim 1.

7. Work data is acquired from the terminal device of the on-site worker, which associates a checklist with predetermined checkpoints for each step of the on-site work process, and images taken of the work content for each checkpoint on the checklist. From the aforementioned captured image of the work data, read the work content information representing the work content. Using a pre-trained model that has been machine-learned, it is determined whether or not the on-site work has been completed at each checkpoint based on the work content information. The process of returning the result of the above determination to the terminal device is as follows: A method for determining the type of task a computer should perform.

8. Work data is acquired from the terminal device of the on-site worker, which associates a checklist with predetermined checkpoints for each step of the on-site work process, and images taken of the work content for each checkpoint on the checklist. From the aforementioned captured image of the work data, read the work content information representing the work content. Using a pre-trained model that has been machine-learned, it is determined whether or not the on-site work has been completed at each checkpoint based on the work content information. The process of returning the result of the above determination to the terminal device is as follows: A program that determines what task to execute on a computer.

Images