Information processing method, information processing system, terminal device, and computer program

Abstract

Provided are an information processing method, an information processing system, a terminal device, and a computer program.　The present invention causes a terminal device of a worker to execute processing for: acquiring a plurality of work data items related to work performed by the worker; and controlling the timing of transmitting the work data items to a server device in accordance with the priority of the acquired work data items.

Description

Information Processing Method, Information Processing System, Terminal Device, and Computer Program

[0001] The present invention relates to an information processing method, an information processing system, a terminal device, and a computer program.

[0002] Patent Document 1 discloses an inspection work guidance system including a voice information output unit that outputs inspection items in an inspection work as voice information, and a voice information input unit that inputs inspection results from an inspector as voice information, which converts the voice information input through the voice information input unit into character information and transmits it to a server device.

[0003] Japanese Patent Application Laid-Open No. 2023-92960

[0004] In Patent Document 1, although it is possible to transmit work results to a server device as character information, it is not possible to control the transmission timing thereof.

[0005] An object of the present disclosure is to provide an information processing method, an information processing system, a terminal device, and a computer program capable of controlling the timing of transmission to a server device according to the priority of work data.

[0006] The information processing method according to the first aspect of the present disclosure acquires a plurality of work data related to work performed by a worker, and controls the timing of transmitting the work data to a server device according to the priority of the acquired work data, which is executed by the worker's terminal device.

[0007] The information processing method according to the second aspect of the present disclosure is the information processing method according to the first aspect, wherein the terminal device immediately transmits work data with relatively high priority and transmits work data with relatively low priority afterwards.

[0008] The information processing method according to the third aspect of the present disclosure is the information processing method according to the first or second aspect, wherein the work data includes data of work results input by the worker, imaging data obtained by imaging the work of the worker, acoustic data obtained by recording sounds at the work site, or equipment data obtained for the work target equipment.

[0009] The information processing method relating to the fourth aspect of this disclosure is configured such that, in the information processing method relating to the third aspect, the priority of the work result data is set higher than the priority of the imaging data, the acoustic data, and the equipment data.

[0010] The information processing method relating to the fifth aspect of this disclosure, in which work recognition is performed immediately based on the imaging data in the information processing method relating to the third aspect, sets the priority of the imaging data higher than the priority of the work result data, the acoustic data, and the equipment data.

[0011] The information processing method relating to the sixth aspect of this disclosure, in which, in the information processing method relating to the third aspect, performs immediate acoustic recognition based on the acoustic data, sets the priority of the acoustic data higher than the priority of the work result data, the imaging data, and the equipment data.

[0012] The information processing method relating to the seventh aspect of this disclosure is an information processing method relating to any one of the second to sixth aspects, wherein the terminal device acquires work data for each of the multiple work procedures and sets the priority of the work data according to the work procedure.

[0013] The information processing method relating to the eighth aspect of this disclosure is an information processing method relating to any one of the third to seventh aspects, wherein the terminal device guides the worker through the work content for each work procedure, and triggers the guidance of the work content to start at least one of imaging the work at each work procedure and recording sounds at the work site.

[0014] The information processing method relating to the ninth aspect of this disclosure is an information processing method relating to any one of the third to eighth aspects, wherein the terminal device recognizes a work scene based on the image data and decides whether or not to transmit the image data according to the recognized work scene.

[0015] The information processing method relating to the tenth aspect of this disclosure is an information processing method relating to any one of the second to ninth aspects, wherein the terminal device detects the communication status and, according to the result of detecting the communication status, decides whether or not to transmit the work data with a relatively low priority.

[0016] The information processing method relating to the eleventh aspect of this disclosure is an information processing method relating to any one of the first to tenth aspects, wherein the terminal device obtains information relating to the transmission status of the work data from the server device and outputs the transmission status of the work data based on the obtained information.

[0017] The information processing system relating to the twelfth aspect of this disclosure comprises a worker's terminal device and a server device that is communicably connected to the terminal device, wherein the terminal device acquires a plurality of work data related to work performed by the worker and controls the timing of transmitting the work data to the server device according to the priority of the acquired work data.

[0018] The terminal device relating to the 13th aspect of this disclosure comprises at least one control unit, the control unit acquires a plurality of work data related to work performed by an operator, and controls the timing of transmitting the acquired work data to a server device according to the priority of the acquired work data.

[0019] The computer program relating to the fourteenth aspect of this disclosure is a computer program that causes a computer to execute a process that acquires multiple work data related to work performed by an operator and controls the timing of transmitting the acquired work data to a server device according to the priority of the acquired work data.

[0020] According to this disclosure, the timing of sending work data to the server device can be controlled according to the priority of the work data.

[0021] This is an explanatory diagram illustrating the outline of the processing performed by the information processing system according to Embodiment 1. This is a block diagram showing the internal configuration of a wearable device. This is a block diagram showing the internal configuration of a server device. This is a conceptual diagram showing an example of a work procedure database. This is an explanatory diagram illustrating an example of voice guidance by a wearable device. This is a flowchart showing the processing procedure of voice guidance performed by the wearable device according to Embodiment 1. This is a flowchart showing the processing procedure of periodic uploads performed by the wearable device according to Embodiment 1. This is a conceptual diagram showing an example of a work history database. This is a schematic diagram showing an upload status confirmation screen. This is an explanatory diagram illustrating an example of setting priorities in Embodiment 2. This is a flowchart showing the processing procedure of voice guidance performed by the wearable device according to Embodiment 2. This is a flowchart showing the processing procedure of periodic uploads performed by the wearable device according to Embodiment 2. This is a flowchart showing the processing procedure performed by the wearable device according to Embodiment 3.

[0022] The information processing system according to the embodiment will be described in detail below with reference to the drawings. (Embodiment 1) Figure 1 is an explanatory diagram illustrating the outline of the processing performed by the information processing system 1 according to Embodiment 1. The information processing system 1 according to the embodiment comprises a wearable device 10 worn by a worker and a server device 20 that manages the work process. The wearable device 10 and the server device 20 are connected to each other so as to be able to communicate with each other via a communication network NW such as the Internet.

[0023] In this embodiment, the worker is, for example, a worker who performs installation, repair, and inspection work related to refrigeration and air conditioning equipment. Refrigeration and air conditioning equipment refers to equipment connected to refrigerant piping and constituting a refrigeration cycle. Refrigeration and air conditioning equipment includes equipment such as outdoor units, indoor units, ventilation systems, heat source equipment, and central control devices that constitute an air conditioning system, and equipment such as refrigerators, heat exchangers, and control devices that constitute a refrigeration system.

[0024] The wearable device 10 is a device having imaging, recording, audio output, and communication functions. The wearable device 10 is worn, for example, around the neck of a worker. Alternatively, the wearable device 10 may be worn on the worker's head, or on another body part such as the shoulder or arm. Furthermore, the wearable device 10 may be a goggle-type camera device, or any device with the above functions, such as a smartphone or action camera, may be used instead of the wearable device 10.

[0025] The wearable device 10 retrieves work procedure manuals from the server device 20 regarding the work to be performed by the worker. The work procedure manual is data that defines the work procedures for tasks such as installation, repair, and inspection, and consists of multiple work procedures. The work procedure manuals are assumed to be created in advance by the work manager or the like for each type of work and each type of equipment, and are registered in the work procedure manual database DB1 (see Figure 4). The wearable device 10 retrieves the necessary work procedure manuals from the server device 20 according to the type of work and the equipment being worked on for the work the worker intends to perform.

[0026] The wearable device 10 provides voice guidance according to the work procedure manual obtained from the server device 20. For example, the wearable device 10 can refer to the work procedure manual and provide voice instructions on the tasks that the worker should perform. The wearable device 10 may also refer to the work procedure manual and provide voice inquiries about the tasks the worker has performed. The worker performs tasks according to the voice guidance output from the wearable device 10, confirms the tasks performed, and responds to inquiries from the wearable device 10 by voice.

[0027] The wearable device 10 starts capturing images and recording audio in response to the output of voice guidance corresponding to each work procedure. Since the wearable device 10 is worn by the worker, the images obtained by the wearable device 10 are from the worker's perspective (i.e., first-person perspective). In this embodiment, the images obtained by the wearable device 10 are videos captured at an appropriate frame rate, but they may also be still images captured at appropriate time intervals. The images captured by the wearable device 10 include the work scene, the equipment being worked on, and various measuring instruments. The sounds recorded by the wearable device 10 include the worker's voice, sounds emitted from the equipment being worked on, ambient sounds, and any other sounds generated at the work site.

[0028] The wearable device 10 terminates imaging and recording when a work procedure is completed. In this embodiment, imaging data and audio data are obtained for each work procedure. The imaging data and audio data for each work procedure are examples of work data in this disclosure.

[0029] The work data may include imaging data and acoustic data for each work procedure, as well as work result data entered by the worker. The work results are entered into the wearable device 10 as voice responses to inquiries from the wearable device 10. The work result data may be voice data or text data converted by speech recognition.

[0030] The work data may include imaging data and acoustic data for each work procedure, as well as equipment data obtained for the equipment being worked on. The equipment data includes measured values ​​such as current, voltage, electrical resistance, power, temperature, airflow, and wind speed measured for the equipment being worked on. The measured values ​​may be values ​​measured by measuring instruments or sensors installed in the equipment being worked on, or values ​​measured by an operator using measuring instruments or sensors. The equipment data may also include status data and error codes output from the equipment being worked on.

[0031] The wearable device 10 uploads the acquired work data to the server device 20 via the communication network NW. At this time, the wearable device 10 controls the timing of uploading the work data to the server device 20 according to the priority of the work data.

[0032] In Embodiment 1, the priority of work result data is set higher than the priority of imaging data, acoustic data, and equipment data. When work result data is input to the wearable device 10, it immediately uploads it to the server device 20, and when imaging data, acoustic data, and equipment data are acquired, it uploads them to the server device 20 retrospectively. For imaging data, acoustic data, and equipment data to be uploaded retrospectively, they should be uploaded to the server device 20 periodically when the radio wave intensity is above a threshold.

[0033] The server device 20 may determine whether the work performed by the worker is appropriate based on the work result data uploaded from the wearable device 10. If the server device 20 determines that the work performed by the worker is appropriate, it can give instructions to the wearable device 10 to prompt the worker to move on to the next work procedure. On the other hand, if the server device 20 determines that the work is not appropriate, it can give instructions to the wearable device 10 and have it output voice guidance prompting the worker to correct the work, thereby guiding the worker to perform the work appropriately.

[0034] In addition to the wearable device 10, the worker may also carry a mobile terminal 30 such as a smartphone. The mobile terminal 30 has an input unit such as a touch panel and a display unit such as an LCD display, and complements some of the functions that the wearable device 10 does not have. The mobile terminal 30 is configured to communicate with the wearable device 10 using short-range wireless communication such as Bluetooth® or ZigBee®. Furthermore, the mobile terminal 30 is configured to communicate with the server device 20 via a communication network NW.

[0035] Figure 2 is a block diagram showing the internal configuration of the wearable device 10. The wearable device 10 includes a control unit 11, a storage unit 12, a communication unit 13, an imaging unit 14, an audio input unit 15, an audio output unit 16, a sensor unit 17, and the like.

[0036] The control unit 11 includes a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), and the like. The ROM in the control unit 11 stores control programs and the like that control the operation of each hardware component of the wearable device 10. The CPU in the control unit 11 reads and executes the control programs and the like stored in the ROM, and controls the operation of each hardware component, thereby making the entire device function as the terminal device of this disclosure. The RAM in the control unit 11 temporarily stores data used during the execution of various processes.

[0037] The control unit 11 may also include functions such as a clock for outputting date and time information, a timer for measuring the elapsed time from the time a measurement start instruction is given until a measurement end instruction is given, and a counter for counting numbers.

[0038] The storage unit 12 is equipped with an auxiliary storage device and stores various types of data, as well as computer programs executed by the control unit 11. The data stored in the storage unit 12 includes the above-mentioned imaging data, acoustic data, processing result data, and equipment data. The computer program (program product) stored in the storage unit 12 includes a transmission control program PG that causes the computer to execute a process to acquire multiple work data and control the timing of transmitting the acquired work data to the server device 20 according to the priority of the work data.

[0039] A computer program including a transmission control program PG may be provided on a non-temporary recording medium RM on which the computer program is recorded in a readable format. The recording medium RM is a portable memory such as a CD-ROM, USB memory, SD card, microSD card, or CompactFlash®. The control unit 11 reads various computer programs from the recording medium RM using a reading device (not shown in the figure) and stores the read computer programs in the storage unit 12. The computer programs stored in the storage unit 12 may also be provided by communication. In this case, the control unit 11 acquires the computer programs by communication via the communication unit 13 and stores the acquired computer programs in the storage unit 12.

[0040] The transmission control program PG may be a single computer program or may consist of multiple computer programs. The transmission control program PG may be executed on a single computer or may be executed collaboratively by multiple computers (e.g., a wearable device 10 and a server device 20). Furthermore, the transmission control program PG may partially utilize existing libraries.

[0041] The communication unit 13 is equipped with a communication module for wireless communication with external devices such as the server device 20. The communication module utilizes a communication module for wireless communication using known mobile communication standards such as 3G, 4G, and 5G, or wireless LAN methods such as Wi-Fi (registered trademark). The communication unit 13 communicates with external devices such as the server device 20 via the communication network NW, transmitting necessary data such as work data, and receiving appropriate data transmitted from the external devices.

[0042] The communication unit 13 may be equipped with a communication module for short-range wireless communication, such as Bluetooth® or ZigBee®, in order to communicate with a mobile terminal 30 such as a smartphone carried by the worker.

[0043] The imaging unit 14 includes an optical lens, an imaging device, a driver circuit, etc. A wide-angle lens is preferably used as the optical lens. The imaging device is a CMOS (Complementary Metal Oxide Semiconductor), a CCD (Charge-Coupled Device), etc., and generates an electrical signal according to the intensity of light imaged through the optical lens. The driver circuit includes a timing generator (TG), etc., sequentially reads out the electrical signal from the imaging device in synchronization with the clock signal output from the TG, and generates moving image data. The moving image data generated by the imaging unit 14 is sent to the control unit 11 and stored in the storage unit 12. Also, the moving image data generated by the imaging unit 14 is transmitted to the server device 20 through the communication unit 13.

[0044] The sound input unit 15 includes a microphone that collects sound, a processing circuit that converts the collected sound into a signal in digital format (acoustic data), etc. The acoustic data generated by the sound input unit 15 is sent to the control unit 11, and after appropriate processing such as noise removal is performed in the control unit 11, it is stored in the storage unit 12. Also, the acoustic data subjected to appropriate processing in the control unit 11 is transmitted to the server device 20 through the communication unit 13.

[0045] The sound output unit 16 includes a speaker that outputs sound. The sound output unit 16 outputs sound based on the acoustic data provided by the control unit 11.

[0046] The sensor unit 17 includes a non-contact sensor for detecting the fingers of an operator, etc. The sensors included in the sensor unit 17 are proximity sensors, gesture sensors, etc. The proximity sensor detects, for example, that the finger of an operator has approached within a predetermined range. The gesture sensor detects, for example, the movement of the finger of an operator. The detection result by the sensor unit 17 is notified to the control unit 11. The control unit 11 may control the operation of the wearable device 10 based on the detection result of the sensor unit 17.

[0047] Figure 3 is a block diagram showing the internal configuration of the server device 20. The server device 20 is a dedicated or general-purpose server device, and includes, for example, a control unit 21, a storage unit 22, a communication unit 23, an operation unit 24, and a display unit 25.

[0048] The control unit 21 includes a CPU, a ROM, a RAM, etc. The ROM included in the control unit 21 stores a control program and the like for controlling the operations of the respective hardware components of the server device 20. The CPU in the control unit 21 reads and executes the control program stored in the ROM and a computer program described later stored in the storage unit 22, and executes a process for controlling the operations of the respective hardware components, thereby causing the entire device to function as the server device 20 of the present disclosure. In the RAM included in the control unit 21, data used during the execution of various processes is temporarily stored.

[0049] In the embodiment, the control unit 21 is configured to include a CPU, a ROM, and a RAM. However, the configuration of the control unit 21 is not limited to the above. The control unit 21 may be, for example, one or a plurality of processing circuits including a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), a DSP (Digital Signal Processor), a quantum processor, a volatile or non-volatile memory, etc. Further, the control unit 21 may include functions such as a clock that outputs date and time information, a timer that measures the elapsed time from when a measurement start instruction is given until a measurement end instruction is given, and a counter that counts numbers.

[0050] The storage unit 22 includes a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive). The storage unit 22 stores various data and various computer programs executed by the control unit 21. Further, the storage unit 22 includes a work procedure manual database DB1 that stores work procedure manuals created for each work type and each device type, and a work history database DB2 that manages the history of work performed by workers. The outline of the work procedure manual database DB1 will be described using FIG. 4. The outline of the work history database DB2 will be described using FIG. 7.

[0051] The communication unit 23 includes a communication module for wireless communication with an external device such as a wearable device 10. The communication module may utilize known mobile communication standards such as 3G, 4G, and 5G, or wireless LAN methods such as Wi-Fi®. The communication unit 23 communicates with the external device, such as the wearable device 10, via a communication network NW, receiving data such as field-of-view images (video data) and transmitting appropriate data to the external device. The communication unit 23 may further include a communication module for short-range wireless communication such as Bluetooth® or ZigBee®.

[0052] The operation unit 24 is equipped with operating devices such as a touch panel, keyboard, and switches, and accepts various inputs and operations from work managers and others. The control unit 21 acquires information input through the operation unit 24 and performs appropriate control based on the various operation information provided by the operation unit 24.

[0053] The display unit 25 is equipped with a display device such as a liquid crystal monitor or an organic EL (Electro-Luminescence) monitor, and displays information that should be notified to the work manager, etc., in response to instructions from the control unit 21.

[0054] The server device 20 may be a single computer, or it may be a computer system composed of multiple computers and peripheral devices. Furthermore, the server device 20 may be a virtualized virtual machine, or it may be a cloud service. Additionally, some of the functions of the server device 20 may be implemented using a combination of SaaS (Software as a Service).

[0055] Figure 4 is a conceptual diagram showing an example of the work procedure database DB1. The work procedure database DB1 is a database that stores work procedure data for each type of work and equipment. In the work procedure database, for example, the procedure number, type, category, and body text are stored in association. The procedure number represents the order of the procedure. The type represents the distinction between guidance and questions. If the type is a question, it indicates a procedure that requires a response from the worker. In Figure 4, the questions exemplified by procedure numbers "02" to "06" all indicate that they can be answered with YES or NO. Questions registered in the work procedure are not limited to those that can be answered with YES or NO; they may also include questions that expect answers through numerical input such as temperature or pressure, or questions that expect answers through text input such as error codes or free-form text. The category represents the work target, such as outdoor unit, indoor unit, controller, or tank. This text is a transcription of the voice guidance output from the wearable device 10, and describes the tasks that the worker should perform in that procedure.

[0056] Figure 4 shows a work procedure manual for a "maintenance inspection" type of work and a "room air conditioner AAA" model. The work procedure manual database DB1 stores such work procedures for each type of work and each model.

[0057] Figure 5 is an explanatory diagram illustrating an example of voice guidance provided by a wearable device 10. Prior to starting work, the worker downloads the necessary work procedure manual from the server device 20 and stores it in the storage unit 12 of the wearable device 10. Specifically, the worker accesses the work procedure manual database DB1 using a mobile terminal 30 and searches for the necessary work procedure manual from the work procedure manual database DB1 according to the type of work and the equipment being worked on. To download the retrieved work procedure manual, the worker sends a download request from the mobile terminal 30 to the server device 20. The wearable device 10 is specified as the download destination. In response to the download request from the mobile terminal 30, the server device 20 reads the corresponding work procedure manual from the work procedure manual database DB1 and sends the read work procedure to the wearable device 10. The wearable device 10 receives the work procedure manual sent from the server device 20 and stores the received work procedure manual in the storage unit 12.

[0058] Alternatively, the work supervisor may give instructions to the server device 20, and the server device 20 may send an appropriate work procedure manual to the wearable device 10 according to the type of work and the equipment being worked on.

[0059] The wearable device 10 provides voice guidance by referring to the work procedure manual downloaded from the server device 20. For example, when providing voice guidance according to the work procedure manual illustrated in Figure 4, the wearable device 10 first reads the text associated with procedure number "01" and outputs the read text as voice guidance from the sound output unit 16. Existing technology is used for the text-to-speech conversion. Since the procedure for procedure number "01" is of the guidance type, a response from the worker is not required, but in the example in Figure 5, the worker is shown to have responded to the voice guidance from the wearable device 10 by saying, "I pressed it."

[0060] The wearable device 10 determines whether the work procedure guided by the voice guidance for procedure number "01" has been completed. For example, the wearable device 10 determines that the work is complete when it receives device data indicating the completion of the work from the target device (in the case of procedure number "01", device data indicating that the start button has been pressed).

[0061] If the wearable device 10 determines that the procedure for procedure number "01" is complete, it moves on to the next procedure, procedure number "02". The wearable device 10 reads the text associated with procedure number "02" as before and outputs the read text as voice guidance from the sound output unit 16. Since the text for procedure number "02" is a question that can be answered with YES or NO, voice guidance such as "Please answer with YES or NO" may be added.

[0062] The wearable device 10 can determine whether or not the work procedure for procedure number "02" has been completed based on the worker's response (voice). If it determines that the work procedure has not been completed, the wearable device 10 may provide additional voice guidance. Figure 5 shows an example where, in response to the voice guidance "Have you cleaned the heat exchanger?", the worker answered "No," and the wearable device 10 then provided additional voice guidance, "Please clean the heat exchanger."

[0063] The wearable device 10 will sequentially provide voice guidance in accordance with the work procedures defined in the work procedure manual, in the same manner as described below.

[0064] In the example shown in Figure 5, the wearable device 10 is configured to determine whether each work procedure has been completed. However, the device data obtained from the work target equipment and the worker's responses may be transmitted to the server device 20, and the server device 20 may determine whether each work procedure has been completed. When the server device 20 determines that the Nth work procedure has been completed, it instructs the wearable device 10 to proceed to the N+1th work procedure. The wearable device 10 then follows the instructions from the server device 20 and provides voice guidance for the next work procedure according to the work procedure manual.

[0065] The following describes the processes performed by the wearable device 10. Figure 6 is a flowchart showing the voice guidance processing procedure performed by the wearable device 10 according to Embodiment 1. The necessary work procedure manual is stored in the storage unit 12 of the wearable device 10. The control unit 11 reads the transmission control program PG from the storage unit 12 and executes it, thereby performing the following processes.

[0066] The control unit 11 first sets the counter value to i=1 (step S101). The control unit 11 refers to the work procedure manual stored in the memory unit 12 and, triggered by the voice guidance for the i-th work procedure, gives instructions to the imaging unit 14 and the sound input unit 15 to start imaging and recording (step S102). The control unit 11 acquires imaging data obtained by imaging the worker's work from the imaging unit 14 and acquires acoustic data obtained by recording sounds at the work site from the sound input unit 15.

[0067] After starting imaging and recording, the control unit 11 outputs voice guidance related to the i-th work procedure from the sound output unit 16 (step S103). The voice guidance includes instructions to inform the worker of the work to be performed, or questions to confirm the work performed by the worker. The control unit 11 outputs one or more voice guidances for each work procedure. The control unit 11 may output only instructions as voice guidance, or only questions as voice guidance. For example, after outputting voice guidance to announce the start of the i-th work procedure, the control unit 11 may output a question as voice guidance to confirm the work performed by the worker. Depending on the answer received from the worker, the control unit 11 may output additional voice guidance. In this embodiment, since the voice guidance is output in step S103 after starting imaging and recording in step S102, the speech from the voice guidance can be recorded as audio data.

[0068] In this embodiment, the control unit 11 is configured to acquire both imaging data and audio data, but it may also be configured to acquire only one of the two. In addition to imaging data and audio data, the control unit 11 may also acquire equipment data obtained about the equipment being worked on.

[0069] The control unit 11 determines whether or not the work result has been entered (step S104). The control unit 11 outputs a question as voice guidance to confirm the work performed by the worker, and if the worker's answer to the voice guidance is entered by voice through the sound input unit 15, the control unit 11 determines that the work result has been entered.

[0070] If the control unit 11 determines that the work result has been entered (S104: YES), it immediately uploads the work result data to the server device 20 (step S105). At this time, the control unit 11 only needs to convert the audio data corresponding to the worker's response into text data and upload the converted text data to the server device 20. Existing speech recognition technology is used for the conversion from audio data to text data. If the system is configured to accept only pre-registered words such as YES, NO, numbers, and error codes as the worker's response, the wearable device 10 can convert the audio data into text data with relatively high accuracy. If the control unit 11 determines that the work result has not been entered (S104: NO), it proceeds to the process in step S106.

[0071] The control unit 11 determines whether the i-th work procedure has been completed (step S106). For example, if the control unit 11 uploads the work result data to the server device 20 and then receives a response from the server device 20 indicating that the work result data has been received, it determines that the i-th work procedure has been completed. Alternatively, if the server device 20, which has received the work result data, instructs the control unit 11 to proceed to the next work procedure, the control unit 11 may determine that the i-th work procedure has been completed. If it determines that the i-th work procedure has not been completed (S106: NO), the control unit 11 returns the process to step S104.

[0072] If the control unit determines that the i-th work procedure has been completed (S106: YES), the control unit 11 terminates the imaging and recording that was started in step S103, and stores the imaging data and acoustic data obtained during that time in the storage unit 12 as the imaging data and acoustic data for the i-th work procedure (step S107).

[0073] Next, the control unit 11 refers to the work procedure manual and determines whether all work procedures have been completed (step S108). If there are any work procedures that have not been completed (S108: NO), the control unit 11 increments the value of the counter by 1 (step S109) and returns the process to step S102. If it is determined that all work procedures have been completed (S108: YES), the control unit 11 terminates the process according to this flowchart.

[0074] In this flowchart, the control unit 11 of the wearable device 10 determines the end of the work procedure in steps S106 and S108. However, the control unit 21 of the server device 20 may determine the end of the work procedure based on the work result data uploaded immediately from the wearable device 10. In this case, the server device 20 can instruct the wearable device 10 to proceed to the next work procedure. The wearable device 10 will follow the instructions from the server device 20 and provide voice guidance for the next work procedure according to the work procedure manual.

[0075] Figure 7 is a flowchart showing the periodic upload processing procedure performed by the wearable device 10 according to Embodiment 1. The control unit 11 performs the periodic upload processing of work data in parallel with the processing of the voice guidance described above.

[0076] The control unit 11 determines whether or not it is time for a periodic upload (step S121). The timing for a periodic upload is defined in advance, for example, by the transmission control program PG. The timing for a periodic upload can be any pre-set timing and does not need to be strictly periodic.

[0077] If the control unit 11 determines that it is not time for a scheduled upload (S121: NO), it waits until it is time for a scheduled upload. If the control unit 11 determines that it is time for a scheduled upload (S121: YES), it sequentially uploads the imaging data and acoustic data stored in the storage unit 12 to the server device 20 for each work procedure (step S122). At this time, the control unit 11 may detect the communication status by the communication unit 13 and transmit the imaging data and acoustic data to the server device 20 only if the communication status is good (for example, if the radio wave strength is above a threshold). If the control unit 11 has acquired equipment data in addition to the imaging data and acoustic data, it may also upload the equipment data to the server device 20.

[0078] The control unit 11 determines whether the upload of imaging data and acoustic data has been completed for all work procedures (step S123). If there is imaging data or acoustic data that has not been uploaded (S123: NO), the control unit 11 returns to step S121. If it is determined that the upload of imaging data and acoustic data has been completed for all work procedures (S123: YES), the control unit 11 terminates the process according to this flowchart.

[0079] The server device 20 registers the work data uploaded from the wearable device 10 into the work history database DB2. Figure 8 is a conceptual diagram showing an example of the work history database DB2. The work history database DB2 stores information about the work performed by the worker, such as associating it with item ID, item name, date and time, worker, work result, imaging data, and acoustic data. The item ID is an identifier used to identify each work procedure shown in the work procedure manual. The item name is the name of each work procedure shown in the work procedure manual. The date and time is the work date and time. Both the work start date and time and the work end date and time may be registered in the date and time. The worker represents the name of the worker. The work result represents the result of the worker's answer to a question output as voice guidance. YES or NO is registered in the work result. Numerical values, error codes, etc., answered by the worker may also be registered in the work result. The imaging data is imaging data for each work procedure uploaded from the wearable device 10. The audio data consists of audio data for each work procedure uploaded from the wearable device 10.

[0080] In the work history database DB2 shown in Figure 8, work data such as work result data, imaging data, and acoustic data are registered. In addition to these, equipment data obtained from the equipment being worked on may also be registered.

[0081] In the work history database DB2 shown in Figure 8, the file names of the imaging data and acoustic data are shown in a simplified manner. However, the file names may be set in a way that makes it clear which work procedure the data pertains to. For example, the file name may include the procedure identifier, procedure identifier, start date and time, end date and time, and the number of times the work has been performed. In this case, the file name can be treated as metadata.

[0082] The worker can access the server device 20 using a wearable device 10 or a mobile terminal 30 to check the status of the work data upload. Figure 9 is a schematic diagram showing the upload status confirmation screen. The worker can use the wearable device 10 to obtain information related to the work data upload status from the server device 20 and transfer the obtained information to the mobile terminal 30, thereby displaying a confirmation screen like the one shown in Figure 9 on the display unit of the mobile terminal 30. If the wearable device 10 has a display unit, the confirmation screen may be displayed on the display unit of the wearable device 10.

[0083] The confirmation screen in Figure 9 displays the item names of the work procedures, the start date and time of the work, and the work results, along with the upload status of the work data for the outdoor unit inspection work. The work results are displayed as text information. Here, "Actual: -" indicates that the upload of the work results for the corresponding work item has not yet been completed. The upload status of the image data and sound data is indicated by icons. In the example in Figure 9, items that have been uploaded are indicated by solid line icons, and items that have not been uploaded are indicated by dashed line icons.

[0084] The worker can check the work results and the upload status of the work data by checking the confirmation screen displayed on the mobile terminal 30.

[0085] In Figure 9, the configuration is shown to display the upload status of work data on the screen of the mobile terminal 30. However, the server device 20 may refer to the work history database DB2, create a work report summarizing the work details for each worker, and output it as an electronic or paper document.

[0086] As described above, in Embodiment 1, since voice guidance is provided for each work procedure using the wearable device 10, the worker can confirm the work procedure without interrupting their work and input the work results by voice. With this configuration, it is possible to improve work efficiency and reduce work errors.

[0087] In Embodiment 1, the timing of uploading work data to the server device 20 can be controlled according to the priority of the work data. In Embodiment 1, since the priority of work result data is set high, when work result data is input, it is uploaded to the server device 20 immediately. When other work data, including imaging data, acoustic data, and equipment data, is input, it is uploaded to the server device 20 afterward. The server device 20 checks the work results that are uploaded immediately, determines whether the work was performed properly, and if it determines that it was not performed properly, it can guide the worker to perform the work properly by outputting voice guidance from the wearable device 10.

[0088] The server device 20 stores the work results, imaging data, acoustic data, etc., associated with each work procedure, so that the imaging data, acoustic data, etc. can be used as a guarantee and evidence of the work results.

[0089] In Embodiment 1, first-person perspective video (image data) and annotations for that data (work content for each work procedure) are automatically collected in the work history database DB2. The data collected in the work history database DB2 can be easily repurposed as training data for video analysis.

[0090] In Embodiment 1, the priority of the work result data was set higher than the priority of the imaging data, acoustic data, and equipment data. Alternatively, the priority of the imaging data may be set higher than the priority of the work result data, acoustic data, and equipment data. In this case, the wearable device 10 may immediately upload the imaging data input through the imaging unit 14 to the server device 20, for example, in units of frames every few seconds. The server device 20 can analyze the imaging data uploaded immediately from the wearable device 10 and perform work recognition or scene recognition. Existing methods are used for analyzing the imaging data and performing work recognition or scene recognition. When work recognition or scene recognition is completed, the server device 20 may notify the wearable device 10 of this. The remaining frames after work recognition or screen recognition is completed may be transmitted retrospectively by periodic uploads.

[0091] Furthermore, the priority of acoustic data may be set higher than that of work result data, imaging data, and equipment data. For example, in work procedures where the surrounding environment is likely to change significantly (such as procedures immediately after moving to or from an outdoor or indoor environment), the priority of acoustic data may be set higher. In this case, the wearable device 10 may immediately upload the acoustic data input through the sound input unit 15 to the server device 20, for example, every few seconds. The server device 20 can analyze the work data immediately uploaded from the wearable device 10 and perform acoustic recognition. Acoustic recognition may be speech recognition that recognizes the worker's voice, or it may be abnormal sound detection to detect abnormal noises emitted from the work target equipment or work environment. Existing methods can be used for acoustic recognition such as speech recognition or abnormal sound detection. If the server device 20 detects an abnormal sound through acoustic recognition, it may instruct the wearable device 10 to output additional voice guidance. The server device 20 may also detect the type and level of noise through acoustic recognition and adjust the speech recognition model and preprocessing parameters.

[0092] (Embodiment 2) Embodiment 2 describes a configuration in which priority is set for each work procedure. The overall configuration of the information processing system 1, the internal configuration of the wearable device 10 and the server device 20 are the same as in Embodiment 1, so their description will be omitted.

[0093] Figure 10 is an explanatory diagram illustrating an example of setting priorities in Embodiment 2. In Embodiment 2, a priority is set for each work procedure. The priority for each work procedure is set, for example, by the work manager and registered in the work procedure database DB1. In the work procedure database DB1 shown in Figure 10, in addition to the procedure number, type, category, and body text mentioned above, a priority item is added. The priority represents the priority when uploading work data to the server device 20.

[0094] In the example shown in Figure 10, the work procedures numbered "01" to "03" are set to a low priority. In this case, the work data obtained from work procedures numbered "01" to "03" is uploaded to the server device 20 retrospectively. That is, this work data is uploaded to the server device 20 at regular intervals, not immediately.

[0095] On the other hand, the work procedures numbered "04" to "05" are set to "high" priority. In this case, the work data obtained from the work procedures numbered "04" to "05" is immediately uploaded to the server device 20. The imaging data and acoustic data should be uploaded sequentially every few seconds.

[0096] For procedure number "06," the priority is set to "high," and the condition for its application is specified as "Model AAA." In this case, the work data obtained by procedure number "06" is uploaded immediately to the server device 20 only if the model of the target equipment is "AAA." If the model of the target equipment is anything other than "AAA," the work data obtained by procedure number "06" is uploaded to the server device 20 retrospectively.

[0097] In the procedure for step number "06," the model of the equipment being worked on is specified as a condition for applying a "high" priority. However, instead of specifying the model of the equipment being worked on, a configuration specifying a measuring instrument may also be used. For example, by specifying an ammeter, voltmeter, electrical resistance meter, power meter, thermometer, anemometer, wind speed meter, etc., the priority of procedures for measuring specific physical quantities can be set relatively higher.

[0098] The wearable device 10 can refer to the priority of each work procedure defined in the work procedure manual obtained from the server device 20 and decide whether to upload the data obtained in each work procedure immediately or retrospectively.

[0099] Figure 11 is a flowchart showing the voice guidance processing procedure performed by the wearable device 10 according to Embodiment 2. The necessary work procedure manual is stored in the storage unit 12 of the wearable device 10. The control unit 11 reads the transmission control program PG from the storage unit 12 and executes it, thereby performing the following processing.

[0100] The control unit 11 first sets the counter value to i=1 (step S201). The control unit 11 refers to the work procedure manual stored in the memory unit 12 and, triggered by the voice guidance for the i-th work procedure, gives instructions to the imaging unit 14 and the sound input unit 15 to start imaging and recording (step S202). The control unit 11 acquires imaging data obtained by imaging the worker's work from the imaging unit 14 and acquires acoustic data obtained by recording sounds at the work site from the sound input unit 15.

[0101] After the control unit 11 starts imaging and recording, it outputs voice guidance related to the i-th work procedure from the sound output unit 16 (step S203). In this embodiment, since the voice guidance is output in step S203 after the imaging and recording are started in step S202, the speech from the voice guidance can be recorded as acoustic data.

[0102] The control unit 11 refers to the work procedure manual and determines whether the priority set for the i-th work procedure is "high" (step S204). If the control unit 11 determines that the priority set for the i-th work procedure is "high" (S204: YES), it immediately uploads the work data obtained from that work procedure (step S205). In other words, when work result data is input, the control unit 11 should immediately upload the work result data to the server device 20, and when imaging data, audio data, and equipment data are input, they should be immediately uploaded to the server device 20 within a few seconds.

[0103] The server device 20 may determine whether the work procedure was performed correctly based on the work data uploaded immediately, and may notify the wearable device 10 of the determination result. If the wearable device 10 is notified by the server device 20 that the work procedure was performed correctly, it may retrospectively upload subsequent work data.

[0104] If the control unit 11 determines that the priority set for the i-th work procedure is not "high" (S204: NO), it will not perform the processing in step S205 and will instead execute the processing from step S206 onwards.

[0105] The control unit 11 determines whether the i-th work procedure has been completed (step S206). For example, if the control unit 11 uploads the work result data to the server device 20 and then receives a response from the server device 20 indicating that the work result data has been received, it determines that the i-th work procedure has been completed. Alternatively, if the server device 20, which has received the work result data, instructs the control unit 11 to proceed to the next work procedure, the control unit 11 may determine that the i-th work procedure has been completed. If it determines that the i-th work procedure has not been completed (S206: NO), the control unit 11 returns the process to step S204.

[0106] If the control unit determines that the i-th work procedure has been completed (S206: YES), the control unit 11 terminates the imaging and recording that was started in step S203, and stores the imaging data and acoustic data obtained during that time in the storage unit 12 as the imaging data and acoustic data for the i-th work procedure (step S207).

[0107] Next, the control unit 11 refers to the work procedure manual and determines whether all work procedures have been completed (step S208). If there are any work procedures that have not been completed (S208: NO), the control unit 11 increments the value of the counter by 1 (step S209) and returns the process to step S202. If it is determined that all work procedures have been completed (S208: YES), the control unit 11 terminates the process according to this flowchart.

[0108] Figure 12 is a flowchart showing the periodic upload processing procedure performed by the wearable device 10 according to Embodiment 2. The control unit 11 performs the periodic upload processing of work data in parallel with the processing of the voice guidance described above.

[0109] The control unit 11 determines whether or not it is time for a periodic upload (step S221). The timing for a periodic upload is defined in advance, for example, by the transmission control program PG. The timing for a periodic upload can be any pre-set timing and does not need to be strictly periodic.

[0110] If the control unit 11 determines that it is not time for a scheduled upload (S221: NO), it waits until it is time for a scheduled upload. If the control unit 11 determines that it is time for a scheduled upload (S221: YES), it sequentially uploads the work data obtained by the work procedure with low priority to the server device 20 (step S222). At this time, the control unit 11 may detect the communication status by the communication unit 13 and send the work data to the server device 20 only if the communication status is good (for example, if the radio wave strength is above a threshold).

[0111] The control unit 11 determines whether the upload of work data has been completed for all work procedures (step S223). If there is any imaging data or acoustic data that has not been uploaded (S223: NO), the control unit 11 returns to step S221. If it is determined that the upload of work data has been completed for all work procedures (S223: YES), the control unit 11 terminates the process according to this flowchart.

[0112] As described above, in Embodiment 2, the priority level for each work procedure is set in advance in the work procedure manual. Therefore, the wearable device 10 can refer to the work procedure manual and immediately upload work data with a relatively high priority to the server device 20, and upload work data with a relatively low priority to the server device 20 retrospectively.

[0113] (Embodiment 3) Embodiment 3 describes a configuration in which a work scene is recognized and whether or not to transmit imaging data according to the recognized work scene. The overall configuration of the information processing system 1 and the internal configuration of the wearable device 10 and server device 20 are the same as in Embodiment 1, so their description will be omitted.

[0114] Figure 13 is a flowchart showing the processing steps performed by the wearable device 10 according to Embodiment 3. The control unit 11, following the same procedure as in Embodiments 1 and 2, triggers voice guidance for each work procedure, gives instructions to the imaging unit 14 and the sound input unit 15, starts imaging and recording, and outputs the voice guidance from the sound output unit 16. The control unit 11 acquires imaging data obtained by imaging the worker's work from the imaging unit 14, and acquires acoustic data obtained by recording sounds at the work site from the sound input unit 15.

[0115] The control unit 11 performs the following processing each time it acquires imaging data from the imaging unit 14. Based on the acquired imaging data, the control unit 11 recognizes the work scene (step S301). Existing methods are used for scene recognition. For example, EgoVLPv2 based on Transformer, or a scene recognition model tuned from EgoVLPv2, can be used. The scene recognition model is configured, for example, to output a scene recognition result that includes either "work" or "non-work" in response to input of imaging data in predetermined units (for example, imaging data in 6-second units).

[0116] The control unit 11 determines whether or not to upload the image data based on the recognition result of the work scene (step S302). The control unit 11 decides to upload the image data if the recognition result of the work scene is "work," such as installation work or inspection work of the equipment to be worked on. The control unit 11 also decides not to upload the image data if the recognition result of the work scene is "non-work," such as taking a break or moving around.

[0117] If it is determined that the image data should be uploaded (S302: YES), the control unit 11 uploads the image data (step S303). The timing of uploading the image data follows the timing described in Embodiment 1 or 2. That is, as described in Embodiment 1, the control unit 11 may upload the image data periodically at a subsequent timing. Alternatively, as described in Embodiment 2, the control unit 11 may upload the image data immediately or periodically according to the priority level set for each work procedure.

[0118] If it is determined in step S302 not to upload the imaging data (S302: NO), the control unit 11 terminates the processing according to this flowchart.

[0119] As described above, in Embodiment 3, since image data from non-working situations such as during breaks or while moving is not uploaded to the server device 20, the load on the wearable device 10 and the server device 20 can be reduced.

[0120] Furthermore, while the application example of the information processing system according to this embodiment is assumed to be a work scenario involving air conditioning equipment, it is of course applicable to a variety of work scenarios, not limited to air conditioning equipment, such as work scenarios involving elevators, or maintenance and inspection work scenarios in chemical plants and power supply facilities.

[0121] The embodiments disclosed herein should be considered in all respects to be illustrative and not restrictive. The scope of the invention is indicated by the claims, not in the sense described above, and all modifications within the sense and scope equivalent to the claims are intended.

[0122] 10 Wearable device 11 Control unit 12 Memory unit 13 Communication unit 14 Imaging unit 15 Sound input unit 16 Sound output unit 17 Sensor unit 20 Server device PG Transmission control program DB1 Work procedure database DB2 Work history database

Claims

1. An information processing method in which the worker's terminal device performs a process to control the timing of transmitting multiple work data related to work performed by the worker, according to the priority of the acquired work data.

2. The information processing method according to claim 1, wherein the terminal device transmits work data of relatively high priority immediately and transmits work data of relatively low priority afterward.

3. The information processing method according to claim 2, wherein the work data includes work result data entered by the worker, imaging data obtained by imaging the work performed by the worker, acoustic data obtained by recording sounds at the work site, or equipment data obtained with respect to the equipment being worked on.

4. The information processing method according to claim 3, wherein the priority of the data of the work results is set higher than the priority of the imaging data, the acoustic data, and the equipment data.

5. The information processing method according to claim 3, wherein, when performing immediate work recognition based on the imaging data, the priority of the imaging data is set higher than the priority of the work result data, the acoustic data, and the equipment data.

6. The information processing method according to claim 3, wherein, when performing immediate acoustic recognition based on the acoustic data, the priority of the acoustic data is set higher than the priority of the work result data, the imaging data, and the equipment data.

7. The information processing method according to claim 2, wherein the terminal device acquires work data for each of the multiple work procedures and sets the priority of the work data according to the work procedure.

8. The information processing method according to claim 3, wherein the terminal device guides the worker through each work procedure, and triggers the guidance of the work procedure to start at least one of imaging the work at each work procedure and recording sounds at the work site.

9. The information processing method according to claim 3, wherein the terminal device recognizes a work scene based on the image data and determines whether or not to transmit the image data according to the recognized work scene.

10. The information processing method according to claim 2, wherein the terminal device detects the communication status and determines whether or not to transmit the work data with a relatively low priority according to the result of detecting the communication status.

11. The information processing method according to claim 1, wherein the terminal device obtains information relating to the transmission status of the work data from the server device, and outputs the transmission status of the work data based on the obtained information.

12. An information processing system comprising a worker's terminal device and a server device that is communicatively connected to the terminal device, wherein the terminal device acquires multiple work data related to work performed by the worker and controls the timing of transmitting the work data to the server device according to the priority of the acquired work data.

13. A terminal device comprising at least one control unit, wherein the control unit acquires a plurality of work data related to work performed by an operator, and controls the timing of transmitting the acquired work data to a server device according to the priority of the work data.

14. A computer program that causes a computer to execute a process to acquire multiple work data related to work performed by an operator, and to control the timing of sending the acquired work data to a server device according to the priority of the work data.

Images